LKML Archive on lore.kernel.org
help / color / mirror / Atom feed
From: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
To: linux-kernel@vger.kernel.org
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
	stable@vger.kernel.org, Jerry Hoemann <jerry.hoemann@hp.com>,
	Jeff Kirsher <jeffrey.t.kirsher@intel.com>,
	Ken Helias <kenhelias@firemail.de>,
	Andrew Morton <akpm@linux-foundation.org>,
	Linus Torvalds <torvalds@linux-foundation.org>,
	Jan Kara <jack@suse.cz>
Subject: [PATCH 3.14 96/98] fsnotify: next_i is freed during fsnotify_unmount_inodes.
Date: Sun, 25 Jan 2015 10:07:54 -0800	[thread overview]
Message-ID: <20150125180717.057498861@linuxfoundation.org> (raw)
In-Reply-To: <20150125180712.859646324@linuxfoundation.org>

3.14-stable review patch.  If anyone has any objections, please let me know.

------------------

From: Jerry Hoemann <jerry.hoemann@hp.com>

commit 6424babfd68dd8a83d9c60a5242d27038856599f upstream.

During file system stress testing on 3.10 and 3.12 based kernels, the
umount command occasionally hung in fsnotify_unmount_inodes in the
section of code:

                spin_lock(&inode->i_lock);
                if (inode->i_state & (I_FREEING|I_WILL_FREE|I_NEW)) {
                        spin_unlock(&inode->i_lock);
                        continue;
                }

As this section of code holds the global inode_sb_list_lock, eventually
the system hangs trying to acquire the lock.

Multiple crash dumps showed:

The inode->i_state == 0x60 and i_count == 0 and i_sb_list would point
back at itself.  As this is not the value of list upon entry to the
function, the kernel never exits the loop.

To help narrow down problem, the call to list_del_init in
inode_sb_list_del was changed to list_del.  This poisons the pointers in
the i_sb_list and causes a kernel to panic if it transverse a freed
inode.

Subsequent stress testing paniced in fsnotify_unmount_inodes at the
bottom of the list_for_each_entry_safe loop showing next_i had become
free.

We believe the root cause of the problem is that next_i is being freed
during the window of time that the list_for_each_entry_safe loop
temporarily releases inode_sb_list_lock to call fsnotify and
fsnotify_inode_delete.

The code in fsnotify_unmount_inodes attempts to prevent the freeing of
inode and next_i by calling __iget.  However, the code doesn't do the
__iget call on next_i

	if i_count == 0 or
	if i_state & (I_FREEING | I_WILL_FREE)

The patch addresses this issue by advancing next_i in the above two cases
until we either find a next_i which we can __iget or we reach the end of
the list.  This makes the handling of next_i more closely match the
handling of the variable "inode."

The time to reproduce the hang is highly variable (from hours to days.) We
ran the stress test on a 3.10 kernel with the proposed patch for a week
without failure.

During list_for_each_entry_safe, next_i is becoming free causing
the loop to never terminate.  Advance next_i in those cases where
__iget is not done.

Signed-off-by: Jerry Hoemann <jerry.hoemann@hp.com>
Cc: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
Cc: Ken Helias <kenhelias@firemail.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Jan Kara <jack@suse.cz>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>

---
 fs/notify/inode_mark.c |   17 +++++++++++------
 1 file changed, 11 insertions(+), 6 deletions(-)

--- a/fs/notify/inode_mark.c
+++ b/fs/notify/inode_mark.c
@@ -288,20 +288,25 @@ void fsnotify_unmount_inodes(struct list
 		spin_unlock(&inode->i_lock);
 
 		/* In case the dropping of a reference would nuke next_i. */
-		if ((&next_i->i_sb_list != list) &&
-		    atomic_read(&next_i->i_count)) {
+		while (&next_i->i_sb_list != list) {
 			spin_lock(&next_i->i_lock);
-			if (!(next_i->i_state & (I_FREEING | I_WILL_FREE))) {
+			if (!(next_i->i_state & (I_FREEING | I_WILL_FREE)) &&
+						atomic_read(&next_i->i_count)) {
 				__iget(next_i);
 				need_iput = next_i;
+				spin_unlock(&next_i->i_lock);
+				break;
 			}
 			spin_unlock(&next_i->i_lock);
+			next_i = list_entry(next_i->i_sb_list.next,
+						struct inode, i_sb_list);
 		}
 
 		/*
-		 * We can safely drop inode_sb_list_lock here because we hold
-		 * references on both inode and next_i.  Also no new inodes
-		 * will be added since the umount has begun.
+		 * We can safely drop inode_sb_list_lock here because either
+		 * we actually hold references on both inode and next_i or
+		 * end of list.  Also no new inodes will be added since the
+		 * umount has begun.
 		 */
 		spin_unlock(&inode_sb_list_lock);
 



  parent reply	other threads:[~2015-01-25 19:08 UTC|newest]

Thread overview: 96+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2015-01-25 18:06 [PATCH 3.14 00/98] 3.14.30-stable review Greg Kroah-Hartman
2015-01-25 18:06 ` [PATCH 3.14 02/98] netlink: Always copy on mmap TX Greg Kroah-Hartman
2015-01-25 18:06 ` [PATCH 3.14 03/98] netlink: Dont reorder loads/stores before marking mmap netlink frame as available Greg Kroah-Hartman
2015-01-25 18:06 ` [PATCH 3.14 04/98] in6: fix conflict with glibc Greg Kroah-Hartman
2015-01-25 18:06 ` [PATCH 3.14 05/98] tg3: tg3_disable_ints using uninitialized mailbox value to disable interrupts Greg Kroah-Hartman
2015-01-25 18:06 ` [PATCH 3.14 07/98] batman-adv: Unify fragment size calculation Greg Kroah-Hartman
2015-01-25 18:06 ` [PATCH 3.14 08/98] batman-adv: avoid NULL dereferences and fix if check Greg Kroah-Hartman
2015-01-25 18:06 ` [PATCH 3.14 09/98] net: Fix stacked vlan offload features computation Greg Kroah-Hartman
2015-01-25 18:06 ` [PATCH 3.14 10/98] net: Reset secmark when scrubbing packet Greg Kroah-Hartman
2015-01-25 18:06 ` [PATCH 3.14 11/98] tcp: Do not apply TSO segment limit to non-TSO packets Greg Kroah-Hartman
2015-01-25 18:06 ` [PATCH 3.14 12/98] alx: fix alx_poll() Greg Kroah-Hartman
2015-01-25 18:06 ` [PATCH 3.14 13/98] team: avoid possible underflow of count_pending value for notify_peers and mcast_rejoin Greg Kroah-Hartman
2015-01-25 18:06 ` [PATCH 3.14 14/98] enic: fix rx skb checksum Greg Kroah-Hartman
2015-01-25 18:06 ` [PATCH 3.14 15/98] net/core: Handle csum for CHECKSUM_COMPLETE VXLAN forwarding Greg Kroah-Hartman
2015-01-25 18:06 ` [PATCH 3.14 16/98] drm/vmwgfx: Fix fence event code Greg Kroah-Hartman
2015-01-25 18:06 ` [PATCH 3.14 17/98] drm/ttm: Avoid memory allocation from shrinker functions Greg Kroah-Hartman
2015-01-25 18:06 ` [PATCH 3.14 18/98] drm/radeon: fix typo in CI dpm disable Greg Kroah-Hartman
2015-01-25 18:06 ` [PATCH 3.14 19/98] drm/radeon: work around a hw bug in MGCG on CIK Greg Kroah-Hartman
2015-01-25 18:06 ` [PATCH 3.14 21/98] drm/radeon: properly filter DP1.2 4k modes on non-DP1.2 hw Greg Kroah-Hartman
2015-01-25 18:06 ` [PATCH 3.14 22/98] drm/i915: Dont complain about stolen conflicts on gen3 Greg Kroah-Hartman
2015-01-25 18:06 ` [PATCH 3.14 23/98] drm/i915: Only warn the first time we attempt to mmio whilst suspended Greg Kroah-Hartman
2015-01-25 18:06 ` [PATCH 3.14 24/98] drm/nv4c/mc: disable msi Greg Kroah-Hartman
2015-01-25 18:06 ` [PATCH 3.14 27/98] ARC: [nsimosci] move peripherals to match model to FPGA Greg Kroah-Hartman
2015-01-25 18:06 ` [PATCH 3.14 28/98] ARC: switch to generic ENTRY/END assembler annotations Greg Kroah-Hartman
2015-01-25 18:06 ` [PATCH 3.14 29/98] cfg80211: dont WARN about two consecutive Country IE hint Greg Kroah-Hartman
2015-01-25 18:06 ` [PATCH 3.14 30/98] cfg80211: avoid mem leak on driver hint set Greg Kroah-Hartman
2015-01-25 18:06 ` [PATCH 3.14 31/98] cfg80211: Fix 160 MHz channels with 80+80 and 160 MHz drivers Greg Kroah-Hartman
2015-01-25 18:06 ` [PATCH 3.14 32/98] hp_accel: Add support for HP ZBook 15 Greg Kroah-Hartman
2015-01-25 18:06 ` [PATCH 3.14 33/98] tick/powerclamp: Remove tick_nohz_idle abuse Greg Kroah-Hartman
2015-01-25 18:06 ` [PATCH 3.14 34/98] genirq: Prevent proc race against freeing of irq descriptors Greg Kroah-Hartman
2015-01-25 18:06 ` [PATCH 3.14 35/98] iscsi-target: Fail connection on short sendmsg writes Greg Kroah-Hartman
2015-01-25 18:06 ` [PATCH 3.14 36/98] Revert "[SCSI] mpt2sas: Remove phys on topology change." Greg Kroah-Hartman
2015-01-25 18:06 ` [PATCH 3.14 37/98] Revert "[SCSI] mpt3sas: Remove phys on topology change" Greg Kroah-Hartman
2015-01-25 18:06 ` [PATCH 3.14 38/98] scsi: blacklist RSOC for Microsoft iSCSI target devices Greg Kroah-Hartman
2015-01-25 18:06 ` [PATCH 3.14 39/98] clk: samsung: Fix double add of syscore ops after driver rebind Greg Kroah-Hartman
2015-01-25 18:06 ` [PATCH 3.14 40/98] clk: Really fix deadlock with mmap_sem Greg Kroah-Hartman
2015-01-26 12:46   ` Luis Henriques
2015-01-26 22:20     ` Greg Kroah-Hartman
2015-01-25 18:06 ` [PATCH 3.14 41/98] clk: Dont try to use a struct clk* after it could have been freed Greg Kroah-Hartman
2015-01-25 18:07 ` [PATCH 3.14 42/98] parisc: fix out-of-register compiler error in ldcw inline assembler function Greg Kroah-Hartman
2015-01-25 18:07 ` [PATCH 3.14 43/98] storvsc: ring buffer failures may result in I/O freeze Greg Kroah-Hartman
2015-01-25 18:07 ` [PATCH 3.14 44/98] net: ethernet: cpsw: fix hangs with interrupts Greg Kroah-Hartman
2015-01-25 18:07 ` [PATCH 3.14 45/98] video/logo: prevent use of logos after they have been freed Greg Kroah-Hartman
2015-01-25 18:07 ` [PATCH 3.14 46/98] [media] smiapp-pll: Correct clock debug prints Greg Kroah-Hartman
2015-01-25 18:07 ` [PATCH 3.14 48/98] [media] smiapp: Take mutex during PLL update in sensor initialisation Greg Kroah-Hartman
2015-01-25 18:07 ` [PATCH 3.14 49/98] [media] sound: simplify au0828 quirk table Greg Kroah-Hartman
2015-01-25 18:07 ` [PATCH 3.14 50/98] [media] sound: Update au0828 quirks table Greg Kroah-Hartman
2015-01-25 18:07 ` [PATCH 3.14 51/98] [media] uvcvideo: Fix destruction order in uvc_delete() Greg Kroah-Hartman
2015-01-25 18:07 ` [PATCH 3.14 52/98] vfio-pci: Fix the check on pci device type in vfio_pci_probe() Greg Kroah-Hartman
2015-01-25 18:07 ` [PATCH 3.14 53/98] drivers: net: cpsw: fix multicast flush in dual emac mode Greg Kroah-Hartman
2015-01-25 18:07 ` [PATCH 3.14 54/98] ftrace/jprobes/x86: Fix conflict between jprobes and function graph tracing Greg Kroah-Hartman
2015-01-25 18:07 ` [PATCH 3.14 55/98] NFSv4.1: Fix client id trunking on Linux Greg Kroah-Hartman
2015-01-25 18:07 ` [PATCH 3.14 56/98] gpiolib: of: Correct error handling in of_get_named_gpiod_flags Greg Kroah-Hartman
2015-01-25 18:07 ` [PATCH 3.14 57/98] gpio: fix memory and reference leaks in gpiochip_add error path Greg Kroah-Hartman
2015-01-25 18:07 ` [PATCH 3.14 58/98] OHCI: add a quirk for ULi M5237 blocking on reset Greg Kroah-Hartman
2015-01-25 18:07 ` [PATCH 3.14 59/98] usb: dwc3: gadget: Fix TRB preparation during SG Greg Kroah-Hartman
2015-01-25 18:07 ` [PATCH 3.14 60/98] usb: dwc3: gadget: Stop TRB preparation after limit is reached Greg Kroah-Hartman
2015-01-25 18:07 ` [PATCH 3.14 61/98] USB: cp210x: fix ID for production CEL MeshConnect USB Stick Greg Kroah-Hartman
2015-01-25 18:07 ` [PATCH 3.14 62/98] USB: cp210x: add IDs for CEL USB sticks and MeshWorks devices Greg Kroah-Hartman
2015-01-25 18:07 ` [PATCH 3.14 63/98] USB: keyspan: fix null-deref at probe Greg Kroah-Hartman
2015-01-25 18:07 ` [PATCH 3.14 64/98] USB: console: fix uninitialised ldisc semaphore Greg Kroah-Hartman
2015-01-25 18:07 ` [PATCH 3.14 65/98] USB: console: fix potential use after free Greg Kroah-Hartman
2015-01-25 18:07 ` [PATCH 3.14 66/98] USB: EHCI: fix initialization bug in iso_stream_schedule() Greg Kroah-Hartman
2015-01-25 18:07 ` [PATCH 3.14 67/98] usb: musb: stuff leak of struct usb_hcd Greg Kroah-Hartman
2015-01-25 18:07 ` [PATCH 3.14 68/98] can: kvaser_usb: Dont free packets when tight on URBs Greg Kroah-Hartman
2015-01-25 18:07 ` [PATCH 3.14 69/98] can: kvaser_usb: Reset all URB tx contexts upon channel close Greg Kroah-Hartman
2015-01-25 18:07 ` [PATCH 3.14 70/98] can: kvaser_usb: Dont send a RESET_CHIP for non-existing channels Greg Kroah-Hartman
2015-01-25 18:07 ` [PATCH 3.14 72/98] Input: I8042 - add Acer Aspire 7738 to the nomux list Greg Kroah-Hartman
2015-01-25 18:07 ` [PATCH 3.14 73/98] ARM: dts: imx25: Fix the SPI1 clocks Greg Kroah-Hartman
2015-01-25 18:07 ` [PATCH 3.14 74/98] ARM: imx6q: drop unnecessary semicolon Greg Kroah-Hartman
2015-01-25 18:07 ` [PATCH 3.14 75/98] ARM: clk-imx6q: fix video divider for rev T0 1.0 Greg Kroah-Hartman
2015-01-25 18:07 ` [PATCH 3.14 76/98] ARM: omap5/dra7xx: Fix frequency typos Greg Kroah-Hartman
2015-01-25 18:07 ` [PATCH 3.14 77/98] ARM: shmobile: sh73a0 legacy: Set .control_parent for all irqpin instances Greg Kroah-Hartman
2015-01-25 18:07 ` [PATCH 3.14 78/98] decompress_bunzip2: off by one in get_next_block() Greg Kroah-Hartman
2015-01-25 18:07 ` [PATCH 3.14 79/98] um: Skip futex_atomic_cmpxchg_inatomic() test Greg Kroah-Hartman
2015-01-25 18:07 ` [PATCH 3.14 80/98] x86, um: actually mark system call tables readonly Greg Kroah-Hartman
2015-01-25 18:07 ` [PATCH 3.14 81/98] LOCKD: Fix a race when initialising nlmsvc_timeout Greg Kroah-Hartman
2015-01-25 18:07 ` [PATCH 3.14 82/98] tcm_loop: Fixup tag handling Greg Kroah-Hartman
2015-01-25 18:07 ` [PATCH 3.14 83/98] tcm_loop: Fix wrong I_T nexus association Greg Kroah-Hartman
2015-01-25 18:07 ` [PATCH 3.14 84/98] vhost-scsi: Add missing virtio-scsi -> TCM attribute conversion Greg Kroah-Hartman
2015-01-25 18:07 ` [PATCH 3.14 85/98] iscsi,iser-target: Initiate termination only once Greg Kroah-Hartman
2015-01-25 18:07 ` [PATCH 3.14 86/98] iser-target: Fix flush + disconnect completion handling Greg Kroah-Hartman
2015-01-25 18:07 ` [PATCH 3.14 87/98] iser-target: Parallelize CM connection establishment Greg Kroah-Hartman
2015-01-25 18:07 ` [PATCH 3.14 88/98] iser-target: Fix connected_handler + teardown flow race Greg Kroah-Hartman
2015-01-25 18:07 ` [PATCH 3.14 89/98] iser-target: Handle ADDR_CHANGE event for listener cm_id Greg Kroah-Hartman
2015-01-25 18:07 ` [PATCH 3.14 90/98] iser-target: Fix implicit termination of connections Greg Kroah-Hartman
2015-01-25 18:07 ` [PATCH 3.14 91/98] bcache: Make sure to pass GFP_WAIT to mempool_alloc() Greg Kroah-Hartman
2015-01-25 18:07 ` [PATCH 3.14 92/98] KVM: nVMX: Disable unrestricted mode if ept=0 Greg Kroah-Hartman
2015-01-25 18:07 ` [PATCH 3.14 93/98] netfilter: ipset: small potential read beyond the end of buffer Greg Kroah-Hartman
2015-01-25 18:07 ` [PATCH 3.14 94/98] net: prevent of emerging cross-namespace symlinks Greg Kroah-Hartman
2015-01-25 18:07 ` [PATCH 3.14 95/98] net: fix creation adjacent device symlinks Greg Kroah-Hartman
2015-01-25 18:07 ` Greg Kroah-Hartman [this message]
2015-01-25 18:07 ` [PATCH 3.14 97/98] s390/3215: fix hanging console issue Greg Kroah-Hartman
2015-01-25 18:07 ` [PATCH 3.14 98/98] s390/3215: fix tty output containing tabs Greg Kroah-Hartman
2015-01-25 21:37 ` [PATCH 3.14 00/98] 3.14.30-stable review Guenter Roeck
2015-01-26 17:43 ` Shuah Khan

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20150125180717.057498861@linuxfoundation.org \
    --to=gregkh@linuxfoundation.org \
    --cc=akpm@linux-foundation.org \
    --cc=jack@suse.cz \
    --cc=jeffrey.t.kirsher@intel.com \
    --cc=jerry.hoemann@hp.com \
    --cc=kenhelias@firemail.de \
    --cc=linux-kernel@vger.kernel.org \
    --cc=stable@vger.kernel.org \
    --cc=torvalds@linux-foundation.org \
    --subject='Re: [PATCH 3.14 96/98] fsnotify: next_i is freed during fsnotify_unmount_inodes.' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).