kernel BUG at fs/inode.c:LINE! (2)

kernel BUG at fs/inode.c:LINE! (2)

[syzbot]

Hello,

syzbot found the following issue on:

HEAD commit:    d012a719 Linux 5.9-rc2
git tree:       upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=15aa650e900000
kernel config:  https://syzkaller.appspot.com/x/.config?x=891ca5711a9f1650
dashboard link: https://syzkaller.appspot.com/bug?extid=c92c93d1f1aaaacdb9db
compiler:       clang version 10.0.0 (https://github.com/llvm/llvm-project/ c2443155a0fb245c8f17f2c1c72b6ea391e86e81)
syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=12ecb939900000
C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=140a19a9900000

The issue was bisected to:

commit a9ed4a6560b8562b7e2e2bed9527e88001f7b682
Author: Marc Zyngier <maz@kernel.org>
Date:   Wed Aug 19 16:12:17 2020 +0000

    epoll: Keep a reference on files added to the check list

bisection log:  https://syzkaller.appspot.com/x/bisect.txt?x=16a50519900000
final oops:     https://syzkaller.appspot.com/x/report.txt?x=15a50519900000
console output: https://syzkaller.appspot.com/x/log.txt?x=11a50519900000

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+c92c93d1f1aaaacdb9db@syzkaller.appspotmail.com
Fixes: a9ed4a6560b8 ("epoll: Keep a reference on files added to the check list")

------------[ cut here ]------------
kernel BUG at fs/inode.c:1668!
invalid opcode: 0000 [#1] PREEMPT SMP KASAN
CPU: 0 PID: 29571 Comm: syz-executor709 Not tainted 5.9.0-rc2-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
RIP: 0010:iput+0x6d8/0x6e0 fs/inode.c:1668
Code: ef ff e9 1a fc ff ff 44 89 e9 80 e1 07 80 c1 03 38 c1 0f 8c c8 fe ff ff 4c 89 ef e8 a2 51 ef ff e9 bb fe ff ff e8 68 77 af ff <0f> 0b 66 0f 1f 44 00 00 55 41 57 41 56 53 48 89 f5 48 89 fb 49 bf
RSP: 0018:ffffc9000e25fda8 EFLAGS: 00010293
RAX: ffffffff81c580b8 RBX: ffff888085112600 RCX: ffff8880a6eea200
RDX: 0000000000000000 RSI: 0000000000000040 RDI: 0000000000000000
RBP: 0000000000000040 R08: ffffffff81c57a40 R09: ffffed10116fe44d
R10: ffffed10116fe44d R11: 0000000000000000 R12: 1ffff11010a224ac
R13: dffffc0000000000 R14: ffff888085112600 R15: ffff888085112560
FS:  0000000001665880(0000) GS:ffff8880ae800000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000020000000 CR3: 000000008f752000 CR4: 00000000001506f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 __sock_release net/socket.c:608 [inline]
 sock_close+0x1c3/0x260 net/socket.c:1277
 __fput+0x34f/0x7b0 fs/file_table.c:281
 task_work_run+0x137/0x1c0 kernel/task_work.c:141
 tracehook_notify_resume include/linux/tracehook.h:188 [inline]
 exit_to_user_mode_loop kernel/entry/common.c:140 [inline]
 exit_to_user_mode_prepare+0xfa/0x1b0 kernel/entry/common.c:167
 syscall_exit_to_user_mode+0x5e/0x1a0 kernel/entry/common.c:242
 entry_SYSCALL_64_after_hwframe+0x44/0xa9
RIP: 0033:0x4058e1
Code: 75 14 b8 03 00 00 00 0f 05 48 3d 01 f0 ff ff 0f 83 04 19 00 00 c3 48 83 ec 08 e8 6a fc ff ff 48 89 04 24 b8 03 00 00 00 0f 05 <48> 8b 3c 24 48 89 c2 e8 b3 fc ff ff 48 89 d0 48 83 c4 08 48 3d 01
RSP: 002b:00007ffc7ca575a0 EFLAGS: 00000293 ORIG_RAX: 0000000000000003
RAX: 0000000000000000 RBX: 0000000000000006 RCX: 00000000004058e1
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000005
RBP: 0000000000000007 R08: 0000000120080522 R09: 0000000120080522
R10: 0000000120080522 R11: 0000000000000293 R12: 00000000006dbc5c
R13: 0000000000000001 R14: 00000000006dbc50 R15: 0000000000000064
Modules linked in:
---[ end trace 35240c511479d576 ]---
RIP: 0010:iput+0x6d8/0x6e0 fs/inode.c:1668
Code: ef ff e9 1a fc ff ff 44 89 e9 80 e1 07 80 c1 03 38 c1 0f 8c c8 fe ff ff 4c 89 ef e8 a2 51 ef ff e9 bb fe ff ff e8 68 77 af ff <0f> 0b 66 0f 1f 44 00 00 55 41 57 41 56 53 48 89 f5 48 89 fb 49 bf
RSP: 0018:ffffc9000e25fda8 EFLAGS: 00010293
RAX: ffffffff81c580b8 RBX: ffff888085112600 RCX: ffff8880a6eea200
RDX: 0000000000000000 RSI: 0000000000000040 RDI: 0000000000000000
RBP: 0000000000000040 R08: ffffffff81c57a40 R09: ffffed10116fe44d
R10: ffffed10116fe44d R11: 0000000000000000 R12: 1ffff11010a224ac
R13: dffffc0000000000 R14: ffff888085112600 R15: ffff888085112560
FS:  0000000001665880(0000) GS:ffff8880ae900000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000020000000 CR3: 000000008f752000 CR4: 00000000001506e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400


---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzkaller@googlegroups.com.

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
For information about bisection process see: https://goo.gl/tpsmEJ#bisection
syzbot can test patches for this issue, for details see:
https://goo.gl/tpsmEJ#testing-patches

Re: kernel BUG at fs/inode.c:LINE! (2)

[Al Viro]

On Fri, Aug 28, 2020 at 06:18:17AM -0700, syzbot wrote:
> Hello,
> 
> syzbot found the following issue on:
> 
> HEAD commit:    d012a719 Linux 5.9-rc2
> git tree:       upstream
> console output: https://syzkaller.appspot.com/x/log.txt?x=15aa650e900000
> kernel config:  https://syzkaller.appspot.com/x/.config?x=891ca5711a9f1650
> dashboard link: https://syzkaller.appspot.com/bug?extid=c92c93d1f1aaaacdb9db
> compiler:       clang version 10.0.0 (https://github.com/llvm/llvm-project/ c2443155a0fb245c8f17f2c1c72b6ea391e86e81)
> syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=12ecb939900000
> C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=140a19a9900000

Trying to reproduce...

Re: kernel BUG at fs/inode.c:LINE! (2)

[Al Viro]

On Fri, Aug 28, 2020 at 04:38:25PM +0100, Al Viro wrote:
> On Fri, Aug 28, 2020 at 06:18:17AM -0700, syzbot wrote:
> > Hello,
> > 
> > syzbot found the following issue on:
> > 
> > HEAD commit:    d012a719 Linux 5.9-rc2
> > git tree:       upstream
> > console output: https://syzkaller.appspot.com/x/log.txt?x=15aa650e900000
> > kernel config:  https://syzkaller.appspot.com/x/.config?x=891ca5711a9f1650
> > dashboard link: https://syzkaller.appspot.com/bug?extid=c92c93d1f1aaaacdb9db
> > compiler:       clang version 10.0.0 (https://github.com/llvm/llvm-project/ c2443155a0fb245c8f17f2c1c72b6ea391e86e81)
> > syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=12ecb939900000
> > C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=140a19a9900000
> 
> Trying to reproduce...

OK, I think I see what's going on.  ep_loop_check_proc() runs into an already
doomed file that has already committed to getting killed (->f_count is already
at 0), but still hadn't gotten through its epitems removal (e.g. has its
eventpoll_release_file() sitting there trying to get epmutex).

Blindly bumping refcount here is worse than useless.  Try this, to verify that
this is what's going on; it's _not_ a proper fix, but it should at least tell
if we have something else going on.

diff --git a/fs/eventpoll.c b/fs/eventpoll.c
index e0decff22ae2..4590223b2de9 100644
--- a/fs/eventpoll.c
+++ b/fs/eventpoll.c
@@ -1995,7 +1995,7 @@ static int ep_loop_check_proc(void *priv, void *cookie, int call_nests)
 			 * during ep_insert().
 			 */
 			if (list_empty(&epi->ffd.file->f_tfile_llink)) {
-				get_file(epi->ffd.file);
+				if (get_file_rcu(epi->ffd.file))
 				list_add(&epi->ffd.file->f_tfile_llink,
 					 &tfile_check_list);
 			}

Re: kernel BUG at fs/inode.c:LINE! (2)

[Al Viro]

On Fri, Aug 28, 2020 at 06:54:13PM +0100, Al Viro wrote:
> On Fri, Aug 28, 2020 at 04:38:25PM +0100, Al Viro wrote:
> > On Fri, Aug 28, 2020 at 06:18:17AM -0700, syzbot wrote:
> > > Hello,
> > > 
> > > syzbot found the following issue on:
> > > 
> > > HEAD commit:    d012a719 Linux 5.9-rc2
> > > git tree:       upstream
> > > console output: https://syzkaller.appspot.com/x/log.txt?x=15aa650e900000
> > > kernel config:  https://syzkaller.appspot.com/x/.config?x=891ca5711a9f1650
> > > dashboard link: https://syzkaller.appspot.com/bug?extid=c92c93d1f1aaaacdb9db
> > > compiler:       clang version 10.0.0 (https://github.com/llvm/llvm-project/ c2443155a0fb245c8f17f2c1c72b6ea391e86e81)
> > > syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=12ecb939900000
> > > C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=140a19a9900000
> > 
> > Trying to reproduce...
> 
> OK, I think I see what's going on.  ep_loop_check_proc() runs into an already
> doomed file that has already committed to getting killed (->f_count is already
> at 0), but still hadn't gotten through its epitems removal (e.g. has its
> eventpoll_release_file() sitting there trying to get epmutex).
> 
> Blindly bumping refcount here is worse than useless.  Try this, to verify that
> this is what's going on; it's _not_ a proper fix, but it should at least tell
> if we have something else going on.

... and what I think is the right way to fix the original race is (on top of
mainline) this:

[PATCH] Use list_empty_careful() in eventpoll_release()

... to avoid races with list_del_init() in clear_tfile_check_list().
Get rid of pinning files on check list in eventpoll.c - it's not needed
there.

Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
---
diff --git a/fs/eventpoll.c b/fs/eventpoll.c
index e0decff22ae2..39eae45bff18 100644
--- a/fs/eventpoll.c
+++ b/fs/eventpoll.c
@@ -1995,7 +1995,6 @@ static int ep_loop_check_proc(void *priv, void *cookie, int call_nests)
 			 * during ep_insert().
 			 */
 			if (list_empty(&epi->ffd.file->f_tfile_llink)) {
-				get_file(epi->ffd.file);
 				list_add(&epi->ffd.file->f_tfile_llink,
 					 &tfile_check_list);
 			}
@@ -2042,7 +2041,6 @@ static void clear_tfile_check_list(void)
 		file = list_first_entry(&tfile_check_list, struct file,
 					f_tfile_llink);
 		list_del_init(&file->f_tfile_llink);
-		fput(file);
 	}
 	INIT_LIST_HEAD(&tfile_check_list);
 }
@@ -2206,7 +2204,6 @@ int do_epoll_ctl(int epfd, int op, int fd, struct epoll_event *epds,
 				if (ep_loop_check(ep, tf.file) != 0)
 					goto error_tgt_fput;
 			} else {
-				get_file(tf.file);
 				list_add(&tf.file->f_tfile_llink,
 							&tfile_check_list);
 			}
diff --git a/include/linux/eventpoll.h b/include/linux/eventpoll.h
index 8f000fada5a4..e2bdefd90cf8 100644
--- a/include/linux/eventpoll.h
+++ b/include/linux/eventpoll.h
@@ -46,11 +46,9 @@ static inline void eventpoll_release(struct file *file)
 	 * Fast check to avoid the get/release of the semaphore. Since
 	 * we're doing this outside the semaphore lock, it might return
 	 * false negatives, but we don't care. It'll help in 99.99% of cases
-	 * to avoid the semaphore lock. False positives simply cannot happen
-	 * because the file in on the way to be removed and nobody ( but
-	 * eventpoll ) has still a reference to this file.
+	 * to avoid the semaphore lock.
 	 */
-	if (likely(list_empty(&file->f_ep_links)))
+	if (likely(list_empty_careful(&file->f_ep_links)))
 		return;
 
 	/*