kernel panic: EXT4-fs (device loop0): panic forced after error

kernel panic: EXT4-fs (device loop0): panic forced after error

[syzbot]

Hello,

syzbot found the following crash on:

HEAD commit:    c1c07416cdd4 Merge tag 'kbuild-fixes-v4.17' of git://git.k..
git tree:       upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=11a8a07b800000
kernel config:  https://syzkaller.appspot.com/x/.config?x=5a1dc06635c10d27
dashboard link: https://syzkaller.appspot.com/bug?extid=a9a45987b8b2daabdc88
compiler:       gcc (GCC) 8.0.1 20180413 (experimental)
syzkaller repro:https://syzkaller.appspot.com/x/repro.syz?x=164fa297800000
C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=13635c37800000

IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: syzbot+a9a45987b8b2daabdc88@syzkaller.appspotmail.com

random: sshd: uninitialized urandom read (32 bytes read)
random: sshd: uninitialized urandom read (32 bytes read)
random: sshd: uninitialized urandom read (32 bytes read)
random: sshd: uninitialized urandom read (32 bytes read)
EXT4-fs error (device loop0): ext4_iget:4756: inode #2: comm  
syz-executor909: root inode unallocated
Kernel panic - not syncing: EXT4-fs (device loop0): panic forced after error

CPU: 1 PID: 4451 Comm: syz-executor909 Not tainted 4.17.0-rc3+ #34
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS  
Google 01/01/2011
Call Trace:
  __dump_stack lib/dump_stack.c:77 [inline]
  dump_stack+0x1b9/0x294 lib/dump_stack.c:113
  panic+0x22f/0x4de kernel/panic.c:184
  ext4_handle_error.part.120.cold.128+0x1d/0x1d fs/ext4/super.c:431
  ext4_handle_error fs/ext4/super.c:408 [inline]
  __ext4_error_inode+0x351/0x730 fs/ext4/super.c:494
  ext4_iget+0xe82/0x3dd0 fs/ext4/inode.c:4756
  ext4_fill_super+0x733c/0xd810 fs/ext4/super.c:4208
  mount_bdev+0x30c/0x3e0 fs/super.c:1164
  ext4_mount+0x34/0x40 fs/ext4/super.c:5740
  mount_fs+0xae/0x328 fs/super.c:1267
  vfs_kern_mount.part.34+0xd4/0x4d0 fs/namespace.c:1037
  vfs_kern_mount fs/namespace.c:1027 [inline]
  do_new_mount fs/namespace.c:2518 [inline]
  do_mount+0x564/0x3070 fs/namespace.c:2848
  ksys_mount+0x12d/0x140 fs/namespace.c:3064
  __do_sys_mount fs/namespace.c:3078 [inline]
  __se_sys_mount fs/namespace.c:3075 [inline]
  __x64_sys_mount+0xbe/0x150 fs/namespace.c:3075
  do_syscall_64+0x1b1/0x800 arch/x86/entry/common.c:287
  entry_SYSCALL_64_after_hwframe+0x49/0xbe
RIP: 0033:0x442dda
RSP: 002b:00007ffdadaf1e38 EFLAGS: 00000297 ORIG_RAX: 00000000000000a5
RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 0000000000442dda
RDX: 0000000020000480 RSI: 0000000020000100 RDI: 00007ffdadaf1e40
RBP: 0000000000000004 R08: 0000000020000440 R09: 000000000000000a
R10: 0000000000000000 R11: 0000000000000297 R12: 0000000000401c40
R13: 0000000000401cd0 R14: 0000000000000000 R15: 0000000000000000
Dumping ftrace buffer:
    (ftrace buffer empty)
Kernel Offset: disabled
Rebooting in 86400 seconds..


---
This bug is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzkaller@googlegroups.com.

syzbot will keep track of this bug report.
If you forgot to add the Reported-by tag, once the fix for this bug is  
merged
into any tree, please reply to this email with:
#syz fix: exact-commit-title
If you want to test a patch for this bug, please reply with:
#syz test: git://repo/address.git branch
and provide the patch inline or as an attachment.
To mark this as a duplicate of another syzbot report, please reply with:
#syz dup: exact-subject-of-another-report
If it's a one-off invalid bug report, please reply with:
#syz invalid
Note: if the crash happens again, it will cause creation of a new bug  
report.
Note: all commands must start from beginning of the line in the email body.

Re: kernel panic: EXT4-fs (device loop0): panic forced after error

[Theodore Y. Ts'o]

On Sat, May 05, 2018 at 05:57:02PM -0700, syzbot wrote:
> Hello,
> 
> syzbot found the following crash on:
> 
> EXT4-fs error (device loop0): ext4_iget:4756: inode #2: comm
> syz-executor909: root inode unallocated
> Kernel panic - not syncing: EXT4-fs (device loop0): panic forced after error

I don't get why syzbot considers this a bug.  It created a corrupted
file system, mounted it as root, and said file system had the flag
which says, "panic if you find a file system corruption".

In what world is this a security bug?  There's a *reason* why I've
always said people who want to containers to be allowed to mount
arbitrary file systems controlled by potentially malicious container
users are insane....

I could mark this as a one-off invalid bug, but if syzkaller is going
to be generating classes of corrupted file systems like this, and are
going to be complaing about how this causes the kernel to crash, then
we have a fundamental syzkaller BUG.

					- Ted

Re: kernel panic: EXT4-fs (device loop0): panic forced after error

[Tetsuo Handa]

On 2018/05/06 11:24, Theodore Y. Ts'o wrote:
> On Sat, May 05, 2018 at 05:57:02PM -0700, syzbot wrote:
>> Hello,
>>
>> syzbot found the following crash on:
>>
>> EXT4-fs error (device loop0): ext4_iget:4756: inode #2: comm
>> syz-executor909: root inode unallocated
>> Kernel panic - not syncing: EXT4-fs (device loop0): panic forced after error
> 
> I don't get why syzbot considers this a bug.  It created a corrupted
> file system, mounted it as root, and said file system had the flag
> which says, "panic if you find a file system corruption".

"panic if file system error occurred (so that we won't continue with
inconsistent state)", doesn't it?

Since syzbot is hitting this error path inside mount() request, calling
panic() when something went wrong inside mount() request might be
overkill. We can recover without shutting down the system, can't we?

> 
> In what world is this a security bug?  There's a *reason* why I've
> always said people who want to containers to be allowed to mount
> arbitrary file systems controlled by potentially malicious container
> users are insane....
> 
> I could mark this as a one-off invalid bug, but if syzkaller is going
> to be generating classes of corrupted file systems like this, and are
> going to be complaing about how this causes the kernel to crash, then
> we have a fundamental syzkaller BUG.
> 
> 					- Ted

If we won't try to recover this case, this specific report would be
marked as "#syz invalid".

Re: kernel panic: EXT4-fs (device loop0): panic forced after error

[Dmitry Vyukov]

On Sun, May 6, 2018 at 7:03 AM, Tetsuo Handa
<penguin-kernel@i-love.sakura.ne.jp> wrote:
> On 2018/05/06 11:24, Theodore Y. Ts'o wrote:
>> On Sat, May 05, 2018 at 05:57:02PM -0700, syzbot wrote:
>>> Hello,
>>>
>>> syzbot found the following crash on:
>>>
>>> EXT4-fs error (device loop0): ext4_iget:4756: inode #2: comm
>>> syz-executor909: root inode unallocated
>>> Kernel panic - not syncing: EXT4-fs (device loop0): panic forced after error
>>
>> I don't get why syzbot considers this a bug.  It created a corrupted
>> file system, mounted it as root, and said file system had the flag
>> which says, "panic if you find a file system corruption".
> In what world is this a security bug?

Hi Ted,

Do you mean why syzbot considers kernel panics as something to report?
Or why syzbot has not understood why this panic is somehow special?
syzbot doesn't report only security problems. It just reports bugs.
E.g. NULL derefs and deadlocks are also not security bugs, but are
reported.


> "panic if file system error occurred (so that we won't continue with
> inconsistent state)", doesn't it?
>
> Since syzbot is hitting this error path inside mount() request, calling
> panic() when something went wrong inside mount() request might be
> overkill. We can recover without shutting down the system, can't we?

Can you please give me background regarding purpose of this flag?
What's the intended use case? And what exactly is corrupted?
If kernel detects that an image is corrupted during mount shouldn't it
just report an error? If I am reading this correctly, any USB cable
that plug into my computer can shut it down. Or maybe even existing
cables can shut down all computers in a country on a remote command.
Am I missing something here?


>> In what world is this a security bug?  There's a *reason* why I've
>> always said people who want to containers to be allowed to mount
>> arbitrary file systems controlled by potentially malicious container
>> users are insane....
>>
>> I could mark this as a one-off invalid bug, but if syzkaller is going
>> to be generating classes of corrupted file systems like this, and are
>> going to be complaing about how this causes the kernel to crash, then
>> we have a fundamental syzkaller BUG.
>>
>>                                       - Ted
>
> If we won't try to recover this case, this specific report would be
> marked as "#syz invalid".
>
> --
> You received this message because you are subscribed to the Google Groups "syzkaller-bugs" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller-bugs+unsubscribe@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller-bugs/d295f575-ba4b-b0ba-a856-405f9d393c9d%40I-love.SAKURA.ne.jp.
> For more options, visit https://groups.google.com/d/optout.

Re: kernel panic: EXT4-fs (device loop0): panic forced after error

[Theodore Y. Ts'o]

On Sun, May 06, 2018 at 11:15:45AM +0200, Dmitry Vyukov wrote:
> >> I don't get why syzbot considers this a bug.  It created a corrupted
> >> file system, mounted it as root, and said file system had the flag
> >> which says, "panic if you find a file system corruption".
> > In what world is this a security bug?
> 
> Do you mean why syzbot considers kernel panics as something to report?
> Or why syzbot has not understood why this panic is somehow special?
> syzbot doesn't report only security problems. It just reports bugs.
> E.g. NULL derefs and deadlocks are also not security bugs, but are
> reported.

EXT4-fs (device loop0): panic forced after error

This is working as intended.  See the tune2fs man page.

     -e error-behavior
              Change the behavior of the kernel code when errors are
              detected.  In all cases, a filesystem error will cause
              e2fsck(8) to check the filesystem on the next boot.
              error-behavior can be one of the following:

                   continue    Continue normal execution.

                   remount-ro  Remount filesystem read-only.

                   panic       Cause a kernel panic.


> Can you please give me background regarding purpose of this flag?
> What's the intended use case? And what exactly is corrupted?

When the file system is corrupted, you might not want to continue
processing.  If you have some kind High Availability system available,
having the system shutdown so the secondary/backup system can take
over for the corrupted file system.  This might be better than
continuing to steer the spaceship, or continuning to control the x-ray
dosage received by a patient, etc., etc.

> If kernel detects that an image is corrupted during mount shouldn't it
> just report an error?

The problem is that this would, in the general case, putting in a full
or partial kernel-mode fsck and requiring it to be run before mount.
Sure, I could add more complexity in the system to bypass the check in
iget() if it is being called from mount, but what if SELinux is
running and it tries to fetch an xattr?  Or what if Syzkaller's
repro.c decides to start reading or writing files, or deleting files
in the directory?

More generally, root can always do stuff which will cause the system
to crash or shut down.  It can kexec a file containing garbage; it can
call the shutdown system call.  Presumably it is not reasonable to
require the kexec(2) system call to validate the image before
transfering control to it?  After all, that would require solving the
Halting Problem.  :-)

> If I am reading this correctly, any USB cable
> that plug into my computer can shut it down. Or maybe even existing
> cables can shut down all computers in a country on a remote command.
> Am I missing something here?

If you have physical access to the machine, you can shut it down.  And
there cases when shutting down a system or even all of the computers
in a cluster is the right thing to do.  Not shutting down the computer
could actually have harmful physical effects (e.g., a transformer
blowing up, a centrifuge tearing itself apart[1], etc.)  The system
administrator should have the choice to set up a file system such that
if an inconsistency is detected.

						- Ted

[1] Although if the centrifuge is in Iran, and you are in the NSA,
that might be considered a feature, not a bug.  Of course, if the
transformer is in the US or Germany, and the entity triggering a bug
which causes it to blow up is sponsored by Russia, then of course
that's a different story.  :-)

Re: kernel panic: EXT4-fs (device loop0): panic forced after error

[Theodore Y. Ts'o]

On Sun, May 06, 2018 at 02:03:57PM +0900, Tetsuo Handa wrote:
> 
> Since syzbot is hitting this error path inside mount() request, calling
> panic() when something went wrong inside mount() request might be
> overkill. We can recover without shutting down the system, can't we?

We could add a full kernel-mode fsck which gets run before mount ---
the question is how much complexity we want to add.  If SELinux is
enabled, then we have to check xattr consinsistency, etc., etc.

> > I could mark this as a one-off invalid bug, but if syzkaller is going
> > to be generating classes of corrupted file systems like this, and are
> > going to be complaing about how this causes the kernel to crash, then
> > we have a fundamental syzkaller BUG.
> > 
> If we won't try to recover this case, this specific report would be
> marked as "#syz invalid".

I can do that for this specific case.  Howevre, I'd rather not have to
mark a large number of reports as invalid, if syz is going to produce
a large number of such things.  Which is why I'm raising the questihon
--- is there any way we can make syz smart enough to not raise false
positvies in this case?

In the future I can see the repro attempting to actually do stuff with
the mounted file system, which is why I want to put my foot down now
before the only answer really is adding a kernel-mode fsck before the
file system is allowed to be mounted.

Root is always going to be able to crash the system.  For example,
suppose syzkaller creates a repros which opens /dev/mem and starts
scribbling all over it.  Would we be happy if it started creating
large number of reports for that class of "bug"?

						- Ted

Re: kernel panic: EXT4-fs (device loop0): panic forced after error

[Tetsuo Handa]

Theodore Y. Ts'o wrote:
> On Sun, May 06, 2018 at 02:03:57PM +0900, Tetsuo Handa wrote:
> > 
> > Since syzbot is hitting this error path inside mount() request, calling
> > panic() when something went wrong inside mount() request might be
> > overkill. We can recover without shutting down the system, can't we?
> 
> We could add a full kernel-mode fsck which gets run before mount ---
> the question is how much complexity we want to add.  If SELinux is
> enabled, then we have to check xattr consinsistency, etc., etc.

You are thinking too complicated. I'm not asking for kernel-mode fsck.
I'm just suggesting that mount() request returns an error to the caller
(and the administrator invokes fsck etc. as needed).

We are fixing bugs which occur during mount operation (e.g.

  https://groups.google.com/d/msg/syzkaller-bugs/Yp4q8n-MijM/yDX3zl1XBQAJ
  https://groups.google.com/d/msg/syzkaller-bugs/4C4oiBX8vZ0/W6pi8NdbBgAJ
  https://groups.google.com/d/msg/syzkaller-bugs/QBnHAQBy2pI/ccf-yL5bBgAJ

). And extX filesystem is different from other filesystems that it invokes
error action specified by errors= parameter rather than return an error to
the caller.

Re: kernel panic: EXT4-fs (device loop0): panic forced after error

[Theodore Y. Ts'o]

On Sun, May 06, 2018 at 11:40:10PM +0900, Tetsuo Handa wrote:
> > We could add a full kernel-mode fsck which gets run before mount ---
> > the question is how much complexity we want to add.  If SELinux is
> > enabled, then we have to check xattr consinsistency, etc., etc.
> 
> You are thinking too complicated. I'm not asking for kernel-mode fsck.

That is the logical outcome of what you are asking for.  There will
*always* be a point after which where we can't atomically unwind the
mount, and we have to proceed.  And after that point, when we detect
an inconsistency all we can do is what the system administrator
requested that we do.  Sure, for this particular case, we can
significantly add more complexity and decrease the maintainability of
the code paths involved.  But there will always be another case
(e.g,. xattr's being read by SELinux or IMA) that will happen during
the mount, and are we expected to catch all of those cases?

We do catch a lot of cases where we refuse the mount and complain that
the file system is badly corrupted.  This just doesn't happen to be
one of them.

> I'm just suggesting that mount() request returns an error to the caller
> (and the administrator invokes fsck etc. as needed).
> 
> We are fixing bugs which occur during mount operation (e.g.
> 
>   https://groups.google.com/d/msg/syzkaller-bugs/Yp4q8n-MijM/yDX3zl1XBQAJ
>   https://groups.google.com/d/msg/syzkaller-bugs/4C4oiBX8vZ0/W6pi8NdbBgAJ
>   https://groups.google.com/d/msg/syzkaller-bugs/QBnHAQBy2pI/ccf-yL5bBgAJ

These are different because there are kernel OOPS or warning messages.
This is neither a kernel OOPS or a WARN_ON or BUG_ON.

> And extX filesystem is different from other filesystems that it invokes
> error action specified by errors= parameter rather than return an error to
> the caller.

Syzkaller (or anyone else) can mount the file system with
errors=continue or errors=remount-ro if it wants to override the
requested behavior of the flag in the superblock which is manipulated
by tune2fs.

						- Ted

Re: kernel panic: EXT4-fs (device loop0): panic forced after error

[Dmitry Vyukov]

On Sun, May 6, 2018 at 10:30 PM, Theodore Y. Ts'o <tytso@mit.edu> wrote:
> On Sun, May 06, 2018 at 11:40:10PM +0900, Tetsuo Handa wrote:
>> > We could add a full kernel-mode fsck which gets run before mount ---
>> > the question is how much complexity we want to add.  If SELinux is
>> > enabled, then we have to check xattr consinsistency, etc., etc.
>>
>> You are thinking too complicated. I'm not asking for kernel-mode fsck.
>
> That is the logical outcome of what you are asking for.  There will
> *always* be a point after which where we can't atomically unwind the
> mount, and we have to proceed.  And after that point, when we detect
> an inconsistency all we can do is what the system administrator
> requested that we do.  Sure, for this particular case, we can
> significantly add more complexity and decrease the maintainability of
> the code paths involved.  But there will always be another case
> (e.g,. xattr's being read by SELinux or IMA) that will happen during
> the mount, and are we expected to catch all of those cases?
>
> We do catch a lot of cases where we refuse the mount and complain that
> the file system is badly corrupted.  This just doesn't happen to be
> one of them.
>
>> I'm just suggesting that mount() request returns an error to the caller
>> (and the administrator invokes fsck etc. as needed).
>>
>> We are fixing bugs which occur during mount operation (e.g.
>>
>>   https://groups.google.com/d/msg/syzkaller-bugs/Yp4q8n-MijM/yDX3zl1XBQAJ
>>   https://groups.google.com/d/msg/syzkaller-bugs/4C4oiBX8vZ0/W6pi8NdbBgAJ
>>   https://groups.google.com/d/msg/syzkaller-bugs/QBnHAQBy2pI/ccf-yL5bBgAJ
>
> These are different because there are kernel OOPS or warning messages.
> This is neither a kernel OOPS or a WARN_ON or BUG_ON.
>
>> And extX filesystem is different from other filesystems that it invokes
>> error action specified by errors= parameter rather than return an error to
>> the caller.
>
> Syzkaller (or anyone else) can mount the file system with
> errors=continue or errors=remount-ro if it wants to override the
> requested behavior of the flag in the superblock which is manipulated
> by tune2fs.


Filed https://github.com/google/syzkaller/issues/599 to always pass
errors=remount-ro when mounting ext4.

Re: kernel panic: EXT4-fs (device loop0): panic forced after error

[Dmitry Vyukov]

On Mon, May 14, 2018 at 11:12 AM, Dmitry Vyukov <dvyukov@google.com> wrote:
> On Sun, May 6, 2018 at 10:30 PM, Theodore Y. Ts'o <tytso@mit.edu> wrote:
>> On Sun, May 06, 2018 at 11:40:10PM +0900, Tetsuo Handa wrote:
>>> > We could add a full kernel-mode fsck which gets run before mount ---
>>> > the question is how much complexity we want to add.  If SELinux is
>>> > enabled, then we have to check xattr consinsistency, etc., etc.
>>>
>>> You are thinking too complicated. I'm not asking for kernel-mode fsck.
>>
>> That is the logical outcome of what you are asking for.  There will
>> *always* be a point after which where we can't atomically unwind the
>> mount, and we have to proceed.  And after that point, when we detect
>> an inconsistency all we can do is what the system administrator
>> requested that we do.  Sure, for this particular case, we can
>> significantly add more complexity and decrease the maintainability of
>> the code paths involved.  But there will always be another case
>> (e.g,. xattr's being read by SELinux or IMA) that will happen during
>> the mount, and are we expected to catch all of those cases?
>>
>> We do catch a lot of cases where we refuse the mount and complain that
>> the file system is badly corrupted.  This just doesn't happen to be
>> one of them.
>>
>>> I'm just suggesting that mount() request returns an error to the caller
>>> (and the administrator invokes fsck etc. as needed).
>>>
>>> We are fixing bugs which occur during mount operation (e.g.
>>>
>>>   https://groups.google.com/d/msg/syzkaller-bugs/Yp4q8n-MijM/yDX3zl1XBQAJ
>>>   https://groups.google.com/d/msg/syzkaller-bugs/4C4oiBX8vZ0/W6pi8NdbBgAJ
>>>   https://groups.google.com/d/msg/syzkaller-bugs/QBnHAQBy2pI/ccf-yL5bBgAJ
>>
>> These are different because there are kernel OOPS or warning messages.
>> This is neither a kernel OOPS or a WARN_ON or BUG_ON.
>>
>>> And extX filesystem is different from other filesystems that it invokes
>>> error action specified by errors= parameter rather than return an error to
>>> the caller.
>>
>> Syzkaller (or anyone else) can mount the file system with
>> errors=continue or errors=remount-ro if it wants to override the
>> requested behavior of the flag in the superblock which is manipulated
>> by tune2fs.
>
>
> Filed https://github.com/google/syzkaller/issues/599 to always pass
> errors=remount-ro when mounting ext4.

This was fixed in syzkaller. With this commit:
https://github.com/google/syzkaller/commit/deb0e69e1028ba3152631c3f1d2fac98c12e33a5
syzkaller should always pass errors=continue when mounting ext2/3/4.

#syz invalid