LKML Archive on lore.kernel.org
help / color / mirror / Atom feed
* [PATCH] btrfs: reset replace target device to allocation state on close
@ 2021-08-20 17:50 Desmond Cheong Zhi Xi
  2021-08-23  7:44 ` Anand Jain
  2021-08-23 11:56 ` David Sterba
  0 siblings, 2 replies; 3+ messages in thread
From: Desmond Cheong Zhi Xi @ 2021-08-20 17:50 UTC (permalink / raw)
  To: clm, josef, dsterba
  Cc: Desmond Cheong Zhi Xi, linux-btrfs, linux-kernel, skhan, gregkh,
	linux-kernel-mentees

This crash was observed with a failed assertion on device close:

  BTRFS: Transaction aborted (error -28)
  WARNING: CPU: 1 PID: 3902 at fs/btrfs/extent-tree.c:2150 btrfs_run_delayed_refs+0x1d2/0x1e0 [btrfs]
  Modules linked in: btrfs blake2b_generic libcrc32c crc32c_intel xor zstd_decompress zstd_compress xxhash lzo_compress lzo_decompress raid6_pq loop
  CPU: 1 PID: 3902 Comm: kworker/u8:4 Not tainted 5.14.0-rc5-default+ #1532
  Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.12.0-59-gc9ba527-rebuilt.opensuse.org 04/01/2014
  Workqueue: events_unbound btrfs_async_reclaim_metadata_space [btrfs]
  RIP: 0010:btrfs_run_delayed_refs+0x1d2/0x1e0 [btrfs]
  RSP: 0018:ffffb7a5452d7d80 EFLAGS: 00010282
  RAX: 0000000000000000 RBX: 0000000000000003 RCX: 0000000000000000
  RDX: 0000000000000001 RSI: ffffffffabee13c4 RDI: 00000000ffffffff
  RBP: ffff97834176a378 R08: 0000000000000001 R09: 0000000000000001
  R10: 0000000000000000 R11: 0000000000000001 R12: ffff97835195d388
  R13: 0000000005b08000 R14: ffff978385484000 R15: 000000000000016c
  FS:  0000000000000000(0000) GS:ffff9783bd800000(0000) knlGS:0000000000000000
  CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  CR2: 000056190d003fe8 CR3: 000000002a81e005 CR4: 0000000000170ea0
  Call Trace:
   flush_space+0x197/0x2f0 [btrfs]
   btrfs_async_reclaim_metadata_space+0x139/0x300 [btrfs]
   process_one_work+0x262/0x5e0
   worker_thread+0x4c/0x320
   ? process_one_work+0x5e0/0x5e0
   kthread+0x144/0x170
   ? set_kthread_struct+0x40/0x40
   ret_from_fork+0x1f/0x30
  irq event stamp: 19334989
  hardirqs last  enabled at (19334997): [<ffffffffab0e0c87>] console_unlock+0x2b7/0x400
  hardirqs last disabled at (19335006): [<ffffffffab0e0d0d>] console_unlock+0x33d/0x400
  softirqs last  enabled at (19334900): [<ffffffffaba0030d>] __do_softirq+0x30d/0x574
  softirqs last disabled at (19334893): [<ffffffffab0721ec>] irq_exit_rcu+0x12c/0x140
  ---[ end trace 45939e308e0dd3c7 ]---
  BTRFS: error (device vdd) in btrfs_run_delayed_refs:2150: errno=-28 No space left
  BTRFS info (device vdd): forced readonly
  BTRFS warning (device vdd): failed setting block group ro: -30
  BTRFS info (device vdd): suspending dev_replace for unmount
  assertion failed: !test_bit(BTRFS_DEV_STATE_REPLACE_TGT, &device->dev_state), in fs/btrfs/volumes.c:1150
  ------------[ cut here ]------------
  kernel BUG at fs/btrfs/ctree.h:3431!
  invalid opcode: 0000 [#1] PREEMPT SMP
  CPU: 1 PID: 3982 Comm: umount Tainted: G        W         5.14.0-rc5-default+ #1532
  Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.12.0-59-gc9ba527-rebuilt.opensuse.org 04/01/2014
  RIP: 0010:assertfail.constprop.0+0x18/0x1a [btrfs]
  RSP: 0018:ffffb7a5454c7db8 EFLAGS: 00010246
  RAX: 0000000000000068 RBX: ffff978364b91c00 RCX: 0000000000000000
  RDX: 0000000000000000 RSI: ffffffffabee13c4 RDI: 00000000ffffffff
  RBP: ffff9783523a4c00 R08: 0000000000000001 R09: 0000000000000001
  R10: 0000000000000000 R11: 0000000000000001 R12: ffff9783523a4d18
  R13: 0000000000000000 R14: 0000000000000004 R15: 0000000000000003
  FS:  00007f61c8f42800(0000) GS:ffff9783bd800000(0000) knlGS:0000000000000000
  CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  CR2: 000056190cffa810 CR3: 0000000030b96002 CR4: 0000000000170ea0
  Call Trace:
   btrfs_close_one_device.cold+0x11/0x55 [btrfs]
   close_fs_devices+0x44/0xb0 [btrfs]
   btrfs_close_devices+0x48/0x160 [btrfs]
   generic_shutdown_super+0x69/0x100
   kill_anon_super+0x14/0x30
   btrfs_kill_super+0x12/0x20 [btrfs]
   deactivate_locked_super+0x2c/0xa0
   cleanup_mnt+0x144/0x1b0
   task_work_run+0x59/0xa0
   exit_to_user_mode_loop+0xe7/0xf0
   exit_to_user_mode_prepare+0xaf/0xf0
   syscall_exit_to_user_mode+0x19/0x50
   do_syscall_64+0x4a/0x90
   entry_SYSCALL_64_after_hwframe+0x44/0xae

This happens when close_ctree is called while a dev_replace hasn't
completed. In close_ctree, we suspend the dev_replace, but keep the
replace target around so that we can resume the dev_replace procedure
when we mount the root again. This is the call trace:

  close_ctree():
    btrfs_dev_replace_suspend_for_unmount();
    btrfs_close_devices():
      btrfs_close_fs_devices():
        btrfs_close_one_device():
          ASSERT(!test_bit(BTRFS_DEV_STATE_REPLACE_TGT,
                 &device->dev_state));

However, since the replace target sticks around, there is a device
with BTRFS_DEV_STATE_REPLACE_TGT set on close, and we fail the
assertion in btrfs_close_one_device.

To fix this, if we come across the replace target device when
closing, we should properly reset it back to allocation state. This
fix also ensures that if a non-target device has a corrupted state and
has the BTRFS_DEV_STATE_REPLACE_TGT bit set, the assertion will still
catch the error.

Reported-by: David Sterba <dsterba@suse.com>
Signed-off-by: Desmond Cheong Zhi Xi <desmondcheongzx@gmail.com>
---
 fs/btrfs/volumes.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/fs/btrfs/volumes.c b/fs/btrfs/volumes.c
index 70f94b75f25a..a5afebb78ecf 100644
--- a/fs/btrfs/volumes.c
+++ b/fs/btrfs/volumes.c
@@ -1130,6 +1130,9 @@ static void btrfs_close_one_device(struct btrfs_device *device)
 		fs_devices->rw_devices--;
 	}
 
+	if (device->devid == BTRFS_DEV_REPLACE_DEVID)
+		clear_bit(BTRFS_DEV_STATE_REPLACE_TGT, &device->dev_state);
+
 	if (test_bit(BTRFS_DEV_STATE_MISSING, &device->dev_state))
 		fs_devices->missing_devices--;
 
-- 
2.25.1


^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: [PATCH] btrfs: reset replace target device to allocation state on close
  2021-08-20 17:50 [PATCH] btrfs: reset replace target device to allocation state on close Desmond Cheong Zhi Xi
@ 2021-08-23  7:44 ` Anand Jain
  2021-08-23 11:56 ` David Sterba
  1 sibling, 0 replies; 3+ messages in thread
From: Anand Jain @ 2021-08-23  7:44 UTC (permalink / raw)
  To: Desmond Cheong Zhi Xi, linux-btrfs
  Cc: linux-kernel, clm, skhan, gregkh, linux-kernel-mentees, josef, dsterba

On 21/08/2021 01:50, Desmond Cheong Zhi Xi wrote:
> This crash was observed with a failed assertion on device close:
> 
>    BTRFS: Transaction aborted (error -28)
>    WARNING: CPU: 1 PID: 3902 at fs/btrfs/extent-tree.c:2150 btrfs_run_delayed_refs+0x1d2/0x1e0 [btrfs]
>    Modules linked in: btrfs blake2b_generic libcrc32c crc32c_intel xor zstd_decompress zstd_compress xxhash lzo_compress lzo_decompress raid6_pq loop
>    CPU: 1 PID: 3902 Comm: kworker/u8:4 Not tainted 5.14.0-rc5-default+ #1532
>    Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.12.0-59-gc9ba527-rebuilt.opensuse.org 04/01/2014
>    Workqueue: events_unbound btrfs_async_reclaim_metadata_space [btrfs]
>    RIP: 0010:btrfs_run_delayed_refs+0x1d2/0x1e0 [btrfs]
>    RSP: 0018:ffffb7a5452d7d80 EFLAGS: 00010282
>    RAX: 0000000000000000 RBX: 0000000000000003 RCX: 0000000000000000
>    RDX: 0000000000000001 RSI: ffffffffabee13c4 RDI: 00000000ffffffff
>    RBP: ffff97834176a378 R08: 0000000000000001 R09: 0000000000000001
>    R10: 0000000000000000 R11: 0000000000000001 R12: ffff97835195d388
>    R13: 0000000005b08000 R14: ffff978385484000 R15: 000000000000016c
>    FS:  0000000000000000(0000) GS:ffff9783bd800000(0000) knlGS:0000000000000000
>    CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>    CR2: 000056190d003fe8 CR3: 000000002a81e005 CR4: 0000000000170ea0
>    Call Trace:
>     flush_space+0x197/0x2f0 [btrfs]
>     btrfs_async_reclaim_metadata_space+0x139/0x300 [btrfs]
>     process_one_work+0x262/0x5e0
>     worker_thread+0x4c/0x320
>     ? process_one_work+0x5e0/0x5e0
>     kthread+0x144/0x170
>     ? set_kthread_struct+0x40/0x40
>     ret_from_fork+0x1f/0x30
>    irq event stamp: 19334989
>    hardirqs last  enabled at (19334997): [<ffffffffab0e0c87>] console_unlock+0x2b7/0x400
>    hardirqs last disabled at (19335006): [<ffffffffab0e0d0d>] console_unlock+0x33d/0x400
>    softirqs last  enabled at (19334900): [<ffffffffaba0030d>] __do_softirq+0x30d/0x574
>    softirqs last disabled at (19334893): [<ffffffffab0721ec>] irq_exit_rcu+0x12c/0x140
>    ---[ end trace 45939e308e0dd3c7 ]---
>    BTRFS: error (device vdd) in btrfs_run_delayed_refs:2150: errno=-28 No space left
>    BTRFS info (device vdd): forced readonly
>    BTRFS warning (device vdd): failed setting block group ro: -30
>    BTRFS info (device vdd): suspending dev_replace for unmount
>    assertion failed: !test_bit(BTRFS_DEV_STATE_REPLACE_TGT, &device->dev_state), in fs/btrfs/volumes.c:1150
>    ------------[ cut here ]------------
>    kernel BUG at fs/btrfs/ctree.h:3431!
>    invalid opcode: 0000 [#1] PREEMPT SMP
>    CPU: 1 PID: 3982 Comm: umount Tainted: G        W         5.14.0-rc5-default+ #1532
>    Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.12.0-59-gc9ba527-rebuilt.opensuse.org 04/01/2014
>    RIP: 0010:assertfail.constprop.0+0x18/0x1a [btrfs]
>    RSP: 0018:ffffb7a5454c7db8 EFLAGS: 00010246
>    RAX: 0000000000000068 RBX: ffff978364b91c00 RCX: 0000000000000000
>    RDX: 0000000000000000 RSI: ffffffffabee13c4 RDI: 00000000ffffffff
>    RBP: ffff9783523a4c00 R08: 0000000000000001 R09: 0000000000000001
>    R10: 0000000000000000 R11: 0000000000000001 R12: ffff9783523a4d18
>    R13: 0000000000000000 R14: 0000000000000004 R15: 0000000000000003
>    FS:  00007f61c8f42800(0000) GS:ffff9783bd800000(0000) knlGS:0000000000000000
>    CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>    CR2: 000056190cffa810 CR3: 0000000030b96002 CR4: 0000000000170ea0
>    Call Trace:
>     btrfs_close_one_device.cold+0x11/0x55 [btrfs]
>     close_fs_devices+0x44/0xb0 [btrfs]
>     btrfs_close_devices+0x48/0x160 [btrfs]
>     generic_shutdown_super+0x69/0x100
>     kill_anon_super+0x14/0x30
>     btrfs_kill_super+0x12/0x20 [btrfs]
>     deactivate_locked_super+0x2c/0xa0
>     cleanup_mnt+0x144/0x1b0
>     task_work_run+0x59/0xa0
>     exit_to_user_mode_loop+0xe7/0xf0
>     exit_to_user_mode_prepare+0xaf/0xf0
>     syscall_exit_to_user_mode+0x19/0x50
>     do_syscall_64+0x4a/0x90
>     entry_SYSCALL_64_after_hwframe+0x44/0xae
> 
> This happens when close_ctree is called while a dev_replace hasn't
> completed. In close_ctree, we suspend the dev_replace, but keep the
> replace target around so that we can resume the dev_replace procedure
> when we mount the root again. This is the call trace:
> 
>    close_ctree():
>      btrfs_dev_replace_suspend_for_unmount();
>      btrfs_close_devices():
>        btrfs_close_fs_devices():
>          btrfs_close_one_device():
>            ASSERT(!test_bit(BTRFS_DEV_STATE_REPLACE_TGT,
>                   &device->dev_state));
> 
> However, since the replace target sticks around, there is a device
> with BTRFS_DEV_STATE_REPLACE_TGT set on close, and we fail the
> assertion in btrfs_close_one_device.
> 
> To fix this, if we come across the replace target device when
> closing, we should properly reset it back to allocation state. This
> fix also ensures that if a non-target device has a corrupted state and
> has the BTRFS_DEV_STATE_REPLACE_TGT bit set, the assertion will still
> catch the error.
> 
> Reported-by: David Sterba <dsterba@suse.com>
> Signed-off-by: Desmond Cheong Zhi Xi <desmondcheongzx@gmail.com>
> ---
>   fs/btrfs/volumes.c | 3 +++
>   1 file changed, 3 insertions(+)
> 
> diff --git a/fs/btrfs/volumes.c b/fs/btrfs/volumes.c
> index 70f94b75f25a..a5afebb78ecf 100644
> --- a/fs/btrfs/volumes.c
> +++ b/fs/btrfs/volumes.c
> @@ -1130,6 +1130,9 @@ static void btrfs_close_one_device(struct btrfs_device *device)
>   		fs_devices->rw_devices--;
>   	}
>   
> +	if (device->devid == BTRFS_DEV_REPLACE_DEVID)
> +		clear_bit(BTRFS_DEV_STATE_REPLACE_TGT, &device->dev_state);
> +



This bug is reminiscent of the commit 321f69f86a0f (btrfs: reset the 
device back to allocation state when removing).
Before this commit, we freed the btrfs_device at every close. And alloc 
a new btrfs_device, so all the dev_state resets back to 0.

Moving on, we still have other dev_state which aren't reset back to 0 at 
the time of close. But it can be a separate cleanup patch when needed.

For now, looks good for me.

Reviewed-by: Anand Jain <anand.jain@oracle.com>

Thanks, Anand



>   	if (test_bit(BTRFS_DEV_STATE_MISSING, &device->dev_state))
>   		fs_devices->missing_devices--;
>   
> 


^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: [PATCH] btrfs: reset replace target device to allocation state on close
  2021-08-20 17:50 [PATCH] btrfs: reset replace target device to allocation state on close Desmond Cheong Zhi Xi
  2021-08-23  7:44 ` Anand Jain
@ 2021-08-23 11:56 ` David Sterba
  1 sibling, 0 replies; 3+ messages in thread
From: David Sterba @ 2021-08-23 11:56 UTC (permalink / raw)
  To: Desmond Cheong Zhi Xi
  Cc: clm, josef, dsterba, linux-btrfs, linux-kernel, skhan, gregkh,
	linux-kernel-mentees

On Sat, Aug 21, 2021 at 01:50:40AM +0800, Desmond Cheong Zhi Xi wrote:
> This crash was observed with a failed assertion on device close:
> 
>   BTRFS: Transaction aborted (error -28)
>   WARNING: CPU: 1 PID: 3902 at fs/btrfs/extent-tree.c:2150 btrfs_run_delayed_refs+0x1d2/0x1e0 [btrfs]
>   Modules linked in: btrfs blake2b_generic libcrc32c crc32c_intel xor zstd_decompress zstd_compress xxhash lzo_compress lzo_decompress raid6_pq loop
>   CPU: 1 PID: 3902 Comm: kworker/u8:4 Not tainted 5.14.0-rc5-default+ #1532
>   Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.12.0-59-gc9ba527-rebuilt.opensuse.org 04/01/2014
>   Workqueue: events_unbound btrfs_async_reclaim_metadata_space [btrfs]
>   RIP: 0010:btrfs_run_delayed_refs+0x1d2/0x1e0 [btrfs]
>   RSP: 0018:ffffb7a5452d7d80 EFLAGS: 00010282
>   RAX: 0000000000000000 RBX: 0000000000000003 RCX: 0000000000000000
>   RDX: 0000000000000001 RSI: ffffffffabee13c4 RDI: 00000000ffffffff
>   RBP: ffff97834176a378 R08: 0000000000000001 R09: 0000000000000001
>   R10: 0000000000000000 R11: 0000000000000001 R12: ffff97835195d388
>   R13: 0000000005b08000 R14: ffff978385484000 R15: 000000000000016c
>   FS:  0000000000000000(0000) GS:ffff9783bd800000(0000) knlGS:0000000000000000
>   CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>   CR2: 000056190d003fe8 CR3: 000000002a81e005 CR4: 0000000000170ea0
>   Call Trace:
>    flush_space+0x197/0x2f0 [btrfs]
>    btrfs_async_reclaim_metadata_space+0x139/0x300 [btrfs]
>    process_one_work+0x262/0x5e0
>    worker_thread+0x4c/0x320
>    ? process_one_work+0x5e0/0x5e0
>    kthread+0x144/0x170
>    ? set_kthread_struct+0x40/0x40
>    ret_from_fork+0x1f/0x30
>   irq event stamp: 19334989
>   hardirqs last  enabled at (19334997): [<ffffffffab0e0c87>] console_unlock+0x2b7/0x400
>   hardirqs last disabled at (19335006): [<ffffffffab0e0d0d>] console_unlock+0x33d/0x400
>   softirqs last  enabled at (19334900): [<ffffffffaba0030d>] __do_softirq+0x30d/0x574
>   softirqs last disabled at (19334893): [<ffffffffab0721ec>] irq_exit_rcu+0x12c/0x140
>   ---[ end trace 45939e308e0dd3c7 ]---
>   BTRFS: error (device vdd) in btrfs_run_delayed_refs:2150: errno=-28 No space left
>   BTRFS info (device vdd): forced readonly
>   BTRFS warning (device vdd): failed setting block group ro: -30
>   BTRFS info (device vdd): suspending dev_replace for unmount
>   assertion failed: !test_bit(BTRFS_DEV_STATE_REPLACE_TGT, &device->dev_state), in fs/btrfs/volumes.c:1150
>   ------------[ cut here ]------------
>   kernel BUG at fs/btrfs/ctree.h:3431!
>   invalid opcode: 0000 [#1] PREEMPT SMP
>   CPU: 1 PID: 3982 Comm: umount Tainted: G        W         5.14.0-rc5-default+ #1532
>   Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.12.0-59-gc9ba527-rebuilt.opensuse.org 04/01/2014
>   RIP: 0010:assertfail.constprop.0+0x18/0x1a [btrfs]
>   RSP: 0018:ffffb7a5454c7db8 EFLAGS: 00010246
>   RAX: 0000000000000068 RBX: ffff978364b91c00 RCX: 0000000000000000
>   RDX: 0000000000000000 RSI: ffffffffabee13c4 RDI: 00000000ffffffff
>   RBP: ffff9783523a4c00 R08: 0000000000000001 R09: 0000000000000001
>   R10: 0000000000000000 R11: 0000000000000001 R12: ffff9783523a4d18
>   R13: 0000000000000000 R14: 0000000000000004 R15: 0000000000000003
>   FS:  00007f61c8f42800(0000) GS:ffff9783bd800000(0000) knlGS:0000000000000000
>   CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>   CR2: 000056190cffa810 CR3: 0000000030b96002 CR4: 0000000000170ea0
>   Call Trace:
>    btrfs_close_one_device.cold+0x11/0x55 [btrfs]
>    close_fs_devices+0x44/0xb0 [btrfs]
>    btrfs_close_devices+0x48/0x160 [btrfs]
>    generic_shutdown_super+0x69/0x100
>    kill_anon_super+0x14/0x30
>    btrfs_kill_super+0x12/0x20 [btrfs]
>    deactivate_locked_super+0x2c/0xa0
>    cleanup_mnt+0x144/0x1b0
>    task_work_run+0x59/0xa0
>    exit_to_user_mode_loop+0xe7/0xf0
>    exit_to_user_mode_prepare+0xaf/0xf0
>    syscall_exit_to_user_mode+0x19/0x50
>    do_syscall_64+0x4a/0x90
>    entry_SYSCALL_64_after_hwframe+0x44/0xae
> 
> This happens when close_ctree is called while a dev_replace hasn't
> completed. In close_ctree, we suspend the dev_replace, but keep the
> replace target around so that we can resume the dev_replace procedure
> when we mount the root again. This is the call trace:
> 
>   close_ctree():
>     btrfs_dev_replace_suspend_for_unmount();
>     btrfs_close_devices():
>       btrfs_close_fs_devices():
>         btrfs_close_one_device():
>           ASSERT(!test_bit(BTRFS_DEV_STATE_REPLACE_TGT,
>                  &device->dev_state));
> 
> However, since the replace target sticks around, there is a device
> with BTRFS_DEV_STATE_REPLACE_TGT set on close, and we fail the
> assertion in btrfs_close_one_device.
> 
> To fix this, if we come across the replace target device when
> closing, we should properly reset it back to allocation state. This
> fix also ensures that if a non-target device has a corrupted state and
> has the BTRFS_DEV_STATE_REPLACE_TGT bit set, the assertion will still
> catch the error.
> 
> Reported-by: David Sterba <dsterba@suse.com>
> Signed-off-by: Desmond Cheong Zhi Xi <desmondcheongzx@gmail.com>

Thanks, added to misc-next.

^ permalink raw reply	[flat|nested] 3+ messages in thread

end of thread, other threads:[~2021-08-23 11:59 UTC | newest]

Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-08-20 17:50 [PATCH] btrfs: reset replace target device to allocation state on close Desmond Cheong Zhi Xi
2021-08-23  7:44 ` Anand Jain
2021-08-23 11:56 ` David Sterba

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).