Dear Miklos, Am 17.07.20 um 15:03 schrieb Miklos Szeredi: > On Fri, Jul 17, 2020 at 2:52 PM Matthew Wilcox wrote: >> >> On Fri, Jul 17, 2020 at 02:39:03PM +0200, Miklos Szeredi wrote: >>> On Fri, Jul 17, 2020 at 10:07 AM Paul Menzel wrote: >>>> [105591.121285] INFO: task ls:21242 blocked for more than 120 seconds. >>>> [105591.121293] Not tainted 5.7.0-1-amd64 #1 Debian 5.7.6-1 >>>> [105591.121295] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. >>>> [105591.121298] ls D 0 21242 778 0x00004004 >>>> [105591.121304] Call Trace: >>>> [105591.121319] __schedule+0x2da/0x770 >>>> [105591.121326] schedule+0x4a/0xb0 >>>> [105591.121339] request_wait_answer+0x122/0x210 [fuse] >>>> [105591.121349] ? finish_wait+0x80/0x80 >>>> [105591.121357] fuse_simple_request+0x198/0x290 [fuse] >>>> [105591.121366] fuse_do_getattr+0xcf/0x2c0 [fuse] >>>> [105591.121376] vfs_statx+0x96/0xe0 >>>> >>>> The `ls` process cannot be killed. The SSHFS issue *Fuse sshfs blocks >>>> standby (Visual Studio Code?)* from 2018 already reported this for Linux >>>> 4.17, and the SSHFS developers asked to report this to the Linux kernel. >>> >>> This is a very old and fundamental issue. Theoretical solution for >>> killing the stuck process exists, but it's not trivial and since the >>> above mentioned workarounds work well in all cases it's not high >>> priority right now. Unfortunately, it become quite a lot more annoying for the user, as it seems to prevent the system from suspending. ``` [ 0.000000] Linux version 5.16.0-1-amd64 (debian-kernel@lists.debian.org) (gcc-11 (Debian 11.2.0-16) 11.2.0, GNU ld (GNU Binutils for Debian) 2.37.90.20220130) #1 SMP PREEMPT Debian 5.16.7-2 (2022-02-09) […] [ 1854.968846] PM: suspend entry (s2idle) [ 1854.974879] Filesystems sync: 0.006 seconds [ 1854.974957] (NULL device *): firmware: direct-loading firmware i915/kbl_dmc_ver1_04.bin [ 1854.975025] (NULL device *): firmware: direct-loading firmware regulatory.db.p7s [ 1854.975027] (NULL device *): firmware: direct-loading firmware regulatory.db [ 1854.975036] (NULL device *): firmware: direct-loading firmware intel/ibt-17-16-1.ddc [ 1854.975226] (NULL device *): firmware: direct-loading firmware intel/ibt-17-16-1.sfi [ 1854.975336] (NULL device *): firmware: direct-loading firmware iwlwifi-9000-pu-b0-jf-b0-46.ucode [ 1855.243187] Freezing user space processes ... [ 1875.251939] Freezing of tasks failed after 20.008 seconds (1 tasks refusing to freeze, wq_busy=0): [ 1875.252120] task:git state:D stack: 0 pid: 4608 ppid: 933 flags:0x00004004 [ 1875.252133] Call Trace: [ 1875.252137] [ 1875.252147] __schedule+0x30a/0x9f0 [ 1875.252164] schedule+0x4e/0xc0 [ 1875.252175] request_wait_answer+0xa1/0x210 [fuse] [ 1875.252197] ? do_wait_intr+0xa0/0xa0 [ 1875.252206] fuse_simple_request+0x19b/0x310 [fuse] [ 1875.252226] fuse_lookup_name+0xef/0x200 [fuse] [ 1875.252246] ? mod_objcg_state+0x100/0x300 [ 1875.252262] fuse_lookup+0x68/0x190 [fuse] [ 1875.252285] __lookup_slow+0x81/0x140 [ 1875.252296] walk_component+0x154/0x1d0 [ 1875.252305] ? fuse_permission+0x34/0x1c0 [fuse] [ 1875.252325] link_path_walk.part.0.constprop.0+0x249/0x380 [ 1875.252335] ? path_init+0x57/0x3f0 [ 1875.252343] path_lookupat+0x3e/0x1b0 [ 1875.252352] filename_lookup+0xcf/0x1d0 [ 1875.252364] ? __wake_up_common+0x7d/0x180 [ 1875.252369] ? __check_object_size+0x136/0x150 [ 1875.252378] ? strncpy_from_user+0x4e/0x140 [ 1875.252388] user_path_at_empty+0x3a/0x50 [ 1875.252397] vfs_statx+0x74/0x130 [ 1875.252407] ? __fput+0xff/0x250 [ 1875.252415] __do_sys_newfstatat+0x31/0x70 [ 1875.252426] ? xfd_validate_state+0x1e/0x80 [ 1875.252434] ? restore_fpregs_from_fpstate+0x41/0xc0 [ 1875.252440] ? fpregs_restore_userregs+0x53/0xd0 [ 1875.252446] ? exit_to_user_mode_prepare+0x1a5/0x210 [ 1875.252454] do_syscall_64+0x38/0xc0 [ 1875.252462] entry_SYSCALL_64_after_hwframe+0x44/0xae [ 1875.252474] RIP: 0033:0x7f377e314aba [ 1875.252482] RSP: 002b:00007ffdd453c338 EFLAGS: 00000246 ORIG_RAX: 0000000000000106 [ 1875.252490] RAX: ffffffffffffffda RBX: 0000563397362124 RCX: 00007f377e314aba [ 1875.252495] RDX: 00007ffdd453c370 RSI: 0000563397b8fed0 RDI: 00000000ffffff9c [ 1875.252499] RBP: 0000563397b8fed0 R08: 00007f377e3f5ba0 R09: 0000563397a8ba90 [ 1875.252502] R10: 0000000000000100 R11: 0000000000000246 R12: 00007ffdd453c530 [ 1875.252506] R13: 00007ffdd453cca0 R14: 00005633973435a0 R15: 0000563397b1dc40 [ 1875.252514] [ 1875.252524] OOM killer enabled. [ 1875.252526] Restarting tasks ... done. [ 1875.322190] PM: suspend exit [ 1875.322355] PM: suspend entry (s2idle) [ 1875.329266] Filesystems sync: 0.006 seconds [ 1875.329656] Freezing user space processes ... [ 1895.339239] Freezing of tasks failed after 20.009 seconds (1 tasks refusing to freeze, wq_busy=0): [ 1895.339423] task:git state:D stack: 0 pid: 4608 ppid: 933 flags:0x00004004 [ 1895.339436] Call Trace: [ 1895.339440] [ 1895.339450] __schedule+0x30a/0x9f0 [ 1895.339467] schedule+0x4e/0xc0 [ 1895.339478] request_wait_answer+0xa1/0x210 [fuse] [ 1895.339500] ? do_wait_intr+0xa0/0xa0 [ 1895.339510] fuse_simple_request+0x19b/0x310 [fuse] [ 1895.339529] fuse_lookup_name+0xef/0x200 [fuse] [ 1895.339549] ? mod_objcg_state+0x100/0x300 [ 1895.339565] fuse_lookup+0x68/0x190 [fuse] [ 1895.339588] __lookup_slow+0x81/0x140 [ 1895.339599] walk_component+0x154/0x1d0 [ 1895.339609] ? fuse_permission+0x34/0x1c0 [fuse] [ 1895.339629] link_path_walk.part.0.constprop.0+0x249/0x380 [ 1895.339638] ? path_init+0x57/0x3f0 [ 1895.339647] path_lookupat+0x3e/0x1b0 [ 1895.339656] filename_lookup+0xcf/0x1d0 [ 1895.339667] ? __wake_up_common+0x7d/0x180 [ 1895.339673] ? __check_object_size+0x136/0x150 [ 1895.339681] ? strncpy_from_user+0x4e/0x140 [ 1895.339691] user_path_at_empty+0x3a/0x50 [ 1895.339701] vfs_statx+0x74/0x130 [ 1895.339711] ? __fput+0xff/0x250 [ 1895.339719] __do_sys_newfstatat+0x31/0x70 [ 1895.339730] ? xfd_validate_state+0x1e/0x80 [ 1895.339738] ? restore_fpregs_from_fpstate+0x41/0xc0 [ 1895.339744] ? fpregs_restore_userregs+0x53/0xd0 [ 1895.339750] ? exit_to_user_mode_prepare+0x1a5/0x210 [ 1895.339759] do_syscall_64+0x38/0xc0 [ 1895.339767] entry_SYSCALL_64_after_hwframe+0x44/0xae [ 1895.339780] RIP: 0033:0x7f377e314aba [ 1895.339787] RSP: 002b:00007ffdd453c338 EFLAGS: 00000246 ORIG_RAX: 0000000000000106 [ 1895.339795] RAX: ffffffffffffffda RBX: 0000563397362124 RCX: 00007f377e314aba [ 1895.339799] RDX: 00007ffdd453c370 RSI: 0000563397b8fed0 RDI: 00000000ffffff9c [ 1895.339803] RBP: 0000563397b8fed0 R08: 00007f377e3f5ba0 R09: 0000563397a8ba90 [ 1895.339807] R10: 0000000000000100 R11: 0000000000000246 R12: 00007ffdd453c530 [ 1895.339811] R13: 00007ffdd453cca0 R14: 00005633973435a0 R15: 0000563397b1dc40 [ 1895.339819] [ 1895.339830] OOM killer enabled. [ 1895.339832] Restarting tasks ... done. [ 1895.401161] PM: suspend exit ``` Can this be worked around? >> What? All you need to do is return -EINTR from fuse_do_getattr() if >> there's a fatal signal. What "fundamental issue"? > > TL;DR: the fundamental issue is not with getattr, but with ops that > hold locks. We could make an exception for ops that do not hold > locks, but it would not be a solution to the problem, and as I said > this is not something we can't live with. > > The fundamental issue is that a task killed while the userspace > filesystem is still performing that operation will release the vfs > lock and allow another op requiring that lock tobe sent to the > userspace filesystem. This may confuse the userspace filesystem > otherwise relying on the locking and quite possibly result in fs > corruption. > > To fix this, we need to add shadow locking somewhere that duplicates > the vfs locks but are only released if userspace finished processing > the request. Best place to put the shadow locks is probably in the > kernel. Kind regards, Paul