LKML Archive on lore.kernel.org
help / color / mirror / Atom feed
* [2.6.22] circular lock detected
@ 2007-08-24 21:00 Folkert van Heusden
  2007-08-26 23:03 ` Michal Piotrowski
  2007-09-02 11:55 ` Andrew Morton
  0 siblings, 2 replies; 11+ messages in thread
From: Folkert van Heusden @ 2007-08-24 21:00 UTC (permalink / raw)
  To: linux-kernel

Hi,

2.6.22 kernel with hyperthreading enabled only ext3 filesystems (2).

[  346.314640] =======================================================
[  346.314758] [ INFO: possible circular locking dependency detected ]
[  346.314815] 2.6.22 #5
[  346.314862] -------------------------------------------------------
[  346.314920] tor/2421 is trying to acquire lock:
[  346.314973]  (tty_mutex){--..}, at: [<c121aa83>] mutex_lock+0x8/0xa
[  346.315193] 
[  346.315195] but task is already holding lock:
[  346.316203]  (&s->s_dquot.dqptr_sem){----}, at: [<c10a485f>] dquot_alloc_space+0x50/0x189
[  346.316457] 
[  346.316458] which lock already depends on the new lock.
[  346.316461] 
[  346.316585] 
[  346.316586] the existing dependency chain (in reverse order) is:
[  346.316675] 
[  346.316676] -> #4 (&s->s_dquot.dqptr_sem){----}:
[  346.316871]        [<c103b2ee>] check_prev_add+0x15b/0x281
[  346.317204]        [<c103b49f>] check_prevs_add+0x8b/0xe8
[  346.317535]        [<c103ce2a>] __lock_acquire+0x65d/0xb51
[  346.317873]        [<c103d786>] lock_acquire+0x62/0x81
[  346.318195]        [<c10341af>] down_read+0x2b/0x3d
[  346.318527]        [<c10a485f>] dquot_alloc_space+0x50/0x189
[  346.318854]        [<c10b5c55>] ext3_new_blocks+0x44b/0x5a2
[  346.319187]        [<c10b81e4>] ext3_alloc_blocks+0x40/0xdf
[  346.319513]        [<c10b82d3>] ext3_alloc_branch+0x50/0x21b
[  346.319850]        [<c10b87b1>] ext3_get_blocks_handle+0x1b8/0x367
[  346.320184]        [<c10b8ada>] ext3_getblk+0x97/0x228
[  346.320530]        [<c10b8c85>] ext3_bread+0x1a/0x78
[  346.320870]        [<c10bde64>] ext3_mkdir+0xf4/0x270
[  346.321188]        [<c107cdf6>] vfs_mkdir+0xb3/0x161
[  346.321506]        [<c107cf30>] sys_mkdirat+0x8c/0xc4
[  346.321820]        [<c107cf88>] sys_mkdir+0x20/0x22
[  346.322131]        [<c1003eea>] syscall_call+0x7/0xb
[  346.322446]        [<ffffffff>] 0xffffffff
[  346.322779] 
[  346.322780] -> #3 (&ei->truncate_mutex){--..}:
[  346.322965]        [<c103b2ee>] check_prev_add+0x15b/0x281
[  346.323278]        [<c103b49f>] check_prevs_add+0x8b/0xe8
[  346.323602]        [<c103ce2a>] __lock_acquire+0x65d/0xb51
[  346.323919]        [<c103d786>] lock_acquire+0x62/0x81
[  346.324234]        [<c121ab04>] __mutex_lock_slowpath+0x75/0x28c
[  346.324857]        [<c121aa83>] mutex_lock+0x8/0xa
[  346.325177]        [<c10ba380>] ext3_truncate+0x170/0x468
[  346.325501]        [<c1060e9a>] vmtruncate+0xa6/0x116
[  346.325828]        [<c1087937>] inode_setattr+0x145/0x16c
[  346.326150]        [<c10bb301>] ext3_setattr+0x150/0x22f
[  346.326479]        [<c1087cb0>] notify_change+0x352/0x386
[  346.326802]        [<c1071af4>] do_truncate+0x5c/0x7f
[  346.327116]        [<c107c2e3>] may_open+0x1ec/0x231
[  346.327434]        [<c107c4a4>] open_namei+0xdd/0x5a2
[  346.327770]        [<c107275b>] do_filp_open+0x2c/0x53
[  346.328090]        [<c1072a90>] do_sys_open+0x52/0xd8
[  346.328408]        [<c1072b32>] sys_open+0x1c/0x1e
[  346.328727]        [<c1003eea>] syscall_call+0x7/0xb
[  346.329042]        [<ffffffff>] 0xffffffff
[  346.329368] 
[  346.329370] -> #2 (&inode->i_alloc_sem){--..}:
[  346.329556]        [<c103b2ee>] check_prev_add+0x15b/0x281
[  346.329879]        [<c103b49f>] check_prevs_add+0x8b/0xe8
[  346.330192]        [<c103ce2a>] __lock_acquire+0x65d/0xb51
[  346.330508]        [<c103d786>] lock_acquire+0x62/0x81
[  346.330826]        [<c1034245>] down_write+0x2b/0x45
[  346.331137]        [<c1087c37>] notify_change+0x2d9/0x386
[  346.331453]        [<c1071af4>] do_truncate+0x5c/0x7f
[  346.331769]        [<c107c2e3>] may_open+0x1ec/0x231
[  346.332087]        [<c107c4a4>] open_namei+0xdd/0x5a2
[  346.332401]        [<c107275b>] do_filp_open+0x2c/0x53
[  346.332720]        [<c1072a90>] do_sys_open+0x52/0xd8
[  346.333037]        [<c1072b32>] sys_open+0x1c/0x1e
[  346.333600]        [<c1003eea>] syscall_call+0x7/0xb
[  346.333912]        [<ffffffff>] 0xffffffff
[  346.334232] 
[  346.334233] -> #1 (&sysfs_inode_imutex_key){--..}:
[  346.334422]        [<c103b2ee>] check_prev_add+0x15b/0x281
[  346.334734]        [<c103b49f>] check_prevs_add+0x8b/0xe8
[  346.335046]        [<c103ce2a>] __lock_acquire+0x65d/0xb51
[  346.335361]        [<c103d786>] lock_acquire+0x62/0x81
[  346.335685]        [<c121ab04>] __mutex_lock_slowpath+0x75/0x28c
[  346.336001]        [<c121aa83>] mutex_lock+0x8/0xa
[  346.336313]        [<c10b2828>] remove_dir+0x2c/0xf1
[  346.336629]        [<c10b28f5>] sysfs_remove_subdir+0x8/0xa
[  346.336941]        [<c10b3ccf>] sysfs_remove_group+0x67/0x78
[  346.337256]        [<c115cc58>] dpm_sysfs_remove+0x12/0x16
[  346.337574]        [<c115c36f>] device_pm_remove+0x21/0x62
[  346.337888]        [<c1156dc5>] device_del+0x82/0x291
[  346.338199]        [<c1156fdf>] device_unregister+0xb/0x15
[  346.338516]        [<c11571da>] device_destroy+0x8d/0x9a
[  346.338827]        [<c1142712>] vcs_remove_sysfs+0x1c/0x3a
[  346.339142]        [<c1149075>] con_close+0x5e/0x6b
[  346.339458]        [<c113aa5e>] release_dev+0x4c4/0x6ce
[  346.339767]        [<c113b0d6>] tty_release+0x12/0x1c
[  346.340078]        [<c1074479>] __fput+0x145/0x16e
[  346.340392]        [<c1074332>] fput+0x19/0x1b
[  346.340701]        [<c1072bb3>] filp_close+0x3c/0x75
[  346.341012]        [<c1072c55>] sys_close+0x69/0xb4
[  346.341325]        [<c1003eea>] syscall_call+0x7/0xb
[  346.341892]        [<ffffffff>] 0xffffffff
[  346.342213] 
[  346.342214] -> #0 (tty_mutex){--..}:
[  346.342397]        [<c103b1c7>] check_prev_add+0x34/0x281
[  346.342708]        [<c103b49f>] check_prevs_add+0x8b/0xe8
[  346.343017]        [<c103ce2a>] __lock_acquire+0x65d/0xb51
[  346.343333]        [<c103d786>] lock_acquire+0x62/0x81
[  346.343646]        [<c121ab04>] __mutex_lock_slowpath+0x75/0x28c
[  346.343956]        [<c121aa83>] mutex_lock+0x8/0xa
[  346.344263]        [<c10a41d5>] print_warning+0x8c/0x15d
[  346.344575]        [<c10a4993>] dquot_alloc_space+0x184/0x189
[  346.344883]        [<c10b5c55>] ext3_new_blocks+0x44b/0x5a2
[  346.345193]        [<c10b81e4>] ext3_alloc_blocks+0x40/0xdf
[  346.345507]        [<c10b82d3>] ext3_alloc_branch+0x50/0x21b
[  346.345816]        [<c10b87b1>] ext3_get_blocks_handle+0x1b8/0x367
[  346.346127]        [<c10b89d8>] ext3_get_block+0x78/0xe3
[  346.346441]        [<c1094802>] __block_prepare_write+0x168/0x415
[  346.346750]        [<c109530e>] block_prepare_write+0x28/0x3b
[  346.347057]        [<c10b8e84>] ext3_prepare_write+0xe3/0x17e
[  346.347369]        [<c10552f1>] generic_file_buffered_write+0x1b4/0x614
[  346.347689]        [<c105598b>] __generic_file_aio_write_nolock+0x23a/0x533
[  346.348002]        [<c1055d8d>] generic_file_aio_write+0x58/0xc4
[  346.348314]        [<c10b6a93>] ext3_file_write+0x2d/0xba
[  346.348627]        [<c107353b>] do_sync_write+0xc7/0x116
[  346.348935]        [<c10736e2>] vfs_write+0x158/0x15d
[  346.349246]        [<c1073788>] sys_write+0x3d/0x64
[  346.349560]        [<c1003eea>] syscall_call+0x7/0xb
[  346.349869]        [<ffffffff>] 0xffffffff
[  346.350438] 
[  346.350440] other info that might help us debug this:
[  346.350442] 
[  346.350562] 3 locks held by tor/2421:
[  346.350604]  #0:  (&inode->i_mutex){--..}, at: [<c121aa83>] mutex_lock+0x8/0xa
[  346.350827]  #1:  (&ei->truncate_mutex){--..}, at: [<c121aa83>] mutex_lock+0x8/0xa
[  346.351049]  #2:  (&s->s_dquot.dqptr_sem){----}, at: [<c10a485f>] dquot_alloc_space+0x50/0x189
[  346.351274] 
[  346.351276] stack backtrace:
[  346.351363]  [<c1004cac>] show_trace_log_lvl+0x1a/0x30
[  346.351446]  [<c1004cd4>] show_trace+0x12/0x14
[  346.351526]  [<c1004dce>] dump_stack+0x16/0x18
[  346.351605]  [<c103ab29>] print_circular_bug_tail+0x6f/0x71
[  346.351686]  [<c103b1c7>] check_prev_add+0x34/0x281
[  346.351764]  [<c103b49f>] check_prevs_add+0x8b/0xe8
[  346.351843]  [<c103ce2a>] __lock_acquire+0x65d/0xb51
[  346.351921]  [<c103d786>] lock_acquire+0x62/0x81
[  346.352002]  [<c121ab04>] __mutex_lock_slowpath+0x75/0x28c
[  346.352083]  [<c121aa83>] mutex_lock+0x8/0xa
[  346.352160]  [<c10a41d5>] print_warning+0x8c/0x15d
[  346.352240]  [<c10a4993>] dquot_alloc_space+0x184/0x189
[  346.352323]  [<c10b5c55>] ext3_new_blocks+0x44b/0x5a2
[  346.352402]  [<c10b81e4>] ext3_alloc_blocks+0x40/0xdf
[  346.352479]  [<c10b82d3>] ext3_alloc_branch+0x50/0x21b
[  346.352558]  [<c10b87b1>] ext3_get_blocks_handle+0x1b8/0x367
[  346.352638]  [<c10b89d8>] ext3_get_block+0x78/0xe3
[  346.352720]  [<c1094802>] __block_prepare_write+0x168/0x415
[  346.352801]  [<c109530e>] block_prepare_write+0x28/0x3b
[  346.352881]  [<c10b8e84>] ext3_prepare_write+0xe3/0x17e
[  346.352961]  [<c10552f1>] generic_file_buffered_write+0x1b4/0x614
[  346.353041]  [<c105598b>] __generic_file_aio_write_nolock+0x23a/0x533
[  346.353121]  [<c1055d8d>] generic_file_aio_write+0x58/0xc4
[  346.353200]  [<c10b6a93>] ext3_file_write+0x2d/0xba
[  346.353281]  [<c107353b>] do_sync_write+0xc7/0x116
[  346.353364]  [<c10736e2>] vfs_write+0x158/0x15d
[  346.353444]  [<c1073788>] sys_write+0x3d/0x64
[  346.353524]  [<c1003eea>] syscall_call+0x7/0xb
[  346.353603]  =======================


Folkert van Heusden

-- 
www.vanheusden.com/multitail - win een vlaai van multivlaai! zorg
ervoor dat multitail opgenomen wordt in Fedora Core, AIX, Solaris of
HP/UX en win een vlaai naar keuze
----------------------------------------------------------------------
Phone: +31-6-41278122, PGP-key: 1F28D8AE, www.vanheusden.com

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [2.6.22] circular lock detected
  2007-08-24 21:00 [2.6.22] circular lock detected Folkert van Heusden
@ 2007-08-26 23:03 ` Michal Piotrowski
  2007-09-02 11:55 ` Andrew Morton
  1 sibling, 0 replies; 11+ messages in thread
From: Michal Piotrowski @ 2007-08-26 23:03 UTC (permalink / raw)
  To: Folkert van Heusden; +Cc: linux-kernel, linux-fsdevel, linux-ext4

Hi,

[Adding fsdevel and linux-ext4 to CC]

On 24/08/07, Folkert van Heusden <folkert@vanheusden.com> wrote:
> Hi,
>
> 2.6.22 kernel with hyperthreading enabled only ext3 filesystems (2).
>
> [  346.314640] =======================================================
> [  346.314758] [ INFO: possible circular locking dependency detected ]
> [  346.314815] 2.6.22 #5
> [  346.314862] -------------------------------------------------------
> [  346.314920] tor/2421 is trying to acquire lock:
> [  346.314973]  (tty_mutex){--..}, at: [<c121aa83>] mutex_lock+0x8/0xa
> [  346.315193]
> [  346.315195] but task is already holding lock:
> [  346.316203]  (&s->s_dquot.dqptr_sem){----}, at: [<c10a485f>] dquot_alloc_space+0x50/0x189
> [  346.316457]
> [  346.316458] which lock already depends on the new lock.
> [  346.316461]
> [  346.316585]
> [  346.316586] the existing dependency chain (in reverse order) is:
> [  346.316675]
> [  346.316676] -> #4 (&s->s_dquot.dqptr_sem){----}:
> [  346.316871]        [<c103b2ee>] check_prev_add+0x15b/0x281
> [  346.317204]        [<c103b49f>] check_prevs_add+0x8b/0xe8
> [  346.317535]        [<c103ce2a>] __lock_acquire+0x65d/0xb51
> [  346.317873]        [<c103d786>] lock_acquire+0x62/0x81
> [  346.318195]        [<c10341af>] down_read+0x2b/0x3d
> [  346.318527]        [<c10a485f>] dquot_alloc_space+0x50/0x189
> [  346.318854]        [<c10b5c55>] ext3_new_blocks+0x44b/0x5a2
> [  346.319187]        [<c10b81e4>] ext3_alloc_blocks+0x40/0xdf
> [  346.319513]        [<c10b82d3>] ext3_alloc_branch+0x50/0x21b
> [  346.319850]        [<c10b87b1>] ext3_get_blocks_handle+0x1b8/0x367
> [  346.320184]        [<c10b8ada>] ext3_getblk+0x97/0x228
> [  346.320530]        [<c10b8c85>] ext3_bread+0x1a/0x78
> [  346.320870]        [<c10bde64>] ext3_mkdir+0xf4/0x270
> [  346.321188]        [<c107cdf6>] vfs_mkdir+0xb3/0x161
> [  346.321506]        [<c107cf30>] sys_mkdirat+0x8c/0xc4
> [  346.321820]        [<c107cf88>] sys_mkdir+0x20/0x22
> [  346.322131]        [<c1003eea>] syscall_call+0x7/0xb
> [  346.322446]        [<ffffffff>] 0xffffffff
> [  346.322779]
> [  346.322780] -> #3 (&ei->truncate_mutex){--..}:
> [  346.322965]        [<c103b2ee>] check_prev_add+0x15b/0x281
> [  346.323278]        [<c103b49f>] check_prevs_add+0x8b/0xe8
> [  346.323602]        [<c103ce2a>] __lock_acquire+0x65d/0xb51
> [  346.323919]        [<c103d786>] lock_acquire+0x62/0x81
> [  346.324234]        [<c121ab04>] __mutex_lock_slowpath+0x75/0x28c
> [  346.324857]        [<c121aa83>] mutex_lock+0x8/0xa
> [  346.325177]        [<c10ba380>] ext3_truncate+0x170/0x468
> [  346.325501]        [<c1060e9a>] vmtruncate+0xa6/0x116
> [  346.325828]        [<c1087937>] inode_setattr+0x145/0x16c
> [  346.326150]        [<c10bb301>] ext3_setattr+0x150/0x22f
> [  346.326479]        [<c1087cb0>] notify_change+0x352/0x386
> [  346.326802]        [<c1071af4>] do_truncate+0x5c/0x7f
> [  346.327116]        [<c107c2e3>] may_open+0x1ec/0x231
> [  346.327434]        [<c107c4a4>] open_namei+0xdd/0x5a2
> [  346.327770]        [<c107275b>] do_filp_open+0x2c/0x53
> [  346.328090]        [<c1072a90>] do_sys_open+0x52/0xd8
> [  346.328408]        [<c1072b32>] sys_open+0x1c/0x1e
> [  346.328727]        [<c1003eea>] syscall_call+0x7/0xb
> [  346.329042]        [<ffffffff>] 0xffffffff
> [  346.329368]
> [  346.329370] -> #2 (&inode->i_alloc_sem){--..}:
> [  346.329556]        [<c103b2ee>] check_prev_add+0x15b/0x281
> [  346.329879]        [<c103b49f>] check_prevs_add+0x8b/0xe8
> [  346.330192]        [<c103ce2a>] __lock_acquire+0x65d/0xb51
> [  346.330508]        [<c103d786>] lock_acquire+0x62/0x81
> [  346.330826]        [<c1034245>] down_write+0x2b/0x45
> [  346.331137]        [<c1087c37>] notify_change+0x2d9/0x386
> [  346.331453]        [<c1071af4>] do_truncate+0x5c/0x7f
> [  346.331769]        [<c107c2e3>] may_open+0x1ec/0x231
> [  346.332087]        [<c107c4a4>] open_namei+0xdd/0x5a2
> [  346.332401]        [<c107275b>] do_filp_open+0x2c/0x53
> [  346.332720]        [<c1072a90>] do_sys_open+0x52/0xd8
> [  346.333037]        [<c1072b32>] sys_open+0x1c/0x1e
> [  346.333600]        [<c1003eea>] syscall_call+0x7/0xb
> [  346.333912]        [<ffffffff>] 0xffffffff
> [  346.334232]
> [  346.334233] -> #1 (&sysfs_inode_imutex_key){--..}:
> [  346.334422]        [<c103b2ee>] check_prev_add+0x15b/0x281
> [  346.334734]        [<c103b49f>] check_prevs_add+0x8b/0xe8
> [  346.335046]        [<c103ce2a>] __lock_acquire+0x65d/0xb51
> [  346.335361]        [<c103d786>] lock_acquire+0x62/0x81
> [  346.335685]        [<c121ab04>] __mutex_lock_slowpath+0x75/0x28c
> [  346.336001]        [<c121aa83>] mutex_lock+0x8/0xa
> [  346.336313]        [<c10b2828>] remove_dir+0x2c/0xf1
> [  346.336629]        [<c10b28f5>] sysfs_remove_subdir+0x8/0xa
> [  346.336941]        [<c10b3ccf>] sysfs_remove_group+0x67/0x78
> [  346.337256]        [<c115cc58>] dpm_sysfs_remove+0x12/0x16
> [  346.337574]        [<c115c36f>] device_pm_remove+0x21/0x62
> [  346.337888]        [<c1156dc5>] device_del+0x82/0x291
> [  346.338199]        [<c1156fdf>] device_unregister+0xb/0x15
> [  346.338516]        [<c11571da>] device_destroy+0x8d/0x9a
> [  346.338827]        [<c1142712>] vcs_remove_sysfs+0x1c/0x3a
> [  346.339142]        [<c1149075>] con_close+0x5e/0x6b
> [  346.339458]        [<c113aa5e>] release_dev+0x4c4/0x6ce
> [  346.339767]        [<c113b0d6>] tty_release+0x12/0x1c
> [  346.340078]        [<c1074479>] __fput+0x145/0x16e
> [  346.340392]        [<c1074332>] fput+0x19/0x1b
> [  346.340701]        [<c1072bb3>] filp_close+0x3c/0x75
> [  346.341012]        [<c1072c55>] sys_close+0x69/0xb4
> [  346.341325]        [<c1003eea>] syscall_call+0x7/0xb
> [  346.341892]        [<ffffffff>] 0xffffffff
> [  346.342213]
> [  346.342214] -> #0 (tty_mutex){--..}:
> [  346.342397]        [<c103b1c7>] check_prev_add+0x34/0x281
> [  346.342708]        [<c103b49f>] check_prevs_add+0x8b/0xe8
> [  346.343017]        [<c103ce2a>] __lock_acquire+0x65d/0xb51
> [  346.343333]        [<c103d786>] lock_acquire+0x62/0x81
> [  346.343646]        [<c121ab04>] __mutex_lock_slowpath+0x75/0x28c
> [  346.343956]        [<c121aa83>] mutex_lock+0x8/0xa
> [  346.344263]        [<c10a41d5>] print_warning+0x8c/0x15d
> [  346.344575]        [<c10a4993>] dquot_alloc_space+0x184/0x189
> [  346.344883]        [<c10b5c55>] ext3_new_blocks+0x44b/0x5a2
> [  346.345193]        [<c10b81e4>] ext3_alloc_blocks+0x40/0xdf
> [  346.345507]        [<c10b82d3>] ext3_alloc_branch+0x50/0x21b
> [  346.345816]        [<c10b87b1>] ext3_get_blocks_handle+0x1b8/0x367
> [  346.346127]        [<c10b89d8>] ext3_get_block+0x78/0xe3
> [  346.346441]        [<c1094802>] __block_prepare_write+0x168/0x415
> [  346.346750]        [<c109530e>] block_prepare_write+0x28/0x3b
> [  346.347057]        [<c10b8e84>] ext3_prepare_write+0xe3/0x17e
> [  346.347369]        [<c10552f1>] generic_file_buffered_write+0x1b4/0x614
> [  346.347689]        [<c105598b>] __generic_file_aio_write_nolock+0x23a/0x533
> [  346.348002]        [<c1055d8d>] generic_file_aio_write+0x58/0xc4
> [  346.348314]        [<c10b6a93>] ext3_file_write+0x2d/0xba
> [  346.348627]        [<c107353b>] do_sync_write+0xc7/0x116
> [  346.348935]        [<c10736e2>] vfs_write+0x158/0x15d
> [  346.349246]        [<c1073788>] sys_write+0x3d/0x64
> [  346.349560]        [<c1003eea>] syscall_call+0x7/0xb
> [  346.349869]        [<ffffffff>] 0xffffffff
> [  346.350438]
> [  346.350440] other info that might help us debug this:
> [  346.350442]
> [  346.350562] 3 locks held by tor/2421:
> [  346.350604]  #0:  (&inode->i_mutex){--..}, at: [<c121aa83>] mutex_lock+0x8/0xa
> [  346.350827]  #1:  (&ei->truncate_mutex){--..}, at: [<c121aa83>] mutex_lock+0x8/0xa
> [  346.351049]  #2:  (&s->s_dquot.dqptr_sem){----}, at: [<c10a485f>] dquot_alloc_space+0x50/0x189
> [  346.351274]
> [  346.351276] stack backtrace:
> [  346.351363]  [<c1004cac>] show_trace_log_lvl+0x1a/0x30
> [  346.351446]  [<c1004cd4>] show_trace+0x12/0x14
> [  346.351526]  [<c1004dce>] dump_stack+0x16/0x18
> [  346.351605]  [<c103ab29>] print_circular_bug_tail+0x6f/0x71
> [  346.351686]  [<c103b1c7>] check_prev_add+0x34/0x281
> [  346.351764]  [<c103b49f>] check_prevs_add+0x8b/0xe8
> [  346.351843]  [<c103ce2a>] __lock_acquire+0x65d/0xb51
> [  346.351921]  [<c103d786>] lock_acquire+0x62/0x81
> [  346.352002]  [<c121ab04>] __mutex_lock_slowpath+0x75/0x28c
> [  346.352083]  [<c121aa83>] mutex_lock+0x8/0xa
> [  346.352160]  [<c10a41d5>] print_warning+0x8c/0x15d
> [  346.352240]  [<c10a4993>] dquot_alloc_space+0x184/0x189
> [  346.352323]  [<c10b5c55>] ext3_new_blocks+0x44b/0x5a2
> [  346.352402]  [<c10b81e4>] ext3_alloc_blocks+0x40/0xdf
> [  346.352479]  [<c10b82d3>] ext3_alloc_branch+0x50/0x21b
> [  346.352558]  [<c10b87b1>] ext3_get_blocks_handle+0x1b8/0x367
> [  346.352638]  [<c10b89d8>] ext3_get_block+0x78/0xe3
> [  346.352720]  [<c1094802>] __block_prepare_write+0x168/0x415
> [  346.352801]  [<c109530e>] block_prepare_write+0x28/0x3b
> [  346.352881]  [<c10b8e84>] ext3_prepare_write+0xe3/0x17e
> [  346.352961]  [<c10552f1>] generic_file_buffered_write+0x1b4/0x614
> [  346.353041]  [<c105598b>] __generic_file_aio_write_nolock+0x23a/0x533
> [  346.353121]  [<c1055d8d>] generic_file_aio_write+0x58/0xc4
> [  346.353200]  [<c10b6a93>] ext3_file_write+0x2d/0xba
> [  346.353281]  [<c107353b>] do_sync_write+0xc7/0x116
> [  346.353364]  [<c10736e2>] vfs_write+0x158/0x15d
> [  346.353444]  [<c1073788>] sys_write+0x3d/0x64
> [  346.353524]  [<c1003eea>] syscall_call+0x7/0xb
> [  346.353603]  =======================
>
>
> Folkert van Heusden
>
> --
> www.vanheusden.com/multitail - win een vlaai van multivlaai! zorg
> ervoor dat multitail opgenomen wordt in Fedora Core, AIX, Solaris of
> HP/UX en win een vlaai naar keuze
> ----------------------------------------------------------------------
> Phone: +31-6-41278122, PGP-key: 1F28D8AE, www.vanheusden.com
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Regards,
Michal

-- 
LOG
http://www.stardust.webpages.pl/log/

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [2.6.22] circular lock detected
  2007-08-24 21:00 [2.6.22] circular lock detected Folkert van Heusden
  2007-08-26 23:03 ` Michal Piotrowski
@ 2007-09-02 11:55 ` Andrew Morton
  2007-09-02 17:11   ` Folkert van Heusden
  2007-09-03 12:27   ` Jan Kara
  1 sibling, 2 replies; 11+ messages in thread
From: Andrew Morton @ 2007-09-02 11:55 UTC (permalink / raw)
  To: Folkert van Heusden; +Cc: linux-kernel, Jan Kara

> On Fri, 24 Aug 2007 23:00:33 +0200 Folkert van Heusden <folkert@vanheusden.com> wrote:
> Hi,
> 
> 2.6.22 kernel with hyperthreading enabled only ext3 filesystems (2).
> 
> [  346.314640] =======================================================
> [  346.314758] [ INFO: possible circular locking dependency detected ]
> [  346.314815] 2.6.22 #5
> [  346.314862] -------------------------------------------------------
> [  346.314920] tor/2421 is trying to acquire lock:
> [  346.314973]  (tty_mutex){--..}, at: [<c121aa83>] mutex_lock+0x8/0xa
> [  346.315193] 
> [  346.315195] but task is already holding lock:
> [  346.316203]  (&s->s_dquot.dqptr_sem){----}, at: [<c10a485f>] dquot_alloc_space+0x50/0x189
> [  346.316457] 
> [  346.316458] which lock already depends on the new lock.
> [  346.316461] 
> [  346.316585] 
> [  346.316586] the existing dependency chain (in reverse order) is:
> [  346.316675] 
> [  346.316676] -> #4 (&s->s_dquot.dqptr_sem){----}:
> [  346.316871]        [<c103b2ee>] check_prev_add+0x15b/0x281
> [  346.317204]        [<c103b49f>] check_prevs_add+0x8b/0xe8
> [  346.317535]        [<c103ce2a>] __lock_acquire+0x65d/0xb51
> [  346.317873]        [<c103d786>] lock_acquire+0x62/0x81
> [  346.318195]        [<c10341af>] down_read+0x2b/0x3d
> [  346.318527]        [<c10a485f>] dquot_alloc_space+0x50/0x189
> [  346.318854]        [<c10b5c55>] ext3_new_blocks+0x44b/0x5a2
> [  346.319187]        [<c10b81e4>] ext3_alloc_blocks+0x40/0xdf
> [  346.319513]        [<c10b82d3>] ext3_alloc_branch+0x50/0x21b
> [  346.319850]        [<c10b87b1>] ext3_get_blocks_handle+0x1b8/0x367
> [  346.320184]        [<c10b8ada>] ext3_getblk+0x97/0x228
> [  346.320530]        [<c10b8c85>] ext3_bread+0x1a/0x78
> [  346.320870]        [<c10bde64>] ext3_mkdir+0xf4/0x270
> [  346.321188]        [<c107cdf6>] vfs_mkdir+0xb3/0x161
> [  346.321506]        [<c107cf30>] sys_mkdirat+0x8c/0xc4
> [  346.321820]        [<c107cf88>] sys_mkdir+0x20/0x22
> [  346.322131]        [<c1003eea>] syscall_call+0x7/0xb
> [  346.322446]        [<ffffffff>] 0xffffffff
> [  346.322779] 
> [  346.322780] -> #3 (&ei->truncate_mutex){--..}:
> [  346.322965]        [<c103b2ee>] check_prev_add+0x15b/0x281
> [  346.323278]        [<c103b49f>] check_prevs_add+0x8b/0xe8
> [  346.323602]        [<c103ce2a>] __lock_acquire+0x65d/0xb51
> [  346.323919]        [<c103d786>] lock_acquire+0x62/0x81
> [  346.324234]        [<c121ab04>] __mutex_lock_slowpath+0x75/0x28c
> [  346.324857]        [<c121aa83>] mutex_lock+0x8/0xa
> [  346.325177]        [<c10ba380>] ext3_truncate+0x170/0x468
> [  346.325501]        [<c1060e9a>] vmtruncate+0xa6/0x116
> [  346.325828]        [<c1087937>] inode_setattr+0x145/0x16c
> [  346.326150]        [<c10bb301>] ext3_setattr+0x150/0x22f
> [  346.326479]        [<c1087cb0>] notify_change+0x352/0x386
> [  346.326802]        [<c1071af4>] do_truncate+0x5c/0x7f
> [  346.327116]        [<c107c2e3>] may_open+0x1ec/0x231
> [  346.327434]        [<c107c4a4>] open_namei+0xdd/0x5a2
> [  346.327770]        [<c107275b>] do_filp_open+0x2c/0x53
> [  346.328090]        [<c1072a90>] do_sys_open+0x52/0xd8
> [  346.328408]        [<c1072b32>] sys_open+0x1c/0x1e
> [  346.328727]        [<c1003eea>] syscall_call+0x7/0xb
> [  346.329042]        [<ffffffff>] 0xffffffff
> [  346.329368] 
> [  346.329370] -> #2 (&inode->i_alloc_sem){--..}:
> [  346.329556]        [<c103b2ee>] check_prev_add+0x15b/0x281
> [  346.329879]        [<c103b49f>] check_prevs_add+0x8b/0xe8
> [  346.330192]        [<c103ce2a>] __lock_acquire+0x65d/0xb51
> [  346.330508]        [<c103d786>] lock_acquire+0x62/0x81
> [  346.330826]        [<c1034245>] down_write+0x2b/0x45
> [  346.331137]        [<c1087c37>] notify_change+0x2d9/0x386
> [  346.331453]        [<c1071af4>] do_truncate+0x5c/0x7f
> [  346.331769]        [<c107c2e3>] may_open+0x1ec/0x231
> [  346.332087]        [<c107c4a4>] open_namei+0xdd/0x5a2
> [  346.332401]        [<c107275b>] do_filp_open+0x2c/0x53
> [  346.332720]        [<c1072a90>] do_sys_open+0x52/0xd8
> [  346.333037]        [<c1072b32>] sys_open+0x1c/0x1e
> [  346.333600]        [<c1003eea>] syscall_call+0x7/0xb
> [  346.333912]        [<ffffffff>] 0xffffffff
> [  346.334232] 
> [  346.334233] -> #1 (&sysfs_inode_imutex_key){--..}:
> [  346.334422]        [<c103b2ee>] check_prev_add+0x15b/0x281
> [  346.334734]        [<c103b49f>] check_prevs_add+0x8b/0xe8
> [  346.335046]        [<c103ce2a>] __lock_acquire+0x65d/0xb51
> [  346.335361]        [<c103d786>] lock_acquire+0x62/0x81
> [  346.335685]        [<c121ab04>] __mutex_lock_slowpath+0x75/0x28c
> [  346.336001]        [<c121aa83>] mutex_lock+0x8/0xa
> [  346.336313]        [<c10b2828>] remove_dir+0x2c/0xf1
> [  346.336629]        [<c10b28f5>] sysfs_remove_subdir+0x8/0xa
> [  346.336941]        [<c10b3ccf>] sysfs_remove_group+0x67/0x78
> [  346.337256]        [<c115cc58>] dpm_sysfs_remove+0x12/0x16
> [  346.337574]        [<c115c36f>] device_pm_remove+0x21/0x62
> [  346.337888]        [<c1156dc5>] device_del+0x82/0x291
> [  346.338199]        [<c1156fdf>] device_unregister+0xb/0x15
> [  346.338516]        [<c11571da>] device_destroy+0x8d/0x9a
> [  346.338827]        [<c1142712>] vcs_remove_sysfs+0x1c/0x3a
> [  346.339142]        [<c1149075>] con_close+0x5e/0x6b
> [  346.339458]        [<c113aa5e>] release_dev+0x4c4/0x6ce
> [  346.339767]        [<c113b0d6>] tty_release+0x12/0x1c
> [  346.340078]        [<c1074479>] __fput+0x145/0x16e
> [  346.340392]        [<c1074332>] fput+0x19/0x1b
> [  346.340701]        [<c1072bb3>] filp_close+0x3c/0x75
> [  346.341012]        [<c1072c55>] sys_close+0x69/0xb4
> [  346.341325]        [<c1003eea>] syscall_call+0x7/0xb
> [  346.341892]        [<ffffffff>] 0xffffffff
> [  346.342213] 
> [  346.342214] -> #0 (tty_mutex){--..}:
> [  346.342397]        [<c103b1c7>] check_prev_add+0x34/0x281
> [  346.342708]        [<c103b49f>] check_prevs_add+0x8b/0xe8
> [  346.343017]        [<c103ce2a>] __lock_acquire+0x65d/0xb51
> [  346.343333]        [<c103d786>] lock_acquire+0x62/0x81
> [  346.343646]        [<c121ab04>] __mutex_lock_slowpath+0x75/0x28c
> [  346.343956]        [<c121aa83>] mutex_lock+0x8/0xa
> [  346.344263]        [<c10a41d5>] print_warning+0x8c/0x15d
> [  346.344575]        [<c10a4993>] dquot_alloc_space+0x184/0x189
> [  346.344883]        [<c10b5c55>] ext3_new_blocks+0x44b/0x5a2
> [  346.345193]        [<c10b81e4>] ext3_alloc_blocks+0x40/0xdf
> [  346.345507]        [<c10b82d3>] ext3_alloc_branch+0x50/0x21b
> [  346.345816]        [<c10b87b1>] ext3_get_blocks_handle+0x1b8/0x367
> [  346.346127]        [<c10b89d8>] ext3_get_block+0x78/0xe3
> [  346.346441]        [<c1094802>] __block_prepare_write+0x168/0x415
> [  346.346750]        [<c109530e>] block_prepare_write+0x28/0x3b
> [  346.347057]        [<c10b8e84>] ext3_prepare_write+0xe3/0x17e
> [  346.347369]        [<c10552f1>] generic_file_buffered_write+0x1b4/0x614
> [  346.347689]        [<c105598b>] __generic_file_aio_write_nolock+0x23a/0x533
> [  346.348002]        [<c1055d8d>] generic_file_aio_write+0x58/0xc4
> [  346.348314]        [<c10b6a93>] ext3_file_write+0x2d/0xba
> [  346.348627]        [<c107353b>] do_sync_write+0xc7/0x116
> [  346.348935]        [<c10736e2>] vfs_write+0x158/0x15d
> [  346.349246]        [<c1073788>] sys_write+0x3d/0x64
> [  346.349560]        [<c1003eea>] syscall_call+0x7/0xb
> [  346.349869]        [<ffffffff>] 0xffffffff
> [  346.350438] 
> [  346.350440] other info that might help us debug this:
> [  346.350442] 
> [  346.350562] 3 locks held by tor/2421:
> [  346.350604]  #0:  (&inode->i_mutex){--..}, at: [<c121aa83>] mutex_lock+0x8/0xa
> [  346.350827]  #1:  (&ei->truncate_mutex){--..}, at: [<c121aa83>] mutex_lock+0x8/0xa
> [  346.351049]  #2:  (&s->s_dquot.dqptr_sem){----}, at: [<c10a485f>] dquot_alloc_space+0x50/0x189
> [  346.351274] 
> [  346.351276] stack backtrace:
> [  346.351363]  [<c1004cac>] show_trace_log_lvl+0x1a/0x30
> [  346.351446]  [<c1004cd4>] show_trace+0x12/0x14
> [  346.351526]  [<c1004dce>] dump_stack+0x16/0x18
> [  346.351605]  [<c103ab29>] print_circular_bug_tail+0x6f/0x71
> [  346.351686]  [<c103b1c7>] check_prev_add+0x34/0x281
> [  346.351764]  [<c103b49f>] check_prevs_add+0x8b/0xe8
> [  346.351843]  [<c103ce2a>] __lock_acquire+0x65d/0xb51
> [  346.351921]  [<c103d786>] lock_acquire+0x62/0x81
> [  346.352002]  [<c121ab04>] __mutex_lock_slowpath+0x75/0x28c
> [  346.352083]  [<c121aa83>] mutex_lock+0x8/0xa
> [  346.352160]  [<c10a41d5>] print_warning+0x8c/0x15d
> [  346.352240]  [<c10a4993>] dquot_alloc_space+0x184/0x189
> [  346.352323]  [<c10b5c55>] ext3_new_blocks+0x44b/0x5a2
> [  346.352402]  [<c10b81e4>] ext3_alloc_blocks+0x40/0xdf
> [  346.352479]  [<c10b82d3>] ext3_alloc_branch+0x50/0x21b
> [  346.352558]  [<c10b87b1>] ext3_get_blocks_handle+0x1b8/0x367
> [  346.352638]  [<c10b89d8>] ext3_get_block+0x78/0xe3
> [  346.352720]  [<c1094802>] __block_prepare_write+0x168/0x415
> [  346.352801]  [<c109530e>] block_prepare_write+0x28/0x3b
> [  346.352881]  [<c10b8e84>] ext3_prepare_write+0xe3/0x17e
> [  346.352961]  [<c10552f1>] generic_file_buffered_write+0x1b4/0x614
> [  346.353041]  [<c105598b>] __generic_file_aio_write_nolock+0x23a/0x533
> [  346.353121]  [<c1055d8d>] generic_file_aio_write+0x58/0xc4
> [  346.353200]  [<c10b6a93>] ext3_file_write+0x2d/0xba
> [  346.353281]  [<c107353b>] do_sync_write+0xc7/0x116
> [  346.353364]  [<c10736e2>] vfs_write+0x158/0x15d
> [  346.353444]  [<c1073788>] sys_write+0x3d/0x64
> [  346.353524]  [<c1003eea>] syscall_call+0x7/0xb
> [  346.353603]  =======================
> 

Has been reported before, but I don't recall whether we fixed it.  Jan,
do you know>?

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [2.6.22] circular lock detected
  2007-09-02 11:55 ` Andrew Morton
@ 2007-09-02 17:11   ` Folkert van Heusden
  2007-09-03 12:27   ` Jan Kara
  1 sibling, 0 replies; 11+ messages in thread
From: Folkert van Heusden @ 2007-09-02 17:11 UTC (permalink / raw)
  To: Andrew Morton; +Cc: linux-kernel, Jan Kara

> > 2.6.22 kernel with hyperthreading enabled only ext3 filesystems (2).
> > 
> > [  346.314640] =======================================================
...
> > [  346.349869]        [<ffffffff>] 0xffffffff
> > [  346.350438] 
> > [  346.350440] other info that might help us debug this:
> > [  346.350442] 
> > [  346.350562] 3 locks held by tor/2421:
> > [  346.350604]  #0:  (&inode->i_mutex){--..}, at: [<c121aa83>] mutex_lock+0x8/0xa
> > [  346.350827]  #1:  (&ei->truncate_mutex){--..}, at: [<c121aa83>] mutex_lock+0x8/0xa
> > [  346.351049]  #2:  (&s->s_dquot.dqptr_sem){----}, at: [<c10a485f>] dquot_alloc_space+0x50/0x189
> > [  346.351274] 
> > [  346.351276] stack backtrace:
...
> > [  346.353603]  =======================
> 
> Has been reported before, but I don't recall whether we fixed it.  Jan,
> do you know>?

It is repeatable. Yesterday I changed from ata to sata and it still
occurs altough in a different proces:

[ 7382.917927] 

Oh and it also seems it takes longer.

[ 7382.917929] =======================================================
[ 7382.918024] [ INFO: possible circular locking dependency detected ]
[ 7382.918072] 2.6.22 #2
[ 7382.918114] -------------------------------------------------------
[ 7382.918159] moo/17202 is trying to acquire lock:
[ 7382.918204]  (tty_mutex){--..}, at: [<c1228353>] mutex_lock+0x8/0xa
[ 7382.918392] 
[ 7382.918393] but task is already holding lock:
[ 7382.918474]  (&s->s_dquot.dqptr_sem){----}, at: [<c10a47a3>] dquot_alloc_space+0x50/0x189
[ 7382.918665] 
[ 7382.918666] which lock already depends on the new lock.
[ 7382.918668] 
[ 7382.918788] 
[ 7382.918789] the existing dependency chain (in reverse order) is:
[ 7382.918873] 
[ 7382.918874] -> #4 (&s->s_dquot.dqptr_sem){----}:
[ 7382.919061]        [<c103b232>] check_prev_add+0x15b/0x281
[ 7382.919378]        [<c103b3e3>] check_prevs_add+0x8b/0xe8
[ 7382.919690]        [<c103cd6e>] __lock_acquire+0x65d/0xb51
[ 7382.920004]        [<c103d6ca>] lock_acquire+0x62/0x81
[ 7382.920322]        [<c10340f3>] down_read+0x2b/0x3d
[ 7382.920635]        [<c10a47a3>] dquot_alloc_space+0x50/0x189
[ 7382.920957]        [<c10b5b99>] ext3_new_blocks+0x44b/0x5a2
[ 7382.921280]        [<c10b8128>] ext3_alloc_blocks+0x40/0xdf
[ 7382.921592]        [<c10b8217>] ext3_alloc_branch+0x50/0x21b
[ 7382.921906]        [<c10b86f5>] ext3_get_blocks_handle+0x1b8/0x367
[ 7382.922222]        [<c10b8a1e>] ext3_getblk+0x97/0x228
[ 7382.922534]        [<c10b8bc9>] ext3_bread+0x1a/0x78
[ 7382.922844]        [<c10bdda8>] ext3_mkdir+0xf4/0x270
[ 7382.923160]        [<c107cd3a>] vfs_mkdir+0xb3/0x161
[ 7382.923474]        [<c107ce74>] sys_mkdirat+0x8c/0xc4
[ 7382.923789]        [<c107cecc>] sys_mkdir+0x20/0x22
[ 7382.924106]        [<c1003eea>] syscall_call+0x7/0xb
[ 7382.924425]        [<ffffffff>] 0xffffffff
[ 7382.924752] 
[ 7382.924753] -> #3 (&ei->truncate_mutex){--..}:
[ 7382.924939]        [<c103b232>] check_prev_add+0x15b/0x281
[ 7382.925257]        [<c103b3e3>] check_prevs_add+0x8b/0xe8
[ 7382.925571]        [<c103cd6e>] __lock_acquire+0x65d/0xb51
[ 7382.925886]        [<c103d6ca>] lock_acquire+0x62/0x81
[ 7382.926202]        [<c12283d4>] __mutex_lock_slowpath+0x75/0x28c
[ 7382.926782]        [<c1228353>] mutex_lock+0x8/0xa
[ 7382.927101]        [<c10ba2c4>] ext3_truncate+0x170/0x468
[ 7382.927414]        [<c1060dde>] vmtruncate+0xa6/0x116
[ 7382.927726]        [<c108787b>] inode_setattr+0x145/0x16c
[ 7382.928041]        [<c10bb245>] ext3_setattr+0x150/0x22f
[ 7382.928358]        [<c1087bf4>] notify_change+0x352/0x386
[ 7382.928672]        [<c1071a38>] do_truncate+0x5c/0x7f
[ 7382.928988]        [<c107c227>] may_open+0x1ec/0x231
[ 7382.929289]        [<c107c3e8>] open_namei+0xdd/0x5a2
[ 7382.929298]        [<c107269f>] do_filp_open+0x2c/0x53
[ 7382.929308]        [<c10729d4>] do_sys_open+0x52/0xd8
[ 7382.929316]        [<c1072a76>] sys_open+0x1c/0x1e
[ 7382.929323]        [<c1003eea>] syscall_call+0x7/0xb
[ 7382.929332]        [<ffffffff>] 0xffffffff
[ 7382.929351] 
[ 7382.929352] -> #2 (&inode->i_alloc_sem){--..}:
[ 7382.929358]        [<c103b232>] check_prev_add+0x15b/0x281
[ 7382.929367]        [<c103b3e3>] check_prevs_add+0x8b/0xe8
[ 7382.929374]        [<c103cd6e>] __lock_acquire+0x65d/0xb51
[ 7382.929382]        [<c103d6ca>] lock_acquire+0x62/0x81
[ 7382.929390]        [<c1034189>] down_write+0x2b/0x45
[ 7382.929401]        [<c1087b7b>] notify_change+0x2d9/0x386
[ 7382.929409]        [<c1071a38>] do_truncate+0x5c/0x7f
[ 7382.929416]        [<c107c227>] may_open+0x1ec/0x231
[ 7382.929423]        [<c107c3e8>] open_namei+0xdd/0x5a2
[ 7382.929430]        [<c107269f>] do_filp_open+0x2c/0x53
[ 7382.929441]        [<c10729d4>] do_sys_open+0x52/0xd8
[ 7382.929449]        [<c1072a76>] sys_open+0x1c/0x1e
[ 7382.929455]        [<c1003eea>] syscall_call+0x7/0xb
[ 7382.929463]        [<ffffffff>] 0xffffffff
[ 7382.929471] 
[ 7382.929472] -> #1 (&sysfs_inode_imutex_key){--..}:
[ 7382.929481]        [<c103b232>] check_prev_add+0x15b/0x281
[ 7382.929489]        [<c103b3e3>] check_prevs_add+0x8b/0xe8
[ 7382.929496]        [<c103cd6e>] __lock_acquire+0x65d/0xb51
[ 7382.929505]        [<c103d6ca>] lock_acquire+0x62/0x81
[ 7382.929512]        [<c12283d4>] __mutex_lock_slowpath+0x75/0x28c
[ 7382.929523]        [<c1228353>] mutex_lock+0x8/0xa
[ 7382.929531]        [<c10b276c>] remove_dir+0x2c/0xf1
[ 7382.929538]        [<c10b2839>] sysfs_remove_subdir+0x8/0xa
[ 7382.929547]        [<c10b3c13>] sysfs_remove_group+0x67/0x78
[ 7382.929554]        [<c115cd9c>] dpm_sysfs_remove+0x12/0x16
[ 7382.929563]        [<c115c4b3>] device_pm_remove+0x21/0x62
[ 7382.929570]        [<c1156f09>] device_del+0x82/0x291
[ 7382.929578]        [<c1157123>] device_unregister+0xb/0x15
[ 7382.929585]        [<c115731e>] device_destroy+0x8d/0x9a
[ 7382.929592]        [<c1142856>] vcs_remove_sysfs+0x1c/0x3a
[ 7382.929600]        [<c11491b9>] con_close+0x5e/0x6b
[ 7382.929607]        [<c113aba2>] release_dev+0x4c4/0x6ce
[ 7382.929614]        [<c113b21a>] tty_release+0x12/0x1c
[ 7382.929620]        [<c10743bd>] __fput+0x145/0x16e
[ 7382.929628]        [<c1074276>] fput+0x19/0x1b
[ 7382.929634]        [<c1072af7>] filp_close+0x3c/0x75
[ 7382.929641]        [<c1072b99>] sys_close+0x69/0xb4
[ 7382.929648]        [<c1003eea>] syscall_call+0x7/0xb
[ 7382.929654]        [<ffffffff>] 0xffffffff
[ 7382.929665] 
[ 7382.929666] -> #0 (tty_mutex){--..}:
[ 7382.929670]        [<c103b10b>] check_prev_add+0x34/0x281
[ 7382.929677]        [<c103b3e3>] check_prevs_add+0x8b/0xe8
[ 7382.929684]        [<c103cd6e>] __lock_acquire+0x65d/0xb51
[ 7382.929691]        [<c103d6ca>] lock_acquire+0x62/0x81
[ 7382.929698]        [<c12283d4>] __mutex_lock_slowpath+0x75/0x28c
[ 7382.929706]        [<c1228353>] mutex_lock+0x8/0xa
[ 7382.929712]        [<c10a4119>] print_warning+0x8c/0x15d
[ 7382.929720]        [<c10a48d7>] dquot_alloc_space+0x184/0x189
[ 7382.929727]        [<c10b5b99>] ext3_new_blocks+0x44b/0x5a2
[ 7382.929735]        [<c10b8128>] ext3_alloc_blocks+0x40/0xdf
[ 7382.929741]        [<c10b8217>] ext3_alloc_branch+0x50/0x21b
[ 7382.929748]        [<c10b86f5>] ext3_get_blocks_handle+0x1b8/0x367
[ 7382.929755]        [<c10b891c>] ext3_get_block+0x78/0xe3
[ 7382.929761]        [<c1094746>] __block_prepare_write+0x168/0x415
[ 7382.929770]        [<c1095252>] block_prepare_write+0x28/0x3b
[ 7382.929777]        [<c10b8dc8>] ext3_prepare_write+0xe3/0x17e
[ 7382.929784]        [<c1055235>] generic_file_buffered_write+0x1b4/0x614
[ 7382.929792]        [<c10558cf>] __generic_file_aio_write_nolock+0x23a/0x533
[ 7382.929800]        [<c1055cd1>] generic_file_aio_write+0x58/0xc4
[ 7382.929807]        [<c10b69d7>] ext3_file_write+0x2d/0xba
[ 7382.929815]        [<c107347f>] do_sync_write+0xc7/0x116
[ 7382.929822]        [<c1073626>] vfs_write+0x158/0x15d
[ 7382.929829]        [<c10736cc>] sys_write+0x3d/0x64
[ 7382.929835]        [<c1003eea>] syscall_call+0x7/0xb
[ 7382.929842]        [<ffffffff>] 0xffffffff
[ 7382.929851] 
[ 7382.929852] other info that might help us debug this:
[ 7382.929853] 
[ 7382.929855] 3 locks held by moo/17202:
[ 7382.929857]  #0:  (&inode->i_mutex){--..}, at: [<c1228353>] mutex_lock+0x8/0xa
[ 7382.929864]  #1:  (&ei->truncate_mutex){--..}, at: [<c1228353>] mutex_lock+0x8/0xa
[ 7382.929871]  #2:  (&s->s_dquot.dqptr_sem){----}, at: [<c10a47a3>] dquot_alloc_space+0x50/0x189
[ 7382.929877] 
[ 7382.929878] stack backtrace:
[ 7382.929880]  [<c1004cac>] show_trace_log_lvl+0x1a/0x30
[ 7382.929885]  [<c1004cd4>] show_trace+0x12/0x14
[ 7382.929888]  [<c1004dce>] dump_stack+0x16/0x18
[ 7382.929892]  [<c103aa6d>] print_circular_bug_tail+0x6f/0x71
[ 7382.929896]  [<c103b10b>] check_prev_add+0x34/0x281
[ 7382.929900]  [<c103b3e3>] check_prevs_add+0x8b/0xe8
[ 7382.929903]  [<c103cd6e>] __lock_acquire+0x65d/0xb51
[ 7382.929907]  [<c103d6ca>] lock_acquire+0x62/0x81
[ 7382.929911]  [<c12283d4>] __mutex_lock_slowpath+0x75/0x28c
[ 7382.929915]  [<c1228353>] mutex_lock+0x8/0xa
[ 7382.929919]  [<c10a4119>] print_warning+0x8c/0x15d
[ 7382.929923]  [<c10a48d7>] dquot_alloc_space+0x184/0x189
[ 7382.929926]  [<c10b5b99>] ext3_new_blocks+0x44b/0x5a2
[ 7382.929931]  [<c10b8128>] ext3_alloc_blocks+0x40/0xdf
[ 7382.929934]  [<c10b8217>] ext3_alloc_branch+0x50/0x21b
[ 7382.929937]  [<c10b86f5>] ext3_get_blocks_handle+0x1b8/0x367
[ 7382.929941]  [<c10b891c>] ext3_get_block+0x78/0xe3
[ 7382.929944]  [<c1094746>] __block_prepare_write+0x168/0x415
[ 7382.929949]  [<c1095252>] block_prepare_write+0x28/0x3b
[ 7382.929953]  [<c10b8dc8>] ext3_prepare_write+0xe3/0x17e
[ 7382.929957]  [<c1055235>] generic_file_buffered_write+0x1b4/0x614
[ 7382.929961]  [<c10558cf>] __generic_file_aio_write_nolock+0x23a/0x533
[ 7382.929966]  [<c1055cd1>] generic_file_aio_write+0x58/0xc4
[ 7382.929970]  [<c10b69d7>] ext3_file_write+0x2d/0xba
[ 7382.929974]  [<c107347f>] do_sync_write+0xc7/0x116
[ 7382.929978]  [<c1073626>] vfs_write+0x158/0x15d
[ 7382.929981]  [<c10736cc>] sys_write+0x3d/0x64
[ 7382.929985]  [<c1003eea>] syscall_call+0x7/0xb
[ 7382.929988]  =======================


Folkert van Heusden

-- 

Multitail - gibkaja utilita po sledovaniju log-fajlov i vyvoda
kommand. Fil'trovanie, raskrašivanie, slijanie, vizual'noe sravnenie,
i t.d.  http://www.vanheusden.com/multitail/
----------------------------------------------------------------------
Phone: +31-6-41278122, PGP-key: 1F28D8AE, www.vanheusden.com

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [2.6.22] circular lock detected
  2007-09-02 11:55 ` Andrew Morton
  2007-09-02 17:11   ` Folkert van Heusden
@ 2007-09-03 12:27   ` Jan Kara
  2007-09-03 12:49     ` Andrew Morton
  1 sibling, 1 reply; 11+ messages in thread
From: Jan Kara @ 2007-09-03 12:27 UTC (permalink / raw)
  To: Andrew Morton; +Cc: Folkert van Heusden, linux-kernel, Jan Kara, Peter Zijlstra

> > On Fri, 24 Aug 2007 23:00:33 +0200 Folkert van Heusden <folkert@vanheusden.com> wrote:
> > Hi,
> > 
> > 2.6.22 kernel with hyperthreading enabled only ext3 filesystems (2).
> > 
> > [  346.314640] =======================================================
> > [  346.314758] [ INFO: possible circular locking dependency detected ]
> > [  346.314815] 2.6.22 #5
> > [  346.314862] -------------------------------------------------------
> > [  346.314920] tor/2421 is trying to acquire lock:
> > [  346.314973]  (tty_mutex){--..}, at: [<c121aa83>] mutex_lock+0x8/0xa
> > [  346.315193] 
> > [  346.315195] but task is already holding lock:
> > [  346.316203]  (&s->s_dquot.dqptr_sem){----}, at: [<c10a485f>] dquot_alloc_space+0x50/0x189
> > [  346.316457] 
> > [  346.316458] which lock already depends on the new lock.
> > [  346.316461] 
> > [  346.316585] 
> > [  346.316586] the existing dependency chain (in reverse order) is:
> > [  346.316675] 
> > [  346.316676] -> #4 (&s->s_dquot.dqptr_sem){----}:
> > [  346.316871]        [<c103b2ee>] check_prev_add+0x15b/0x281
> > [  346.317204]        [<c103b49f>] check_prevs_add+0x8b/0xe8
> > [  346.317535]        [<c103ce2a>] __lock_acquire+0x65d/0xb51
> > [  346.317873]        [<c103d786>] lock_acquire+0x62/0x81
> > [  346.318195]        [<c10341af>] down_read+0x2b/0x3d
> > [  346.318527]        [<c10a485f>] dquot_alloc_space+0x50/0x189
> > [  346.318854]        [<c10b5c55>] ext3_new_blocks+0x44b/0x5a2
> > [  346.319187]        [<c10b81e4>] ext3_alloc_blocks+0x40/0xdf
> > [  346.319513]        [<c10b82d3>] ext3_alloc_branch+0x50/0x21b
> > [  346.319850]        [<c10b87b1>] ext3_get_blocks_handle+0x1b8/0x367
> > [  346.320184]        [<c10b8ada>] ext3_getblk+0x97/0x228
> > [  346.320530]        [<c10b8c85>] ext3_bread+0x1a/0x78
> > [  346.320870]        [<c10bde64>] ext3_mkdir+0xf4/0x270
> > [  346.321188]        [<c107cdf6>] vfs_mkdir+0xb3/0x161
> > [  346.321506]        [<c107cf30>] sys_mkdirat+0x8c/0xc4
> > [  346.321820]        [<c107cf88>] sys_mkdir+0x20/0x22
> > [  346.322131]        [<c1003eea>] syscall_call+0x7/0xb
> > [  346.322446]        [<ffffffff>] 0xffffffff
> > [  346.322779] 
> > [  346.322780] -> #3 (&ei->truncate_mutex){--..}:
> > [  346.322965]        [<c103b2ee>] check_prev_add+0x15b/0x281
> > [  346.323278]        [<c103b49f>] check_prevs_add+0x8b/0xe8
> > [  346.323602]        [<c103ce2a>] __lock_acquire+0x65d/0xb51
> > [  346.323919]        [<c103d786>] lock_acquire+0x62/0x81
> > [  346.324234]        [<c121ab04>] __mutex_lock_slowpath+0x75/0x28c
> > [  346.324857]        [<c121aa83>] mutex_lock+0x8/0xa
> > [  346.325177]        [<c10ba380>] ext3_truncate+0x170/0x468
> > [  346.325501]        [<c1060e9a>] vmtruncate+0xa6/0x116
> > [  346.325828]        [<c1087937>] inode_setattr+0x145/0x16c
> > [  346.326150]        [<c10bb301>] ext3_setattr+0x150/0x22f
> > [  346.326479]        [<c1087cb0>] notify_change+0x352/0x386
> > [  346.326802]        [<c1071af4>] do_truncate+0x5c/0x7f
> > [  346.327116]        [<c107c2e3>] may_open+0x1ec/0x231
> > [  346.327434]        [<c107c4a4>] open_namei+0xdd/0x5a2
> > [  346.327770]        [<c107275b>] do_filp_open+0x2c/0x53
> > [  346.328090]        [<c1072a90>] do_sys_open+0x52/0xd8
> > [  346.328408]        [<c1072b32>] sys_open+0x1c/0x1e
> > [  346.328727]        [<c1003eea>] syscall_call+0x7/0xb
> > [  346.329042]        [<ffffffff>] 0xffffffff
> > [  346.329368] 
> > [  346.329370] -> #2 (&inode->i_alloc_sem){--..}:
> > [  346.329556]        [<c103b2ee>] check_prev_add+0x15b/0x281
> > [  346.329879]        [<c103b49f>] check_prevs_add+0x8b/0xe8
> > [  346.330192]        [<c103ce2a>] __lock_acquire+0x65d/0xb51
> > [  346.330508]        [<c103d786>] lock_acquire+0x62/0x81
> > [  346.330826]        [<c1034245>] down_write+0x2b/0x45
> > [  346.331137]        [<c1087c37>] notify_change+0x2d9/0x386
> > [  346.331453]        [<c1071af4>] do_truncate+0x5c/0x7f
> > [  346.331769]        [<c107c2e3>] may_open+0x1ec/0x231
> > [  346.332087]        [<c107c4a4>] open_namei+0xdd/0x5a2
> > [  346.332401]        [<c107275b>] do_filp_open+0x2c/0x53
> > [  346.332720]        [<c1072a90>] do_sys_open+0x52/0xd8
> > [  346.333037]        [<c1072b32>] sys_open+0x1c/0x1e
> > [  346.333600]        [<c1003eea>] syscall_call+0x7/0xb
> > [  346.333912]        [<ffffffff>] 0xffffffff
> > [  346.334232] 
> > [  346.334233] -> #1 (&sysfs_inode_imutex_key){--..}:
> > [  346.334422]        [<c103b2ee>] check_prev_add+0x15b/0x281
> > [  346.334734]        [<c103b49f>] check_prevs_add+0x8b/0xe8
> > [  346.335046]        [<c103ce2a>] __lock_acquire+0x65d/0xb51
> > [  346.335361]        [<c103d786>] lock_acquire+0x62/0x81
> > [  346.335685]        [<c121ab04>] __mutex_lock_slowpath+0x75/0x28c
> > [  346.336001]        [<c121aa83>] mutex_lock+0x8/0xa
> > [  346.336313]        [<c10b2828>] remove_dir+0x2c/0xf1
> > [  346.336629]        [<c10b28f5>] sysfs_remove_subdir+0x8/0xa
> > [  346.336941]        [<c10b3ccf>] sysfs_remove_group+0x67/0x78
> > [  346.337256]        [<c115cc58>] dpm_sysfs_remove+0x12/0x16
> > [  346.337574]        [<c115c36f>] device_pm_remove+0x21/0x62
> > [  346.337888]        [<c1156dc5>] device_del+0x82/0x291
> > [  346.338199]        [<c1156fdf>] device_unregister+0xb/0x15
> > [  346.338516]        [<c11571da>] device_destroy+0x8d/0x9a
> > [  346.338827]        [<c1142712>] vcs_remove_sysfs+0x1c/0x3a
> > [  346.339142]        [<c1149075>] con_close+0x5e/0x6b
> > [  346.339458]        [<c113aa5e>] release_dev+0x4c4/0x6ce
> > [  346.339767]        [<c113b0d6>] tty_release+0x12/0x1c
> > [  346.340078]        [<c1074479>] __fput+0x145/0x16e
> > [  346.340392]        [<c1074332>] fput+0x19/0x1b
> > [  346.340701]        [<c1072bb3>] filp_close+0x3c/0x75
> > [  346.341012]        [<c1072c55>] sys_close+0x69/0xb4
> > [  346.341325]        [<c1003eea>] syscall_call+0x7/0xb
> > [  346.341892]        [<ffffffff>] 0xffffffff
> > [  346.342213] 
> > [  346.342214] -> #0 (tty_mutex){--..}:
> > [  346.342397]        [<c103b1c7>] check_prev_add+0x34/0x281
> > [  346.342708]        [<c103b49f>] check_prevs_add+0x8b/0xe8
> > [  346.343017]        [<c103ce2a>] __lock_acquire+0x65d/0xb51
> > [  346.343333]        [<c103d786>] lock_acquire+0x62/0x81
> > [  346.343646]        [<c121ab04>] __mutex_lock_slowpath+0x75/0x28c
> > [  346.343956]        [<c121aa83>] mutex_lock+0x8/0xa
> > [  346.344263]        [<c10a41d5>] print_warning+0x8c/0x15d
> > [  346.344575]        [<c10a4993>] dquot_alloc_space+0x184/0x189
> > [  346.344883]        [<c10b5c55>] ext3_new_blocks+0x44b/0x5a2
> > [  346.345193]        [<c10b81e4>] ext3_alloc_blocks+0x40/0xdf
> > [  346.345507]        [<c10b82d3>] ext3_alloc_branch+0x50/0x21b
> > [  346.345816]        [<c10b87b1>] ext3_get_blocks_handle+0x1b8/0x367
> > [  346.346127]        [<c10b89d8>] ext3_get_block+0x78/0xe3
> > [  346.346441]        [<c1094802>] __block_prepare_write+0x168/0x415
> > [  346.346750]        [<c109530e>] block_prepare_write+0x28/0x3b
> > [  346.347057]        [<c10b8e84>] ext3_prepare_write+0xe3/0x17e
> > [  346.347369]        [<c10552f1>] generic_file_buffered_write+0x1b4/0x614
> > [  346.347689]        [<c105598b>] __generic_file_aio_write_nolock+0x23a/0x533
> > [  346.348002]        [<c1055d8d>] generic_file_aio_write+0x58/0xc4
> > [  346.348314]        [<c10b6a93>] ext3_file_write+0x2d/0xba
> > [  346.348627]        [<c107353b>] do_sync_write+0xc7/0x116
> > [  346.348935]        [<c10736e2>] vfs_write+0x158/0x15d
> > [  346.349246]        [<c1073788>] sys_write+0x3d/0x64
> > [  346.349560]        [<c1003eea>] syscall_call+0x7/0xb
> > [  346.349869]        [<ffffffff>] 0xffffffff
> > [  346.350438] 
> > [  346.350440] other info that might help us debug this:
> > [  346.350442] 
> > [  346.350562] 3 locks held by tor/2421:
> > [  346.350604]  #0:  (&inode->i_mutex){--..}, at: [<c121aa83>] mutex_lock+0x8/0xa
> > [  346.350827]  #1:  (&ei->truncate_mutex){--..}, at: [<c121aa83>] mutex_lock+0x8/0xa
> > [  346.351049]  #2:  (&s->s_dquot.dqptr_sem){----}, at: [<c10a485f>] dquot_alloc_space+0x50/0x189
> > [  346.351274] 
> > [  346.351276] stack backtrace:
> > [  346.351363]  [<c1004cac>] show_trace_log_lvl+0x1a/0x30
> > [  346.351446]  [<c1004cd4>] show_trace+0x12/0x14
> > [  346.351526]  [<c1004dce>] dump_stack+0x16/0x18
> > [  346.351605]  [<c103ab29>] print_circular_bug_tail+0x6f/0x71
> > [  346.351686]  [<c103b1c7>] check_prev_add+0x34/0x281
> > [  346.351764]  [<c103b49f>] check_prevs_add+0x8b/0xe8
> > [  346.351843]  [<c103ce2a>] __lock_acquire+0x65d/0xb51
> > [  346.351921]  [<c103d786>] lock_acquire+0x62/0x81
> > [  346.352002]  [<c121ab04>] __mutex_lock_slowpath+0x75/0x28c
> > [  346.352083]  [<c121aa83>] mutex_lock+0x8/0xa
> > [  346.352160]  [<c10a41d5>] print_warning+0x8c/0x15d
> > [  346.352240]  [<c10a4993>] dquot_alloc_space+0x184/0x189
> > [  346.352323]  [<c10b5c55>] ext3_new_blocks+0x44b/0x5a2
> > [  346.352402]  [<c10b81e4>] ext3_alloc_blocks+0x40/0xdf
> > [  346.352479]  [<c10b82d3>] ext3_alloc_branch+0x50/0x21b
> > [  346.352558]  [<c10b87b1>] ext3_get_blocks_handle+0x1b8/0x367
> > [  346.352638]  [<c10b89d8>] ext3_get_block+0x78/0xe3
> > [  346.352720]  [<c1094802>] __block_prepare_write+0x168/0x415
> > [  346.352801]  [<c109530e>] block_prepare_write+0x28/0x3b
> > [  346.352881]  [<c10b8e84>] ext3_prepare_write+0xe3/0x17e
> > [  346.352961]  [<c10552f1>] generic_file_buffered_write+0x1b4/0x614
> > [  346.353041]  [<c105598b>] __generic_file_aio_write_nolock+0x23a/0x533
> > [  346.353121]  [<c1055d8d>] generic_file_aio_write+0x58/0xc4
> > [  346.353200]  [<c10b6a93>] ext3_file_write+0x2d/0xba
> > [  346.353281]  [<c107353b>] do_sync_write+0xc7/0x116
> > [  346.353364]  [<c10736e2>] vfs_write+0x158/0x15d
> > [  346.353444]  [<c1073788>] sys_write+0x3d/0x64
> > [  346.353524]  [<c1003eea>] syscall_call+0x7/0xb
> > [  346.353603]  =======================
> > 
> Has been reported before, but I don't recall whether we fixed it.  Jan,
> do you know>?
  I think we at least found a solution: Teach lockdep that I_MUTEX for
different filesystems is different. Peter Zilstra wrote a patch for that
and Folkert even confirmed that it fixes the problem for him. I'm not
sure what happened with the patch afterwards though. Adding Peter to CC
:).

								Honza
-- 
Jan Kara <jack@suse.cz>
SuSE CR Labs

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [2.6.22] circular lock detected
  2007-09-03 12:27   ` Jan Kara
@ 2007-09-03 12:49     ` Andrew Morton
  2007-09-03 14:01       ` Jan Kara
  0 siblings, 1 reply; 11+ messages in thread
From: Andrew Morton @ 2007-09-03 12:49 UTC (permalink / raw)
  To: Jan Kara; +Cc: folkert, linux-kernel, jack, a.p.zijlstra

> On Mon, 3 Sep 2007 14:27:02 +0200 Jan Kara <jack@suse.cz> wrote:
> > > On Fri, 24 Aug 2007 23:00:33 +0200 Folkert van Heusden <folkert@vanheusden.com> wrote:
> > > Hi,
> > > 
> > > 2.6.22 kernel with hyperthreading enabled only ext3 filesystems (2).
> > > 
> > > [  346.314640] =======================================================
> > > [  346.314758] [ INFO: possible circular locking dependency detected ]
> > > [  346.314815] 2.6.22 #5
> > > [  346.314862] -------------------------------------------------------
> > > [  346.314920] tor/2421 is trying to acquire lock:
> > > [  346.314973]  (tty_mutex){--..}, at: [<c121aa83>] mutex_lock+0x8/0xa
> > > [  346.315193] 
> > > [  346.315195] but task is already holding lock:
> > > [  346.316203]  (&s->s_dquot.dqptr_sem){----}, at: [<c10a485f>] dquot_alloc_space+0x50/0x189
> > > [  346.316457] 
> > > [  346.316458] which lock already depends on the new lock.
>
> ...
>
> > > [  346.353603]  =======================
> > > 
> > Has been reported before, but I don't recall whether we fixed it.  Jan,
> > do you know>?
>   I think we at least found a solution: Teach lockdep that I_MUTEX for
> different filesystems is different. Peter Zilstra wrote a patch for that
> and Folkert even confirmed that it fixes the problem for him. I'm not
> sure what happened with the patch afterwards though. Adding Peter to CC
> :).

But this is a tty_lock-versus-dqptr_sem ranking error.  Unrelated to i_mutex?

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [2.6.22] circular lock detected
  2007-09-03 12:49     ` Andrew Morton
@ 2007-09-03 14:01       ` Jan Kara
  2007-09-25  8:02         ` Peter Zijlstra
  0 siblings, 1 reply; 11+ messages in thread
From: Jan Kara @ 2007-09-03 14:01 UTC (permalink / raw)
  To: Andrew Morton; +Cc: folkert, linux-kernel, a.p.zijlstra

On Mon 03-09-07 05:49:59, Andrew Morton wrote:
> > On Mon, 3 Sep 2007 14:27:02 +0200 Jan Kara <jack@suse.cz> wrote:
> > > > On Fri, 24 Aug 2007 23:00:33 +0200 Folkert van Heusden <folkert@vanheusden.com> wrote:
> > > > Hi,
> > > > 
> > > > 2.6.22 kernel with hyperthreading enabled only ext3 filesystems (2).
> > > > 
> > > > [  346.314640] =======================================================
> > > > [  346.314758] [ INFO: possible circular locking dependency detected ]
> > > > [  346.314815] 2.6.22 #5
> > > > [  346.314862] -------------------------------------------------------
> > > > [  346.314920] tor/2421 is trying to acquire lock:
> > > > [  346.314973]  (tty_mutex){--..}, at: [<c121aa83>] mutex_lock+0x8/0xa
> > > > [  346.315193] 
> > > > [  346.315195] but task is already holding lock:
> > > > [  346.316203]  (&s->s_dquot.dqptr_sem){----}, at: [<c10a485f>] dquot_alloc_space+0x50/0x189
> > > > [  346.316457] 
> > > > [  346.316458] which lock already depends on the new lock.
> >
> > ...
> >
> > > > [  346.353603]  =======================
> > > > 
> > > Has been reported before, but I don't recall whether we fixed it.  Jan,
> > > do you know>?
> >   I think we at least found a solution: Teach lockdep that I_MUTEX for
> > different filesystems is different. Peter Zilstra wrote a patch for that
> > and Folkert even confirmed that it fixes the problem for him. I'm not
> > sure what happened with the patch afterwards though. Adding Peter to CC
> > :).
> 
> But this is a tty_lock-versus-dqptr_sem ranking error.  Unrelated to i_mutex?
  The final report is for this ranking but the locking chain (if I understand it
right) is:
tty_mutex (con_close) -> i_mutex (sysfs: remove_subdir)
i_mutex (do_truncate) -> i_alloc_sem (notify_change) -> truncate_mutex (ext3_truncate)
truncate_mutex (ext3_get_blocks_handle) -> dqptr_sem (dquot_alloc_space)

So it complains about tty_mutex vs dqptr_sem (I don't know why it does not
complain about tty_mutex vs i_mutex) but the wrong link in the chain is
that i_mutex from remove_subdir() [sysfs] and i_mutex from do_truncate()
[ext3] are different and should never depend on each other...

									Honza

-- 
Jan Kara <jack@suse.cz>
SuSE CR Labs

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [2.6.22] circular lock detected
  2007-09-03 14:01       ` Jan Kara
@ 2007-09-25  8:02         ` Peter Zijlstra
  2007-09-25  8:46           ` Jan Kara
  0 siblings, 1 reply; 11+ messages in thread
From: Peter Zijlstra @ 2007-09-25  8:02 UTC (permalink / raw)
  To: Jan Kara; +Cc: Andrew Morton, folkert, linux-kernel

On Mon, 3 Sep 2007 16:01:35 +0200 Jan Kara <jack@suse.cz> wrote:

> On Mon 03-09-07 05:49:59, Andrew Morton wrote:
> > > On Mon, 3 Sep 2007 14:27:02 +0200 Jan Kara <jack@suse.cz> wrote:
> > > > > On Fri, 24 Aug 2007 23:00:33 +0200 Folkert van Heusden <folkert@vanheusden.com> wrote:

> > > > Has been reported before, but I don't recall whether we fixed it.  Jan,
> > > > do you know>?
> > >   I think we at least found a solution: Teach lockdep that I_MUTEX for
> > > different filesystems is different. Peter Zilstra wrote a patch for that
> > > and Folkert even confirmed that it fixes the problem for him. I'm not
> > > sure what happened with the patch afterwards though. Adding Peter to CC
> > > :).
> > 
> > But this is a tty_lock-versus-dqptr_sem ranking error.  Unrelated to i_mutex?
>   The final report is for this ranking but the locking chain (if I understand it
> right) is:
> tty_mutex (con_close) -> i_mutex (sysfs: remove_subdir)
> i_mutex (do_truncate) -> i_alloc_sem (notify_change) -> truncate_mutex (ext3_truncate)
> truncate_mutex (ext3_get_blocks_handle) -> dqptr_sem (dquot_alloc_space)
> 
> So it complains about tty_mutex vs dqptr_sem (I don't know why it does not
> complain about tty_mutex vs i_mutex) but the wrong link in the chain is
> that i_mutex from remove_subdir() [sysfs] and i_mutex from do_truncate()
> [ext3] are different and should never depend on each other...
> 

Found it again.

---

Give each filesystem its own inode lock class. The various filesystems have
different locking order wrt the inode locks; esp. the pseudo filesystems
differ from the rest.

Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
---
 fs/inode.c         |   12 +++++++++---
 include/linux/fs.h |    5 +++++
 2 files changed, 14 insertions(+), 3 deletions(-)

Index: linux-2.6/fs/inode.c
===================================================================
--- linux-2.6.orig/fs/inode.c
+++ linux-2.6/fs/inode.c
@@ -142,6 +142,15 @@ static struct inode *alloc_inode(struct 
 			return NULL;
 		}
 
+		spin_lock_init(&inode->i_lock);
+		lockdep_set_class(&inode->i_lock, &sb->s_type->i_lock_key);
+
+		mutex_init(&inode->i_mutex);
+		lockdep_set_class(&inode->i_mutex, &sb->s_type->i_mutex_key);
+
+		init_rwsem(&inode->i_alloc_sem);
+		lockdep_set_class(&inode->i_alloc_sem, &sb->s_type->i_alloc_sem_key);
+
 		mapping->a_ops = &empty_aops;
  		mapping->host = inode;
 		mapping->flags = 0;
@@ -190,8 +199,6 @@ void inode_init_once(struct inode *inode
 	INIT_HLIST_NODE(&inode->i_hash);
 	INIT_LIST_HEAD(&inode->i_dentry);
 	INIT_LIST_HEAD(&inode->i_devices);
-	mutex_init(&inode->i_mutex);
-	init_rwsem(&inode->i_alloc_sem);
 	INIT_RADIX_TREE(&inode->i_data.page_tree, GFP_ATOMIC);
 	rwlock_init(&inode->i_data.tree_lock);
 	spin_lock_init(&inode->i_data.i_mmap_lock);
@@ -199,7 +206,6 @@ void inode_init_once(struct inode *inode
 	spin_lock_init(&inode->i_data.private_lock);
 	INIT_RAW_PRIO_TREE_ROOT(&inode->i_data.i_mmap);
 	INIT_LIST_HEAD(&inode->i_data.i_mmap_nonlinear);
-	spin_lock_init(&inode->i_lock);
 	i_size_ordered_init(inode);
 #ifdef CONFIG_INOTIFY
 	INIT_LIST_HEAD(&inode->inotify_watches);
Index: linux-2.6/include/linux/fs.h
===================================================================
--- linux-2.6.orig/include/linux/fs.h
+++ linux-2.6/include/linux/fs.h
@@ -1302,8 +1302,13 @@ struct file_system_type {
 	struct module *owner;
 	struct file_system_type * next;
 	struct list_head fs_supers;
+
 	struct lock_class_key s_lock_key;
 	struct lock_class_key s_umount_key;
+
+	struct lock_class_key i_lock_key;
+	struct lock_class_key i_mutex_key;
+	struct lock_class_key i_alloc_sem_key;
 };
 
 extern int get_sb_bdev(struct file_system_type *fs_type,


^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [2.6.22] circular lock detected
  2007-09-25  8:02         ` Peter Zijlstra
@ 2007-09-25  8:46           ` Jan Kara
  2008-01-04  7:40             ` Simon Arlott
  0 siblings, 1 reply; 11+ messages in thread
From: Jan Kara @ 2007-09-25  8:46 UTC (permalink / raw)
  To: Peter Zijlstra; +Cc: Andrew Morton, folkert, linux-kernel

On Tue 25-09-07 10:02:43, Peter Zijlstra wrote:
> On Mon, 3 Sep 2007 16:01:35 +0200 Jan Kara <jack@suse.cz> wrote:
> 
> > On Mon 03-09-07 05:49:59, Andrew Morton wrote:
> > > > On Mon, 3 Sep 2007 14:27:02 +0200 Jan Kara <jack@suse.cz> wrote:
> > > > > > On Fri, 24 Aug 2007 23:00:33 +0200 Folkert van Heusden <folkert@vanheusden.com> wrote:
> 
> > > > > Has been reported before, but I don't recall whether we fixed it.  Jan,
> > > > > do you know>?
> > > >   I think we at least found a solution: Teach lockdep that I_MUTEX for
> > > > different filesystems is different. Peter Zilstra wrote a patch for that
> > > > and Folkert even confirmed that it fixes the problem for him. I'm not
> > > > sure what happened with the patch afterwards though. Adding Peter to CC
> > > > :).
> > > 
> > > But this is a tty_lock-versus-dqptr_sem ranking error.  Unrelated to i_mutex?
> >   The final report is for this ranking but the locking chain (if I understand it
> > right) is:
> > tty_mutex (con_close) -> i_mutex (sysfs: remove_subdir)
> > i_mutex (do_truncate) -> i_alloc_sem (notify_change) -> truncate_mutex (ext3_truncate)
> > truncate_mutex (ext3_get_blocks_handle) -> dqptr_sem (dquot_alloc_space)
> > 
> > So it complains about tty_mutex vs dqptr_sem (I don't know why it does not
> > complain about tty_mutex vs i_mutex) but the wrong link in the chain is
> > that i_mutex from remove_subdir() [sysfs] and i_mutex from do_truncate()
> > [ext3] are different and should never depend on each other...
> > 
> 
> Found it again.
  Cool, thanks Peter. Andrew, would you put it into -mm? This should take care of
the false lockdep warnings from the quota code. If I recall correctly, one
of the reporters even confirmed it fixes the problem for him.
  The patch looks fine. You can add:
Signed-off-by: Jan Kara <jack@suse.cz>

									Honza

> Give each filesystem its own inode lock class. The various filesystems have
> different locking order wrt the inode locks; esp. the pseudo filesystems
> differ from the rest.
> 
> Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
> ---
>  fs/inode.c         |   12 +++++++++---
>  include/linux/fs.h |    5 +++++
>  2 files changed, 14 insertions(+), 3 deletions(-)
> 
> Index: linux-2.6/fs/inode.c
> ===================================================================
> --- linux-2.6.orig/fs/inode.c
> +++ linux-2.6/fs/inode.c
> @@ -142,6 +142,15 @@ static struct inode *alloc_inode(struct 
>  			return NULL;
>  		}
>  
> +		spin_lock_init(&inode->i_lock);
> +		lockdep_set_class(&inode->i_lock, &sb->s_type->i_lock_key);
> +
> +		mutex_init(&inode->i_mutex);
> +		lockdep_set_class(&inode->i_mutex, &sb->s_type->i_mutex_key);
> +
> +		init_rwsem(&inode->i_alloc_sem);
> +		lockdep_set_class(&inode->i_alloc_sem, &sb->s_type->i_alloc_sem_key);
> +
>  		mapping->a_ops = &empty_aops;
>   		mapping->host = inode;
>  		mapping->flags = 0;
> @@ -190,8 +199,6 @@ void inode_init_once(struct inode *inode
>  	INIT_HLIST_NODE(&inode->i_hash);
>  	INIT_LIST_HEAD(&inode->i_dentry);
>  	INIT_LIST_HEAD(&inode->i_devices);
> -	mutex_init(&inode->i_mutex);
> -	init_rwsem(&inode->i_alloc_sem);
>  	INIT_RADIX_TREE(&inode->i_data.page_tree, GFP_ATOMIC);
>  	rwlock_init(&inode->i_data.tree_lock);
>  	spin_lock_init(&inode->i_data.i_mmap_lock);
> @@ -199,7 +206,6 @@ void inode_init_once(struct inode *inode
>  	spin_lock_init(&inode->i_data.private_lock);
>  	INIT_RAW_PRIO_TREE_ROOT(&inode->i_data.i_mmap);
>  	INIT_LIST_HEAD(&inode->i_data.i_mmap_nonlinear);
> -	spin_lock_init(&inode->i_lock);
>  	i_size_ordered_init(inode);
>  #ifdef CONFIG_INOTIFY
>  	INIT_LIST_HEAD(&inode->inotify_watches);
> Index: linux-2.6/include/linux/fs.h
> ===================================================================
> --- linux-2.6.orig/include/linux/fs.h
> +++ linux-2.6/include/linux/fs.h
> @@ -1302,8 +1302,13 @@ struct file_system_type {
>  	struct module *owner;
>  	struct file_system_type * next;
>  	struct list_head fs_supers;
> +
>  	struct lock_class_key s_lock_key;
>  	struct lock_class_key s_umount_key;
> +
> +	struct lock_class_key i_lock_key;
> +	struct lock_class_key i_mutex_key;
> +	struct lock_class_key i_alloc_sem_key;
>  };
>  
>  extern int get_sb_bdev(struct file_system_type *fs_type,
> 
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [2.6.22] circular lock detected
  2007-09-25  8:46           ` Jan Kara
@ 2008-01-04  7:40             ` Simon Arlott
  2008-01-04  9:21               ` Peter Zijlstra
  0 siblings, 1 reply; 11+ messages in thread
From: Simon Arlott @ 2008-01-04  7:40 UTC (permalink / raw)
  To: Jan Kara; +Cc: Peter Zijlstra, Andrew Morton, folkert, linux-kernel

On 25/09/07 09:46, Jan Kara wrote:
> On Tue 25-09-07 10:02:43, Peter Zijlstra wrote:
>> On Mon, 3 Sep 2007 16:01:35 +0200 Jan Kara <jack@suse.cz> wrote:
>> 
>> > On Mon 03-09-07 05:49:59, Andrew Morton wrote:
>> > > > On Mon, 3 Sep 2007 14:27:02 +0200 Jan Kara <jack@suse.cz> wrote:
>> > > > > > On Fri, 24 Aug 2007 23:00:33 +0200 Folkert van Heusden <folkert@vanheusden.com> wrote:
>> 
>> > > > > Has been reported before, but I don't recall whether we fixed it.  Jan,
>> > > > > do you know>?
>> > > >   I think we at least found a solution: Teach lockdep that I_MUTEX for
>> > > > different filesystems is different. Peter Zilstra wrote a patch for that
>> > > > and Folkert even confirmed that it fixes the problem for him. I'm not
>> > > > sure what happened with the patch afterwards though. Adding Peter to CC
>> > > > :).
>> > > 
>> > > But this is a tty_lock-versus-dqptr_sem ranking error.  Unrelated to i_mutex?
>> >   The final report is for this ranking but the locking chain (if I understand it
>> > right) is:
>> > tty_mutex (con_close) -> i_mutex (sysfs: remove_subdir)
>> > i_mutex (do_truncate) -> i_alloc_sem (notify_change) -> truncate_mutex (ext3_truncate)
>> > truncate_mutex (ext3_get_blocks_handle) -> dqptr_sem (dquot_alloc_space)
>> > 
>> > So it complains about tty_mutex vs dqptr_sem (I don't know why it does not
>> > complain about tty_mutex vs i_mutex) but the wrong link in the chain is
>> > that i_mutex from remove_subdir() [sysfs] and i_mutex from do_truncate()
>> > [ext3] are different and should never depend on each other...
>> > 
>> 
>> Found it again.
>   Cool, thanks Peter. Andrew, would you put it into -mm? This should take care of
> the false lockdep warnings from the quota code. If I recall correctly, one
> of the reporters even confirmed it fixes the problem for him.
>   The patch looks fine. You can add:
> Signed-off-by: Jan Kara <jack@suse.cz>
> 
> 									Honza
> 
>> Give each filesystem its own inode lock class. The various filesystems have
>> different locking order wrt the inode locks; esp. the pseudo filesystems
>> differ from the rest.
>> 
>> Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>

This patch still doesn't exist in 2.6.23.9 and the warning isn't fixed...

03 00:31:34 [1905219.899008] =======================================================
03 00:31:34 [1905219.907136] [ INFO: possible circular locking dependency detected ]
03 00:31:34 [1905219.913569] 2.6.23.9-git #43
03 00:31:34 [1905219.916624] -------------------------------------------------------
03 00:31:34 [1905219.923059] bacula-sd/32639 is trying to acquire lock:
03 00:31:34 [1905219.928364]  (tty_mutex){--..}, at: [<80389fd7>] mutex_lock+0x1c/0x1f
03 00:31:34 [1905219.935047]
03 00:31:34 [1905219.935049] but task is already holding lock:
03 00:31:34 [1905219.941238]  (&s->s_dquot.dqptr_sem){----}, at: [<801839bf>] dquot_alloc_space+0x42/0x15b
03 00:31:34 [1905219.949646]
03 00:31:34 [1905219.949649] which lock already depends on the new lock.
03 00:31:34 [1905219.949652]
03 00:31:34 [1905219.958367]
03 00:31:34 [1905219.958369] the existing dependency chain (in reverse order) is:
03 00:31:34 [1905219.966204]
03 00:31:34 [1905219.966206] -> #4 (&s->s_dquot.dqptr_sem){----}:
03 00:31:34 [1905219.972714]        [<80131c5b>] __lock_acquire+0x9ba/0xb96
03 00:31:34 [1905219.978511]        [<80132200>] lock_acquire+0x5d/0x75
03 00:31:34 [1905219.983957]        [<8012ac3c>] down_read+0x3a/0x4b
03 00:31:34 [1905219.989252]        [<801839bf>] dquot_alloc_space+0x42/0x15b
03 00:31:34 [1905219.995217]        [<801920e1>] ext3_new_blocks+0x83/0x5ba
03 00:31:34 [1905220.001009]        [<80195111>] ext3_get_blocks_handle+0x386/0x822
03 00:31:34 [1905220.007495]        [<80195889>] ext3_get_block+0xac/0xc5
03 00:31:34 [1905220.013113]        [<80174e54>] __block_prepare_write+0x151/0x3c6
03 00:31:34 [1905220.019603]        [<801750ed>] block_prepare_write+0x24/0x32
03 00:31:34 [1905220.025654]        [<80196a43>] ext3_prepare_write+0x98/0x153
03 00:31:34 [1905220.031705]        [<8013cbbe>] generic_file_buffered_write+0x219/0x57c
03 00:31:34 [1905220.038630]        [<8013d384>] __generic_file_aio_write_nolock+0x463/0x4b3
03 00:31:34 [1905220.045894]        [<8013d42a>] generic_file_aio_write+0x56/0xb4
03 00:31:34 [1905220.052292]        [<8019311b>] ext3_file_write+0x27/0x99
03 00:31:34 [1905220.057996]        [<80157742>] do_sync_write+0xc4/0x101
03 00:31:34 [1905220.063615]        [<80157f03>] vfs_write+0xaf/0x138
03 00:31:34 [1905220.068885]        [<8015840f>] sys_write+0x3d/0x61
03 00:31:34 [1905220.074071]        [<80102702>] sysenter_past_esp+0x5f/0x99
03 00:31:34 [1905220.080046]        [<ffffffff>] 0xffffffff
03 00:31:34 [1905220.084456]
03 00:31:34 [1905220.084458] -> #3 (&ei->truncate_mutex){--..}:
03 00:31:34 [1905220.090781]        [<80131c5b>] __lock_acquire+0x9ba/0xb96
03 00:31:34 [1905220.096573]        [<80132200>] lock_acquire+0x5d/0x75
03 00:31:34 [1905220.102020]        [<80389e25>] __mutex_lock_slowpath+0xdb/0x271
03 00:31:34 [1905220.108418]        [<80389fd7>] mutex_lock+0x1c/0x1f
03 00:31:34 [1905220.113690]        [<80196110>] ext3_truncate+0x49e/0x7ec
03 00:31:34 [1905220.119397]        [<80148496>] vmtruncate+0x161/0x187
03 00:31:34 [1905220.124847]        [<80169794>] inode_setattr+0x73/0x149
03 00:31:34 [1905220.130470]        [<8019720c>] ext3_setattr+0x185/0x1e1
03 00:31:34 [1905220.136088]        [<8016999a>] notify_change+0x130/0x2d2
03 00:31:34 [1905220.141883]        [<80156c7b>] do_truncate+0x60/0x79
03 00:31:34 [1905220.147245]        [<8015f377>] may_open+0x1b9/0x201
03 00:31:34 [1905220.152522]        [<801612e3>] open_namei+0x240/0x539
03 00:31:34 [1905220.157965]        [<8015651b>] do_filp_open+0x26/0x3b
03 00:31:34 [1905220.163411]        [<80156575>] do_sys_open+0x45/0xc9
03 00:31:34 [1905220.168857]        [<80156631>] sys_open+0x1c/0x1e
03 00:31:34 [1905220.173955]        [<80102702>] sysenter_past_esp+0x5f/0x99
03 00:31:34 [1905220.179830]        [<ffffffff>] 0xffffffff
03 00:31:34 [1905220.184233]
03 00:31:34 [1905220.184235] -> #2 (&inode->i_alloc_sem){--..}:
03 00:31:34 [1905220.190559]        [<80131c5b>] __lock_acquire+0x9ba/0xb96
03 00:31:34 [1905220.196436]        [<80132200>] lock_acquire+0x5d/0x75
03 00:31:34 [1905220.201884]        [<8012acd6>] down_write+0x3a/0x53
03 00:31:34 [1905220.207152]        [<80169952>] notify_change+0xe8/0x2d2
03 00:31:34 [1905220.212772]        [<80156c7b>] do_truncate+0x60/0x79
03 00:31:34 [1905220.218130]        [<8015f377>] may_open+0x1b9/0x201
03 00:31:34 [1905220.223485]        [<801612e3>] open_namei+0x240/0x539
03 00:31:34 [1905220.228926]        [<8015651b>] do_filp_open+0x26/0x3b
03 00:31:34 [1905220.234372]        [<80156575>] do_sys_open+0x45/0xc9
03 00:31:34 [1905220.239727]        [<80156631>] sys_open+0x1c/0x1e
03 00:31:34 [1905220.244819]        [<80102702>] sysenter_past_esp+0x5f/0x99
03 00:31:34 [1905220.250760]        [<ffffffff>] 0xffffffff
03 00:31:34 [1905220.255156]
03 00:31:34 [1905220.255158] -> #1 (&inode->i_mutex){--..}:
03 00:31:34 [1905220.261135]        [<80131c5b>] __lock_acquire+0x9ba/0xb96
03 00:31:34 [1905220.268005]        [<80132200>] lock_acquire+0x5d/0x75
03 00:31:34 [1905220.273448]        [<80389e25>] __mutex_lock_slowpath+0xdb/0x271
03 00:31:34 [1905220.279752]        [<80389fd7>] mutex_lock+0x1c/0x1f
03 00:31:34 [1905220.285024]        [<80190cf6>] get_node+0x1d/0x46
03 00:31:34 [1905220.290119]        [<80190e84>] devpts_get_tty+0xb/0x3b
03 00:31:34 [1905220.295646]        [<80214fb5>] init_dev+0x21/0x4a2
03 00:31:34 [1905220.300837]        [<80217c95>] ptmx_open+0xfd/0x1bc
03 00:31:34 [1905220.306102]        [<80159f91>] chrdev_open+0x14a/0x180
03 00:31:34 [1905220.311618]        [<801563b2>] __dentry_open+0xfd/0x1af
03 00:31:34 [1905220.317232]        [<801564e5>] nameidata_to_filp+0x27/0x37
03 00:31:34 [1905220.323097]        [<80156528>] do_filp_open+0x33/0x3b
03 00:31:34 [1905220.328535]        [<80156575>] do_sys_open+0x45/0xc9
03 00:31:34 [1905220.333874]        [<80156631>] sys_open+0x1c/0x1e
03 00:31:34 [1905220.338981]        [<80102702>] sysenter_past_esp+0x5f/0x99
03 00:31:34 [1905220.344855]        [<ffffffff>] 0xffffffff
03 00:31:34 [1905220.349256]
03 00:31:34 [1905220.349258] -> #0 (tty_mutex){--..}:
03 00:31:34 [1905220.354717]        [<80131b4b>] __lock_acquire+0x8aa/0xb96
03 00:31:34 [1905220.360505]        [<80132200>] lock_acquire+0x5d/0x75
03 00:31:34 [1905220.365956]        [<80389e25>] __mutex_lock_slowpath+0xdb/0x271
03 00:31:34 [1905220.372258]        [<80389fd7>] mutex_lock+0x1c/0x1f
03 00:31:34 [1905220.377517]        [<80182ea9>] print_warning+0x92/0x137
03 00:31:34 [1905220.383139]        [<80183ab8>] dquot_alloc_space+0x13b/0x15b
03 00:31:34 [1905220.389176]        [<801920e1>] ext3_new_blocks+0x83/0x5ba
03 00:31:34 [1905220.394961]        [<80195111>] ext3_get_blocks_handle+0x386/0x822
03 00:31:34 [1905220.401429]        [<80195889>] ext3_get_block+0xac/0xc5
03 00:31:34 [1905220.407047]        [<80174e54>] __block_prepare_write+0x151/0x3c6
03 00:31:34 [1905220.413455]        [<801750ed>] block_prepare_write+0x24/0x32
03 00:31:34 [1905220.419506]        [<80196a43>] ext3_prepare_write+0x98/0x153
03 00:31:34 [1905220.425553]        [<8013cbbe>] generic_file_buffered_write+0x219/0x57c
03 00:31:34 [1905220.432465]        [<8013d384>] __generic_file_aio_write_nolock+0x463/0x4b3
03 00:31:34 [1905220.439724]        [<8013d42a>] generic_file_aio_write+0x56/0xb4
03 00:31:34 [1905220.446027]        [<8019311b>] ext3_file_write+0x27/0x99
03 00:31:34 [1905220.451728]        [<80157742>] do_sync_write+0xc4/0x101
03 00:31:34 [1905220.457338]        [<80157f03>] vfs_write+0xaf/0x138
03 00:31:34 [1905220.462600]        [<8015840f>] sys_write+0x3d/0x61
03 00:31:34 [1905220.467771]        [<80102702>] sysenter_past_esp+0x5f/0x99
03 00:31:34 [1905220.473639]        [<ffffffff>] 0xffffffff
03 00:31:34 [1905220.478040]
03 00:31:34 [1905220.478043] other info that might help us debug this:
03 00:31:34 [1905220.478046]
03 00:31:34 [1905220.486588] 3 locks held by bacula-sd/32639:
03 00:31:34 [1905220.491025]  #0:  (&inode->i_mutex){--..}, at: [<80389fd7>] mutex_lock+0x1c/0x1f
03 00:31:34 [1905220.498668]  #1:  (&ei->truncate_mutex){--..}, at: [<80389fd7>] mutex_lock+0x1c/0x1f
03 00:31:34 [1905220.506659]  #2:  (&s->s_dquot.dqptr_sem){----}, at: [<801839bf>] dquot_alloc_space+0x42/0x15b
03 00:31:34 [1905220.515511]
03 00:31:34 [1905220.515513] stack backtrace:
03 00:31:34 [1905220.520230]  [<80103694>] show_trace_log_lvl+0x1a/0x2f
03 00:31:34 [1905220.525555]  [<8010421b>] show_trace+0x12/0x14
03 00:31:34 [1905220.530185]  [<80104232>] dump_stack+0x15/0x17
03 00:31:34 [1905220.534817]  [<801303c4>] print_circular_bug_tail+0x5e/0x67
03 00:31:34 [1905220.540580]  [<80131b4b>] __lock_acquire+0x8aa/0xb96
03 00:31:34 [1905220.545731]  [<80132200>] lock_acquire+0x5d/0x75
03 00:31:34 [1905220.550536]  [<80389e25>] __mutex_lock_slowpath+0xdb/0x271
03 00:31:34 [1905220.556203]  [<80389fd7>] mutex_lock+0x1c/0x1f
03 00:31:34 [1905220.560838]  [<80182ea9>] print_warning+0x92/0x137
03 00:31:34 [1905220.565819]  [<80183ab8>] dquot_alloc_space+0x13b/0x15b
03 00:31:34 [1905220.571230]  [<801920e1>] ext3_new_blocks+0x83/0x5ba
03 00:31:34 [1905220.576381]  [<80195111>] ext3_get_blocks_handle+0x386/0x822
03 00:31:34 [1905220.582226]  [<80195889>] ext3_get_block+0xac/0xc5
03 00:31:34 [1905220.587205]  [<80174e54>] __block_prepare_write+0x151/0x3c6
03 00:31:34 [1905220.592963]  [<801750ed>] block_prepare_write+0x24/0x32
03 00:31:34 [1905220.598372]  [<80196a43>] ext3_prepare_write+0x98/0x153
03 00:31:34 [1905220.603785]  [<8013cbbe>] generic_file_buffered_write+0x219/0x57c
03 00:31:34 [1905220.610063]  [<8013d384>] __generic_file_aio_write_nolock+0x463/0x4b3
03 00:31:34 [1905220.616687]  [<8013d42a>] generic_file_aio_write+0x56/0xb4
03 00:31:34 [1905220.622362]  [<8019311b>] ext3_file_write+0x27/0x99
03 00:31:34 [1905220.627426]  [<80157742>] do_sync_write+0xc4/0x101
03 00:31:34 [1905220.632401]  [<80157f03>] vfs_write+0xaf/0x138
03 00:31:34 [1905220.637033]  [<8015840f>] sys_write+0x3d/0x61
03 00:31:34 [1905220.641574]  [<80102702>] sysenter_past_esp+0x5f/0x99
03 00:31:34 [1905220.646811]  =======================

>> ---
>>  fs/inode.c         |   12 +++++++++---
>>  include/linux/fs.h |    5 +++++
>>  2 files changed, 14 insertions(+), 3 deletions(-)
>> 
>> Index: linux-2.6/fs/inode.c
>> ===================================================================
>> --- linux-2.6.orig/fs/inode.c
>> +++ linux-2.6/fs/inode.c
>> @@ -142,6 +142,15 @@ static struct inode *alloc_inode(struct 
>>  			return NULL;
>>  		}
>>  
>> +		spin_lock_init(&inode->i_lock);
>> +		lockdep_set_class(&inode->i_lock, &sb->s_type->i_lock_key);
>> +
>> +		mutex_init(&inode->i_mutex);
>> +		lockdep_set_class(&inode->i_mutex, &sb->s_type->i_mutex_key);
>> +
>> +		init_rwsem(&inode->i_alloc_sem);
>> +		lockdep_set_class(&inode->i_alloc_sem, &sb->s_type->i_alloc_sem_key);
>> +
>>  		mapping->a_ops = &empty_aops;
>>   		mapping->host = inode;
>>  		mapping->flags = 0;
>> @@ -190,8 +199,6 @@ void inode_init_once(struct inode *inode
>>  	INIT_HLIST_NODE(&inode->i_hash);
>>  	INIT_LIST_HEAD(&inode->i_dentry);
>>  	INIT_LIST_HEAD(&inode->i_devices);
>> -	mutex_init(&inode->i_mutex);
>> -	init_rwsem(&inode->i_alloc_sem);
>>  	INIT_RADIX_TREE(&inode->i_data.page_tree, GFP_ATOMIC);
>>  	rwlock_init(&inode->i_data.tree_lock);
>>  	spin_lock_init(&inode->i_data.i_mmap_lock);
>> @@ -199,7 +206,6 @@ void inode_init_once(struct inode *inode
>>  	spin_lock_init(&inode->i_data.private_lock);
>>  	INIT_RAW_PRIO_TREE_ROOT(&inode->i_data.i_mmap);
>>  	INIT_LIST_HEAD(&inode->i_data.i_mmap_nonlinear);
>> -	spin_lock_init(&inode->i_lock);
>>  	i_size_ordered_init(inode);
>>  #ifdef CONFIG_INOTIFY
>>  	INIT_LIST_HEAD(&inode->inotify_watches);
>> Index: linux-2.6/include/linux/fs.h
>> ===================================================================
>> --- linux-2.6.orig/include/linux/fs.h
>> +++ linux-2.6/include/linux/fs.h
>> @@ -1302,8 +1302,13 @@ struct file_system_type {
>>  	struct module *owner;
>>  	struct file_system_type * next;
>>  	struct list_head fs_supers;
>> +
>>  	struct lock_class_key s_lock_key;
>>  	struct lock_class_key s_umount_key;
>> +
>> +	struct lock_class_key i_lock_key;
>> +	struct lock_class_key i_mutex_key;
>> +	struct lock_class_key i_alloc_sem_key;
>>  };
>>  
>>  extern int get_sb_bdev(struct file_system_type *fs_type,
>> 


-- 
Simon Arlott

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [2.6.22] circular lock detected
  2008-01-04  7:40             ` Simon Arlott
@ 2008-01-04  9:21               ` Peter Zijlstra
  0 siblings, 0 replies; 11+ messages in thread
From: Peter Zijlstra @ 2008-01-04  9:21 UTC (permalink / raw)
  To: Simon Arlott; +Cc: Jan Kara, Andrew Morton, folkert, linux-kernel

[-- Attachment #1: Type: text/plain, Size: 2478 bytes --]


On Fri, 2008-01-04 at 07:40 +0000, Simon Arlott wrote:
> On 25/09/07 09:46, Jan Kara wrote:
> > On Tue 25-09-07 10:02:43, Peter Zijlstra wrote:
> >> On Mon, 3 Sep 2007 16:01:35 +0200 Jan Kara <jack@suse.cz> wrote:
> >> 
> >> > On Mon 03-09-07 05:49:59, Andrew Morton wrote:
> >> > > > On Mon, 3 Sep 2007 14:27:02 +0200 Jan Kara <jack@suse.cz> wrote:
> >> > > > > > On Fri, 24 Aug 2007 23:00:33 +0200 Folkert van Heusden <folkert@vanheusden.com> wrote:
> >> 
> >> > > > > Has been reported before, but I don't recall whether we fixed it.  Jan,
> >> > > > > do you know>?
> >> > > >   I think we at least found a solution: Teach lockdep that I_MUTEX for
> >> > > > different filesystems is different. Peter Zilstra wrote a patch for that
> >> > > > and Folkert even confirmed that it fixes the problem for him. I'm not
> >> > > > sure what happened with the patch afterwards though. Adding Peter to CC
> >> > > > :).
> >> > > 
> >> > > But this is a tty_lock-versus-dqptr_sem ranking error.  Unrelated to i_mutex?
> >> >   The final report is for this ranking but the locking chain (if I understand it
> >> > right) is:
> >> > tty_mutex (con_close) -> i_mutex (sysfs: remove_subdir)
> >> > i_mutex (do_truncate) -> i_alloc_sem (notify_change) -> truncate_mutex (ext3_truncate)
> >> > truncate_mutex (ext3_get_blocks_handle) -> dqptr_sem (dquot_alloc_space)
> >> > 
> >> > So it complains about tty_mutex vs dqptr_sem (I don't know why it does not
> >> > complain about tty_mutex vs i_mutex) but the wrong link in the chain is
> >> > that i_mutex from remove_subdir() [sysfs] and i_mutex from do_truncate()
> >> > [ext3] are different and should never depend on each other...
> >> > 
> >> 
> >> Found it again.
> >   Cool, thanks Peter. Andrew, would you put it into -mm? This should take care of
> > the false lockdep warnings from the quota code. If I recall correctly, one
> > of the reporters even confirmed it fixes the problem for him.
> >   The patch looks fine. You can add:
> > Signed-off-by: Jan Kara <jack@suse.cz>
> > 
> > 									Honza
> > 
> >> Give each filesystem its own inode lock class. The various filesystems have
> >> different locking order wrt the inode locks; esp. the pseudo filesystems
> >> differ from the rest.
> >> 
> >> Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
> 
> This patch still doesn't exist in 2.6.23.9 and the warning isn't fixed...

It would have been if you'd used .24-rc :-)

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

^ permalink raw reply	[flat|nested] 11+ messages in thread

end of thread, other threads:[~2008-01-04  9:22 UTC | newest]

Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2007-08-24 21:00 [2.6.22] circular lock detected Folkert van Heusden
2007-08-26 23:03 ` Michal Piotrowski
2007-09-02 11:55 ` Andrew Morton
2007-09-02 17:11   ` Folkert van Heusden
2007-09-03 12:27   ` Jan Kara
2007-09-03 12:49     ` Andrew Morton
2007-09-03 14:01       ` Jan Kara
2007-09-25  8:02         ` Peter Zijlstra
2007-09-25  8:46           ` Jan Kara
2008-01-04  7:40             ` Simon Arlott
2008-01-04  9:21               ` Peter Zijlstra

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).