Many processes stuck in blk_congestion_wait

Many processes stuck in blk_congestion_wait

[Sev Binello]

We are experiencing numerous hangs/slow downs on our WS4 U5 nfs client 
machines (not seen on WS3).
Many processes end up in a D state for long periods of time
and are in  blk_congestion_wait().

Time in this state can vary,
not sure what triggers them to come out, some don't
But time in this state can be quite lengthy.
Machines are running 2.6.9-55.0.9.ELsmp

Below is a sample trace, they all look pretty much the same....

Any clues how to debug this ?
Thanks
-Sev

-- 

Sev Binello
Brookhaven National Laboratory
Upton, New York
631-344-5647
sev@bnl.gov

Jan 23 12:39:27 acnmcr4p kernel: RampEditor    D C02D6EB4  1288 
21855      1         21857 22139 (NOTLB)
Jan 23 12:39:27 acnmcr4p kernel: f6e5ec94 00200086 65cc8b98 c02d6eb4 
f56a9430 e3923830 00200200 00000000 Jan 23 12:39:27 acnmcr4p kernel: 
c35c4860 00000000 c35bc780 c35bbde0 00000000 00000000 0d1d4140 001553b9 
Jan 23 12:39:27 acnmcr4p kernel:e3923830 e8fd93f0 e8fd955c 00000000 
00200246 65cef381 65cef381 f6e5ed04 Jan 23 12:39:27 acnmcr4p kernel: 
Call Trace:
Jan 23 12:39:27 acnmcr4p kernel:  [<c02d6eb4>] common_interrupt+0x18/0x20
Jan 23 12:39:27 acnmcr4p kernel:  [<c02d4c5a>] schedule_timeout+0x139/0x154
Jan 23 12:39:27 acnmcr4p kernel:  [<c012a73a>] process_timeout+0x0/0x5
Jan 23 12:39:27 acnmcr4p kernel:  [<c02d4b17>] 
io_schedule_timeout+0x26/0x30
Jan 23 12:39:27 acnmcr4p kernel:  [<c02267f8>] 
blk_congestion_wait+0x64/0x78
Jan 23 12:39:27 acnmcr4p kernel:  [<c012052d>] 
autoremove_wake_function+0x0/0x2d
Jan 23 12:39:27 acnmcr4p kernel:  [<c0145038>] 
get_writeback_state+0x30/0x35
Jan 23 12:39:27 acnmcr4p kernel:  [<c012052d>] 
autoremove_wake_function+0x0/0x2d
Jan 23 12:39:27 acnmcr4p kernel:  [<c0145291>] 
balance_dirty_pages+0xbe/0x11c
Jan 23 12:39:27 acnmcr4p kernel:  [<f8f86105>] 
nfs_commit_write+0x43/0x72 [nfs]
Jan 23 12:39:27 acnmcr4p kernel:  [<c0142517>] 
generic_file_buffered_write+0x41f/0x501
Jan 23 12:39:27 acnmcr4p kernel:  [<c01ac299>] inode_has_perm+0x4c/0x54
Jan 23 12:39:27 acnmcr4p kernel:  [<c011dd34>] move_tasks+0x19d/0x202
Jan 23 12:39:27 acnmcr4p kernel:  [<c0142982>] 
__generic_file_aio_write_nolock+0x389/0x3b7
Jan 23 12:39:27 acnmcr4p kernel:  [<c01429e9>] 
generic_file_aio_write_nolock+0x39/0x7f
Jan 23 12:39:27 acnmcr4p kernel:  [<c0142bd3>] 
generic_file_aio_write+0x72/0xc6
Jan 23 12:39:27 acnmcr4p kernel:  [<f8f86212>] nfs_file_write+0xde/0xf9 
[nfs]
Jan 23 12:39:27 acnmcr4p kernel:  [<c015b5b8>] do_sync_write+0x9e/0xcb
Jan 23 12:39:27 acnmcr4p kernel:  [<c016be58>] poll_freewait+0x33/0x38
Jan 23 12:39:27 acnmcr4p kernel:  [<c01adf3a>] 
selinux_file_permission+0x117/0x120
Jan 23 12:39:27 acnmcr4p kernel:  [<c012052d>] 
autoremove_wake_function+0x0/0x2d
Jan 23 12:39:27 acnmcr4p kernel:  [<c015b69b>] vfs_write+0xb6/0xe2
Jan 23 12:39:27 acnmcr4p kernel:  [<c015b765>] sys_write+0x3c/0x62
Jan 23 12:39:27 acnmcr4p kernel:  [<c02d64db>] syscall_call+0x7/0xb
Jan 23 12:39:27 acnmcr4p kernel:  [<c02d007b>] 
unix_dgram_sendmsg+0x23c/0x45d

Re: Many processes stuck in blk_congestion_wait

[Fengguang Wu]

On Wed, Jan 23, 2008 at 05:19:14PM -0500, Sev Binello wrote:
> We are experiencing numerous hangs/slow downs on our WS4 U5 nfs client 
> machines (not seen on WS3).
> Many processes end up in a D state for long periods of time
> and are in  blk_congestion_wait().
> 
> Time in this state can vary,
> not sure what triggers them to come out, some don't
> But time in this state can be quite lengthy.

> Machines are running 2.6.9-55.0.9.ELsmp

It's a pretty old kernel.  If possible, a kernel upgrade may well fix
your problem.