LKML Archive on lore.kernel.org
help / color / mirror / Atom feed
* system hang on latest git
@ 2008-01-29 19:15 Luck, Tony
  2008-01-29 20:11 ` Jens Axboe
  0 siblings, 1 reply; 6+ messages in thread
From: Luck, Tony @ 2008-01-29 19:15 UTC (permalink / raw)
  To: LKML; +Cc: linux-ia64

[-- Attachment #1: Type: text/plain, Size: 2838 bytes --]

I pulled Linus' tree this morning (git head = 0ba6c33bcddc64a54b5f1c25a696c4767dc76292)
and built for ia64 (using arch/ia64/configs/tiger_defconfig).   System booted
OK, but when I stressed it a little (building another kernel with "make -j32")
it hung.

The console has a bunch (98) of warnings about tasks blocked for more than 120
seconds like this:
INFO: task grep:9168 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001ed8afbf0 bsp=e0000001ed8a1280
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001ed8afc20 bsp=e0000001ed8a1208
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001ed8afca0 bsp=e0000001ed8a11c8
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001ed8afca0 bsp=e0000001ed8a1190
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001ed8afca0 bsp=e0000001ed8a1158
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001ed8afca0 bsp=e0000001ed8a1130
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001ed8afcc0 bsp=e0000001ed8a1100
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001ed8afcc0 bsp=e0000001ed8a10b0
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001ed8afcc0 bsp=e0000001ed8a1088
 [<a0000001000d6c20>] do_generic_mapping_read+0x780/0x7a0
                                sp=e0000001ed8afce0 bsp=e0000001ed8a0fe0
 [<a0000001000db250>] generic_file_aio_read+0x290/0x340
                                sp=e0000001ed8afce0 bsp=e0000001ed8a0f80
 [<a00000010012c990>] do_sync_read+0x170/0x200
                                sp=e0000001ed8afd10 bsp=e0000001ed8a0f40
 [<a00000010012cbd0>] vfs_read+0x1b0/0x2e0
                                sp=e0000001ed8afe20 bsp=e0000001ed8a0ef0
 [<a00000010012d250>] sys_read+0x70/0xe0
                                sp=e0000001ed8afe20 bsp=e0000001ed8a0e78
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001ed8afe30 bsp=e0000001ed8a0e78


[The stack trace has several variations ... some from sys_read(), some from
sys_open(), some from sys_execve(), some from sys_mmap() etc. 84/98 stack
traces pass through the touch_atime->__mark_inode_dirty path ... all 98
are attached]

A quick dig into processor state shows 8 cpus are idle.  7 are spinning
in __spin_lock_irq() from __make_request() and one is in spin_lock() from
as_merged_requests().

-Tony


[-- Attachment #2: traces.txt --]
[-- Type: text/plain, Size: 233883 bytes --]

"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001b00bfca0 bsp=e0000001b00b0e30
 [<a000000100704cf0>] io_schedule+0x50/0x80
                                sp=e0000001b00bfcd0 bsp=e0000001b00b0e10
 [<a000000100179d60>] sync_buffer+0x100/0x120
                                sp=e0000001b00bfcd0 bsp=e0000001b00b0de8
 [<a000000100705130>] __wait_on_bit+0xd0/0x180
                                sp=e0000001b00bfcd0 bsp=e0000001b00b0d98
 [<a0000001007052c0>] out_of_line_wait_on_bit+0xe0/0x100
                                sp=e0000001b00bfcd0 bsp=e0000001b00b0d50
 [<a000000100179b80>] __wait_on_buffer+0x40/0x60
                                sp=e0000001b00bfd50 bsp=e0000001b00b0d30
 [<a000000100253250>] journal_commit_transaction+0x1110/0x1fc0
                                sp=e0000001b00bfd50 bsp=e0000001b00b0c90
 [<a000000100258c60>] kjournald+0x180/0x420
                                sp=e0000001b00bfdc0 bsp=e0000001b00b0c30
 [<a0000001000a7c40>] kthread+0xa0/0x120
                                sp=e0000001b00bfe30 bsp=e0000001b00b0c00
 [<a000000100013970>] kernel_thread_helper+0x30/0x60
                                sp=e0000001b00bfe30 bsp=e0000001b00b0bd0
 [<a0000001000089c0>] start_kernel_thread+0x20/0x40
                                sp=e0000001b00bfe30 bsp=e0000001b00b0bd0
INFO: task make:8649 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001e97afb40 bsp=e0000001e97a0d40
 [<a000000100704e60>] schedule_timeout+0x40/0x180
                                sp=e0000001e97afb70 bsp=e0000001e97a0d10
 [<a000000100704450>] wait_for_common+0x1b0/0x300
                                sp=e0000001e97afba0 bsp=e0000001e97a0cc0
 [<a0000001007045e0>] wait_for_completion+0x40/0x60
                                sp=e0000001e97afc00 bsp=e0000001e97a0ca0
 [<a000000100077240>] do_fork+0x420/0x480
                                sp=e0000001e97afc00 bsp=e0000001e97a0c50
 [<a000000100009d80>] sys_clone+0x60/0x80
                                sp=e0000001e97afc20 bsp=e0000001e97a0c00
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001e97afe30 bsp=e0000001e97a0c00
 [<a000000000010740>] __kernel_syscall_via_break+0x0/0x20
                                sp=e0000001e97b0000 bsp=e0000001e97a0c00
INFO: task cc1:8839 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001edc4fbf0 bsp=e0000001edc41268
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001edc4fc20 bsp=e0000001edc411e8
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001edc4fca0 bsp=e0000001edc411b0
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001edc4fca0 bsp=e0000001edc41178
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001edc4fca0 bsp=e0000001edc41140
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001edc4fca0 bsp=e0000001edc41118
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001edc4fcc0 bsp=e0000001edc410e8
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001edc4fcc0 bsp=e0000001edc41098
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001edc4fcc0 bsp=e0000001edc41070
 [<a0000001000d6c20>] do_generic_mapping_read+0x780/0x7a0
                                sp=e0000001edc4fce0 bsp=e0000001edc40fc8
 [<a0000001000db250>] generic_file_aio_read+0x290/0x340
                                sp=e0000001edc4fce0 bsp=e0000001edc40f68
 [<a00000010012c990>] do_sync_read+0x170/0x200
                                sp=e0000001edc4fd10 bsp=e0000001edc40f28
 [<a00000010012cbd0>] vfs_read+0x1b0/0x2e0
                                sp=e0000001edc4fe20 bsp=e0000001edc40ed8
 [<a00000010012d250>] sys_read+0x70/0xe0
                                sp=e0000001edc4fe20 bsp=e0000001edc40e60
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001edc4fe30 bsp=e0000001edc40e60
 [<a000000000010740>] __kernel_syscall_via_break+0x0/0x20
                                sp=e0000001edc50000 bsp=e0000001edc40e60
INFO: task sh:9107 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001eea8f6d0 bsp=e0000001eea81448
 [<a000000100704cf0>] io_schedule+0x50/0x80
                                sp=e0000001eea8f700 bsp=e0000001eea81428
 [<a000000100179d60>] sync_buffer+0x100/0x120
                                sp=e0000001eea8f700 bsp=e0000001eea81408
 [<a000000100705130>] __wait_on_bit+0xd0/0x180
                                sp=e0000001eea8f700 bsp=e0000001eea813b0
 [<a0000001007052c0>] out_of_line_wait_on_bit+0xe0/0x100
                                sp=e0000001eea8f700 bsp=e0000001eea81368
 [<a000000100179b80>] __wait_on_buffer+0x40/0x60
                                sp=e0000001eea8f780 bsp=e0000001eea81348
 [<a000000100184370>] __bread+0x190/0x220
                                sp=e0000001eea8f780 bsp=e0000001eea81318
 [<a00000010022a6e0>] ext3_get_branch+0xe0/0x220
                                sp=e0000001eea8f780 bsp=e0000001eea812c0
 [<a00000010022f130>] ext3_get_blocks_handle+0x130/0x1560
                                sp=e0000001eea8f780 bsp=e0000001eea811d8
 [<a0000001002306f0>] ext3_get_block+0x190/0x200
                                sp=e0000001eea8f850 bsp=e0000001eea81180
 [<a00000010018dfd0>] do_mpage_readpage+0x310/0xb80
                                sp=e0000001eea8f850 bsp=e0000001eea810d8
 [<a00000010018e9b0>] mpage_readpages+0x170/0x240
                                sp=e0000001eea8fc50 bsp=e0000001eea81088
 [<a00000010022be10>] ext3_readpages+0x30/0x60
                                sp=e0000001eea8fcd0 bsp=e0000001eea81050
 [<a0000001000e7120>] __do_page_cache_readahead+0x2e0/0x4c0
                                sp=e0000001eea8fcd0 bsp=e0000001eea80fe0
 [<a0000001000e7a40>] ondemand_readahead+0x460/0x4a0
                                sp=e0000001eea8fce0 bsp=e0000001eea80f80
 [<a0000001000e7ad0>] page_cache_sync_readahead+0x50/0x80
                                sp=e0000001eea8fce0 bsp=e0000001eea80f40
 [<a0000001000d6560>] do_generic_mapping_read+0xc0/0x7a0
                                sp=e0000001eea8fce0 bsp=e0000001eea80ea0
 [<a0000001000db250>] generic_file_aio_read+0x290/0x340
                                sp=e0000001eea8fce0 bsp=e0000001eea80e40
 [<a00000010012c990>] do_sync_read+0x170/0x200
                                sp=e0000001eea8fd10 bsp=e0000001eea80e00
 [<a00000010012cbd0>] vfs_read+0x1b0/0x2e0
                                sp=e0000001eea8fe20 bsp=e0000001eea80da8
 [<a000000100136800>] kernel_read+0x60/0xa0
                                sp=e0000001eea8fe20 bsp=e0000001eea80d68
 [<a000000100138140>] prepare_binprm+0x1e0/0x200
                                sp=e0000001eea8fe30 bsp=e0000001eea80d40
 [<a000000100138c50>] do_execve+0x150/0x320
                                sp=e0000001eea8fe30 bsp=e0000001eea80cd8
 [<a000000100013760>] sys_execve+0x60/0xc0
                                sp=e0000001eea8fe30 bsp=e0000001eea80ca0
 [<a000000100009b90>] ia64_execve+0x30/0x140
                                sp=e0000001eea8fe30 bsp=e0000001eea80c50
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001eea8fe30 bsp=e0000001eea80c50
 [<a000000000010740>] __kernel_syscall_via_break+0x0/0x20
                                sp=e0000001eea90000 bsp=e0000001eea80c50
INFO: task gcc:9122 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001ecfdfb80 bsp=e0000001ecfd10c0
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001ecfdfbb0 bsp=e0000001ecfd1048
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001ecfdfc30 bsp=e0000001ecfd1010
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001ecfdfc30 bsp=e0000001ecfd0fd0
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001ecfdfc30 bsp=e0000001ecfd0f98
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001ecfdfc30 bsp=e0000001ecfd0f70
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001ecfdfc50 bsp=e0000001ecfd0f40
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001ecfdfc50 bsp=e0000001ecfd0ef0
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001ecfdfc50 bsp=e0000001ecfd0ec8
 [<a00000010013f100>] __link_path_walk+0x820/0x21a0
                                sp=e0000001ecfdfc70 bsp=e0000001ecfd0e48
 [<a000000100140b40>] link_path_walk+0xc0/0x260
                                sp=e0000001ecfdfc90 bsp=e0000001ecfd0e10
 [<a00000010013e570>] path_walk+0x30/0x60
                                sp=e0000001ecfdfd20 bsp=e0000001ecfd0de0
 [<a000000100141200>] do_path_lookup+0x520/0x580
                                sp=e0000001ecfdfd20 bsp=e0000001ecfd0d90
 [<a000000100141de0>] __user_walk_fd+0x60/0xc0
                                sp=e0000001ecfdfd30 bsp=e0000001ecfd0d58
 [<a000000100134350>] vfs_stat_fd+0x30/0xa0
                                sp=e0000001ecfdfd30 bsp=e0000001ecfd0d28
 [<a0000001001348d0>] sys_newstat+0x30/0x80
                                sp=e0000001ecfdfdc0 bsp=e0000001ecfd0cc8
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001ecfdfe30 bsp=e0000001ecfd0cc8
 [<a000000000010740>] __kernel_syscall_via_break+0x0/0x20
                                sp=e0000001ecfe0000 bsp=e0000001ecfd0cc8
INFO: task sh:9128 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001ef28fbf0 bsp=e0000001ef281138
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001ef28fc20 bsp=e0000001ef2810c0
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001ef28fca0 bsp=e0000001ef281088
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001ef28fca0 bsp=e0000001ef281050
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001ef28fca0 bsp=e0000001ef281018
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001ef28fca0 bsp=e0000001ef280fe8
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001ef28fcc0 bsp=e0000001ef280fb8
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001ef28fcc0 bsp=e0000001ef280f68
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001ef28fcc0 bsp=e0000001ef280f40
 [<a0000001000d6c20>] do_generic_mapping_read+0x780/0x7a0
                                sp=e0000001ef28fce0 bsp=e0000001ef280ea0
 [<a0000001000db250>] generic_file_aio_read+0x290/0x340
                                sp=e0000001ef28fce0 bsp=e0000001ef280e40
 [<a00000010012c990>] do_sync_read+0x170/0x200
                                sp=e0000001ef28fd10 bsp=e0000001ef280e00
 [<a00000010012cbd0>] vfs_read+0x1b0/0x2e0
                                sp=e0000001ef28fe20 bsp=e0000001ef280da8
 [<a000000100136800>] kernel_read+0x60/0xa0
                                sp=e0000001ef28fe20 bsp=e0000001ef280d68
 [<a000000100138140>] prepare_binprm+0x1e0/0x200
                                sp=e0000001ef28fe30 bsp=e0000001ef280d40
 [<a000000100138c50>] do_execve+0x150/0x320
                                sp=e0000001ef28fe30 bsp=e0000001ef280cd8
 [<a000000100013760>] sys_execve+0x60/0xc0
                                sp=e0000001ef28fe30 bsp=e0000001ef280ca0
 [<a000000100009b90>] ia64_execve+0x30/0x140
                                sp=e0000001ef28fe30 bsp=e0000001ef280c50
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001ef28fe30 bsp=e0000001ef280c50
 [<a000000000010740>] __kernel_syscall_via_break+0x0/0x20
                                sp=e0000001ef290000 bsp=e0000001ef280c50
INFO: task sh:9136 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001e5bdfbf0 bsp=e0000001e5bd1138
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001e5bdfc20 bsp=e0000001e5bd10c0
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001e5bdfca0 bsp=e0000001e5bd1088
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001e5bdfca0 bsp=e0000001e5bd1050
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001e5bdfca0 bsp=e0000001e5bd1018
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001e5bdfca0 bsp=e0000001e5bd0fe8
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001e5bdfcc0 bsp=e0000001e5bd0fb8
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001e5bdfcc0 bsp=e0000001e5bd0f68
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001e5bdfcc0 bsp=e0000001e5bd0f40
 [<a0000001000d6c20>] do_generic_mapping_read+0x780/0x7a0
                                sp=e0000001e5bdfce0 bsp=e0000001e5bd0ea0
 [<a0000001000db250>] generic_file_aio_read+0x290/0x340
                                sp=e0000001e5bdfce0 bsp=e0000001e5bd0e40
 [<a00000010012c990>] do_sync_read+0x170/0x200
                                sp=e0000001e5bdfd10 bsp=e0000001e5bd0e00
 [<a00000010012cbd0>] vfs_read+0x1b0/0x2e0
                                sp=e0000001e5bdfe20 bsp=e0000001e5bd0da8
 [<a000000100136800>] kernel_read+0x60/0xa0
                                sp=e0000001e5bdfe20 bsp=e0000001e5bd0d68
 [<a000000100138140>] prepare_binprm+0x1e0/0x200
                                sp=e0000001e5bdfe30 bsp=e0000001e5bd0d40
 [<a000000100138c50>] do_execve+0x150/0x320
                                sp=e0000001e5bdfe30 bsp=e0000001e5bd0cd8
 [<a000000100013760>] sys_execve+0x60/0xc0
                                sp=e0000001e5bdfe30 bsp=e0000001e5bd0ca0
 [<a000000100009b90>] ia64_execve+0x30/0x140
                                sp=e0000001e5bdfe30 bsp=e0000001e5bd0c50
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001e5bdfe30 bsp=e0000001e5bd0c50
 [<a000000000010740>] __kernel_syscall_via_break+0x0/0x20
                                sp=e0000001e5be0000 bsp=e0000001e5bd0c50
INFO: task mv:9143 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001b2aafbe0 bsp=e0000001b2aa12a8
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001b2aafc10 bsp=e0000001b2aa1230
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001b2aafc90 bsp=e0000001b2aa11f0
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001b2aafc90 bsp=e0000001b2aa11b8
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001b2aafc90 bsp=e0000001b2aa1180
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001b2aafc90 bsp=e0000001b2aa1158
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001b2aafcb0 bsp=e0000001b2aa1128
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001b2aafcb0 bsp=e0000001b2aa10d8
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001b2aafcb0 bsp=e0000001b2aa10b0
 [<a00000010013ffa0>] __link_path_walk+0x16c0/0x21a0
                                sp=e0000001b2aafcd0 bsp=e0000001b2aa1030
 [<a000000100140b40>] link_path_walk+0xc0/0x260
                                sp=e0000001b2aafcf0 bsp=e0000001b2aa0ff0
 [<a00000010013e570>] path_walk+0x30/0x60
                                sp=e0000001b2aafd80 bsp=e0000001b2aa0fc8
 [<a000000100141200>] do_path_lookup+0x520/0x580
                                sp=e0000001b2aafd80 bsp=e0000001b2aa0f78
 [<a0000001001415e0>] __path_lookup_intent_open+0x80/0x120
                                sp=e0000001b2aafd90 bsp=e0000001b2aa0f28
 [<a0000001001416c0>] path_lookup_open+0x40/0x60
                                sp=e0000001b2aafd90 bsp=e0000001b2aa0ee8
 [<a000000100142b60>] open_namei+0xc0/0xf40
                                sp=e0000001b2aafd90 bsp=e0000001b2aa0e48
 [<a00000010012b150>] do_filp_open+0x50/0xc0
                                sp=e0000001b2aafda0 bsp=e0000001b2aa0e10
 [<a00000010012b9b0>] do_sys_open+0x90/0x1c0
                                sp=e0000001b2aafe30 bsp=e0000001b2aa0db8
 [<a00000010012bb30>] sys_open+0x50/0x80
                                sp=e0000001b2aafe30 bsp=e0000001b2aa0d60
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001b2aafe30 bsp=e0000001b2aa0d60
INFO: task mv:9150 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001ecfafbf0 bsp=e0000001ecfa1168
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001ecfafc20 bsp=e0000001ecfa10f0
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001ecfafca0 bsp=e0000001ecfa10b8
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001ecfafca0 bsp=e0000001ecfa1080
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001ecfafca0 bsp=e0000001ecfa1048
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001ecfafca0 bsp=e0000001ecfa1020
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001ecfafcc0 bsp=e0000001ecfa0fe8
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001ecfafcc0 bsp=e0000001ecfa0f98
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001ecfafcc0 bsp=e0000001ecfa0f70
 [<a0000001000d6c20>] do_generic_mapping_read+0x780/0x7a0
                                sp=e0000001ecfafce0 bsp=e0000001ecfa0ed0
 [<a0000001000db250>] generic_file_aio_read+0x290/0x340
                                sp=e0000001ecfafce0 bsp=e0000001ecfa0e70
 [<a00000010012c990>] do_sync_read+0x170/0x200
                                sp=e0000001ecfafd10 bsp=e0000001ecfa0e30
 [<a00000010012cbd0>] vfs_read+0x1b0/0x2e0
                                sp=e0000001ecfafe20 bsp=e0000001ecfa0dd8
 [<a00000010012d250>] sys_read+0x70/0xe0
                                sp=e0000001ecfafe20 bsp=e0000001ecfa0d60
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001ecfafe30 bsp=e0000001ecfa0d60
INFO: task sh:9154 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001eb41fbf0 bsp=e0000001eb411138
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001eb41fc20 bsp=e0000001eb4110c0
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001eb41fca0 bsp=e0000001eb411088
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001eb41fca0 bsp=e0000001eb411050
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001eb41fca0 bsp=e0000001eb411018
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001eb41fca0 bsp=e0000001eb410fe8
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001eb41fcc0 bsp=e0000001eb410fb8
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001eb41fcc0 bsp=e0000001eb410f68
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001eb41fcc0 bsp=e0000001eb410f40
 [<a0000001000d6c20>] do_generic_mapping_read+0x780/0x7a0
                                sp=e0000001eb41fce0 bsp=e0000001eb410ea0
 [<a0000001000db250>] generic_file_aio_read+0x290/0x340
                                sp=e0000001eb41fce0 bsp=e0000001eb410e40
 [<a00000010012c990>] do_sync_read+0x170/0x200
                                sp=e0000001eb41fd10 bsp=e0000001eb410e00
 [<a00000010012cbd0>] vfs_read+0x1b0/0x2e0
                                sp=e0000001eb41fe20 bsp=e0000001eb410da8
 [<a000000100136800>] kernel_read+0x60/0xa0
                                sp=e0000001eb41fe20 bsp=e0000001eb410d68
 [<a000000100138140>] prepare_binprm+0x1e0/0x200
                                sp=e0000001eb41fe30 bsp=e0000001eb410d40
 [<a000000100138c50>] do_execve+0x150/0x320
                                sp=e0000001eb41fe30 bsp=e0000001eb410cd8
 [<a000000100013760>] sys_execve+0x60/0xc0
                                sp=e0000001eb41fe30 bsp=e0000001eb410ca0
 [<a000000100009b90>] ia64_execve+0x30/0x140
                                sp=e0000001eb41fe30 bsp=e0000001eb410c50
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001eb41fe30 bsp=e0000001eb410c50
 [<a000000000010740>] __kernel_syscall_via_break+0x0/0x20
                                sp=e0000001eb420000 bsp=e0000001eb410c50
INFO: task sh:9155 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001ecdffbf0 bsp=e0000001ecdf1138
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001ecdffc20 bsp=e0000001ecdf10c0
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001ecdffca0 bsp=e0000001ecdf1088
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001ecdffca0 bsp=e0000001ecdf1050
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001ecdffca0 bsp=e0000001ecdf1018
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001ecdffca0 bsp=e0000001ecdf0fe8
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001ecdffcc0 bsp=e0000001ecdf0fb8
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001ecdffcc0 bsp=e0000001ecdf0f68
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001ecdffcc0 bsp=e0000001ecdf0f40
 [<a0000001000d6c20>] do_generic_mapping_read+0x780/0x7a0
                                sp=e0000001ecdffce0 bsp=e0000001ecdf0ea0
 [<a0000001000db250>] generic_file_aio_read+0x290/0x340
                                sp=e0000001ecdffce0 bsp=e0000001ecdf0e40
 [<a00000010012c990>] do_sync_read+0x170/0x200
                                sp=e0000001ecdffd10 bsp=e0000001ecdf0e00
 [<a00000010012cbd0>] vfs_read+0x1b0/0x2e0
                                sp=e0000001ecdffe20 bsp=e0000001ecdf0da8
 [<a000000100136800>] kernel_read+0x60/0xa0
                                sp=e0000001ecdffe20 bsp=e0000001ecdf0d68
 [<a000000100138140>] prepare_binprm+0x1e0/0x200
                                sp=e0000001ecdffe30 bsp=e0000001ecdf0d40
 [<a000000100138c50>] do_execve+0x150/0x320
                                sp=e0000001ecdffe30 bsp=e0000001ecdf0cd8
 [<a000000100013760>] sys_execve+0x60/0xc0
                                sp=e0000001ecdffe30 bsp=e0000001ecdf0ca0
 [<a000000100009b90>] ia64_execve+0x30/0x140
                                sp=e0000001ecdffe30 bsp=e0000001ecdf0c50
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001ecdffe30 bsp=e0000001ecdf0c50
 [<a000000000010740>] __kernel_syscall_via_break+0x0/0x20
                                sp=e0000001ece00000 bsp=e0000001ecdf0c50
INFO: task gcc:9156 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001ee6cfbf0 bsp=e0000001ee6c11e0
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001ee6cfc20 bsp=e0000001ee6c1168
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001ee6cfca0 bsp=e0000001ee6c1130
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001ee6cfca0 bsp=e0000001ee6c10f8
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001ee6cfca0 bsp=e0000001ee6c10c0
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001ee6cfca0 bsp=e0000001ee6c1098
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001ee6cfcc0 bsp=e0000001ee6c1068
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001ee6cfcc0 bsp=e0000001ee6c1018
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001ee6cfcc0 bsp=e0000001ee6c0fe8
 [<a0000001000d6c20>] do_generic_mapping_read+0x780/0x7a0
                                sp=e0000001ee6cfce0 bsp=e0000001ee6c0f48
 [<a0000001000db250>] generic_file_aio_read+0x290/0x340
                                sp=e0000001ee6cfce0 bsp=e0000001ee6c0ee8
 [<a00000010012c990>] do_sync_read+0x170/0x200
                                sp=e0000001ee6cfd10 bsp=e0000001ee6c0ea8
 [<a00000010012cbd0>] vfs_read+0x1b0/0x2e0
                                sp=e0000001ee6cfe20 bsp=e0000001ee6c0e58
 [<a000000100136800>] kernel_read+0x60/0xa0
                                sp=e0000001ee6cfe20 bsp=e0000001ee6c0e18
 [<a000000100138140>] prepare_binprm+0x1e0/0x200
                                sp=e0000001ee6cfe30 bsp=e0000001ee6c0de8
 [<a000000100138c50>] do_execve+0x150/0x320
                                sp=e0000001ee6cfe30 bsp=e0000001ee6c0d80
 [<a000000100013760>] sys_execve+0x60/0xc0
                                sp=e0000001ee6cfe30 bsp=e0000001ee6c0d48
 [<a000000100009b90>] ia64_execve+0x30/0x140
                                sp=e0000001ee6cfe30 bsp=e0000001ee6c0cf8
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001ee6cfe30 bsp=e0000001ee6c0cf8
 [<a000000000010740>] __kernel_syscall_via_break+0x0/0x20
                                sp=e0000001ee6d0000 bsp=e0000001ee6c0cf8
INFO: task sh:9157 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001ee41fbf0 bsp=e0000001ee411138
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001ee41fc20 bsp=e0000001ee4110c0
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001ee41fca0 bsp=e0000001ee411088
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001ee41fca0 bsp=e0000001ee411050
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001ee41fca0 bsp=e0000001ee411018
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001ee41fca0 bsp=e0000001ee410fe8
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001ee41fcc0 bsp=e0000001ee410fb8
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001ee41fcc0 bsp=e0000001ee410f68
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001ee41fcc0 bsp=e0000001ee410f40
 [<a0000001000d6c20>] do_generic_mapping_read+0x780/0x7a0
                                sp=e0000001ee41fce0 bsp=e0000001ee410ea0
 [<a0000001000db250>] generic_file_aio_read+0x290/0x340
                                sp=e0000001ee41fce0 bsp=e0000001ee410e40
 [<a00000010012c990>] do_sync_read+0x170/0x200
                                sp=e0000001ee41fd10 bsp=e0000001ee410e00
 [<a00000010012cbd0>] vfs_read+0x1b0/0x2e0
                                sp=e0000001ee41fe20 bsp=e0000001ee410da8
 [<a000000100136800>] kernel_read+0x60/0xa0
                                sp=e0000001ee41fe20 bsp=e0000001ee410d68
 [<a000000100138140>] prepare_binprm+0x1e0/0x200
                                sp=e0000001ee41fe30 bsp=e0000001ee410d40
 [<a000000100138c50>] do_execve+0x150/0x320
                                sp=e0000001ee41fe30 bsp=e0000001ee410cd8
 [<a000000100013760>] sys_execve+0x60/0xc0
                                sp=e0000001ee41fe30 bsp=e0000001ee410ca0
 [<a000000100009b90>] ia64_execve+0x30/0x140
                                sp=e0000001ee41fe30 bsp=e0000001ee410c50
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001ee41fe30 bsp=e0000001ee410c50
 [<a000000000010740>] __kernel_syscall_via_break+0x0/0x20
                                sp=e0000001ee420000 bsp=e0000001ee410c50
INFO: task sh:9158 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001ee38fbf0 bsp=e0000001ee381138
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001ee38fc20 bsp=e0000001ee3810c0
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001ee38fca0 bsp=e0000001ee381088
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001ee38fca0 bsp=e0000001ee381050
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001ee38fca0 bsp=e0000001ee381018
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001ee38fca0 bsp=e0000001ee380fe8
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001ee38fcc0 bsp=e0000001ee380fb8
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001ee38fcc0 bsp=e0000001ee380f68
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001ee38fcc0 bsp=e0000001ee380f40
 [<a0000001000d6c20>] do_generic_mapping_read+0x780/0x7a0
                                sp=e0000001ee38fce0 bsp=e0000001ee380ea0
 [<a0000001000db250>] generic_file_aio_read+0x290/0x340
                                sp=e0000001ee38fce0 bsp=e0000001ee380e40
 [<a00000010012c990>] do_sync_read+0x170/0x200
                                sp=e0000001ee38fd10 bsp=e0000001ee380e00
 [<a00000010012cbd0>] vfs_read+0x1b0/0x2e0
                                sp=e0000001ee38fe20 bsp=e0000001ee380da8
 [<a000000100136800>] kernel_read+0x60/0xa0
                                sp=e0000001ee38fe20 bsp=e0000001ee380d68
 [<a000000100138140>] prepare_binprm+0x1e0/0x200
                                sp=e0000001ee38fe30 bsp=e0000001ee380d40
 [<a000000100138c50>] do_execve+0x150/0x320
                                sp=e0000001ee38fe30 bsp=e0000001ee380cd8
 [<a000000100013760>] sys_execve+0x60/0xc0
                                sp=e0000001ee38fe30 bsp=e0000001ee380ca0
 [<a000000100009b90>] ia64_execve+0x30/0x140
                                sp=e0000001ee38fe30 bsp=e0000001ee380c50
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001ee38fe30 bsp=e0000001ee380c50
 [<a000000000010740>] __kernel_syscall_viX%É•…­­0x0/0x20
                                sp=e0000001ee390000 bsp=e0000001ee380c50
INFO: task sh:9159 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001ef04fbc0 bsp=e0000001ef041298
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001ef04fbf0 bsp=e0000001ef041220
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001ef04fc70 bsp=e0000001ef0411e0
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001ef04fc70 bsp=e0000001ef0411a8
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001ef04fc70 bsp=e0000001ef041170
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001ef04fc70 bsp=e0000001ef041148
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001ef04fc90 bsp=e0000001ef041118
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001ef04fc90 bsp=e0000001ef0410c8
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001ef04fc90 bsp=e0000001ef0410a0
 [<a00000010013ffa0>] __link_path_walk+0x16c0/0x21a0
                                sp=e0000001ef04fcb0 bsp=e0000001ef041020
 [<a000000100140b40>] link_path_walk+0xc0/0x260
                                sp=e0000001ef04fcd0 bsp=e0000001ef040fe0
 [<a00000010013e570>] path_walk+0x30/0x60
                                sp=e0000001ef04fd60 bsp=e0000001ef040fb8
 [<a000000100141200>] do_path_lookup+0x520/0x580
                                sp=e0000001ef04fd60 bsp=e0000001ef040f68
 [<a0000001001415e0>] __path_lookup_intent_open+0x80/0x120
                                sp=e0000001ef04fd70 bsp=e0000001ef040f18
 [<a0000001001416c0>] path_lookup_open+0x40/0x60
                                sp=e0000001ef04fd70 bsp=e0000001ef040ed8
 [<a000000100136640>] open_exec+0x40/0x1a0
                                sp=e0000001ef04fd70 bsp=e0000001ef040eb8
 [<a000000100198c90>] load_elf_binary+0x530/0x2700
                                sp=e0000001ef04fe00 bsp=e0000001ef040da0
 [<a000000100138710>] search_binary_handler+0x1b0/0x5a0
                                sp=e0000001ef04fe30 bsp=e0000001ef040d40
 [<a000000100138d40>] do_execve+0x240/0x320
                                sp=e0000001ef04fe30 bsp=e0000001ef040cd8
 [<a000000100013760>] sys_execve+0x60/0xc0
                                sp=e0000001ef04fe30 bsp=e0000001ef040ca0
 [<a000000100009b90>] ia64_execve+0x30/0x140
                                sp=e0000001ef04fe30 bsp=e0000001ef040c50
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001ef04fe30 bsp=e0000001ef040c50
 [<a000000000010740>] __kernel_syscall_via_break+0x0/0x20
                                sp=e0000001ef050000 bsp=e0000001ef040c50
INFO: task sh:9160 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001ea03fbc0 bsp=e0000001ea031288
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001ea03fbf0 bsp=e0000001ea031210
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001ea03fc70 bsp=e0000001ea0311d0
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001ea03fc70 bsp=e0000001ea031198
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001ea03fc70 bsp=e0000001ea031160
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001ea03fc70 bsp=e0000001ea031138
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001ea03fc90 bsp=e0000001ea031108
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001ea03fc90 bsp=e0000001ea0310b8
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001ea03fc90 bsp=e0000001ea031090
 [<a0000001000d6c20>] do_generic_mapping_read+0x780/0x7a0
                                sp=e0000001ea03fcb0 bsp=e0000001ea030fe8
 [<a0000001000db250>] generic_file_aio_read+0x290/0x340
                                sp=e0000001ea03fcb0 bsp=e0000001ea030f88
 [<a00000010012c990>] do_sync_read+0x170/0x200
                                sp=e0000001ea03fce0 bsp=e0000001ea030f48
 [<a00000010012cbd0>] vfs_read+0x1b0/0x2e0
                                sp=e0000001ea03fdf0 bsp=e0000001ea030ef8
 [<a000000100136800>] kernel_read+0x60/0xa0
                                sp=e0000001ea03fdf0 bsp=e0000001ea030eb8
 [<a000000100198d00>] load_elf_binary+0x5a0/0x2700
                                sp=e0000001ea03fe00 bsp=e0000001ea030da0
 [<a000000100138710>] search_binary_handler+0x1b0/0x5a0
                                sp=e0000001ea03fe30 bsp=e0000001ea030d40
 [<a000000100138d40>] do_execve+0x240/0x320
                                sp=e0000001ea03fe30 bsp=e0000001ea030cd8
 [<a000000100013760>] sys_execve+0x60/0xc0
                                sp=e0000001ea03fe30 bsp=e0000001ea030ca0
 [<a000000100009b90>] ia64_execve+0x30/0x140
                                sp=e0000001ea03fe30 bsp=e0000001ea030c50
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001ea03fe30 bsp=e0000001ea030c50
 [<a000000000010740>] __kernel_syscall_via_break+0x0/0x20
                                sp=e0000001ea040000 bsp=e0000001ea030c50
INFO: task objdump:9161 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001ee78fd10 bsp=e0000001ee781240
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001ee78fd40 bsp=e0000001ee7811c0
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001ee78fdc0 bsp=e0000001ee781188
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001ee78fdc0 bsp=e0000001ee781150
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001ee78fdc0 bsp=e0000001ee781118
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001ee78fdc0 bsp=e0000001ee7810f0
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001ee78fde0 bsp=e0000001ee7810c0
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001ee78fde0 bsp=e0000001ee781070
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001ee78fde0 bsp=e0000001ee781048
 [<a0000001000d79b0>] generic_file_mmap+0x90/0x100
                                sp=e0000001ee78fe00 bsp=e0000001ee781020
 [<a000000100101dd0>] mmap_region+0x3b0/0x7a0
                                sp=e0000001ee78fe00 bsp=e0000001ee780f68
 [<a000000100102820>] do_mmap_pgoff+0x660/0x680
                                sp=e0000001ee78fe30 bsp=e0000001ee780ef8
 [<a000000100033000>] sys_mmap+0x1a0/0x260
                                sp=e0000001ee78fe30 bsp=e0000001ee780e70
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001ee78fe30 bsp=e0000001ee780e70
INFO: task grep:9162 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001eb39fbe0 bsp=e0000001eb3912a8
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001eb39fc10 bsp=e0000001eb391230
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001eb39fc90 bsp=e0000001eb3911f0
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001eb39fc90 bsp=e0000001eb3911b8
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001eb39fc90 bsp=e0000001eb391180
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001eb39fc90 bsp=e0000001eb391158
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001eb39fcb0 bsp=e0000001eb391128
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001eb39fcb0 bsp=e0000001eb3910d8
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001eb39fcb0 bsp=e0000001eb3910b0
 [<a00000010013ffa0>] __link_path_walk+0x16c0/0x21a0
                                sp=e0000001eb39fcd0 bsp=e0000001eb391030
 [<a000000100140b40>] link_path_walk+0xc0/0x260
                                sp=e0000001eb39fcf0 bsp=e0000001eb390ff0
 [<a00000010013e570>] path_walk+0x30/0x60
                                sp=e0000001eb39fd80 bsp=e0000001eb390fc8
 [<a000000100141200>] do_path_lookup+0x520/0x580
                                sp=e0000001eb39fd80 bsp=e0000001eb390f78
 [<a0000001001415e0>] __path_lookup_intent_open+0x80/0x120
                                sp=e0000001eb39fd90 bsp=e0000001eb390f28
 [<a0000001001416c0>] path_lookup_open+0x40/0x60
                                sp=e0000001eb39fd90 bsp=e0000001eb390ee8
 [<a000000100142b60>] open_namei+0xc0/0xf40
                                sp=e0000001eb39fd90 bsp=e0000001eb390e48
 [<a00000010012b150>] do_filp_open+0x50/0xc0
                                sp=e0000001eb39fda0 bsp=e0000001eb390e10
 [<a00000010012b9b0>] do_sys_open+0x90/0x1c0
                                sp=e0000001eb39fe30 bsp=e0000001eb390db8
 [<a00000010012bb30>] sys_open+0x50/0x80
                                sp=e0000001eb39fe30 bsp=e0000001eb390d60
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001eb39fe30 bsp=e0000001eb390d60
INFO: task objdump:9163 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001e99bfbf0 bsp=e0000001e99b1168
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001e99bfc20 bsp=e0000001e99b10f0
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001e99bfca0 bsp=e0000001e99b10b8
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001e99bfca0 bsp=e0000001e99b1080
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001e99bfca0 bsp=e0000001e99b1048
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001e99bfca0 bsp=e0000001e99b1020
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001e99bfcc0 bsp=e0000001e99b0fe8
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001e99bfcc0 bsp=e0000001e99b0f98
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001e99bfcc0 bsp=e0000001e99b0f70
 [<a0000001000d6c20>] do_generic_mapping_read+0x780/0x7a0
                                sp=e0000001e99bfce0 bsp=e0000001e99b0ed0
 [<a0000001000db250>] generic_file_aio_read+0x290/0x340
                                sp=e0000001e99bfce0 bsp=e0000001e99b0e70
 [<a00000010012c990>] do_sync_read+0x170/0x200
                                sp=e0000001e99bfd10 bsp=e0000001e99b0e30
 [<a00000010012cbd0>] vfs_read+0x1b0/0x2e0
                                sp=e0000001e99bfe20 bsp=e0000001e99b0dd8
 [<a00000010012d250>] sys_read+0x70/0xe0
                                sp=e0000001e99bfe20 bsp=e0000001e99b0d60
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001e99bfe30 bsp=e0000001e99b0d60
INFO: task grep:9164 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001ed63fbf0 bsp=e0000001ed631168
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001ed63fc20 bsp=e0000001ed6310f0
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001ed63fca0 bsp=e0000001ed6310b8
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001ed63fca0 bsp=e0000001ed631080
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001ed63fca0 bsp=e0000001ed631048
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001ed63fca0 bsp=e0000001ed631020
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001ed63fcc0 bsp=e0000001ed630fe8
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001ed63fcc0 bsp=e0000001ed630f98
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001ed63fcc0 bsp=e0000001ed630f70
 [<a0000001000d6c20>] do_generic_mapping_read+0x780/0x7a0
                                sp=e0000001ed63fce0 bsp=e0000001ed630ed0
 [<a0000001000db250>] generic_file_aio_read+0x290/0x340
                                sp=e0000001ed63fce0 bsp=e0000001ed630e70
 [<a00000010012c990>] do_sync_read+0x170/0x200
                                sp=e0000001ed63fd10 bsp=e0000001ed630e30
 [<a00000010012cbd0>] vfs_read+0x1b0/0x2e0
                                sp=e0000001ed63fe20 bsp=e0000001ed630dd8
 [<a00000010012d250>] sys_read+0x70/0xe0
                                sp=e0000001ed63fe20 bsp=e0000001ed630d60
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001ed63fe30 bsp=e0000001ed630d60
INFO: task objdump:9165 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001ee18fbf0 bsp=e0000001ee181280
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001ee18fc20 bsp=e0000001ee181208
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001ee18fca0 bsp=e0000001ee1811c8
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001ee18fca0 bsp=e0000001ee181190
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001ee18fca0 bsp=e0000001ee181158
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001ee18fca0 bsp=e0000001ee181130
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001ee18fcc0 bsp=e0000001ee181100
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001ee18fcc0 bsp=e0000001ee1810b0
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001ee18fcc0 bsp=e0000001ee181088
 [<a0000001000d6c20>] do_generic_mapping_read+0x780/0x7a0
                                sp=e0000001ee18fce0 bsp=e0000001ee180fe0
 [<a0000001000db250>] generic_file_aio_read+0x290/0x340
                                sp=e0000001ee18fce0 bsp=e0000001ee180f80
 [<a00000010012c990>] do_sync_read+0x170/0x200
                                sp=e0000001ee18fd10 bsp=e0000001ee180f40
 [<a00000010012cbd0>] vfs_read+0x1b0/0x2e0
                                sp=e0000001ee18fe20 bsp=e0000001ee180ef0
 [<a00000010012d250>] sys_read+0x70/0xe0
                                sp=e0000001ee18fe20 bsp=e0000001ee180e78
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001ee18fe30 bsp=e0000001ee180e78
INFO: task grep:9166 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001eb50fbe0 bsp=e0000001eb5013b8
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001eb50fc10 bsp=e0000001eb501340
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001eb50fc90 bsp=e0000001eb501308
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001eb50fc90 bsp=e0000001eb5012d0
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001eb50fc90 bsp=e0000001eb501298
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001eb50fc90 bsp=e0000001eb501270
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001eb50fcb0 bsp=e0000001eb501240
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001eb50fcb0 bsp=e0000001eb5011e8
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001eb50fcb0 bsp=e0000001eb5011c0
 [<a00000010013ffa0>] __link_path_walk+0x16c0/0x21a0
                                sp=e0000001eb50fcd0 bsp=e0000001eb501140
 [<a000000100140b40>] link_path_walk+0xc0/0x260
                                sp=e0000001eb50fcf0 bsp=e0000001eb501108
 [<a00000010013e570>] path_walk+0x30/0x60
                                sp=e0000001eb50fd80 bsp=e0000001eb5010e0
 [<a000000100141200>] do_path_lookup+0x520/0x580
                                sp=e0000001eb50fd80 bsp=e0000001eb501090
 [<a0000001001415e0>] __path_lookup_intent_open+0x80/0x120
                                sp=e0000001eb50fd90 bsp=e0000001eb501040
 [<a0000001001416c0>] path_lookup_open+0x40/0x60
                                sp=e0000001eb50fd90 bsp=e0000001eb501000
 [<a000000100142b60>] open_namei+0xc0/0xf40
                                sp=e0000001eb50fd90 bsp=e0000001eb500f58
 [<a00000010012b150>] do_filp_open+0x50/0xc0
                                sp=e0000001eb50fda0 bsp=e0000001eb500f20
 [<a00000010012b9b0>] do_sys_open+0x90/0x1c0
                                sp=e0000001eb50fe30 bsp=e0000001eb500ed0
 [<a00000010012bb30>] sys_open+0x50/0x80
                                sp=e0000001eb50fe30 bsp=e0000001eb500e78
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001eb50fe30 bsp=e0000001eb500e78
INFO: task objdump:9167 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001b7c7fbe0 bsp=e0000001b7c713b8
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001b7c7fc10 bsp=e0000001b7c71340
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001b7c7fc90 bsp=e0000001b7c71308
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001b7c7fc90 bsp=e0000001b7c712d0
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001b7c7fc90 bsp=e0000001b7c71298
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001b7c7fc90 bsp=e0000001b7c71270
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001b7c7fcb0 bsp=e0000001b7c71240
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001b7c7fcb0 bsp=e0000001b7c711e8
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001b7c7fcb0 bsp=e0000001b7c711c0
 [<a00000010013ffa0>] __link_path_walk+0x16c0/0x21a0
                                sp=e0000001b7c7fcd0 bsp=e0000001b7c71140
 [<a000000100140b40>] link_path_walk+0xc0/0x260
                                sp=e0000001b7c7fcf0 bsp=e0000001b7c71108
 [<a00000010013e570>] path_walk+0x30/0x60
                                sp=e0000001b7c7fd80 bsp=e0000001b7c710e0
 [<a000000100141200>] do_path_lookup+0x520/0x580
                                sp=e0000001b7c7fd80 bsp=e0000001b7c71090
 [<a0000001001415e0>] __path_lookup_intent_open+0x80/0x120
                                sp=e0000001b7c7fd90 bsp=e0000001b7c71040
 [<a0000001001416c0>] path_lookup_open+0x40/0x60
                                sp=e0000001b7c7fd90 bsp=e0000001b7c71000
 [<a000000100142b60>] open_namei+0xc0/0xf40
                                sp=e0000001b7c7fd90 bsp=e0000001b7c70f58
 [<a00000010012b150>] do_filp_open+0x50/0xc0
                                sp=e0000001b7c7fda0 bsp=e0000001b7c70f20
 [<a00000010012b9b0>] do_sys_open+0x90/0x1c0
                                sp=e0000001b7c7fe30 bsp=e0000001b7c70ed0
 [<a00000010012bb30>] sys_open+0x50/0x80
                                sp=e0000001b7c7fe30 bsp=e0000001b7c70e78
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001b7c7fe30 bsp=e0000001b7c70e78
INFO: task grep:9168 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001ed8afbf0 bsp=e0000001ed8a1280
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001ed8afc20 bsp=e0000001ed8a1208
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001ed8afca0 bsp=e0000001ed8a11c8
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001ed8afca0 bsp=e0000001ed8a1190
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001ed8afca0 bsp=e0000001ed8a1158
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001ed8afca0 bsp=e0000001ed8a1130
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001ed8afcc0 bsp=e0000001ed8a1100
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001ed8afcc0 bsp=e0000001ed8a10b0
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001ed8afcc0 bsp=e0000001ed8a1088
 [<a0000001000d6c20>] do_generic_mapping_read+0x780/0x7a0
                                sp=e0000001ed8afce0 bsp=e0000001ed8a0fe0
 [<a0000001000db250>] generic_file_aio_read+0x290/0x340
                                sp=e0000001ed8afce0 bsp=e0000001ed8a0f80
 [<a00000010012c990>] do_sync_read+0x170/0x200
                                sp=e0000001ed8afd10 bsp=e0000001ed8a0f40
 [<a00000010012cbd0>] vfs_read+0x1b0/0x2e0
                                sp=e0000001ed8afe20 bsp=e0000001ed8a0ef0
 [<a00000010012d250>] sys_read+0x70/0xe0
                                sp=e0000001ed8afe20 bsp=e0000001ed8a0e78
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001ed8afe30 bsp=e0000001ed8a0e78
INFO: task kjournald:1272 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001b00bfca0 bsp=e0000001b00b0e30
 [<a000000100704cf0>] io_schedule+0x50/0x80
                                sp=e0000001b00bfcd0 bsp=e0000001b00b0e10
 [<a000000100179d60>] sync_buffer+0x100/0x120
                                sp=e0000001b00bfcd0 bsp=e0000001b00b0de8
 [<a000000100705130>] __wait_on_bit+0xd0/0x180
                                sp=e0000001b00bfcd0 bsp=e0000001b00b0d98
 [<a0000001007052c0>] out_of_line_wait_on_bit+0xe0/0x100
                                sp=e0000001b00bfcd0 bsp=e0000001b00b0d50
 [<a000000100179b80>] __wait_on_buffer+0x40/0x60
                                sp=e0000001b00bfd50 bsp=e0000001b00b0d30
 [<a000000100253250>] journal_commit_transaction+0x1110/0x1fc0
                                sp=e0000001b00bfd50 bsp=e0000001b00b0c90
 [<a000000100258c60>] kjournald+0x180/0x420
                                sp=e0000001b00bfdc0 bsp=e0000001b00b0c30
 [<a0000001000a7c40>] kthread+0xa0/0x120
                                sp=e0000001b00bfe30 bsp=e0000001b00b0c00
 [<a000000100013970>] kernel_thread_helper+0x30/0x60
                                sp=e0000001b00bfe30 bsp=e0000001b00b0bd0
 [<a0000001000089c0>] start_kernel_thread+0x20/0x40
                                sp=e0000001b00bfe30 bsp=e0000001b00b0bd0
INFO: task syslogd:4887 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001bdf2fcd0 bsp=e0000001bdf20f70
 [<a000000100259240>] log_wait_commit+0x180/0x220
                                sp=e0000001bdf2fd00 bsp=e0000001bdf20f28
 [<a00000010024d350>] journal_stop+0x3f0/0x460
                                sp=e0000001bdf2fd60 bsp=e0000001bdf20ed0
 [<a00000010024d440>] journal_force_commit+0x80/0xa0
                                sp=e0000001bdf2fd70 bsp=e0000001bdf20eb0
 [<a000000100241240>] ext3_force_commit+0x60/0x80
                                sp=e0000001bdf2fd70 bsp=e0000001bdf20e90
 [<a00000010022d5e0>] ext3_write_inode+0xa0/0xc0
                                sp=e0000001bdf2fd70 bsp=e0000001bdf20e68
 [<a000000100171870>] __writeback_single_inode+0x510/0x7c0
                                sp=e0000001bdf2fd70 bsp=e0000001bdf20e10
 [<a000000100171de0>] sync_inode+0x40/0x80
                                sp=e0000001bdf2fdf0 bsp=e0000001bdf20dd8
 [<a000000100227bd0>] ext3_sync_file+0x1b0/0x1e0
                                sp=e0000001bdf2fdf0 bsp=e0000001bdf20db0
 [<a0000001001788b0>] do_fsync+0xf0/0x160
                                sp=e0000001bdf2fe30 bsp=e0000001bdf20d70
 [<a000000100178970>] __do_fsync+0x50/0x80
                                sp=e0000001bdf2fe30 bsp=e0000001bdf20d40
 [<a0000001001789d0>] sys_fsync+0x30/0x60
                                sp=e0000001bdf2fe30 bsp=e0000001bdf20ce8
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001bdf2fe30 bsp=e0000001bdf20ce8
 [<a000000000010740>] __kernel_syscall_via_break+0x0/0x20
                                sp=e0000001bdf30000 bsp=e0000001bdf20ce8
INFO: task make:8649 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001e97afb40 bsp=e0000001e97a0d40
 [<a000000100704e60>] schedule_timeout+0x40/0x180
                                sp=e0000001e97afb70 bsp=e0000001e97a0d10
 [<a000000100704450>] wait_for_common+0x1b0/0x300
                                sp=e0000001e97afba0 bsp=e0000001e97a0cc0
 [<a0000001007045e0>] wait_for_completion+0x40/0x60
                                sp=e0000001e97afc00 bsp=e0000001e97a0ca0
 [<a000000100077240>] do_fork+0x420/0x480
                                sp=e0000001e97afc00 bsp=e0000001e97a0c50
 [<a000000100009d80>] sys_clone+0x60/0x80
                                sp=e0000001e97afc20 bsp=e0000001e97a0c00
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001e97afe30 bsp=e0000001e97a0c00
 [<a000000000010740>] __kernel_syscall_via_break+0x0/0x20
                                sp=e0000001e97b0000 bsp=e0000001e97a0c00
INFO: task cc1:8839 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001edc4fbf0 bsp=e0000001edc41268
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001edc4fc20 bsp=e0000001edc411e8
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001edc4fca0 bsp=e0000001edc411b0
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001edc4fca0 bsp=e0000001edc41178
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001edc4fca0 bsp=e0000001edc41140
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001edc4fca0 bsp=e0000001edc41118
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001edc4fcc0 bsp=e0000001edc410e8
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001edc4fcc0 bsp=e0000001edc41098
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001edc4fcc0 bsp=e0000001edc41070
 [<a0000001000d6c20>] do_generic_mapping_read+0x780/0x7a0
                                sp=e0000001edc4fce0 bsp=e0000001edc40fc8
 [<a0000001000db250>] generic_file_aio_read+0x290/0x340
                                sp=e0000001edc4fce0 bsp=e0000001edc40f68
 [<a00000010012c990>] do_sync_read+0x170/0x200
                                sp=e0000001edc4fd10 bsp=e0000001edc40f28
 [<a00000010012cbd0>] vfs_read+0x1b0/0x2e0
                                sp=e0000001edc4fe20 bsp=e0000001edc40ed8
 [<a00000010012d250>] sys_read+0x70/0xe0
                                sp=e0000001edc4fe20 bsp=e0000001edc40e60
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001edc4fe30 bsp=e0000001edc40e60
 [<a000000000010740>] __kernel_syscall_via_break+0x0/0x20
                                sp=e0000001edc50000 bsp=e0000001edc40e60
INFO: task sh:9107 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001eea8f6d0 bsp=e0000001eea81448
 [<a000000100704cf0>] io_schedule+0x50/0x80
                                sp=e0000001eea8f700 bsp=e0000001eea81428
 [<a000000100179d60>] sync_buffer+0x100/0x120
                                sp=e0000001eea8f700 bsp=e0000001eea81408
 [<a000000100705130>] __wait_on_bit+0xd0/0x180
                                sp=e0000001eea8f700 bsp=e0000001eea813b0
 [<a0000001007052c0>] out_of_line_wait_on_bit+0xe0/0x100
                                sp=e0000001eea8f700 bsp=e0000001eea81368
 [<a000000100179b80>] __wait_on_buffer+0x40/0x60
                                sp=e0000001eea8f780 bsp=e0000001eea81348
 [<a000000100184370>] __bread+0x190/0x220
                                sp=e0000001eea8f780 bsp=e0000001eea81318
 [<a00000010022a6e0>] ext3_get_branch+0xe0/0x220
                                sp=e0000001eea8f780 bsp=e0000001eea812c0
 [<a00000010022f130>] ext3_get_blocks_handle+0x130/0x1560
                                sp=e0000001eea8f780 bsp=e0000001eea811d8
 [<a0000001002306f0>] ext3_get_block+0x190/0x200
                                sp=e0000001eea8f850 bsp=e0000001eea81180
 [<a00000010018dfd0>] do_mpage_readpage+0x310/0xb80
                                sp=e0000001eea8f850 bsp=e0000001eea810d8
 [<a00000010018e9b0>] mpage_readpages+0x170/0x240
                                sp=e0000001eea8fc50 bsp=e0000001eea81088
 [<a00000010022be10>] ext3_readpages+0x30/0x60
                                sp=e0000001eea8fcd0 bsp=e0000001eea81050
 [<a0000001000e7120>] __do_page_cache_readahead+0x2e0/0x4c0
                                sp=e0000001eea8fcd0 bsp=e0000001eea80fe0
 [<a0000001000e7a40>] ondemand_readahead+0x460/0x4a0
                                sp=e0000001eea8fce0 bsp=e0000001eea80f80
 [<a0000001000e7ad0>] page_cache_sync_readahead+0x50/0x80
                                sp=e0000001eea8fce0 bsp=e0000001eea80f40
 [<a0000001000d6560>] do_generic_mapping_read+0xc0/0x7a0
                                sp=e0000001eea8fce0 bsp=e0000001eea80ea0
 [<a0000001000db250>] generic_file_aio_read+0x290/0x340
                                sp=e0000001eea8fce0 bsp=e0000001eea80e40
 [<a00000010012c990>] do_sync_read+0x170/0x200
                                sp=e0000001eea8fd10 bsp=e0000001eea80e00
 [<a00000010012cbd0>] vfs_read+0x1b0/0x2e0
                                sp=e0000001eea8fe20 bsp=e0000001eea80da8
 [<a000000100136800>] kernel_read+0x60/0xa0
                                sp=e0000001eea8fe20 bsp=e0000001eea80d68
 [<a000000100138140>] prepare_binprm+0x1e0/0x200
                                sp=e0000001eea8fe30 bsp=e0000001eea80d40
 [<a000000100138c50>] do_execve+0x150/0x320
                                sp=e0000001eea8fe30 bsp=e0000001eea80cd8
 [<a000000100013760>] sys_execve+0x60/0xc0
                                sp=e0000001eea8fe30 bsp=e0000001eea80ca0
 [<a000000100009b90>] ia64_execve+0x30/0x140
                                sp=e0000001eea8fe30 bsp=e0000001eea80c50
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001eea8fe30 bsp=e0000001eea80c50
 [<a000000000010740>] __kernel_syscall_via_break+0x0/0x20
                                sp=e0000001eea90000 bsp=e0000001eea80c50
INFO: task gcc:9122 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001ecfdfb80 bsp=e0000001ecfd10c0
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001ecfdfbb0 bsp=e0000001ecfd1048
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001ecfdfc30 bsp=e0000001ecfd1010
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001ecfdfc30 bsp=e0000001ecfd0fd0
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001ecfdfc30 bsp=e0000001ecfd0f98
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001ecfdfc30 bsp=e0000001ecfd0f70
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001ecfdfc50 bsp=e0000001ecfd0f40
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001ecfdfc50 bsp=e0000001ecfd0ef0
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001ecfdfc50 bsp=e0000001ecfd0ec8
 [<a00000010013f100>] __link_path_walk+0x820/0x21a0
                                sp=e0000001ecfdfc70 bsp=e0000001ecfd0e48
 [<a000000100140b40>] link_path_walk+0xc0/0x260
                                sp=e0000001ecfdfc90 bsp=e0000001ecfd0e10
 [<a00000010013e570>] path_walk+0x30/0x60
                                sp=e0000001ecfdfd20 bsp=e0000001ecfd0de0
 [<a000000100141200>] do_path_lookup+0x520/0x580
                                sp=e0000001ecfdfd20 bsp=e0000001ecfd0d90
 [<a000000100141de0>] __user_walk_fd+0x60/0xc0
                                sp=e0000001ecfdfd30 bsp=e0000001ecfd0d58
 [<a000000100134350>] vfs_stat_fd+0x30/0xa0
                                sp=e0000001ecfdfd30 bsp=e0000001ecfd0d28
 [<a0000001001348d0>] sys_newstat+0x30/0x80
                                sp=e0000001ecfdfdc0 bsp=e0000001ecfd0cc8
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001ecfdfe30 bsp=e0000001ecfd0cc8
 [<a000000000010740>] __kernel_syscall_via_break+0x0/0x20
                                sp=e0000001ecfe0000 bsp=e0000001ecfd0cc8
INFO: task sh:9128 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001ef28fbf0 bsp=e0000001ef281138
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001ef28fc20 bsp=e0000001ef2810c0
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001ef28fca0 bsp=e0000001ef281088
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001ef28fca0 bsp=e0000001ef281050
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001ef28fca0 bsp=e0000001ef281018
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001ef28fca0 bsp=e0000001ef280fe8
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001ef28fcc0 bsp=e0000001ef280fb8
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001ef28fcc0 bsp=e0000001ef280f68
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001ef28fcc0 bsp=e0000001ef280f40
 [<a0000001000d6c20>] do_generic_mapping_read+0x780/0x7a0
                                sp=e0000001ef28fce0 bsp=e0000001ef280ea0
 [<a0000001000db250>] generic_file_aio_read+0x290/0x340
                                sp=e0000001ef28fce0 bsp=e0000001ef280e40
 [<a00000010012c990>] do_sync_read+0x170/0x200
                                sp=e0000001ef28fd10 bsp=e0000001ef280e00
 [<a00000010012cbd0>] vfs_read+0x1b0/0x2e0
                                sp=e0000001ef28fe20 bsp=e0000001ef280da8
 [<a000000100136800>] kernel_read+0x60/0xa0
                                sp=e0000001ef28fe20 bsp=e0000001ef280d68
 [<a000000100138140>] prepare_binprm+0x1e0/0x200
                                sp=e0000001ef28fe30 bsp=e0000001ef280d40
 [<a000000100138c50>] do_execve+0x150/0x320
                                sp=e0000001ef28fe30 bsp=e0000001ef280cd8
 [<a000000100013760>] sys_execve+0x60/0xc0
                                sp=e0000001ef28fe30 bsp=e0000001ef280ca0
 [<a000000100009b90>] ia64_execve+0x30/0x140
                                sp=e0000001ef28fe30 bsp=e0000001ef280c50
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001ef28fe30 bsp=e0000001ef280c50
 [<a000000000010740>] __kernel_syscall_via_break+0x0/0x20
                                sp=e0000001ef290000 bsp=e0000001ef280c50
INFO: task sh:9136 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001e5bdfbf0 bsp=e0000001e5bd1138
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001e5bdfc20 bsp=e0000001e5bd10c0
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001e5bdfca0 bsp=e0000001e5bd1088
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001e5bdfca0 bsp=e0000001e5bd1050
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001e5bdfca0 bsp=e0000001e5bd1018
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001e5bdfca0 bsp=e0000001e5bd0fe8
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001e5bdfcc0 bsp=e0000001e5bd0fb8
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001e5bdfcc0 bsp=e0000001e5bd0f68
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001e5bdfcc0 bsp=e0000001e5bd0f40
 [<a0000001000d6c20>] do_generic_mapping_read+0x780/0x7a0
                                sp=e0000001e5bdfce0 bsp=e0000001e5bd0ea0
 [<a0000001000db250>] generic_file_aio_read+0x290/0x340
                                sp=e0000001e5bdfce0 bsp=e0000001e5bd0e40
 [<a00000010012c990>] do_sync_read+0x170/0x200
                                sp=e0000001e5bdfd10 bsp=e0000001e5bd0e00
 [<a00000010012cbd0>] vfs_read+0x1b0/0x2e0
                                sp=e0000001e5bdfe20 bsp=e0000001e5bd0da8
 [<a000000100136800>] kernel_read+0x60/0xa0
                                sp=e0000001e5bdfe20 bsp=e0000001e5bd0d68
 [<a000000100138140>] prepare_binprm+0x1e0/0x200
                                sp=e0000001e5bdfe30 bsp=e0000001e5bd0d40
 [<a000000100138c50>] do_execve+0x150/0x320
                                sp=e0000001e5bdfe30 bsp=e0000001e5bd0cd8
 [<a000000100013760>] sys_execve+0x60/0xc0
                                sp=e0000001e5bdfe30 bsp=e0000001e5bd0ca0
 [<a000000100009b90>] ia64_execve+0x30/0x140
                                sp=e0000001e5bdfe30 bsp=e0000001e5bd0c50
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001e5bdfe30 bsp=e0000001e5bd0c50
 [<a000000000010740>] __kernel_syscall_via_break+0x0/0x20
                                sp=e0000001e5be0000 bsp=e0000001e5bd0c50
INFO: task mv:9143 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001b2aafbe0 bsp=e0000001b2aa12a8
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001b2aafc10 bsp=e0000001b2aa1230
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001b2aafc90 bsp=e0000001b2aa11f0
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001b2aafc90 bsp=e0000001b2aa11b8
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001b2aafc90 bsp=e0000001b2aa1180
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001b2aafc90 bsp=e0000001b2aa1158
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001b2aafcb0 bsp=e0000001b2aa1128
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001b2aafcb0 bsp=e0000001b2aa10d8
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001b2aafcb0 bsp=e0000001b2aa10b0
 [<a00000010013ffa0>] __link_path_walk+0x16c0/0x21a0
                                sp=e0000001b2aafcd0 bsp=e0000001b2aa1030
 [<a000000100140b40>] link_path_walk+0xc0/0x260
                                sp=e0000001b2aafcf0 bsp=e0000001b2aa0ff0
 [<a00000010013e570>] path_walk+0x30/0x60
                                sp=e0000001b2aafd80 bsp=e0000001b2aa0fc8
 [<a000000100141200>] do_path_lookup+0x520/0x580
                                sp=e0000001b2aafd80 bsp=e0000001b2aa0f78
 [<a0000001001415e0>] __path_lookup_intent_open+0x80/0x120
                                sp=e0000001b2aafd90 bsp=e0000001b2aa0f28
 [<a0000001001416c0>] path_lookup_open+0x40/0x60
                                sp=e0000001b2aafd90 bsp=e0000001b2aa0ee8
 [<a000000100142b60>] open_namei+0xc0/0xf40
                                sp=e0000001b2aafd90 bsp=e0000001b2aa0e48
 [<a00000010012b150>] do_filp_open+0x50/0xc0
                                sp=e0000001b2aafda0 bsp=e0000001b2aa0e10
 [<a00000010012b9b0>] do_sys_open+0x90/0x1c0
                                sp=e0000001b2aafe30 bsp=e0000001b2aa0db8
 [<a00000010012bb30>] sys_open+0x50/0x80
                                sp=e0000001b2aafe30 bsp=e0000001b2aa0d60
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001b2aafe30 bsp=e0000001b2aa0d60
INFO: task mv:9150 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001ecfafbf0 bsp=e0000001ecfa1168
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001ecfafc20 bsp=e0000001ecfa10f0
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001ecfafca0 bsp=e0000001ecfa10b8
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001ecfafca0 bsp=e0000001ecfa1080
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001ecfafca0 bsp=e0000001ecfa1048
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001ecfafca0 bsp=e0000001ecfa1020
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001ecfafcc0 bsp=e0000001ecfa0fe8
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001ecfafcc0 bsp=e0000001ecfa0f98
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001ecfafcc0 bsp=e0000001ecfa0f70
 [<a0000001000d6c20>] do_generic_mapping_read+0x780/0x7a0
                                sp=e0000001ecfafce0 bsp=e0000001ecfa0ed0
 [<a0000001000db250>] generic_file_aio_read+0x290/0x340
                                sp=e0000001ecfafce0 bsp=e0000001ecfa0e70
 [<a00000010012c990>] do_sync_read+0x170/0x200
                                sp=e0000001ecfafd10 bsp=e0000001ecfa0e30
 [<a00000010012cbd0>] vfs_read+0x1b0/0x2e0
                                sp=e0000001ecfafe20 bsp=e0000001ecfa0dd8
 [<a00000010012d250>] sys_read+0x70/0xe0
                                sp=e0000001ecfafe20 bsp=e0000001ecfa0d60
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001ecfafe30 bsp=e0000001ecfa0d60
INFO: task sh:9154 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001eb41fbf0 bsp=e0000001eb411138
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001eb41fc20 bsp=e0000001eb4110c0
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001eb41fca0 bsp=e0000001eb411088
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001eb41fca0 bsp=e0000001eb411050
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001eb41fca0 bsp=e0000001eb411018
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001eb41fca0 bsp=e0000001eb410fe8
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001eb41fcc0 bsp=e0000001eb410fb8
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001eb41fcc0 bsp=e0000001eb410f68
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001eb41fcc0 bsp=e0000001eb410f40
 [<a0000001000d6c20>] do_generic_mapping_read+0x780/0x7a0
                                sp=e0000001eb41fce0 bsp=e0000001eb410ea0
 [<a0000001000db250>] generic_file_aio_read+0x290/0x340
                                sp=e0000001eb41fce0 bsp=e0000001eb410e40
 [<a00000010012c990>] do_sync_read+0x170/0x200
                                sp=e0000001eb41fd10 bsp=e0000001eb410e00
 [<a00000010012cbd0>] vfs_read+0x1b0/0x2e0
                                sp=e0000001eb41fe20 bsp=e0000001eb410da8
 [<a000000100136800>] kernel_read+0x60/0xa0
                                sp=e0000001eb41fe20 bsp=e0000001eb410d68
 [<a000000100138140>] prepare_binprm+0x1e0/0x200
                                sp=e0000001eb41fe30 bsp=e0000001eb410d40
 [<a000000100138c50>] do_execve+0x150/0x320
                                sp=e0000001eb41fe30 bsp=e0000001eb410cd8
 [<a000000100013760>] sys_execve+0x60/0xc0
                                sp=e0000001eb41fe30 bsp=e0000001eb410ca0
 [<a000000100009b90>] ia64_execve+0x30/0x140
                                sp=e0000001eb41fe30 bsp=e0000001eb410c50
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001eb41fe30 bsp=e0000001eb410c50
 [<a000000000010740>] __kernel_syscall_via_break+0x0/0x20
                                sp=e0000001eb420000 bsp=e0000001eb410c50
INFO: task sh:9155 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001ecdffbf0 bsp=e0000001ecdf1138
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001ecdffc20 bsp=e0000001ecdf10c0
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001ecdffca0 bsp=e0000001ecdf1088
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001ecdffca0 bsp=e0000001ecdf1050
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001ecdffca0 bsp=e0000001ecdf1018
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001ecdffca0 bsp=e0000001ecdf0fe8
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001ecdffcc0 bsp=e0000001ecdf0fb8
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001ecdffcc0 bsp=e0000001ecdf0f68
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001ecdffcc0 bsp=e0000001ecdf0f40
 [<a0000001000d6c20>] do_generic_mapping_read+0x780/0x7a0
                                sp=e0000001ecdffce0 bsp=e0000001ecdf0ea0
 [<a0000001000db250>] generic_file_aio_read+0x290/0x340
                                sp=e0000001ecdffce0 bsp=e0000001ecdf0e40
 [<a00000010012c990>] do_sync_read+0x170/0x200
                                sp=e0000001ecdffd10 bsp=e0000001ecdf0e00
 [<a00000010012cbd0>] vfs_read+0x1b0/0x2e0
                                sp=e0000001ecdffe20 bsp=e0000001ecdf0da8
 [<a000000100136800>] kernel_read+0x60/0xa0
                                sp=e0000001ecdffe20 bsp=e0000001ecdf0d68
 [<a000000100138140>] prepare_binprm+0x1e0/0x200
                                sp=e0000001ecdffe30 bsp=e0000001ecdf0d40
 [<a000000100138c50>] do_execve+0x150/0x320
                                sp=e0000001ecdffe30 bsp=e0000001ecdf0cd8
 [<a000000100013760>] sys_execve+0x60/0xc0
                                sp=e0000001ecdffe30 bsp=e0000001ecdf0ca0
 [<a000000100009b90>] ia64_execve+0x30/0x140
                                sp=e0000001ecdffe30 bsp=e0000001ecdf0c50
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001ecdffe30 bsp=e0000001ecdf0c50
 [<a000000000010740>] __kernel_syscall_via_break+0x0/0x20
                                sp=e0000001ece00000 bsp=e0000001ecdf0c50
INFO: task gcc:9156 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001ee6cfbf0 bsp=e0000001ee6c11e0
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001ee6cfc20 bsp=e0000001ee6c1168
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001ee6cfca0 bsp=e0000001ee6c1130
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001ee6cfca0 bsp=e0000001ee6c10f8
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001ee6cfca0 bsp=e0000001ee6c10c0
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001ee6cfca0 bsp=e0000001ee6c1098
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001ee6cfcc0 bsp=e0000001ee6c1068
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001ee6cfcc0 bsp=e0000001ee6c1018
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001ee6cfcc0 bsp=e0000001ee6c0fe8
 [<a0000001000d6c20>] do_generic_mapping_read+0x780/0x7a0
                                sp=e0000001ee6cfce0 bsp=e0000001ee6c0f48
 [<a0000001000db250>] generic_file_aio_read+0x290/0x340
                                sp=e0000001ee6cfce0 bsp=e0000001ee6c0ee8
 [<a00000010012c990>] do_sync_read+0x170/0x200
                                sp=e0000001ee6cfd10 bsp=e0000001ee6c0ea8
 [<a00000010012cbd0>] vfs_read+0x1b0/0x2e0
                                sp=e0000001ee6cfe20 bsp=e0000001ee6c0e58
 [<a000000100136800>] kernel_read+0x60/0xa0
                                sp=e0000001ee6cfe20 bsp=e0000001ee6c0e18
 [<a000000100138140>] prepare_binprm+0x1e0/0x200
                                sp=e0000001ee6cfe30 bsp=e0000001ee6c0de8
 [<a000000100138c50>] do_execve+0x150/0x320
                                sp=e0000001ee6cfe30 bsp=e0000001ee6c0d80
 [<a000000100013760>] sys_execve+0x60/0xc0
                                sp=e0000001ee6cfe30 bsp=e0000001ee6c0d48
 [<a000000100009b90>] ia64_execve+0x30/0x140
                                sp=e0000001ee6cfe30 bsp=e0000001ee6c0cf8
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001ee6cfe30 bsp=e0000001ee6c0cf8
 [<a000000000010740>] __kernel_syscall_via_break+0x0/0x20
                                sp=e0000001ee6d0000 bsp=e0000001ee6c0cf8
INFO: task sh:9157 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001ee41fbf0 bsp=e0000001ee411138
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001ee41fc20 bsp=e0000001ee4110c0
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001ee41fca0 bsp=e0000001ee411088
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001ee41fca0 bsp=e0000001ee411050
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001ee41fca0 bsp=e0000001ee411018
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001ee41fca0 bsp=e0000001ee410fe8
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001ee41fcc0 bsp=e0000001ee410fb8
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001ee41fcc0 bsp=e0000001ee410f68
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001ee41fcc0 bsp=e0000001ee410f40
 [<a0000001000d6c20>] do_generic_mapping_read+0x780/0x7a0
                                sp=e0000001ee41fce0 bsp=e0000001ee410ea0
 [<a0000001000db250>] generic_file_aio_read+0x290/0x340
                                sp=e0000001ee41fce0 bsp=e0000001ee410e40
 [<a00000010012c990>] do_sync_read+0x170/0x200
                                sp=e0000001ee41fd10 bsp=e0000001ee410e00
 [<a00000010012cbd0>] vfs_read+0x1b0/0x2e0
                                sp=e0000001ee41fe20 bsp=e0000001ee410da8
 [<a000000100136800>] kernel_read+0x60/0xa0
                                sp=e0000001ee41fe20 bsp=e0000001ee410d68
 [<a000000100138140>] prepare_binprm+0x1e0/0x200
                                sp=e0000001ee41fe30 bsp=e0000001ee410d40
 [<a000000100138c50>] do_execve+0x150/0x320
                                sp=e0000001ee41fe30 bsp=e0000001ee410cd8
 [<a000000100013760>] sys_execve+0x60/0xc0
                                sp=e0000001ee41fe30 bsp=e0000001ee410ca0
 [<a000000100009b90>] ia64_execve+0x30/0x140
                                sp=e0000001ee41fe30 bsp=e0000001ee410c50
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001ee41fe30 bsp=e0000001ee410c50
 [<a000000000010740>] __kernel_syscall_via_break+0x0/0x20
                                sp=e0000001ee420000 bsp=e0000001ee410c50
INFO: task sh:9158 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001ee38fbf0 bsp=e0000001ee381138
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001ee38fc20 bsp=e0000001ee3810c0
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001ee38fca0 bsp=e0000001ee381088
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001ee38fca0 bsp=e0000001ee381050
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001ee38fca0 bsp=e0000001ee381018
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001ee38fca0 bsp=e0000001ee380fe8
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001ee38fcc0 bsp=e0000001ee380fb8
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001ee38fcc0 bsp=e0000001ee380f68
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001ee38fcc0 bsp=e0000001ee380f40
 [<a0000001000d6c20>] do_generic_mapping_read+0x780/0x7a0
                                sp=e0000001ee38fce0 bsp=e0000001ee380ea0
 [<a0000001000db250>] generic_file_aio_read+0x290/0x340
                                sp=e0000001ee38fce0 bsp=e0000001ee380e40
 [<a00000010012c990>] do_sync_read+0x170/0x200
                                sp=e0000001ee38fd10 bsp=e0000001ee380e00
 [<a00000010012cbd0>] vfs_read+0x1b0/0x2e0
                                sp=e0000001ee38fe20 bsp=e0000001ee380da8
 [<a000000100136800>] kernel_read+0x60/0xa0
                                sp=e0000001ee38fe20 bsp=e0000001ee380d68
 [<a000000100138140>] prepare_binprm+0x1e0/0x200
                                sp=e0000001ee38fe30 bsp=e0000001ee380d40
 [<a000000100138c50>] do_execve+0x150/0x320
                                sp=e0000001ee38fe30 bsp=e0000001ee380cd8
 [<a000000100013760>] sys_execve+0x60/0xc0
                                sp=e0000001ee38fe30 bsp=e0000001ee380ca0
 [<a000000100009b90>] ia64_execve+0x30/0x140
                                sp=e0000001ee38fe30 bsp=e0000001ee380c50
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001ee38fe30 bsp=e0000001ee380c50
 [<a000000000010740>] __kernel_syscall_via_break+0x0/0x20
                                sp=e0000001ee390000 bsp=e0000001ee380c50
INFO: task sh:9159 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001ef04fbc0 bsp=e0000001ef041298
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001ef04fbf0 bsp=e0000001ef041220
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001ef04fc70 bsp=e0000001ef0411e0
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001ef04fc70 bsp=e0000001ef0411a8
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001ef04fc70 bsp=e0000001ef041170
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001ef04fc70 bsp=e0000001ef041148
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001ef04fc90 bsp=e0000001ef041118
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001ef04fc90 bsp=e0000001ef0410c8
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001ef04fc90 bsp=e0000001ef0410a0
 [<a00000010013ffa0>] __link_path_walk+0x16c0/0x21a0
                                sp=e0000001ef04fcb0 bsp=e0000001ef041020
 [<a000000100140b40>] link_path_walk+0xc0/0x260
                                sp=e0000001ef04fcd0 bsp=e0000001ef040fe0
 [<a00000010013e570>] path_walk+0x30/0x60
                                sp=e0000001ef04fd60 bsp=e0000001ef040fb8
 [<a000000100141200>] do_path_lookup+0x520/0x580
                                sp=e0000001ef04fd60 bsp=e0000001ef040f68
 [<a0000001001415e0>] __path_lookup_intent_open+0x80/0x120
                                sp=e0000001ef04fd70 bsp=e0000001ef040f18
 [<a0000001001416c0>] path_lookup_open+0x40/0x60
                                sp=e0000001ef04fd70 bsp=e0000001ef040ed8
 [<a000000100136640>] open_exec+0x40/0x1a0
                                sp=e0000001ef04fd70 bsp=e0000001ef040eb8
 [<a000000100198c90>] load_elf_binary+0x530/0x2700
                                sp=e0000001ef04fe00 bsp=e0000001ef040da0
 [<a000000100138710>] search_binary_handler+0x1b0/0x5a0
                                sp=e0000001ef04fe30 bsp=e0000001ef040d40
 [<a000000100138d40>] do_execve+0x240/0x320
                                sp=e0000001ef04fe30 bsp=e0000001ef040cd8
 [<a000000100013760>] sys_execve+0x60/0xc0
                                sp=e0000001ef04fe30 bsp=e0000001ef040ca0
 [<a000000100009b90>] ia64_execve+0x30/0x140
                                sp=e0000001ef04fe30 bsp=e0000001ef040c50
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001ef04fe30 bsp=e0000001ef040c50
 [<a000000000010740>] __kernel_syscall_via_break+0x0/0x20
                                sp=e0000001ef050000 bsp=e0000001ef040c50
INFO: task sh:9160 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001ea03fbc0 bsp=e0000001ea031288
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001ea03fbf0 bsp=e0000001ea031210
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001ea03fc70 bsp=e0000001ea0311d0
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001ea03fc70 bsp=e0000001ea031198
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001ea03fc70 bsp=e0000001ea031160
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001ea03fc70 bsp=e0000001ea031138
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001ea03fc90 bsp=e0000001ea031108
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001ea03fc90 bsp=e0000001ea0310b8
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001ea03fc90 bsp=e0000001ea031090
 [<a0000001000d6c20>] do_generic_mapping_read+0x780/0x7a0
                                sp=e0000001ea03fcb0 bsp=e0000001ea030fe8
 [<a0000001000db250>] generic_file_aio_read+0x290/0x340
                                sp=e0000001ea03fcb0 bsp=e0000001ea030f88
 [<a00000010012c990>] do_sync_read+0x170/0x200
                                sp=e0000001ea03fce0 bsp=e0000001ea030f48
 [<a00000010012cbd0>] vfs_read+0x1b0/0x2e0
                                sp=e0000001ea03fdf0 bsp=e0000001ea030ef8
 [<a000000100136800>] kernel_read+0x60/0xa0
                                sp=e0000001ea03fdf0 bsp=e0000001ea030eb8
 [<a000000100198d00>] load_elf_binary+0x5a0/0x2700
                                sp=e0000001ea03fe00 bsp=e0000001ea030da0
 [<a000000100138710>] search_binary_handler+0x1b0/0x5a0
                                sp=e0000001ea03fe30 bsp=e0000001ea030d40
 [<a000000100138d40>] do_execve+0x240/0x320
                                sp=e0000001ea03fe30 bsp=e0000001ea030cd8
 [<a000000100013760>] sys_execve+0x60/0xc0
                                sp=e0000001ea03fe30 bsp=e0000001ea030ca0
 [<a000000100009b90>] ia64_execve+0x30/0x140
                                sp=e0000001ea03fe30 bsp=e0000001ea030c50
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001ea03fe30 bsp=e0000001ea030c50
 [<a000000000010740>] __kernel_syscall_via_break+0x0/0x20
                                sp=e0000001ea040000 bsp=e0000001ea030c50
INFO: task objdump:9161 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001ee78fd10 bsp=e0000001ee781240
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001ee78fd40 bsp=e0000001ee7811c0
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001ee78fdc0 bsp=e0000001ee781188
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001ee78fdc0 bsp=e0000001ee781150
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001ee78fdc0 bsp=e0000001ee781118
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001ee78fdc0 bsp=e0000001ee7810f0
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001ee78fde0 bsp=e0000001ee7810c0
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001ee78fde0 bsp=e0000001ee781070
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001ee78fde0 bsp=e0000001ee781048
 [<a0000001000d79b0>] generic_file_mmap+0x90/0x100
                                sp=e0000001ee78fe00 bsp=e0000001ee781020
 [<a000000100101dd0>] mmap_region+0x3b0/0x7a0
                                sp=e0000001ee78fe00 bsp=e0000001ee780f68
 [<a000000100102820>] do_mmap_pgoff+0x660/0x680
                                sp=e0000001ee78fe30 bsp=e0000001ee780ef8
 [<a000000100033000>] sys_mmap+0x1a0/0x260
                                sp=e0000001ee78fe30 bsp=e0000001ee780e70
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001ee78fe30 bsp=e0000001ee780e70
INFO: task grep:9162 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001eb39fbe0 bsp=e0000001eb3912a8
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001eb39fc10 bsp=e0000001eb391230
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001eb39fc90 bsp=e0000001eb3911f0
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001eb39fc90 bsp=e0000001eb3911b8
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001eb39fc90 bsp=e0000001eb391180
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001eb39fc90 bsp=e0000001eb391158
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001eb39fcb0 bsp=e0000001eb391128
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001eb39fcb0 bsp=e0000001eb3910d8
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001eb39fcb0 bsp=e0000001eb3910b0
 [<a00000010013ffa0>] __link_path_walk+0x16c0/0x21a0
                                sp=e0000001eb39fcd0 bsp=e0000001eb391030
 [<a000000100140b40>] link_path_walk+0xc0/0x260
                                sp=e0000001eb39fcf0 bsp=e0000001eb390ff0
 [<a00000010013e570>] path_walk+0x30/0x60
                                sp=e0000001eb39fd80 bsp=e0000001eb390fc8
 [<a000000100141200>] do_path_lookup+0x520/0x580
                                sp=e0000001eb39fd80 bsp=e0000001eb390f78
 [<a0000001001415e0>] __path_lookup_intent_open+0x80/0x120
                                sp=e0000001eb39fd90 bsp=e0000001eb390f28
 [<a0000001001416c0>] path_lookup_open+0x40/0x60
                                sp=e0000001eb39fd90 bsp=e0000001eb390ee8
 [<a000000100142b60>] open_namei+0xc0/0xf40
                                sp=e0000001eb39fd90 bsp=e0000001eb390e48
 [<a00000010012b150>] do_filp_open+0x50/0xc0
                                sp=e0000001eb39fda0 bsp=e0000001eb390e10
 [<a00000010012b9b0>] do_sys_open+0x90/0x1c0
                                sp=e0000001eb39fe30 bsp=e0000001eb390db8
 [<a00000010012bb30>] sys_open+0x50/0x80
                                sp=e0000001eb39fe30 bsp=e0000001eb390d60
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001eb39fe30 bsp=e0000001eb390d60
INFO: task objdump:9163 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001e99bfbf0 bsp=e0000001e99b1168
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001e99bfc20 bsp=e0000001e99b10f0
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001e99bfca0 bsp=e0000001e99b10b8
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001e99bfca0 bsp=e0000001e99b1080
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001e99bfca0 bsp=e0000001e99b1048
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001e99bfca0 bsp=e0000001e99b1020
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001e99bfcc0 bsp=e0000001e99b0fe8
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001e99bfcc0 bsp=e0000001e99b0f98
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001e99bfcc0 bsp=e0000001e99b0f70
 [<a0000001000d6c20>] do_generic_mapping_read+0x780/0x7a0
                                sp=e0000001e99bfce0 bsp=e0000001e99b0ed0
 [<a0000001000db250>] generic_file_aio_read+0x290/0x340
                                sp=e0000001e99bfce0 bsp=e0000001e99b0e70
 [<a00000010012c990>] do_sync_read+0x170/0x200
                                sp=e0000001e99bfd10 bsp=e0000001e99b0e30
 [<a00000010012cbd0>] vfs_read+0x1b0/0x2e0
                                sp=e0000001e99bfe20 bsp=e0000001e99b0dd8
 [<a00000010012d250>] sys_read+0x70/0xe0
                                sp=e0000001e99bfe20 bsp=e0000001e99b0d60
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001e99bfe30 bsp=e0000001e99b0d60
INFO: task grep:9164 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001ed63fbf0 bsp=e0000001ed631168
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001ed63fc20 bsp=e0000001ed6310f0
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001ed63fca0 bsp=e0000001ed6310b8
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001ed63fca0 bsp=e0000001ed631080
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001ed63fca0 bsp=e0000001ed631048
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001ed63fca0 bsp=e0000001ed631020
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001ed63fcc0 bsp=e0000001ed630fe8
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001ed63fcc0 bsp=e0000001ed630f98
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001ed63fcc0 bsp=e0000001ed630f70
 [<a0000001000d6c20>] do_generic_mapping_read+0x780/0x7a0
                                sp=e0000001ed63fce0 bsp=e0000001ed630ed0
 [<a0000001000db250>] generic_file_aio_read+0x290/0x340
                                sp=e0000001ed63fce0 bsp=e0000001ed630e70
 [<a00000010012c990>] do_sync_read+0x170/0x200
                                sp=e0000001ed63fd10 bsp=e0000001ed630e30
 [<a00000010012cbd0>] vfs_read+0x1b0/0x2e0
                                sp=e0000001ed63fe20 bsp=e0000001ed630dd8
 [<a00000010012d250>] sys_read+0x70/0xe0
                                sp=e0000001ed63fe20 bsp=e0000001ed630d60
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001ed63fe30 bsp=e0000001ed630d60
INFO: task objdump:9165 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001ee18fbf0 bsp=e0000001ee181280
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001ee18fc20 bsp=e0000001ee181208
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001ee18fca0 bsp=e0000001ee1811c8
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001ee18fca0 bsp=e0000001ee181190
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001ee18fca0 bsp=e0000001ee181158
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001ee18fca0 bsp=e0000001ee181130
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001ee18fcc0 bsp=e0000001ee181100
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001ee18fcc0 bsp=e0000001ee1810b0
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001ee18fcc0 bsp=e0000001ee181088
 [<a0000001000d6c20>] do_generic_mapping_read+0x780/0x7a0
                                sp=e0000001ee18fce0 bsp=e0000001ee180fe0
 [<a0000001000db250>] generic_file_aio_read+0x290/0x340
                                sp=e0000001ee18fce0 bsp=e0000001ee180f80
 [<a00000010012c990>] do_sync_read+0x170/0x200
                                sp=e0000001ee18fd10 bsp=e0000001ee180f40
 [<a00000010012cbd0>] vfs_read+0x1b0/0x2e0
                                sp=e0000001ee18fe20 bsp=e0000001ee180ef0
 [<a00000010012d250>] sys_read+0x70/0xe0
                                sp=e0000001ee18fe20 bsp=e0000001ee180e78
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001ee18fe30 bsp=e0000001ee180e78
INFO: task grep:9166 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001eb50fbe0 bsp=e0000001eb5013b8
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001eb50fc10 bsp=e0000001eb501340
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001eb50fc90 bsp=e0000001eb501308
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001eb50fc90 bsp=e0000001eb5012d0
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001eb50fc90 bsp=e0000001eb501298
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001eb50fc90 bsp=e0000001eb501270
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001eb50fcb0 bsp=e0000001eb501240
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001eb50fcb0 bsp=e0000001eb5011e8
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001eb50fcb0 bsp=e0000001eb5011c0
 [<a00000010013ffa0>] __link_path_walk+0x16c0/0x21a0
                                sp=e0000001eb50fcd0 bsp=e0000001eb501140
 [<a000000100140b40>] link_path_walk+0xc0/0x260
                                sp=e0000001eb50fcf0 bsp=e0000001eb501108
 [<a00000010013e570>] path_walk+0x30/0x60
                                sp=e0000001eb50fd80 bsp=e0000001eb5010e0
 [<a000000100141200>] do_path_lookup+0x520/0x580
                                sp=e0000001eb50fd80 bsp=e0000001eb501090
 [<a0000001001415e0>] __path_lookup_intent_open+0x80/0x120
                                sp=e0000001eb50fd90 bsp=e0000001eb501040
 [<a0000001001416c0>] path_lookup_open+0x40/0x60
                                sp=e0000001eb50fd90 bsp=e0000001eb501000
 [<a000000100142b60>] open_namei+0xc0/0xf40
                                sp=e0000001eb50fd90 bsp=e0000001eb500f58
 [<a00000010012b150>] do_filp_open+0x50/0xc0
                                sp=e0000001eb50fda0 bsp=e0000001eb500f20
 [<a00000010012b9b0>] do_sys_open+0x90/0x1c0
                                sp=e0000001eb50fe30 bsp=e0000001eb500ed0
 [<a00000010012bb30>] sys_open+0x50/0x80
                                sp=e0000001eb50fe30 bsp=e0000001eb500e78
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001eb50fe30 bsp=e0000001eb500e78
INFO: task objdump:9167 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001b7c7fbe0 bsp=e0000001b7c713b8
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001b7c7fc10 bsp=e0000001b7c71340
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001b7c7fc90 bsp=e0000001b7c71308
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001b7c7fc90 bsp=e0000001b7c712d0
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001b7c7fc90 bsp=e0000001b7c71298
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001b7c7fc90 bsp=e0000001b7c71270
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001b7c7fcb0 bsp=e0000001b7c71240
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001b7c7fcb0 bsp=e0000001b7c711e8
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001b7c7fcb0 bsp=e0000001b7c711c0
 [<a00000010013ffa0>] __link_path_walk+0x16c0/0x21a0
                                sp=e0000001b7c7fcd0 bsp=e0000001b7c71140
 [<a000000100140b40>] link_path_walk+0xc0/0x260
                                sp=e0000001b7c7fcf0 bsp=e0000001b7c71108
 [<a00000010013e570>] path_walk+0x30/0x60
                                sp=e0000001b7c7fd80 bsp=e0000001b7c710e0
 [<a000000100141200>] do_path_lookup+0x520/0x580
                                sp=e0000001b7c7fd80 bsp=e0000001b7c71090
 [<a0000001001415e0>] __path_lookup_intent_open+0x80/0x120
                                sp=e0000001b7c7fd90 bsp=e0000001b7c71040
 [<a0000001001416c0>] path_lookup_open+0x40/0x60
                                sp=e0000001b7c7fd90 bsp=e0000001b7c71000
 [<a000000100142b60>] open_namei+0xc0/0xf40
                                sp=e0000001b7c7fd90 bsp=e0000001b7c70f58
 [<a00000010012b150>] do_filp_open+0x50/0xc0
                                sp=e0000001b7c7fda0 bsp=e0000001b7c70f20
 [<a00000010012b9b0>] do_sys_open+0x90/0x1c0
                                sp=e0000001b7c7fe30 bsp=e0000001b7c70ed0
 [<a00000010012bb30>] sys_open+0x50/0x80
                                sp=e0000001b7c7fe30 bsp=e0000001b7c70e78
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001b7c7fe30 bsp=e0000001b7c70e78
INFO: task grep:9168 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001ed8afbf0 bsp=e0000001ed8a1280
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001ed8afc20 bsp=e0000001ed8a1208
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001ed8afca0 bsp=e0000001ed8a11c8
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001ed8afca0 bsp=e0000001ed8a1190
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001ed8afca0 bsp=e0000001ed8a1158
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001ed8afca0 bsp=e0000001ed8a1130
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001ed8afcc0 bsp=e0000001ed8a1100
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001ed8afcc0 bsp=e0000001ed8a10b0
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001ed8afcc0 bsp=e0000001ed8a1088
 [<a0000001000d6c20>] do_generic_mapping_read+0x780/0x7a0
                                sp=e0000001ed8afce0 bsp=e0000001ed8a0fe0
 [<a0000001000db250>] generic_file_aio_read+0x290/0x340
                                sp=e0000001ed8afce0 bsp=e0000001ed8a0f80
 [<a00000010012c990>] do_sync_read+0x170/0x200
                                sp=e0000001ed8afd10 bsp=e0000001ed8a0f40
 [<a00000010012cbd0>] vfs_read+0x1b0/0x2e0
                                sp=e0000001ed8afe20 bsp=e0000001ed8a0ef0
 [<a00000010012d250>] sys_read+0x70/0xe0
                                sp=e0000001ed8afe20 bsp=e0000001ed8a0e78
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001ed8afe30 bsp=e0000001ed8a0e78
INFO: task kjournald:1272 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001b00bfca0 bsp=e0000001b00b0e30
 [<a000000100704cf0>] io_schedule+0x50/0x80
                                sp=e0000001b00bfcd0 bsp=e0000001b00b0e10
 [<a000000100179d60>] sync_buffer+0x100/0x120
                                sp=e0000001b00bfcd0 bsp=e0000001b00b0de8
 [<a000000100705130>] __wait_on_bit+0xd0/0x180
                                sp=e0000001b00bfcd0 bsp=e0000001b00b0d98
 [<a0000001007052c0>] out_of_line_wait_on_bit+0xe0/0x100
                                sp=e0000001b00bfcd0 bsp=e0000001b00b0d50
 [<a000000100179b80>] __wait_on_buffer+0x40/0x60
                                sp=e0000001b00bfd50 bsp=e0000001b00b0d30
 [<a000000100253250>] journal_commit_transaction+0x1110/0x1fc0
                                sp=e0000001b00bfd50 bsp=e0000001b00b0c90
 [<a000000100258c60>] kjournald+0x180/0x420
                                sp=e0000001b00bfdc0 bsp=e0000001b00b0c30
 [<a0000001000a7c40>] kthread+0xa0/0x120
                                sp=e0000001b00bfe30 bsp=e0000001b00b0c00
 [<a000000100013970>] kernel_thread_helper+0x30/0x60
                                sp=e0000001b00bfe30 bsp=e0000001b00b0bd0
 [<a0000001000089c0>] start_kernel_thread+0x20/0x40
                                sp=e0000001b00bfe30 bsp=e0000001b00b0bd0
INFO: task syslogd:4887 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001bdf2fcd0 bsp=e0000001bdf20f70
 [<a000000100259240>] log_wait_commit+0x180/0x220
                                sp=e0000001bdf2fd00 bsp=e0000001bdf20f28
 [<a00000010024d350>] journal_stop+0x3f0/0x460
                                sp=e0000001bdf2fd60 bsp=e0000001bdf20ed0
 [<a00000010024d440>] journal_force_commit+0x80/0xa0
                                sp=e0000001bdf2fd70 bsp=e0000001bdf20eb0
 [<a000000100241240>] ext3_force_commit+0x60/0x80
                                sp=e0000001bdf2fd70 bsp=e0000001bdf20e90
 [<a00000010022d5e0>] ext3_write_inode+0xa0/0xc0
                                sp=e0000001bdf2fd70 bsp=e0000001bdf20e68
 [<a000000100171870>] __writeback_single_inode+0x510/0x7c0
                                sp=e0000001bdf2fd70 bsp=e0000001bdf20e10
 [<a000000100171de0>] sync_inode+0x40/0x80
                                sp=e0000001bdf2fdf0 bsp=e0000001bdf20dd8
 [<a000000100227bd0>] ext3_sync_file+0x1b0/0x1e0
                                sp=e0000001bdf2fdf0 bsp=e0000001bdf20db0
 [<a0000001001788b0>] do_fsync+0xf0/0x160
                                sp=e0000001bdf2fe30 bsp=e0000001bdf20d70
 [<a000000100178970>] __do_fsync+0x50/0x80
                                sp=e0000001bdf2fe30 bsp=e0000001bdf20d40
 [<a0000001001789d0>] sys_fsync+0x30/0x60
                                sp=e0000001bdf2fe30 bsp=e0000001bdf20ce8
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001bdf2fe30 bsp=e0000001bdf20ce8
 [<a000000000010740>] __kernel_syscall_via_break+0x0/0x20
                                sp=e0000001bdf30000 bsp=e0000001bdf20ce8
INFO: task make:8649 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001e97afb40 bsp=e0000001e97a0d40
 [<a000000100704e60>] schedule_timeout+0x40/0x180
                                sp=e0000001e97afb70 bsp=e0000001e97a0d10
 [<a000000100704450>] wait_for_common+0x1b0/0x300
                                sp=e0000001e97afba0 bsp=e0000001e97a0cc0
 [<a0000001007045e0>] wait_for_completion+0x40/0x60
                                sp=e0000001e97afc00 bsp=e0000001e97a0ca0
 [<a000000100077240>] do_fork+0x420/0x480
                                sp=e0000001e97afc00 bsp=e0000001e97a0c50
 [<a000000100009d80>] sys_clone+0x60/0x80
                                sp=e0000001e97afc20 bsp=e0000001e97a0c00
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001e97afe30 bsp=e0000001e97a0c00
 [<a000000000010740>] __kernel_syscall_via_break+0x0/0x20
                                sp=e0000001e97b0000 bsp=e0000001e97a0c00
INFO: task cc1:8839 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001edc4fbf0 bsp=e0000001edc41268
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001edc4fc20 bsp=e0000001edc411e8
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001edc4fca0 bsp=e0000001edc411b0
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001edc4fca0 bsp=e0000001edc41178
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001edc4fca0 bsp=e0000001edc41140
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001edc4fca0 bsp=e0000001edc41118
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001edc4fcc0 bsp=e0000001edc410e8
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001edc4fcc0 bsp=e0000001edc41098
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001edc4fcc0 bsp=e0000001edc41070
 [<a0000001000d6c20>] do_generic_mapping_read+0x780/0x7a0
                                sp=e0000001edc4fce0 bsp=e0000001edc40fc8
 [<a0000001000db250>] generic_file_aio_read+0x290/0x340
                                sp=e0000001edc4fce0 bsp=e0000001edc40f68
 [<a00000010012c990>] do_sync_read+0x170/0x200
                                sp=e0000001edc4fd10 bsp=e0000001edc40f28
 [<a00000010012cbd0>] vfs_read+0x1b0/0x2e0
                                sp=e0000001edc4fe20 bsp=e0000001edc40ed8
 [<a00000010012d250>] sys_read+0x70/0xe0
                                sp=e0000001edc4fe20 bsp=e0000001edc40e60
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001edc4fe30 bsp=e0000001edc40e60
 [<a000000000010740>] __kernel_syscall_via_break+0x0/0x20
                                sp=e0000001edc50000 bsp=e0000001edc40e60
INFO: task sh:9107 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001eea8f6d0 bsp=e0000001eea81448
 [<a000000100704cf0>] io_schedule+0x50/0x80
                                sp=e0000001eea8f700 bsp=e0000001eea81428
 [<a000000100179d60>] sync_buffer+0x100/0x120
                                sp=e0000001eea8f700 bsp=e0000001eea81408
 [<a000000100705130>] __wait_on_bit+0xd0/0x180
                                sp=e0000001eea8f700 bsp=e0000001eea813b0
 [<a0000001007052c0>] out_of_line_wait_on_bit+0xe0/0x100
                                sp=e0000001eea8f700 bsp=e0000001eea81368
 [<a000000100179b80>] __wait_on_buffer+0x40/0x60
                                sp=e0000001eea8f780 bsp=e0000001eea81348
 [<a000000100184370>] __bread+0x190/0x220
                                sp=e0000001eea8f780 bsp=e0000001eea81318
 [<a00000010022a6e0>] ext3_get_branch+0xe0/0x220
                                sp=e0000001eea8f780 bsp=e0000001eea812c0
 [<a00000010022f130>] ext3_get_blocks_handle+0x130/0x1560
                                sp=e0000001eea8f780 bsp=e0000001eea811d8
 [<a0000001002306f0>] ext3_get_block+0x190/0x200
                                sp=e0000001eea8f850 bsp=e0000001eea81180
 [<a00000010018dfd0>] do_mpage_readpage+0x310/0xb80
                                sp=e0000001eea8f850 bsp=e0000001eea810d8
 [<a00000010018e9b0>] mpage_readpages+0x170/0x240
                                sp=e0000001eea8fc50 bsp=e0000001eea81088
 [<a00000010022be10>] ext3_readpages+0x30/0x60
                                sp=e0000001eea8fcd0 bsp=e0000001eea81050
 [<a0000001000e7120>] __do_page_cache_readahead+0x2e0/0x4c0
                                sp=e0000001eea8fcd0 bsp=e0000001eea80fe0
 [<a0000001000e7a40>] ondemand_readahead+0x460/0x4a0
                                sp=e0000001eea8fce0 bsp=e0000001eea80f80
 [<a0000001000e7ad0>] page_cache_sync_readahead+0x50/0x80
                                sp=e0000001eea8fce0 bsp=e0000001eea80f40
 [<a0000001000d6560>] do_generic_mapping_read+0xc0/0x7a0
                                sp=e0000001eea8fce0 bsp=e0000001eea80ea0
 [<a0000001000db250>] generic_file_aio_read+0x290/0x340
                                sp=e0000001eea8fce0 bsp=e0000001eea80e40
 [<a00000010012c990>] do_sync_read+0x170/0x200
                                sp=e0000001eea8fd10 bsp=e0000001eea80e00
 [<a00000010012cbd0>] vfs_read+0x1b0/0x2e0
                                sp=e0000001eea8fe20 bsp=e0000001eea80da8
 [<a000000100136800>] kernel_read+0x60/0xa0
                                sp=e0000001eea8fe20 bsp=e0000001eea80d68
 [<a000000100138140>] prepare_binprm+0x1e0/0x200
                                sp=e0000001eea8fe30 bsp=e0000001eea80d40
 [<a000000100138c50>] do_execve+0x150/0x320
                                sp=e0000001eea8fe30 bsp=e0000001eea80cd8
 [<a000000100013760>] sys_execve+0x60/0xc0
                                sp=e0000001eea8fe30 bsp=e0000001eea80ca0
 [<a000000100009b90>] ia64_execve+0x30/0x140
                                sp=e0000001eea8fe30 bsp=e0000001eea80c50
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001eea8fe30 bsp=e0000001eea80c50
 [<a000000000010740>] __kernel_syscall_via_break+0x0/0x20
                                sp=e0000001eea90000 bsp=e0000001eea80c50
INFO: task gcc:9122 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001ecfdfb80 bsp=e0000001ecfd10c0
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001ecfdfbb0 bsp=e0000001ecfd1048
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001ecfdfc30 bsp=e0000001ecfd1010
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001ecfdfc30 bsp=e0000001ecfd0fd0
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001ecfdfc30 bsp=e0000001ecfd0f98
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001ecfdfc30 bsp=e0000001ecfd0f70
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001ecfdfc50 bsp=e0000001ecfd0f40
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001ecfdfc50 bsp=e0000001ecfd0ef0
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001ecfdfc50 bsp=e0000001ecfd0ec8
 [<a00000010013f100>] __link_path_walk+0x820/0x21a0
                                sp=e0000001ecfdfc70 bsp=e0000001ecfd0e48
 [<a000000100140b40>] link_path_walk+0xc0/0x260
                                sp=e0000001ecfdfc90 bsp=e0000001ecfd0e10
 [<a00000010013e570>] path_walk+0x30/0x60
                                sp=e0000001ecfdfd20 bsp=e0000001ecfd0de0
 [<a000000100141200>] do_path_lookup+0x520/0x580
                                sp=e0000001ecfdfd20 bsp=e0000001ecfd0d90
 [<a000000100141de0>] __user_walk_fd+0x60/0xc0
                                sp=e0000001ecfdfd30 bsp=e0000001ecfd0d58
 [<a000000100134350>] vfs_stat_fd+0x30/0xa0
                                sp=e0000001ecfdfd30 bsp=e0000001ecfd0d28
 [<a0000001001348d0>] sys_newstat+0x30/0x80
                                sp=e0000001ecfdfdc0 bsp=e0000001ecfd0cc8
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001ecfdfe30 bsp=e0000001ecfd0cc8
 [<a000000000010740>] __kernel_syscall_via_break+0x0/0x20
                                sp=e0000001ecfe0000 bsp=e0000001ecfd0cc8
INFO: task sh:9128 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001ef28fbf0 bsp=e0000001ef281138
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001ef28fc20 bsp=e0000001ef2810c0
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001ef28fca0 bsp=e0000001ef281088
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001ef28fca0 bsp=e0000001ef281050
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001ef28fca0 bsp=e0000001ef281018
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001ef28fca0 bsp=e0000001ef280fe8
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001ef28fcc0 bsp=e0000001ef280fb8
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001ef28fcc0 bsp=e0000001ef280f68
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001ef28fcc0 bsp=e0000001ef280f40
 [<a0000001000d6c20>] do_generic_mapping_read+0x780/0x7a0
                                sp=e0000001ef28fce0 bsp=e0000001ef280ea0
 [<a0000001000db250>] generic_file_aio_read+0x290/0x340
                                sp=e0000001ef28fce0 bsp=e0000001ef280e40
 [<a00000010012c990>] do_sync_read+0x170/0x200
                                sp=e0000001ef28fd10 bsp=e0000001ef280e00
 [<a00000010012cbd0>] vfs_read+0x1b0/0x2e0
                                sp=e0000001ef28fe20 bsp=e0000001ef280da8
 [<a000000100136800>] kernel_read+0x60/0xa0
                                sp=e0000001ef28fe20 bsp=e0000001ef280d68
 [<a000000100138140>] prepare_binprm+0x1e0/0x200
                                sp=e0000001ef28fe30 bsp=e0000001ef280d40
 [<a000000100138c50>] do_execve+0x150/0x320
                                sp=e0000001ef28fe30 bsp=e0000001ef280cd8
 [<a000000100013760>] sys_execve+0x60/0xc0
                                sp=e0000001ef28fe30 bsp=e0000001ef280ca0
 [<a000000100009b90>] ia64_execve+0x30/0x140
                                sp=e0000001ef28fe30 bsp=e0000001ef280c50
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001ef28fe30 bsp=e0000001ef280c50
 [<a000000000010740>] __kernel_syscall_via_break+0x0/0x20
                                sp=e0000001ef290000 bsp=e0000001ef280c50
INFO: task sh:9136 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001e5bdfbf0 bsp=e0000001e5bd1138
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001e5bdfc20 bsp=e0000001e5bd10c0
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001e5bdfca0 bsp=e0000001e5bd1088
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001e5bdfca0 bsp=e0000001e5bd1050
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001e5bdfca0 bsp=e0000001e5bd1018
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001e5bdfca0 bsp=e0000001e5bd0fe8
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001e5bdfcc0 bsp=e0000001e5bd0fb8
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001e5bdfcc0 bsp=e0000001e5bd0f68
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001e5bdfcc0 bsp=e0000001e5bd0f40
 [<a0000001000d6c20>] do_generic_mapping_read+0x780/0x7a0
                                sp=e0000001e5bdfce0 bsp=e0000001e5bd0ea0
 [<a0000001000db250>] generic_file_aio_read+0x290/0x340
                                sp=e0000001e5bdfce0 bsp=e0000001e5bd0e40
 [<a00000010012c990>] do_sync_read+0x170/0x200
                                sp=e0000001e5bdfd10 bsp=e0000001e5bd0e00
 [<a00000010012cbd0>] vfs_read+0x1b0/0x2e0
                                sp=e0000001e5bdfe20 bsp=e0000001e5bd0da8
 [<a000000100136800>] kernel_read+0x60/0xa0
                                sp=e0000001e5bdfe20 bsp=e0000001e5bd0d68
 [<a000000100138140>] prepare_binprm+0x1e0/0x200
                                sp=e0000001e5bdfe30 bsp=e0000001e5bd0d40
 [<a000000100138c50>] do_execve+0x150/0x320
                                sp=e0000001e5bdfe30 bsp=e0000001e5bd0cd8
 [<a000000100013760>] sys_execve+0x60/0xc0
                                sp=e0000001e5bdfe30 bsp=e0000001e5bd0ca0
 [<a000000100009b90>] ia64_execve+0x30/0x140
                                sp=e0000001e5bdfe30 bsp=e0000001e5bd0c50
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001e5bdfe30 bsp=e0000001e5bd0c50
 [<a000000000010740>] __kernel_syscall_via_break+0x0/0x20
                                sp=e0000001e5be0000 bsp=e0000001e5bd0c50
INFO: task mv:9143 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001b2aafbe0 bsp=e0000001b2aa12a8
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001b2aafc10 bsp=e0000001b2aa1230
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001b2aafc90 bsp=e0000001b2aa11f0
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001b2aafc90 bsp=e0000001b2aa11b8
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001b2aafc90 bsp=e0000001b2aa1180
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001b2aafc90 bsp=e0000001b2aa1158
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001b2aafcb0 bsp=e0000001b2aa1128
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001b2aafcb0 bsp=e0000001b2aa10d8
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001b2aafcb0 bsp=e0000001b2aa10b0
 [<a00000010013ffa0>] __link_path_walk+0x16c0/0x21a0
                                sp=e0000001b2aafcd0 bsp=e0000001b2aa1030
 [<a000000100140b40>] link_path_walk+0xc0/0x260
                                sp=e0000001b2aafcf0 bsp=e0000001b2aa0ff0
 [<a00000010013e570>] path_walk+0x30/0x60
                                sp=e0000001b2aafd80 bsp=e0000001b2aa0fc8
 [<a000000100141200>] do_path_lookup+0x520/0x580
                                sp=e0000001b2aafd80 bsp=e0000001b2aa0f78
 [<a0000001001415e0>] __path_lookup_intent_open+0x80/0x120
                                sp=e0000001b2aafd90 bsp=e0000001b2aa0f28
 [<a0000001001416c0>] path_lookup_open+0x40/0x60
                                sp=e0000001b2aafd90 bsp=e0000001b2aa0ee8
 [<a000000100142b60>] open_namei+0xc0/0xf40
                                sp=e0000001b2aafd90 bsp=e0000001b2aa0e48
 [<a00000010012b150>] do_filp_open+0x50/0xc0
                                sp=e0000001b2aafda0 bsp=e0000001b2aa0e10
 [<a00000010012b9b0>] do_sys_open+0x90/0x1c0
                                sp=e0000001b2aafe30 bsp=e0000001b2aa0db8
 [<a00000010012bb30>] sys_open+0x50/0x80
                                sp=e0000001b2aafe30 bsp=e0000001b2aa0d60
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001b2aafe30 bsp=e0000001b2aa0d60
INFO: task mv:9150 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001ecfafbf0 bsp=e0000001ecfa1168
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001ecfafc20 bsp=e0000001ecfa10f0
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001ecfafca0 bsp=e0000001ecfa10b8
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001ecfafca0 bsp=e0000001ecfa1080
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001ecfafca0 bsp=e0000001ecfa1048
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001ecfafca0 bsp=e0000001ecfa1020
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001ecfafcc0 bsp=e0000001ecfa0fe8
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001ecfafcc0 bsp=e0000001ecfa0f98
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001ecfafcc0 bsp=e0000001ecfa0f70
 [<a0000001000d6c20>] do_generic_mapping_read+0x780/0x7a0
                                sp=e0000001ecfafce0 bsp=e0000001ecfa0ed0
 [<a0000001000db250>] generic_file_aio_read+0x290/0x340
                                sp=e0000001ecfafce0 bsp=e0000001ecfa0e70
 [<a00000010012c990>] do_sync_read+0x170/0x200
                                sp=e0000001ecfafd10 bsp=e0000001ecfa0e30
 [<a00000010012cbd0>] vfs_read+0x1b0/0x2e0
                                sp=e0000001ecfafe20 bsp=e0000001ecfa0dd8
 [<a00000010012d250>] sys_read+0x70/0xe0
                                sp=e0000001ecfafe20 bsp=e0000001ecfa0d60
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001ecfafe30 bsp=e0000001ecfa0d60
INFO: task sh:9154 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001eb41fbf0 bsp=e0000001eb411138
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001eb41fc20 bsp=e0000001eb4110c0
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001eb41fca0 bsp=e0000001eb411088
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001eb41fca0 bsp=e0000001eb411050
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001eb41fca0 bsp=e0000001eb411018
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001eb41fca0 bsp=e0000001eb410fe8
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001eb41fcc0 bsp=e0000001eb410fb8
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001eb41fcc0 bsp=e0000001eb410f68
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001eb41fcc0 bsp=e0000001eb410f40
 [<a0000001000d6c20>] do_generic_mapping_read+0x780/0x7a0
                                sp=e0000001eb41fce0 bsp=e0000001eb410ea0
 [<a0000001000db250>] generic_file_aio_read+0x290/0x340
                                sp=e0000001eb41fce0 bsp=e0000001eb410e40
 [<a00000010012c990>] do_sync_read+0x170/0x200
                                sp=e0000001eb41fd10 bsp=e0000001eb410e00
 [<a00000010012cbd0>] vfs_read+0x1b0/0x2e0
                                sp=e0000001eb41fe20 bsp=e0000001eb410da8
 [<a000000100136800>] kernel_read+0x60/0xa0
                                sp=e0000001eb41fe20 bsp=e0000001eb410d68
 [<a000000100138140>] prepare_binprm+0x1e0/0x200
                                sp=e0000001eb41fe30 bsp=e0000001eb410d40
 [<a000000100138c50>] do_execve+0x150/0x320
                                sp=e0000001eb41fe30 bsp=e0000001eb410cd8
 [<a000000100013760>] sys_execve+0x60/0xc0
                                sp=e0000001eb41fe30 bsp=e0000001eb410ca0
 [<a000000100009b90>] ia64_execve+0x30/0x140
                                sp=e0000001eb41fe30 bsp=e0000001eb410c50
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001eb41fe30 bsp=e0000001eb410c50
 [<a000000000010740>] __kernel_syscall_via_break+0x0/0x20
                                sp=e0000001eb420000 bsp=e0000001eb410c50
INFO: task sh:9155 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001ecdffbf0 bsp=e0000001ecdf1138
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001ecdffc20 bsp=e0000001ecdf10c0
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001ecdffca0 bsp=e0000001ecdf1088
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001ecdffca0 bsp=e0000001ecdf1050
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001ecdffca0 bsp=e0000001ecdf1018
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001ecdffca0 bsp=e0000001ecdf0fe8
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001ecdffcc0 bsp=e0000001ecdf0fb8
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001ecdffcc0 bsp=e0000001ecdf0f68
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001ecdffcc0 bsp=e0000001ecdf0f40
 [<a0000001000d6c20>] do_generic_mapping_read+0x780/0x7a0
                                sp=e0000001ecdffce0 bsp=e0000001ecdf0ea0
 [<a0000001000db250>] generic_file_aio_read+0x290/0x340
                                sp=e0000001ecdffce0 bsp=e0000001ecdf0e40
 [<a00000010012c990>] do_sync_read+0x170/0x200
                                sp=e0000001ecdffd10 bsp=e0000001ecdf0e00
 [<a00000010012cbd0>] vfs_read+0x1b0/0x2e0
                                sp=e0000001ecdffe20 bsp=e0000001ecdf0da8
 [<a000000100136800>] kernel_read+0x60/0xa0
                                sp=e0000001ecdffe20 bsp=e0000001ecdf0d68
 [<a000000100138140>] prepare_binprm+0x1e0/0x200
                                sp=e0000001ecdffe30 bsp=e0000001ecdf0d40
 [<a000000100138c50>] do_execve+0x150/0x320
                                sp=e0000001ecdffe30 bsp=e0000001ecdf0cd8
 [<a000000100013760>] sys_execve+0x60/0xc0
                                sp=e0000001ecdffe30 bsp=e0000001ecdf0ca0
 [<a000000100009b90>] ia64_execve+0x30/0x140
                                sp=e0000001ecdffe30 bsp=e0000001ecdf0c50
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001ecdffe30 bsp=e0000001ecdf0c50
 [<a000000000010740>] __kernel_syscall_via_break+0x0/0x20
                                sp=e0000001ece00000 bsp=e0000001ecdf0c50
INFO: task gcc:9156 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001ee6cfbf0 bsp=e0000001ee6c11e0
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001ee6cfc20 bsp=e0000001ee6c1168
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001ee6cfca0 bsp=e0000001ee6c1130
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001ee6cfca0 bsp=e0000001ee6c10f8
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001ee6cfca0 bsp=e0000001ee6c10c0
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001ee6cfca0 bsp=e0000001ee6c1098
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001ee6cfcc0 bsp=e0000001ee6c1068
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001ee6cfcc0 bsp=e0000001ee6c1018
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001ee6cfcc0 bsp=e0000001ee6c0fe8
 [<a0000001000d6c20>] do_generic_mapping_read+0x780/0x7a0
                                sp=e0000001ee6cfce0 bsp=e0000001ee6c0f48
 [<a0000001000db250>] generic_file_aio_read+0x290/0x340
                                sp=e0000001ee6cfce0 bsp=e0000001ee6c0ee8
 [<a00000010012c990>] do_sync_read+0x170/0x200
                                sp=e0000001ee6cfd10 bsp=e0000001ee6c0ea8
 [<a00000010012cbd0>] vfs_read+0x1b0/0x2e0
                                sp=e0000001ee6cfe20 bsp=e0000001ee6c0e58
 [<a000000100136800>] kernel_read+0x60/0xa0
                                sp=e0000001ee6cfe20 bsp=e0000001ee6c0e18
 [<a000000100138140>] prepare_binprm+0x1e0/0x200
                                sp=e0000001ee6cfe30 bsp=e0000001ee6c0de8
 [<a000000100138c50>] do_execve+0x150/0x320
                                sp=e0000001ee6cfe30 bsp=e0000001ee6c0d80
 [<a000000100013760>] sys_execve+0x60/0xc0
                                sp=e0000001ee6cfe30 bsp=e0000001ee6c0d48
 [<a000000100009b90>] ia64_execve+0x30/0x140
                                sp=e0000001ee6cfe30 bsp=e0000001ee6c0cf8
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001ee6cfe30 bsp=e0000001ee6c0cf8
 [<a000000000010740>] __kernel_syscall_via_break+0x0/0x20
                                sp=e0000001ee6d0000 bsp=e0000001ee6c0cf8
INFO: task sh:9157 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001ee41fbf0 bsp=e0000001ee411138
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001ee41fc20 bsp=e0000001ee4110c0
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001ee41fca0 bsp=e0000001ee411088
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001ee41fca0 bsp=e0000001ee411050
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001ee41fca0 bsp=e0000001ee411018
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001ee41fca0 bsp=e0000001ee410fe8
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001ee41fcc0 bsp=e0000001ee410fb8
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001ee41fcc0 bsp=e0000001ee410f68
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001ee41fcc0 bsp=e0000001ee410f40
 [<a0000001000d6c20>] do_generic_mapping_read+0x780/0x7a0
                                sp=e0000001ee41fce0 bsp=e0000001ee410ea0
 [<a0000001000db250>] generic_file_aio_read+0x290/0x340
                                sp=e0000001ee41fce0 bsp=e0000001ee410e40
 [<a00000010012c990>] do_sync_read+0x170/0x200
                                sp=e0000001ee41fd10 bsp=e0000001ee410e00
 [<a00000010012cbd0>] vfs_read+0x1b0/0x2e0
                                sp=e0000001ee41fe20 bsp=e0000001ee410da8
 [<a000000100136800>] kernel_read+0x60/0xa0
                                sp=e0000001ee41fe20 bsp=e0000001ee410d68
 [<a000000100138140>] prepare_binprm+0x1e0/0x200
                                sp=e0000001ee41fe30 bsp=e0000001ee410d40
 [<a000000100138c50>] do_execve+0x150/0x320
                                sp=e0000001ee41fe30 bsp=e0000001ee410cd8
 [<a000000100013760>] sys_execve+0x60/0xc0
                                sp=e0000001ee41fe30 bsp=e0000001ee410ca0
 [<a000000100009b90>] ia64_execve+0x30/0x140
                                sp=e0000001ee41fe30 bsp=e0000001ee410c50
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001ee41fe30 bsp=e0000001ee410c50
 [<a000000000010740>] __kernel_syscall_via_break+0x0/0x20
                                sp=e0000001ee420000 bsp=e0000001ee410c50
INFO: task sh:9158 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001ee38fbf0 bsp=e0000001ee381138
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001ee38fc20 bsp=e0000001ee3810c0
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001ee38fca0 bsp=e0000001ee381088
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001ee38fca0 bsp=e0000001ee381050
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001ee38fca0 bsp=e0000001ee381018
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001ee38fca0 bsp=e0000001ee380fe8
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001ee38fcc0 bsp=e0000001ee380fb8
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001ee38fcc0 bsp=e0000001ee380f68
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001ee38fcc0 bsp=e0000001ee380f40
 [<a0000001000d6c20>] do_generic_mapping_read+0x780/0x7a0
                                sp=e0000001ee38fce0 bsp=e0000001ee380ea0
 [<a0000001000db250>] generic_file_aio_read+0x290/0x340
                                sp=e0000001ee38fce0 bsp=e0000001ee380e40
 [<a00000010012c990>] do_sync_read+0x170/0x200
                                sp=e0000001ee38fd10 bsp=e0000001ee380e00
 [<a00000010012cbd0>] vfs_read+0x1b0/0x2e0
                                sp=e0000001ee38fe20 bsp=e0000001ee380da8
 [<a000000100136800>] kernel_read+0x60/0xa0
                                sp=e0000001ee38fe20 bsp=e0000001ee380d68
 [<a000000100138140>] prepare_binprm+0x1e0/0x200
                                sp=e0000001ee38fe30 bsp=e0000001ee380d40
 [<a000000100138c50>] do_execve+0x150/0x320
                                sp=e0000001ee38fe30 bsp=e0000001ee380cd8
 [<a000000100013760>] sys_execve+0x60/0xc0
                                sp=e0000001ee38fe30 bsp=e0000001ee380ca0
 [<a000000100009b90>] ia64_execve+0x30/0x140
                                sp=e0000001ee38fe30 bsp=e0000001ee380c50
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001ee38fe30 bsp=e0000001ee380c50
 [<a000000000010740>] __kernel_syscall_via_break+0x0/0x20
                                sp=e0000001ee390000 bsp=e0000001ee380c50
INFO: task sh:9159 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001ef04fbc0 bsp=e0000001ef041298
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001ef04fbf0 bsp=e0000001ef041220
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001ef04fc70 bsp=e0000001ef0411e0
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001ef04fc70 bsp=e0000001ef0411a8
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001ef04fc70 bsp=e0000001ef041170
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001ef04fc70 bsp=e0000001ef041148
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001ef04fc90 bsp=e0000001ef041118
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001ef04fc90 bsp=e0000001ef0410c8
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001ef04fc90 bsp=e0000001ef0410a0
 [<a00000010013ffa0>] __link_path_walk+0x16c0/0x21a0
                                sp=e0000001ef04fcb0 bsp=e0000001ef041020
 [<a000000100140b40>] link_path_walk+0xc0/0x260
                                sp=e0000001ef04fcd0 bsp=e0000001ef040fe0
 [<a00000010013e570>] path_walk+0x30/0x60
                                sp=e0000001ef04fd60 bsp=e0000001ef040fb8
 [<a000000100141200>] do_path_lookup+0x520/0x580
                                sp=e0000001ef04fd60 bsp=e0000001ef040f68
 [<a0000001001415e0>] __path_lookup_intent_open+0x80/0x120
                                sp=e0000001ef04fd70 bsp=e0000001ef040f18
 [<a0000001001416c0>] path_lookup_open+0x40/0x60
                                sp=e0000001ef04fd70 bsp=e0000001ef040ed8
 [<a000000100136640>] open_exec+0x40/0x1a0
                                sp=e0000001ef04fd70 bsp=e0000001ef040eb8
 [<a000000100198c90>] load_elf_binary+0x530/0x2700
                                sp=e0000001ef04fe00 bsp=e0000001ef040da0
 [<a000000100138710>] search_binary_handler+0x1b0/0x5a0
                                sp=e0000001ef04fe30 bsp=e0000001ef040d40
 [<a000000100138d40>] do_execve+0x240/0x320
                                sp=e0000001ef04fe30 bsp=e0000001ef040cd8
 [<a000000100013760>] sys_execve+0x60/0xc0
                                sp=e0000001ef04fe30 bsp=e0000001ef040ca0
 [<a000000100009b90>] ia64_execve+0x30/0x140
                                sp=e0000001ef04fe30 bsp=e0000001ef040c50
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001ef04fe30 bsp=e0000001ef040c50
 [<a000000000010740>] __kernel_syscall_via_break+0x0/0x20
                                sp=e0000001ef050000 bsp=e0000001ef040c50
INFO: task sh:9160 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001ea03fbc0 bsp=e0000001ea031288
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001ea03fbf0 bsp=e0000001ea031210
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001ea03fc70 bsp=e0000001ea0311d0
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001ea03fc70 bsp=e0000001ea031198
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001ea03fc70 bsp=e0000001ea031160
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001ea03fc70 bsp=e0000001ea031138
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001ea03fc90 bsp=e0000001ea031108
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001ea03fc90 bsp=e0000001ea0310b8
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001ea03fc90 bsp=e0000001ea031090
 [<a0000001000d6c20>] do_generic_mapping_read+0x780/0x7a0
                                sp=e0000001ea03fcb0 bsp=e0000001ea030fe8
 [<a0000001000db250>] generic_file_aio_read+0x290/0x340
                                sp=e0000001ea03fcb0 bsp=e0000001ea030f88
 [<a00000010012c990>] do_sync_read+0x170/0x200
                                sp=e0000001ea03fce0 bsp=e0000001ea030f48
 [<a00000010012cbd0>] vfs_read+0x1b0/0x2e0
                                sp=e0000001ea03fdf0 bsp=e0000001ea030ef8
 [<a000000100136800>] kernel_read+0x60/0xa0
                                sp=e0000001ea03fdf0 bsp=e0000001ea030eb8
 [<a000000100198d00>] load_elf_binary+0x5a0/0x2700
                                sp=e0000001ea03fe00 bsp=e0000001ea030da0
 [<a000000100138710>] search_binary_handler+0x1b0/0x5a0
                                sp=e0000001ea03fe30 bsp=e0000001ea030d40
 [<a000000100138d40>] do_execve+0x240/0x320
                                sp=e0000001ea03fe30 bsp=e0000001ea030cd8
 [<a000000100013760>] sys_execve+0x60/0xc0
                                sp=e0000001ea03fe30 bsp=e0000001ea030ca0
 [<a000000100009b90>] ia64_execve+0x30/0x140
                                sp=e0000001ea03fe30 bsp=e0000001ea030c50
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001ea03fe30 bsp=e0000001ea030c50
 [<a000000000010740>] __kernel_syscall_via_break+0x0/0x20
                                sp=e0000001ea040000 bsp=e0000001ea030c50
INFO: task objdump:9161 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001ee78fd10 bsp=e0000001ee781240
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001ee78fd40 bsp=e0000001ee7811c0
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001ee78fdc0 bsp=e0000001ee781188
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001ee78fdc0 bsp=e0000001ee781150
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001ee78fdc0 bsp=e0000001ee781118
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001ee78fdc0 bsp=e0000001ee7810f0
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001ee78fde0 bsp=e0000001ee7810c0
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001ee78fde0 bsp=e0000001ee781070
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001ee78fde0 bsp=e0000001ee781048
 [<a0000001000d79b0>] generic_file_mmap+0x90/0x100
                                sp=e0000001ee78fe00 bsp=e0000001ee781020
 [<a000000100101dd0>] mmap_region+0x3b0/0x7a0
                                sp=e0000001ee78fe00 bsp=e0000001ee780f68
 [<a000000100102820>] do_mmap_pgoff+0x660/0x680
                                sp=e0000001ee78fe30 bsp=e0000001ee780ef8
 [<a000000100033000>] sys_mmap+0x1a0/0x260
                                sp=e0000001ee78fe30 bsp=e0000001ee780e70
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001ee78fe30 bsp=e0000001ee780e70
INFO: task grep:9162 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001eb39fbe0 bsp=e0000001eb3912a8
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001eb39fc10 bsp=e0000001eb391230
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001eb39fc90 bsp=e0000001eb3911f0
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001eb39fc90 bsp=e0000001eb3911b8
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001eb39fc90 bsp=e0000001eb391180
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001eb39fc90 bsp=e0000001eb391158
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001eb39fcb0 bsp=e0000001eb391128
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001eb39fcb0 bsp=e0000001eb3910d8
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001eb39fcb0 bsp=e0000001eb3910b0
 [<a00000010013ffa0>] __link_path_walk+0x16c0/0x21a0
                                sp=e0000001eb39fcd0 bsp=e0000001eb391030
 [<a000000100140b40>] link_path_walk+0xc0/0x260
                                sp=e0000001eb39fcf0 bsp=e0000001eb390ff0
 [<a00000010013e570>] path_walk+0x30/0x60
                                sp=e0000001eb39fd80 bsp=e0000001eb390fc8
 [<a000000100141200>] do_path_lookup+0x520/0x580
                                sp=e0000001eb39fd80 bsp=e0000001eb390f78
 [<a0000001001415e0>] __path_lookup_intent_open+0x80/0x120
                                sp=e0000001eb39fd90 bsp=e0000001eb390f28
 [<a0000001001416c0>] path_lookup_open+0x40/0x60
                                sp=e0000001eb39fd90 bsp=e0000001eb390ee8
 [<a000000100142b60>] open_namei+0xc0/0xf40
                                sp=e0000001eb39fd90 bsp=e0000001eb390e48
 [<a00000010012b150>] do_filp_open+0x50/0xc0
                                sp=e0000001eb39fda0 bsp=e0000001eb390e10
 [<a00000010012b9b0>] do_sys_open+0x90/0x1c0
                                sp=e0000001eb39fe30 bsp=e0000001eb390db8
 [<a00000010012bb30>] sys_open+0x50/0x80
                                sp=e0000001eb39fe30 bsp=e0000001eb390d60
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001eb39fe30 bsp=e0000001eb390d60
INFO: task objdump:9163 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001e99bfbf0 bsp=e0000001e99b1168
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001e99bfc20 bsp=e0000001e99b10f0
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001e99bfca0 bsp=e0000001e99b10b8
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001e99bfca0 bsp=e0000001e99b1080
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001e99bfca0 bsp=e0000001e99b1048
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001e99bfca0 bsp=e0000001e99b1020
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001e99bfcc0 bsp=e0000001e99b0fe8
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001e99bfcc0 bsp=e0000001e99b0f98
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001e99bfcc0 bsp=e0000001e99b0f70
 [<a0000001000d6c20>] do_generic_mapping_read+0x780/0x7a0
                                sp=e0000001e99bfce0 bsp=e0000001e99b0ed0
 [<a0000001000db250>] generic_file_aio_read+0x290/0x340
                                sp=e0000001e99bfce0 bsp=e0000001e99b0e70
 [<a00000010012c990>] do_sync_read+0x170/0x200
                                sp=e0000001e99bfd10 bsp=e0000001e99b0e30
 [<a00000010012cbd0>] vfs_read+0x1b0/0x2e0
                                sp=e0000001e99bfe20 bsp=e0000001e99b0dd8
 [<a00000010012d250>] sys_read+0x70/0xe0
                                sp=e0000001e99bfe20 bsp=e0000001e99b0d60
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001e99bfe30 bsp=e0000001e99b0d60
INFO: task grep:9164 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001ed63fbf0 bsp=e0000001ed631168
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001ed63fc20 bsp=e0000001ed6310f0
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001ed63fca0 bsp=e0000001ed6310b8
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001ed63fca0 bsp=e0000001ed631080
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001ed63fca0 bsp=e0000001ed631048
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001ed63fca0 bsp=e0000001ed631020
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001ed63fcc0 bsp=e0000001ed630fe8
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001ed63fcc0 bsp=e0000001ed630f98
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001ed63fcc0 bsp=e0000001ed630f70
 [<a0000001000d6c20>] do_generic_mapping_read+0x780/0x7a0
                                sp=e0000001ed63fce0 bsp=e0000001ed630ed0
 [<a0000001000db250>] generic_file_aio_read+0x290/0x340
                                sp=e0000001ed63fce0 bsp=e0000001ed630e70
 [<a00000010012c990>] do_sync_read+0x170/0x200
                                sp=e0000001ed63fd10 bsp=e0000001ed630e30
 [<a00000010012cbd0>] vfs_read+0x1b0/0x2e0
                                sp=e0000001ed63fe20 bsp=e0000001ed630dd8
 [<a00000010012d250>] sys_read+0x70/0xe0
                                sp=e0000001ed63fe20 bsp=e0000001ed630d60
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001ed63fe30 bsp=e0000001ed630d60
INFO: task objdump:9165 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001ee18fbf0 bsp=e0000001ee181280
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001ee18fc20 bsp=e0000001ee181208
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001ee18fca0 bsp=e0000001ee1811c8
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001ee18fca0 bsp=e0000001ee181190
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001ee18fca0 bsp=e0000001ee181158
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001ee18fca0 bsp=e0000001ee181130
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001ee18fcc0 bsp=e0000001ee181100
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001ee18fcc0 bsp=e0000001ee1810b0
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001ee18fcc0 bsp=e0000001ee181088
 [<a0000001000d6c20>] do_generic_mapping_read+0x780/0x7a0
                                sp=e0000001ee18fce0 bsp=e0000001ee180fe0
 [<a0000001000db250>] generic_file_aio_read+0x290/0x340
                                sp=e0000001ee18fce0 bsp=e0000001ee180f80
 [<a00000010012c990>] do_sync_read+0x170/0x200
                                sp=e0000001ee18fd10 bsp=e0000001ee180f40
 [<a00000010012cbd0>] vfs_read+0x1b0/0x2e0
                                sp=e0000001ee18fe20 bsp=e0000001ee180ef0
 [<a00000010012d250>] sys_read+0x70/0xe0
                                sp=e0000001ee18fe20 bsp=e0000001ee180e78
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001ee18fe30 bsp=e0000001ee180e78
INFO: task grep:9166 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001eb50fbe0 bsp=e0000001eb5013b8
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001eb50fc10 bsp=e0000001eb501340
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001eb50fc90 bsp=e0000001eb501308
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001eb50fc90 bsp=e0000001eb5012d0
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001eb50fc90 bsp=e0000001eb501298
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001eb50fc90 bsp=e0000001eb501270
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001eb50fcb0 bsp=e0000001eb501240
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001eb50fcb0 bsp=e0000001eb5011e8
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001eb50fcb0 bsp=e0000001eb5011c0
 [<a00000010013ffa0>] __link_path_walk+0x16c0/0x21a0
                                sp=e0000001eb50fcd0 bsp=e0000001eb501140
 [<a000000100140b40>] link_path_walk+0xc0/0x260
                                sp=e0000001eb50fcf0 bsp=e0000001eb501108
 [<a00000010013e570>] path_walk+0x30/0x60
                                sp=e0000001eb50fd80 bsp=e0000001eb5010e0
 [<a000000100141200>] do_path_lookup+0x520/0x580
                                sp=e0000001eb50fd80 bsp=e0000001eb501090
 [<a0000001001415e0>] __path_lookup_intent_open+0x80/0x120
                                sp=e0000001eb50fd90 bsp=e0000001eb501040
 [<a0000001001416c0>] path_lookup_open+0x40/0x60
                                sp=e0000001eb50fd90 bsp=e0000001eb501000
 [<a000000100142b60>] open_namei+0xc0/0xf40
                                sp=e0000001eb50fd90 bsp=e0000001eb500f58
 [<a00000010012b150>] do_filp_open+0x50/0xc0
                                sp=e0000001eb50fda0 bsp=e0000001eb500f20
 [<a00000010012b9b0>] do_sys_open+0x90/0x1c0
                                sp=e0000001eb50fe30 bsp=e0000001eb500ed0
 [<a00000010012bb30>] sys_open+0x50/0x80
                                sp=e0000001eb50fe30 bsp=e0000001eb500e78
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001eb50fe30 bsp=e0000001eb500e78
INFO: task objdump:9167 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001b7c7fbe0 bsp=e0000001b7c713b8
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001b7c7fc10 bsp=e0000001b7c71340
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001b7c7fc90 bsp=e0000001b7c71308
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001b7c7fc90 bsp=e0000001b7c712d0
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001b7c7fc90 bsp=e0000001b7c71298
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001b7c7fc90 bsp=e0000001b7c71270
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001b7c7fcb0 bsp=e0000001b7c71240
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001b7c7fcb0 bsp=e0000001b7c711e8
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001b7c7fcb0 bsp=e0000001b7c711c0
 [<a00000010013ffa0>] __link_path_walk+0x16c0/0x21a0
                                sp=e0000001b7c7fcd0 bsp=e0000001b7c71140
 [<a000000100140b40>] link_path_walk+0xc0/0x260
                                sp=e0000001b7c7fcf0 bsp=e0000001b7c71108
 [<a00000010013e570>] path_walk+0x30/0x60
                                sp=e0000001b7c7fd80 bsp=e0000001b7c710e0
 [<a000000100141200>] do_path_lookup+0x520/0x580
                                sp=e0000001b7c7fd80 bsp=e0000001b7c71090
 [<a0000001001415e0>] __path_lookup_intent_open+0x80/0x120
                                sp=e0000001b7c7fd90 bsp=e0000001b7c71040
 [<a0000001001416c0>] path_lookup_open+0x40/0x60
                                sp=e0000001b7c7fd90 bsp=e0000001b7c71000
 [<a000000100142b60>] open_namei+0xc0/0xf40
                                sp=e0000001b7c7fd90 bsp=e0000001b7c70f58
 [<a00000010012b150>] do_filp_open+0x50/0xc0
                                sp=e0000001b7c7fda0 bsp=e0000001b7c70f20
 [<a00000010012b9b0>] do_sys_open+0x90/0x1c0
                                sp=e0000001b7c7fe30 bsp=e0000001b7c70ed0
 [<a00000010012bb30>] sys_open+0x50/0x80
                                sp=e0000001b7c7fe30 bsp=e0000001b7c70e78
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001b7c7fe30 bsp=e0000001b7c70e78
INFO: task grep:9168 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001ed8afbf0 bsp=e0000001ed8a1280
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001ed8afc20 bsp=e0000001ed8a1208
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001ed8afca0 bsp=e0000001ed8a11c8
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001ed8afca0 bsp=e0000001ed8a1190
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001ed8afca0 bsp=e0000001ed8a1158
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001ed8afca0 bsp=e0000001ed8a1130
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001ed8afcc0 bsp=e0000001ed8a1100
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001ed8afcc0 bsp=e0000001ed8a10b0
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001ed8afcc0 bsp=e0000001ed8a1088
 [<a0000001000d6c20>] do_generic_mapping_read+0x780/0x7a0
                                sp=e0000001ed8afce0 bsp=e0000001ed8a0fe0
 [<a0000001000db250>] generic_file_aio_read+0x290/0x340
                                sp=e0000001ed8afce0 bsp=e0000001ed8a0f80
 [<a00000010012c990>] do_sync_read+0x170/0x200
                                sp=e0000001ed8afd10 bsp=e0000001ed8a0f40
 [<a00000010012cbd0>] vfs_read+0x1b0/0x2e0
                                sp=e0000001ed8afe20 bsp=e0000001ed8a0ef0
 [<a00000010012d250>] sys_read+0x70/0xe0
                                sp=e0000001ed8afe20 bsp=e0000001ed8a0e78
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001ed8afe30 bsp=e0000001ed8a0e78
INFO: task kjournald:1272 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001b00bfca0 bsp=e0000001b00b0e30
 [<a000000100704cf0>] io_schedule+0x50/0x80
                                sp=e0000001b00bfcd0 bsp=e0000001b00b0e10
 [<a000000100179d60>] sync_buffer+0x100/0x120
                                sp=e0000001b00bfcd0 bsp=e0000001b00b0de8
 [<a000000100705130>] __wait_on_bit+0xd0/0x180
                                sp=e0000001b00bfcd0 bsp=e0000001b00b0d98
 [<a0000001007052c0>] out_of_line_wait_on_bit+0xe0/0x100
                                sp=e0000001b00bfcd0 bsp=e0000001b00b0d50
 [<a000000100179b80>] __wait_on_buffer+0x40/0x60
                                sp=e0000001b00bfd50 bsp=e0000001b00b0d30
 [<a000000100253250>] journal_commit_transaction+0x1110/0x1fc0
                                sp=e0000001b00bfd50 bsp=e0000001b00b0c90
 [<a000000100258c60>] kjournald+0x180/0x420
                                sp=e0000001b00bfdc0 bsp=e0000001b00b0c30
 [<a0000001000a7c40>] kthread+0xa0/0x120
                                sp=e0000001b00bfe30 bsp=e0000001b00b0c00
 [<a000000100013970>] kernel_thread_helper+0x30/0x60
                                sp=e0000001b00bfe30 bsp=e0000001b00b0bd0
 [<a0000001000089c0>] start_kernel_thread+0x20/0x40
                                sp=e0000001b00bfe30 bsp=e0000001b00b0bd0
INFO: task syslogd:4887 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001bdf2fcd0 bsp=e0000001bdf20f70
 [<a000000100259240>] log_wait_commit+0x180/0x220
                                sp=e0000001bdf2fd00 bsp=e0000001bdf20f28
 [<a00000010024d350>] journal_stop+0x3f0/0x460
                                sp=e0000001bdf2fd60 bsp=e0000001bdf20ed0
 [<a00000010024d440>] journal_force_commit+0x80/0xa0
                                sp=e0000001bdf2fd70 bsp=e0000001bdf20eb0
 [<a000000100241240>] ext3_force_commit+0x60/0x80
                                sp=e0000001bdf2fd70 bsp=e0000001bdf20e90
 [<a00000010022d5e0>] ext3_write_inode+0xa0/0xc0
                                sp=e0000001bdf2fd70 bsp=e0000001bdf20e68
 [<a000000100171870>] __writeback_single_inode+0x510/0x7c0
                                sp=e0000001bdf2fd70 bsp=e0000001bdf20e10
 [<a000000100171de0>] sync_inode+0x40/0x80
                                sp=e0000001bdf2fdf0 bsp=e0000001bdf20dd8
 [<a000000100227bd0>] ext3_sync_file+0x1b0/0x1e0
                                sp=e0000001bdf2fdf0 bsp=e0000001bdf20db0
 [<a0000001001788b0>] do_fsync+0xf0/0x160
                                sp=e0000001bdf2fe30 bsp=e0000001bdf20d70
 [<a000000100178970>] __do_fsync+0x50/0x80
                                sp=e0000001bdf2fe30 bsp=e0000001bdf20d40
 [<a0000001001789d0>] sys_fsync+0x30/0x60
                                sp=e0000001bdf2fe30 bsp=e0000001bdf20ce8
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001bdf2fe30 bsp=e0000001bdf20ce8
 [<a000000000010740>] __kernel_syscall_via_break+0x0/0x20
                                sp=e0000001bdf30000 bsp=e0000001bdf20ce8
INFO: task make:8649 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001e97afb40 bsp=e0000001e97a0d40
 [<a000000100704e60>] schedule_timeout+0x40/0x180
                                sp=e0000001e97afb70 bsp=e0000001e97a0d10
 [<a000000100704450>] wait_for_common+0x1b0/0x300
                                sp=e0000001e97afba0 bsp=e0000001e97a0cc0
 [<a0000001007045e0>] wait_for_completion+0x40/0x60
                                sp=e0000001e97afc00 bsp=e0000001e97a0ca0
 [<a000000100077240>] do_fork+0x420/0x480
                                sp=e0000001e97afc00 bsp=e0000001e97a0c50
 [<a000000100009d80>] sys_clone+0x60/0x80
                                sp=e0000001e97afc20 bsp=e0000001e97a0c00
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001e97afe30 bsp=e0000001e97a0c00
 [<a000000000010740>] __kernel_syscall_via_break+0x0/0x20
                                sp=e0000001e97b0000 bsp=e0000001e97a0c00
INFO: task cc1:8839 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001edc4fbf0 bsp=e0000001edc41268
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001edc4fc20 bsp=e0000001edc411e8
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001edc4fca0 bsp=e0000001edc411b0
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001edc4fca0 bsp=e0000001edc41178
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001edc4fca0 bsp=e0000001edc41140
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001edc4fca0 bsp=e0000001edc41118
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001edc4fcc0 bsp=e0000001edc410e8
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001edc4fcc0 bsp=e0000001edc41098
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001edc4fcc0 bsp=e0000001edc41070
 [<a0000001000d6c20>] do_generic_mapping_read+0x780/0x7a0
                                sp=e0000001edc4fce0 bsp=e0000001edc40fc8
 [<a0000001000db250>] generic_file_aio_read+0x290/0x340
                                sp=e0000001edc4fce0 bsp=e0000001edc40f68
 [<a00000010012c990>] do_sync_read+0x170/0x200
                                sp=e0000001edc4fd10 bsp=e0000001edc40f28
 [<a00000010012cbd0>] vfs_read+0x1b0/0x2e0
                                sp=e0000001edc4fe20 bsp=e0000001edc40ed8
 [<a00000010012d250>] sys_read+0x70/0xe0
                                sp=e0000001edc4fe20 bsp=e0000001edc40e60
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001edc4fe30 bsp=e0000001edc40e60
 [<a000000000010740>] __kernel_syscall_via_break+0x0/0x20
                                sp=e0000001edc50000 bsp=e0000001edc40e60
INFO: task sh:9107 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001eea8f6d0 bsp=e0000001eea81448
 [<a000000100704cf0>] io_schedule+0x50/0x80
                                sp=e0000001eea8f700 bsp=e0000001eea81428
 [<a000000100179d60>] sync_buffer+0x100/0x120
                                sp=e0000001eea8f700 bsp=e0000001eea81408
 [<a000000100705130>] __wait_on_bit+0xd0/0x180
                                sp=e0000001eea8f700 bsp=e0000001eea813b0
 [<a0000001007052c0>] out_of_line_wait_on_bit+0xe0/0x100
                                sp=e0000001eea8f700 bsp=e0000001eea81368
 [<a000000100179b80>] __wait_on_buffer+0x40/0x60
                                sp=e0000001eea8f780 bsp=e0000001eea81348
 [<a000000100184370>] __bread+0x190/0x220
                                sp=e0000001eea8f780 bsp=e0000001eea81318
 [<a00000010022a6e0>] ext3_get_branch+0xe0/0x220
                                sp=e0000001eea8f780 bsp=e0000001eea812c0
 [<a00000010022f130>] ext3_get_blocks_handle+0x130/0x1560
                                sp=e0000001eea8f780 bsp=e0000001eea811d8
 [<a0000001002306f0>] ext3_get_block+0x190/0x200
                                sp=e0000001eea8f850 bsp=e0000001eea81180
 [<a00000010018dfd0>] do_mpage_readpage+0x310/0xb80
                                sp=e0000001eea8f850 bsp=e0000001eea810d8
 [<a00000010018e9b0>] mpage_readpages+0x170/0x240
                                sp=e0000001eea8fc50 bsp=e0000001eea81088
 [<a00000010022be10>] ext3_readpages+0x30/0x60
                                sp=e0000001eea8fcd0 bsp=e0000001eea81050
 [<a0000001000e7120>] __do_page_cache_readahead+0x2e0/0x4c0
                                sp=e0000001eea8fcd0 bsp=e0000001eea80fe0
 [<a0000001000e7a40>] ondemand_readahead+0x460/0x4a0
                                sp=e0000001eea8fce0 bsp=e0000001eea80f80
 [<a0000001000e7ad0>] page_cache_sync_readahead+0x50/0x80
                                sp=e0000001eea8fce0 bsp=e0000001eea80f40
 [<a0000001000d6560>] do_generic_mapping_read+0xc0/0x7a0
                                sp=e0000001eea8fce0 bsp=e0000001eea80ea0
 [<a0000001000db250>] generic_file_aio_read+0x290/0x340
                                sp=e0000001eea8fce0 bsp=e0000001eea80e40
 [<a00000010012c990>] do_sync_read+0x170/0x200
                                sp=e0000001eea8fd10 bsp=e0000001eea80e00
 [<a00000010012cbd0>] vfs_read+0x1b0/0x2e0
                                sp=e0000001eea8fe20 bsp=e0000001eea80da8
 [<a000000100136800>] kernel_read+0x60/0xa0
                                sp=e0000001eea8fe20 bsp=e0000001eea80d68
 [<a000000100138140>] prepare_binprm+0x1e0/0x200
                                sp=e0000001eea8fe30 bsp=e0000001eea80d40
 [<a000000100138c50>] do_execve+0x150/0x320
                                sp=e0000001eea8fe30 bsp=e0000001eea80cd8
 [<a000000100013760>] sys_execve+0x60/0xc0
                                sp=e0000001eea8fe30 bsp=e0000001eea80ca0
 [<a000000100009b90>] ia64_execve+0x30/0x140
                                sp=e0000001eea8fe30 bsp=e0000001eea80c50
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001eea8fe30 bsp=e0000001eea80c50
 [<a000000000010740>] __kernel_syscall_via_break+0x0/0x20
                                sp=e0000001eea90000 bsp=e0000001eea80c50
INFO: task gcc:9122 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001ecfdfb80 bsp=e0000001ecfd10c0
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001ecfdfbb0 bsp=e0000001ecfd1048
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001ecfdfc30 bsp=e0000001ecfd1010
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001ecfdfc30 bsp=e0000001ecfd0fd0
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001ecfdfc30 bsp=e0000001ecfd0f98
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001ecfdfc30 bsp=e0000001ecfd0f70
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001ecfdfc50 bsp=e0000001ecfd0f40
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001ecfdfc50 bsp=e0000001ecfd0ef0
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001ecfdfc50 bsp=e0000001ecfd0ec8
 [<a00000010013f100>] __link_path_walk+0x820/0x21a0
                                sp=e0000001ecfdfc70 bsp=e0000001ecfd0e48
 [<a000000100140b40>] link_path_walk+0xc0/0x260
                                sp=e0000001ecfdfc90 bsp=e0000001ecfd0e10
 [<a00000010013e570>] path_walk+0x30/0x60
                                sp=e0000001ecfdfd20 bsp=e0000001ecfd0de0
 [<a000000100141200>] do_path_lookup+0x520/0x580
                                sp=e0000001ecfdfd20 bsp=e0000001ecfd0d90
 [<a000000100141de0>] __user_walk_fd+0x60/0xc0
                                sp=e0000001ecfdfd30 bsp=e0000001ecfd0d58
 [<a000000100134350>] vfs_stat_fd+0x30/0xa0
                                sp=e0000001ecfdfd30 bsp=e0000001ecfd0d28
 [<a0000001001348d0>] sys_newstat+0x30/0x80
                                sp=e0000001ecfdfdc0 bsp=e0000001ecfd0cc8
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001ecfdfe30 bsp=e0000001ecfd0cc8
 [<a000000000010740>] __kernel_syscall_via_break+0x0/0x20
                                sp=e0000001ecfe0000 bsp=e0000001ecfd0cc8
INFO: task sh:9128 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001ef28fbf0 bsp=e0000001ef281138
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001ef28fc20 bsp=e0000001ef2810c0
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001ef28fca0 bsp=e0000001ef281088
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001ef28fca0 bsp=e0000001ef281050
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001ef28fca0 bsp=e0000001ef281018
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001ef28fca0 bsp=e0000001ef280fe8
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001ef28fcc0 bsp=e0000001ef280fb8
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001ef28fcc0 bsp=e0000001ef280f68
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001ef28fcc0 bsp=e0000001ef280f40
 [<a0000001000d6c20>] do_generic_mapping_read+0x780/0x7a0
                                sp=e0000001ef28fce0 bsp=e0000001ef280ea0
 [<a0000001000db250>] generic_file_aio_read+0x290/0x340
                                sp=e0000001ef28fce0 bsp=e0000001ef280e40
 [<a00000010012c990>] do_sync_read+0x170/0x200
                                sp=e0000001ef28fd10 bsp=e0000001ef280e00
 [<a00000010012cbd0>] vfs_read+0x1b0/0x2e0
                                sp=e0000001ef28fe20 bsp=e0000001ef280da8
 [<a000000100136800>] kernel_read+0x60/0xa0
                                sp=e0000001ef28fe20 bsp=e0000001ef280d68
 [<a000000100138140>] prepare_binprm+0x1e0/0x200
                                sp=e0000001ef28fe30 bsp=e0000001ef280d40
 [<a000000100138c50>] do_execve+0x150/0x320
                                sp=e0000001ef28fe30 bsp=e0000001ef280cd8
 [<a000000100013760>] sys_execve+0x60/0xc0
                                sp=e0000001ef28fe30 bsp=e0000001ef280ca0
 [<a000000100009b90>] ia64_execve+0x30/0x140
                                sp=e0000001ef28fe30 bsp=e0000001ef280c50
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001ef28fe30 bsp=e0000001ef280c50
 [<a000000000010740>] __kernel_syscall_via_break+0x0/0x20
                                sp=e0000001ef290000 bsp=e0000001ef280c50
INFO: task sh:9136 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001e5bdfbf0 bsp=e0000001e5bd1138
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001e5bdfc20 bsp=e0000001e5bd10c0
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001e5bdfca0 bsp=e0000001e5bd1088
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001e5bdfca0 bsp=e0000001e5bd1050
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001e5bdfca0 bsp=e0000001e5bd1018
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001e5bdfca0 bsp=e0000001e5bd0fe8
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001e5bdfcc0 bsp=e0000001e5bd0fb8
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001e5bdfcc0 bsp=e0000001e5bd0f68
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001e5bdfcc0 bsp=e0000001e5bd0f40
 [<a0000001000d6c20>] do_generic_mapping_read+0x780/0x7a0
                                sp=e0000001e5bdfce0 bsp=e0000001e5bd0ea0
 [<a0000001000db250>] generic_file_aio_read+0x290/0x340
                                sp=e0000001e5bdfce0 bsp=e0000001e5bd0e40
 [<a00000010012c990>] do_sync_read+0x170/0x200
                                sp=e0000001e5bdfd10 bsp=e0000001e5bd0e00
 [<a00000010012cbd0>] vfs_read+0x1b0/0x2e0
                                sp=e0000001e5bdfe20 bsp=e0000001e5bd0da8
 [<a000000100136800>] kernel_read+0x60/0xa0
                                sp=e0000001e5bdfe20 bsp=e0000001e5bd0d68
 [<a000000100138140>] prepare_binprm+0x1e0/0x200
                                sp=e0000001e5bdfe30 bsp=e0000001e5bd0d40
 [<a000000100138c50>] do_execve+0x150/0x320
                                sp=e0000001e5bdfe30 bsp=e0000001e5bd0cd8
 [<a000000100013760>] sys_execve+0x60/0xc0
                                sp=e0000001e5bdfe30 bsp=e0000001e5bd0ca0
 [<a000000100009b90>] ia64_execve+0x30/0x140
                                sp=e0000001e5bdfe30 bsp=e0000001e5bd0c50
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001e5bdfe30 bsp=e0000001e5bd0c50
 [<a000000000010740>] __kernel_syscall_via_break+0x0/0x20
                                sp=e0000001e5be0000 bsp=e0000001e5bd0c50
INFO: task mv:9143 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001b2aafbe0 bsp=e0000001b2aa12a8
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001b2aafc10 bsp=e0000001b2aa1230
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001b2aafc90 bsp=e0000001b2aa11f0
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001b2aafc90 bsp=e0000001b2aa11b8
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001b2aafc90 bsp=e0000001b2aa1180
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001b2aafc90 bsp=e0000001b2aa1158
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001b2aafcb0 bsp=e0000001b2aa1128
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001b2aafcb0 bsp=e0000001b2aa10d8
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001b2aafcb0 bsp=e0000001b2aa10b0
 [<a00000010013ffa0>] __link_path_walk+0x16c0/0x21a0
                                sp=e0000001b2aafcd0 bsp=e0000001b2aa1030
 [<a000000100140b40>] link_path_walk+0xc0/0x260
                                sp=e0000001b2aafcf0 bsp=e0000001b2aa0ff0
 [<a00000010013e570>] path_walk+0x30/0x60
                                sp=e0000001b2aafd80 bsp=e0000001b2aa0fc8
 [<a000000100141200>] do_path_lookup+0x520/0x580
                                sp=e0000001b2aafd80 bsp=e0000001b2aa0f78
 [<a0000001001415e0>] __path_lookup_intent_open+0x80/0x120
                                sp=e0000001b2aafd90 bsp=e0000001b2aa0f28
 [<a0000001001416c0>] path_lookup_open+0x40/0x60
                                sp=e0000001b2aafd90 bsp=e0000001b2aa0ee8
 [<a000000100142b60>] open_namei+0xc0/0xf40
                                sp=e0000001b2aafd90 bsp=e0000001b2aa0e48
 [<a00000010012b150>] do_filp_open+0x50/0xc0
                                sp=e0000001b2aafda0 bsp=e0000001b2aa0e10
 [<a00000010012b9b0>] do_sys_open+0x90/0x1c0
                                sp=e0000001b2aafe30 bsp=e0000001b2aa0db8
 [<a00000010012bb30>] sys_open+0x50/0x80
                                sp=e0000001b2aafe30 bsp=e0000001b2aa0d60
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001b2aafe30 bsp=e0000001b2aa0d60
INFO: task mv:9150 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001ecfafbf0 bsp=e0000001ecfa1168
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001ecfafc20 bsp=e0000001ecfa10f0
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001ecfafca0 bsp=e0000001ecfa10b8
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001ecfafca0 bsp=e0000001ecfa1080
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001ecfafca0 bsp=e0000001ecfa1048
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001ecfafca0 bsp=e0000001ecfa1020
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001ecfafcc0 bsp=e0000001ecfa0fe8
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001ecfafcc0 bsp=e0000001ecfa0f98
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001ecfafcc0 bsp=e0000001ecfa0f70
 [<a0000001000d6c20>] do_generic_mapping_read+0x780/0x7a0
                                sp=e0000001ecfafce0 bsp=e0000001ecfa0ed0
 [<a0000001000db250>] generic_file_aio_read+0x290/0x340
                                sp=e0000001ecfafce0 bsp=e0000001ecfa0e70
 [<a00000010012c990>] do_sync_read+0x170/0x200
                                sp=e0000001ecfafd10 bsp=e0000001ecfa0e30
 [<a00000010012cbd0>] vfs_read+0x1b0/0x2e0
                                sp=e0000001ecfafe20 bsp=e0000001ecfa0dd8
 [<a00000010012d250>] sys_read+0x70/0xe0
                                sp=e0000001ecfafe20 bsp=e0000001ecfa0d60
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001ecfafe30 bsp=e0000001ecfa0d60
INFO: task sh:9154 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001eb41fbf0 bsp=e0000001eb411138
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001eb41fc20 bsp=e0000001eb4110c0
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001eb41fca0 bsp=e0000001eb411088
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001eb41fca0 bsp=e0000001eb411050
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001eb41fca0 bsp=e0000001eb411018
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001eb41fca0 bsp=e0000001eb410fe8
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001eb41fcc0 bsp=e0000001eb410fb8
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001eb41fcc0 bsp=e0000001eb410f68
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001eb41fcc0 bsp=e0000001eb410f40
 [<a0000001000d6c20>] do_generic_mapping_read+0x780/0x7a0
                                sp=e0000001eb41fce0 bsp=e0000001eb410ea0
 [<a0000001000db250>] generic_file_aio_read+0x290/0x340
                                sp=e0000001eb41fce0 bsp=e0000001eb410e40
 [<a00000010012c990>] do_sync_read+0x170/0x200
                                sp=e0000001eb41fd10 bsp=e0000001eb410e00
 [<a00000010012cbd0>] vfs_read+0x1b0/0x2e0
                                sp=e0000001eb41fe20 bsp=e0000001eb410da8
 [<a000000100136800>] kernel_read+0x60/0xa0
                                sp=e0000001eb41fe20 bsp=e0000001eb410d68
 [<a000000100138140>] prepare_binprm+0x1e0/0x200
                                sp=e0000001eb41fe30 bsp=e0000001eb410d40
 [<a000000100138c50>] do_execve+0x150/0x320
                                sp=e0000001eb41fe30 bsp=e0000001eb410cd8
 [<a000000100013760>] sys_execve+0x60/0xc0
                                sp=e0000001eb41fe30 bsp=e0000001eb410ca0
 [<a000000100009b90>] ia64_execve+0x30/0x140
                                sp=e0000001eb41fe30 bsp=e0000001eb410c50
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001eb41fe30 bsp=e0000001eb410c50
 [<a000000000010740>] __kernel_syscall_via_break+0x0/0x20
                                sp=e0000001eb420000 bsp=e0000001eb410c50
INFO: task sh:9155 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001ecdffbf0 bsp=e0000001ecdf1138
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001ecdffc20 bsp=e0000001ecdf10c0
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001ecdffca0 bsp=e0000001ecdf1088
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001ecdffca0 bsp=e0000001ecdf1050
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001ecdffca0 bsp=e0000001ecdf1018
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001ecdffca0 bsp=e0000001ecdf0fe8
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001ecdffcc0 bsp=e0000001ecdf0fb8
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001ecdffcc0 bsp=e0000001ecdf0f68
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001ecdffcc0 bsp=e0000001ecdf0f40
 [<a0000001000d6c20>] do_generic_mapping_read+0x780/0x7a0
                                sp=e0000001ecdffce0 bsp=e0000001ecdf0ea0
 [<a0000001000db250>] generic_file_aio_read+0x290/0x340
                                sp=e0000001ecdffce0 bsp=e0000001ecdf0e40
 [<a00000010012c990>] do_sync_read+0x170/0x200
                                sp=e0000001ecdffd10 bsp=e0000001ecdf0e00
 [<a00000010012cbd0>] vfs_read+0x1b0/0x2e0
                                sp=e0000001ecdffe20 bsp=e0000001ecdf0da8
 [<a000000100136800>] kernel_read+0x60/0xa0
                                sp=e0000001ecdffe20 bsp=e0000001ecdf0d68
 [<a000000100138140>] prepare_binprm+0x1e0/0x200
                                sp=e0000001ecdffe30 bsp=e0000001ecdf0d40
 [<a000000100138c50>] do_execve+0x150/0x320
                                sp=e0000001ecdffe30 bsp=e0000001ecdf0cd8
 [<a000000100013760>] sys_execve+0x60/0xc0
                                sp=e0000001ecdffe30 bsp=e0000001ecdf0ca0
 [<a000000100009b90>] ia64_execve+0x30/0x140
                                sp=e0000001ecdffe30 bsp=e0000001ecdf0c50
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001ecdffe30 bsp=e0000001ecdf0c50
 [<a000000000010740>] __kernel_syscall_via_break+0x0/0x20
                                sp=e0000001ece00000 bsp=e0000001ecdf0c50
INFO: task gcc:9156 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001ee6cfbf0 bsp=e0000001ee6c11e0
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001ee6cfc20 bsp=e0000001ee6c1168
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001ee6cfca0 bsp=e0000001ee6c1130
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001ee6cfca0 bsp=e0000001ee6c10f8
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001ee6cfca0 bsp=e0000001ee6c10c0
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001ee6cfca0 bsp=e0000001ee6c1098
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001ee6cfcc0 bsp=e0000001ee6c1068
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001ee6cfcc0 bsp=e0000001ee6c1018
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001ee6cfcc0 bsp=e0000001ee6c0fe8
 [<a0000001000d6c20>] do_generic_mapping_read+0x780/0x7a0
                                sp=e0000001ee6cfce0 bsp=e0000001ee6c0f48
 [<a0000001000db250>] generic_file_aio_read+0x290/0x340
                                sp=e0000001ee6cfce0 bsp=e0000001ee6c0ee8
 [<a00000010012c990>] do_sync_read+0x170/0x200
                                sp=e0000001ee6cfd10 bsp=e0000001ee6c0ea8
 [<a00000010012cbd0>] vfs_read+0x1b0/0x2e0
                                sp=e0000001ee6cfe20 bsp=e0000001ee6c0e58
 [<a000000100136800>] kernel_read+0x60/0xa0
                                sp=e0000001ee6cfe20 bsp=e0000001ee6c0e18
 [<a000000100138140>] prepare_binprm+0x1e0/0x200
                                sp=e0000001ee6cfe30 bsp=e0000001ee6c0de8
 [<a000000100138c50>] do_execve+0x150/0x320
                                sp=e0000001ee6cfe30 bsp=e0000001ee6c0d80
 [<a000000100013760>] sys_execve+0x60/0xc0
                                sp=e0000001ee6cfe30 bsp=e0000001ee6c0d48
 [<a000000100009b90>] ia64_execve+0x30/0x140
                                sp=e0000001ee6cfe30 bsp=e0000001ee6c0cf8
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001ee6cfe30 bsp=e0000001ee6c0cf8
 [<a000000000010740>] __kernel_syscall_via_break+0x0/0x20
                                sp=e0000001ee6d0000 bsp=e0000001ee6c0cf8
INFO: task sh:9157 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001ee41fbf0 bsp=e0000001ee411138
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001ee41fc20 bsp=e0000001ee4110c0
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001ee41fca0 bsp=e0000001ee411088
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001ee41fca0 bsp=e0000001ee411050
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001ee41fca0 bsp=e0000001ee411018
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001ee41fca0 bsp=e0000001ee410fe8
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001ee41fcc0 bsp=e0000001ee410fb8
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001ee41fcc0 bsp=e0000001ee410f68
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001ee41fcc0 bsp=e0000001ee410f40
 [<a0000001000d6c20>] do_generic_mapping_read+0x780/0x7a0
                                sp=e0000001ee41fce0 bsp=e0000001ee410ea0
 [<a0000001000db250>] generic_file_aio_read+0x290/0x340
                                sp=e0000001ee41fce0 bsp=e0000001ee410e40
 [<a00000010012c990>] do_sync_read+0x170/0x200
                                sp=e0000001ee41fd10 bsp=e0000001ee410e00
 [<a00000010012cbd0>] vfs_read+0x1b0/0x2e0
                                sp=e0000001ee41fe20 bsp=e0000001ee410da8
 [<a000000100136800>] kernel_read+0x60/0xa0
                                sp=e0000001ee41fe20 bsp=e0000001ee410d68
 [<a000000100138140>] prepare_binprm+0x1e0/0x200
                                sp=e0000001ee41fe30 bsp=e0000001ee410d40
 [<a000000100138c50>] do_execve+0x150/0x320
                                sp=e0000001ee41fe30 bsp=e0000001ee410cd8
 [<a000000100013760>] sys_execve+0x60/0xc0
                                sp=e0000001ee41fe30 bsp=e0000001ee410ca0
 [<a000000100009b90>] ia64_execve+0x30/0x140
                                sp=e0000001ee41fe30 bsp=e0000001ee410c50
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001ee41fe30 bsp=e0000001ee410c50
 [<a000000000010740>] __kernel_syscall_via_break+0x0/0x20
                                sp=e0000001ee420000 bsp=e0000001ee410c50
INFO: task sh:9158 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001ee38fbf0 bsp=e0000001ee381138
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001ee38fc20 bsp=e0000001ee3810c0
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001ee38fca0 bsp=e0000001ee381088
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001ee38fca0 bsp=e0000001ee381050
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001ee38fca0 bsp=e0000001ee381018
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001ee38fca0 bsp=e0000001ee380fe8
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001ee38fcc0 bsp=e0000001ee380fb8
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001ee38fcc0 bsp=e0000001ee380f68
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001ee38fcc0 bsp=e0000001ee380f40
 [<a0000001000d6c20>] do_generic_mapping_read+0x780/0x7a0
                                sp=e0000001ee38fce0 bsp=e0000001ee380ea0
 [<a0000001000db250>] generic_file_aio_read+0x290/0x340
                                sp=e0000001ee38fce0 bsp=e0000001ee380e40
 [<a00000010012c990>] do_sync_read+0x170/0x200
                                sp=e0000001ee38fd10 bsp=e0000001ee380e00
 [<a00000010012cbd0>] vfs_read+0x1b0/0x2e0
                                sp=e0000001ee38fe20 bsp=e0000001ee380da8
 [<a000000100136800>] kernel_read+0x60/0xa0
                                sp=e0000001ee38fe20 bsp=e0000001ee380d68
 [<a000000100138140>] prepare_binprm+0x1e0/0x200
                                sp=e0000001ee38fe30 bsp=e0000001ee380d40
 [<a000000100138c50>] do_execve+0x150/0x320
                                sp=e0000001ee38fe30 bsp=e0000001ee380cd8
 [<a000000100013760>] sys_execve+0x60/0xc0
                                sp=e0000001ee38fe30 bsp=e0000001ee380ca0
 [<a000000100009b90>] ia64_execve+0x30/0x140
                                sp=e0000001ee38fe30 bsp=e0000001ee380c50
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001ee38fe30 bsp=e0000001ee380c50
 [<a000000000010740>] __kernel_syscall_via_break+0x0/0x20
                                sp=e0000001ee390000 bsp=e0000001ee380c50
INFO: task sh:9159 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001ef04fbc0 bsp=e0000001ef041298
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001ef04fbf0 bsp=e0000001ef041220
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001ef04fc70 bsp=e0000001ef0411e0
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001ef04fc70 bsp=e0000001ef0411a8
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001ef04fc70 bsp=e0000001ef041170
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001ef04fc70 bsp=e0000001ef041148
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001ef04fc90 bsp=e0000001ef041118
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001ef04fc90 bsp=e0000001ef0410c8
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001ef04fc90 bsp=e0000001ef0410a0
 [<a00000010013ffa0>] __link_path_walk+0x16c0/0x21a0
                                sp=e0000001ef04fcb0 bsp=e0000001ef041020
 [<a000000100140b40>] link_path_walk+0xc0/0x260
                                sp=e0000001ef04fcd0 bsp=e0000001ef040fe0
 [<a00000010013e570>] path_walk+0x30/0x60
                                sp=e0000001ef04fd60 bsp=e0000001ef040fb8
 [<a000000100141200>] do_path_lookup+0x520/0x580
                                sp=e0000001ef04fd60 bsp=e0000001ef040f68
 [<a0000001001415e0>] __path_lookup_intent_open+0x80/0x120
                                sp=e0000001ef04fd70 bsp=e0000001ef040f18
 [<a0000001001416c0>] path_lookup_open+0x40/0x60
                                sp=e0000001ef04fd70 bsp=e0000001ef040ed8
 [<a000000100136640>] open_exec+0x40/0x1a0
                                sp=e0000001ef04fd70 bsp=e0000001ef040eb8
 [<a000000100198c90>] load_elf_binary+0x530/0x2700
                                sp=e0000001ef04fe00 bsp=e0000001ef040da0
 [<a000000100138710>] search_binary_handler+0x1b0/0x5a0
                                sp=e0000001ef04fe30 bsp=e0000001ef040d40
 [<a000000100138d40>] do_execve+0x240/0x320
                                sp=e0000001ef04fe30 bsp=e0000001ef040cd8
 [<a000000100013760>] sys_execve+0x60/0xc0
                                sp=e0000001ef04fe30 bsp=e0000001ef040ca0
 [<a000000100009b90>] ia64_execve+0x30/0x140
                                sp=e0000001ef04fe30 bsp=e0000001ef040c50
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001ef04fe30 bsp=e0000001ef040c50
 [<a000000000010740>] __kernel_syscall_via_break+0x0/0x20
                                sp=e0000001ef050000 bsp=e0000001ef040c50
INFO: task sh:9160 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001ea03fbc0 bsp=e0000001ea031288
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001ea03fbf0 bsp=e0000001ea031210
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001ea03fc70 bsp=e0000001ea0311d0
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001ea03fc70 bsp=e0000001ea031198
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001ea03fc70 bsp=e0000001ea031160
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001ea03fc70 bsp=e0000001ea031138
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001ea03fc90 bsp=e0000001ea031108
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001ea03fc90 bsp=e0000001ea0310b8
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001ea03fc90 bsp=e0000001ea031090
 [<a0000001000d6c20>] do_generic_mapping_read+0x780/0x7a0
                                sp=e0000001ea03fcb0 bsp=e0000001ea030fe8
 [<a0000001000db250>] generic_file_aio_read+0x290/0x340
                                sp=e0000001ea03fcb0 bsp=e0000001ea030f88
 [<a00000010012c990>] do_sync_read+0x170/0x200
                                sp=e0000001ea03fce0 bsp=e0000001ea030f48
 [<a00000010012cbd0>] vfs_read+0x1b0/0x2e0
                                sp=e0000001ea03fdf0 bsp=e0000001ea030ef8
 [<a000000100136800>] kernel_read+0x60/0xa0
                                sp=e0000001ea03fdf0 bsp=e0000001ea030eb8
 [<a000000100198d00>] load_elf_binary+0x5a0/0x2700
                                sp=e0000001ea03fe00 bsp=e0000001ea030da0
 [<a000000100138710>] search_binary_handler+0x1b0/0x5a0
                                sp=e0000001ea03fe30 bsp=e0000001ea030d40
 [<a000000100138d40>] do_execve+0x240/0x320
                                sp=e0000001ea03fe30 bsp=e0000001ea030cd8
 [<a000000100013760>] sys_execve+0x60/0xc0
                                sp=e0000001ea03fe30 bsp=e0000001ea030ca0
 [<a000000100009b90>] ia64_execve+0x30/0x140
                                sp=e0000001ea03fe30 bsp=e0000001ea030c50
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001ea03fe30 bsp=e0000001ea030c50
 [<a000000000010740>] __kernel_syscall_via_break+0x0/0x20
                                sp=e0000001ea040000 bsp=e0000001ea030c50
INFO: task objdump:9161 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001ee78fd10 bsp=e0000001ee781240
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001ee78fd40 bsp=e0000001ee7811c0
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001ee78fdc0 bsp=e0000001ee781188
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001ee78fdc0 bsp=e0000001ee781150
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001ee78fdc0 bsp=e0000001ee781118
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001ee78fdc0 bsp=e0000001ee7810f0
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001ee78fde0 bsp=e0000001ee7810c0
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001ee78fde0 bsp=e0000001ee781070
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001ee78fde0 bsp=e0000001ee781048
 [<a0000001000d79b0>] generic_file_mmap+0x90/0x100
                                sp=e0000001ee78fe00 bsp=e0000001ee781020
 [<a000000100101dd0>] mmap_region+0x3b0/0x7a0
                                sp=e0000001ee78fe00 bsp=e0000001ee780f68
 [<a000000100102820>] do_mmap_pgoff+0x660/0x680
                                sp=e0000001ee78fe30 bsp=e0000001ee780ef8
 [<a000000100033000>] sys_mmap+0x1a0/0x260
                                sp=e0000001ee78fe30 bsp=e0000001ee780e70
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001ee78fe30 bsp=e0000001ee780e70
INFO: task grep:9162 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001eb39fbe0 bsp=e0000001eb3912a8
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001eb39fc10 bsp=e0000001eb391230
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001eb39fc90 bsp=e0000001eb3911f0
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001eb39fc90 bsp=e0000001eb3911b8
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001eb39fc90 bsp=e0000001eb391180
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001eb39fc90 bsp=e0000001eb391158
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001eb39fcb0 bsp=e0000001eb391128
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001eb39fcb0 bsp=e0000001eb3910d8
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001eb39fcb0 bsp=e0000001eb3910b0
 [<a00000010013ffa0>] __link_path_walk+0x16c0/0x21a0
                                sp=e0000001eb39fcd0 bsp=e0000001eb391030
 [<a000000100140b40>] link_path_walk+0xc0/0x260
                                sp=e0000001eb39fcf0 bsp=e0000001eb390ff0
 [<a00000010013e570>] path_walk+0x30/0x60
                                sp=e0000001eb39fd80 bsp=e0000001eb390fc8
 [<a000000100141200>] do_path_lookup+0x520/0x580
                                sp=e0000001eb39fd80 bsp=e0000001eb390f78
 [<a0000001001415e0>] __path_lookup_intent_open+0x80/0x120
                                sp=e0000001eb39fd90 bsp=e0000001eb390f28
 [<a0000001001416c0>] path_lookup_open+0x40/0x60
                                sp=e0000001eb39fd90 bsp=e0000001eb390ee8
 [<a000000100142b60>] open_namei+0xc0/0xf40
                                sp=e0000001eb39fd90 bsp=e0000001eb390e48
 [<a00000010012b150>] do_filp_open+0x50/0xc0
                                sp=e0000001eb39fda0 bsp=e0000001eb390e10
 [<a00000010012b9b0>] do_sys_open+0x90/0x1c0
                                sp=e0000001eb39fe30 bsp=e0000001eb390db8
 [<a00000010012bb30>] sys_open+0x50/0x80
                                sp=e0000001eb39fe30 bsp=e0000001eb390d60
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001eb39fe30 bsp=e0000001eb390d60
INFO: task objdump:9163 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001e99bfbf0 bsp=e0000001e99b1168
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001e99bfc20 bsp=e0000001e99b10f0
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001e99bfca0 bsp=e0000001e99b10b8
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001e99bfca0 bsp=e0000001e99b1080
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001e99bfca0 bsp=e0000001e99b1048
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001e99bfca0 bsp=e0000001e99b1020
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001e99bfcc0 bsp=e0000001e99b0fe8
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001e99bfcc0 bsp=e0000001e99b0f98
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001e99bfcc0 bsp=e0000001e99b0f70
 [<a0000001000d6c20>] do_generic_mapping_read+0x780/0x7a0
                                sp=e0000001e99bfce0 bsp=e0000001e99b0ed0
 [<a0000001000db250>] generic_file_aio_read+0x290/0x340
                                sp=e0000001e99bfce0 bsp=e0000001e99b0e70
 [<a00000010012c990>] do_sync_read+0x170/0x200
                                sp=e0000001e99bfd10 bsp=e0000001e99b0e30
 [<a00000010012cbd0>] vfs_read+0x1b0/0x2e0
                                sp=e0000001e99bfe20 bsp=e0000001e99b0dd8
 [<a00000010012d250>] sys_read+0x70/0xe0
                                sp=e0000001e99bfe20 bsp=e0000001e99b0d60
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001e99bfe30 bsp=e0000001e99b0d60
INFO: task grep:9164 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001ed63fbf0 bsp=e0000001ed631168
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001ed63fc20 bsp=e0000001ed6310f0
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001ed63fca0 bsp=e0000001ed6310b8
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001ed63fca0 bsp=e0000001ed631080
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001ed63fca0 bsp=e0000001ed631048
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001ed63fca0 bsp=e0000001ed631020
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001ed63fcc0 bsp=e0000001ed630fe8
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001ed63fcc0 bsp=e0000001ed630f98
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001ed63fcc0 bsp=e0000001ed630f70
 [<a0000001000d6c20>] do_generic_mapping_read+0x780/0x7a0
                                sp=e0000001ed63fce0 bsp=e0000001ed630ed0
 [<a0000001000db250>] generic_file_aio_read+0x290/0x340
                                sp=e0000001ed63fce0 bsp=e0000001ed630e70
 [<a00000010012c990>] do_sync_read+0x170/0x200
                                sp=e0000001ed63fd10 bsp=e0000001ed630e30
 [<a00000010012cbd0>] vfs_read+0x1b0/0x2e0
                                sp=e0000001ed63fe20 bsp=e0000001ed630dd8
 [<a00000010012d250>] sys_read+0x70/0xe0
                                sp=e0000001ed63fe20 bsp=e0000001ed630d60
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001ed63fe30 bsp=e0000001ed630d60
INFO: task objdump:9165 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001ee18fbf0 bsp=e0000001ee181280
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001ee18fc20 bsp=e0000001ee181208
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001ee18fca0 bsp=e0000001ee1811c8
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001ee18fca0 bsp=e0000001ee181190
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001ee18fca0 bsp=e0000001ee181158
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001ee18fca0 bsp=e0000001ee181130
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001ee18fcc0 bsp=e0000001ee181100
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001ee18fcc0 bsp=e0000001ee1810b0
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001ee18fcc0 bsp=e0000001ee181088
 [<a0000001000d6c20>] do_generic_mapping_read+0x780/0x7a0
                                sp=e0000001ee18fce0 bsp=e0000001ee180fe0
 [<a0000001000db250>] generic_file_aio_read+0x290/0x340
                                sp=e0000001ee18fce0 bsp=e0000001ee180f80
 [<a00000010012c990>] do_sync_read+0x170/0x200
                                sp=e0000001ee18fd10 bsp=e0000001ee180f40
 [<a00000010012cbd0>] vfs_read+0x1b0/0x2e0
                                sp=e0000001ee18fe20 bsp=e0000001ee180ef0
 [<a00000010012d250>] sys_read+0x70/0xe0
                                sp=e0000001ee18fe20 bsp=e0000001ee180e78
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001ee18fe30 bsp=e0000001ee180e78
INFO: task grep:9166 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001eb50fbe0 bsp=e0000001eb5013b8
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001eb50fc10 bsp=e0000001eb501340
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001eb50fc90 bsp=e0000001eb501308
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001eb50fc90 bsp=e0000001eb5012d0
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001eb50fc90 bsp=e0000001eb501298
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001eb50fc90 bsp=e0000001eb501270
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001eb50fcb0 bsp=e0000001eb501240
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001eb50fcb0 bsp=e0000001eb5011e8
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001eb50fcb0 bsp=e0000001eb5011c0
 [<a00000010013ffa0>] __link_path_walk+0x16c0/0x21a0
                                sp=e0000001eb50fcd0 bsp=e0000001eb501140
 [<a000000100140b40>] link_path_walk+0xc0/0x260
                                sp=e0000001eb50fcf0 bsp=e0000001eb501108
 [<a00000010013e570>] path_walk+0x30/0x60
                                sp=e0000001eb50fd80 bsp=e0000001eb5010e0
 [<a000000100141200>] do_path_lookup+0x520/0x580
                                sp=e0000001eb50fd80 bsp=e0000001eb501090
 [<a0000001001415e0>] __path_lookup_intent_open+0x80/0x120
                                sp=e0000001eb50fd90 bsp=e0000001eb501040
 [<a0000001001416c0>] path_lookup_open+0x40/0x60
                                sp=e0000001eb50fd90 bsp=e0000001eb501000
 [<a000000100142b60>] open_namei+0xc0/0xf40
                                sp=e0000001eb50fd90 bsp=e0000001eb500f58
 [<a00000010012b150>] do_filp_open+0x50/0xc0
                                sp=e0000001eb50fda0 bsp=e0000001eb500f20
 [<a00000010012b9b0>] do_sys_open+0x90/0x1c0
                                sp=e0000001eb50fe30 bsp=e0000001eb500ed0
 [<a00000010012bb30>] sys_open+0x50/0x80
                                sp=e0000001eb50fe30 bsp=e0000001eb500e78
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001eb50fe30 bsp=e0000001eb500e78
INFO: task objdump:9167 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001b7c7fbe0 bsp=e0000001b7c713b8
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001b7c7fc10 bsp=e0000001b7c71340
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001b7c7fc90 bsp=e0000001b7c71308
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001b7c7fc90 bsp=e0000001b7c712d0
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001b7c7fc90 bsp=e0000001b7c71298
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001b7c7fc90 bsp=e0000001b7c71270
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001b7c7fcb0 bsp=e0000001b7c71240
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001b7c7fcb0 bsp=e0000001b7c711e8
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001b7c7fcb0 bsp=e0000001b7c711c0
 [<a00000010013ffa0>] __link_path_walk+0x16c0/0x21a0
                                sp=e0000001b7c7fcd0 bsp=e0000001b7c71140
 [<a000000100140b40>] link_path_walk+0xc0/0x260
                                sp=e0000001b7c7fcf0 bsp=e0000001b7c71108
 [<a00000010013e570>] path_walk+0x30/0x60
                                sp=e0000001b7c7fd80 bsp=e0000001b7c710e0
 [<a000000100141200>] do_path_lookup+0x520/0x580
                                sp=e0000001b7c7fd80 bsp=e0000001b7c71090
 [<a0000001001415e0>] __path_lookup_intent_open+0x80/0x120
                                sp=e0000001b7c7fd90 bsp=e0000001b7c71040
 [<a0000001001416c0>] path_lookup_open+0x40/0x60
                                sp=e0000001b7c7fd90 bsp=e0000001b7c71000
 [<a000000100142b60>] open_namei+0xc0/0xf40
                                sp=e0000001b7c7fd90 bsp=e0000001b7c70f58
 [<a00000010012b150>] do_filp_open+0x50/0xc0
                                sp=e0000001b7c7fda0 bsp=e0000001b7c70f20
 [<a00000010012b9b0>] do_sys_open+0x90/0x1c0
                                sp=e0000001b7c7fe30 bsp=e0000001b7c70ed0
 [<a00000010012bb30>] sys_open+0x50/0x80
                                sp=e0000001b7c7fe30 bsp=e0000001b7c70e78
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001b7c7fe30 bsp=e0000001b7c70e78
INFO: task grep:9168 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Call Trace:
 [<a000000100704120>] schedule+0x11c0/0x1340
                                sp=e0000001ed8afbf0 bsp=e0000001ed8a1280
 [<a00000010024e720>] do_get_write_access+0x660/0xbe0
                                sp=e0000001ed8afc20 bsp=e0000001ed8a1208
 [<a00000010024f060>] journal_get_write_access+0x40/0x80
                                sp=e0000001ed8afca0 bsp=e0000001ed8a11c8
 [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
                                sp=e0000001ed8afca0 bsp=e0000001ed8a1190
 [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
                                sp=e0000001ed8afca0 bsp=e0000001ed8a1158
 [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
                                sp=e0000001ed8afca0 bsp=e0000001ed8a1130
 [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
                                sp=e0000001ed8afcc0 bsp=e0000001ed8a1100
 [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
                                sp=e0000001ed8afcc0 bsp=e0000001ed8a10b0
 [<a00000010015b570>] touch_atime+0x310/0x340
                                sp=e0000001ed8afcc0 bsp=e0000001ed8a1088
 [<a0000001000d6c20>] do_generic_mapping_read+0x780/0x7a0
                                sp=e0000001ed8afce0 bsp=e0000001ed8a0fe0
 [<a0000001000db250>] generic_file_aio_read+0x290/0x340
                                sp=e0000001ed8afce0 bsp=e0000001ed8a0f80
 [<a00000010012c990>] do_sync_read+0x170/0x200
                                sp=e0000001ed8afd10 bsp=e0000001ed8a0f40
 [<a00000010012cbd0>] vfs_read+0x1b0/0x2e0
                                sp=e0000001ed8afe20 bsp=e0000001ed8a0ef0
 [<a00000010012d250>] sys_read+0x70/0xe0
                                sp=e0000001ed8afe20 bsp=e0000001ed8a0e78
 [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000001ed8afe30 bsp=e0000001ed8a0e78

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: system hang on latest git
  2008-01-29 19:15 system hang on latest git Luck, Tony
@ 2008-01-29 20:11 ` Jens Axboe
  2008-01-29 20:16   ` Jens Axboe
  0 siblings, 1 reply; 6+ messages in thread
From: Jens Axboe @ 2008-01-29 20:11 UTC (permalink / raw)
  To: Luck, Tony; +Cc: LKML, linux-ia64

On Tue, Jan 29 2008, Luck, Tony wrote:
> I pulled Linus' tree this morning (git head = 0ba6c33bcddc64a54b5f1c25a696c4767dc76292)
> and built for ia64 (using arch/ia64/configs/tiger_defconfig).   System booted
> OK, but when I stressed it a little (building another kernel with "make -j32")
> it hung.
> 
> The console has a bunch (98) of warnings about tasks blocked for more than 120
> seconds like this:
> INFO: task grep:9168 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> 
> Call Trace:
>  [<a000000100704120>] schedule+0x11c0/0x1340
>                                 sp=e0000001ed8afbf0 bsp=e0000001ed8a1280
>  [<a00000010024e720>] do_get_write_access+0x660/0xbe0
>                                 sp=e0000001ed8afc20 bsp=e0000001ed8a1208
>  [<a00000010024f060>] journal_get_write_access+0x40/0x80
>                                 sp=e0000001ed8afca0 bsp=e0000001ed8a11c8
>  [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
>                                 sp=e0000001ed8afca0 bsp=e0000001ed8a1190
>  [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
>                                 sp=e0000001ed8afca0 bsp=e0000001ed8a1158
>  [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
>                                 sp=e0000001ed8afca0 bsp=e0000001ed8a1130
>  [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
>                                 sp=e0000001ed8afcc0 bsp=e0000001ed8a1100
>  [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
>                                 sp=e0000001ed8afcc0 bsp=e0000001ed8a10b0
>  [<a00000010015b570>] touch_atime+0x310/0x340
>                                 sp=e0000001ed8afcc0 bsp=e0000001ed8a1088
>  [<a0000001000d6c20>] do_generic_mapping_read+0x780/0x7a0
>                                 sp=e0000001ed8afce0 bsp=e0000001ed8a0fe0
>  [<a0000001000db250>] generic_file_aio_read+0x290/0x340
>                                 sp=e0000001ed8afce0 bsp=e0000001ed8a0f80
>  [<a00000010012c990>] do_sync_read+0x170/0x200
>                                 sp=e0000001ed8afd10 bsp=e0000001ed8a0f40
>  [<a00000010012cbd0>] vfs_read+0x1b0/0x2e0
>                                 sp=e0000001ed8afe20 bsp=e0000001ed8a0ef0
>  [<a00000010012d250>] sys_read+0x70/0xe0
>                                 sp=e0000001ed8afe20 bsp=e0000001ed8a0e78
>  [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
>                                 sp=e0000001ed8afe30 bsp=e0000001ed8a0e78
> 
> 
> [The stack trace has several variations ... some from sys_read(), some from
> sys_open(), some from sys_execve(), some from sys_mmap() etc. 84/98 stack
> traces pass through the touch_atime->__mark_inode_dirty path ... all 98
> are attached]
> 
> A quick dig into processor state shows 8 cpus are idle.  7 are spinning
> in __spin_lock_irq() from __make_request() and one is in spin_lock() from
> as_merged_requests().

Looks like a deadlock on queue lock and ioc lock, but I don't see
immediately what the problem is. I can't stick around for longer
tonight, but I'll get to the bottom of this tomorrow.

-- 
Jens Axboe


^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: system hang on latest git
  2008-01-29 20:11 ` Jens Axboe
@ 2008-01-29 20:16   ` Jens Axboe
  2008-01-29 21:46     ` Olof Johansson
  0 siblings, 1 reply; 6+ messages in thread
From: Jens Axboe @ 2008-01-29 20:16 UTC (permalink / raw)
  To: Luck, Tony; +Cc: LKML, linux-ia64

On Tue, Jan 29 2008, Jens Axboe wrote:
> On Tue, Jan 29 2008, Luck, Tony wrote:
> > I pulled Linus' tree this morning (git head = 0ba6c33bcddc64a54b5f1c25a696c4767dc76292)
> > and built for ia64 (using arch/ia64/configs/tiger_defconfig).   System booted
> > OK, but when I stressed it a little (building another kernel with "make -j32")
> > it hung.
> > 
> > The console has a bunch (98) of warnings about tasks blocked for more than 120
> > seconds like this:
> > INFO: task grep:9168 blocked for more than 120 seconds.
> > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > 
> > Call Trace:
> >  [<a000000100704120>] schedule+0x11c0/0x1340
> >                                 sp=e0000001ed8afbf0 bsp=e0000001ed8a1280
> >  [<a00000010024e720>] do_get_write_access+0x660/0xbe0
> >                                 sp=e0000001ed8afc20 bsp=e0000001ed8a1208
> >  [<a00000010024f060>] journal_get_write_access+0x40/0x80
> >                                 sp=e0000001ed8afca0 bsp=e0000001ed8a11c8
> >  [<a000000100245db0>] __ext3_journal_get_write_access+0x30/0xa0
> >                                 sp=e0000001ed8afca0 bsp=e0000001ed8a1190
> >  [<a00000010022dea0>] ext3_reserve_inode_write+0x80/0x120
> >                                 sp=e0000001ed8afca0 bsp=e0000001ed8a1158
> >  [<a00000010022df70>] ext3_mark_inode_dirty+0x30/0x80
> >                                 sp=e0000001ed8afca0 bsp=e0000001ed8a1130
> >  [<a000000100232530>] ext3_dirty_inode+0xd0/0x120
> >                                 sp=e0000001ed8afcc0 bsp=e0000001ed8a1100
> >  [<a000000100170e20>] __mark_inode_dirty+0xa0/0x3e0
> >                                 sp=e0000001ed8afcc0 bsp=e0000001ed8a10b0
> >  [<a00000010015b570>] touch_atime+0x310/0x340
> >                                 sp=e0000001ed8afcc0 bsp=e0000001ed8a1088
> >  [<a0000001000d6c20>] do_generic_mapping_read+0x780/0x7a0
> >                                 sp=e0000001ed8afce0 bsp=e0000001ed8a0fe0
> >  [<a0000001000db250>] generic_file_aio_read+0x290/0x340
> >                                 sp=e0000001ed8afce0 bsp=e0000001ed8a0f80
> >  [<a00000010012c990>] do_sync_read+0x170/0x200
> >                                 sp=e0000001ed8afd10 bsp=e0000001ed8a0f40
> >  [<a00000010012cbd0>] vfs_read+0x1b0/0x2e0
> >                                 sp=e0000001ed8afe20 bsp=e0000001ed8a0ef0
> >  [<a00000010012d250>] sys_read+0x70/0xe0
> >                                 sp=e0000001ed8afe20 bsp=e0000001ed8a0e78
> >  [<a00000010000a4a0>] ia64_ret_from_syscall+0x0/0x20
> >                                 sp=e0000001ed8afe30 bsp=e0000001ed8a0e78
> > 
> > 
> > [The stack trace has several variations ... some from sys_read(), some from
> > sys_open(), some from sys_execve(), some from sys_mmap() etc. 84/98 stack
> > traces pass through the touch_atime->__mark_inode_dirty path ... all 98
> > are attached]
> > 
> > A quick dig into processor state shows 8 cpus are idle.  7 are spinning
> > in __spin_lock_irq() from __make_request() and one is in spin_lock() from
> > as_merged_requests().
> 
> Looks like a deadlock on queue lock and ioc lock, but I don't see
> immediately what the problem is. I can't stick around for longer
> tonight, but I'll get to the bottom of this tomorrow.

Actually, can you try this? It has a known race but nothing to worry
about, and it removes ioc->lock from irq context.

diff --git a/block/as-iosched.c b/block/as-iosched.c
index b201d16..585aad2 100644
--- a/block/as-iosched.c
+++ b/block/as-iosched.c
@@ -235,10 +235,8 @@ static void as_put_io_context(struct request *rq)
 	aic = RQ_IOC(rq)->aic;
 
 	if (rq_is_sync(rq) && aic) {
-		spin_lock(&aic->lock);
 		set_bit(AS_TASK_IORUNNING, &aic->state);
 		aic->last_end_request = jiffies;
-		spin_unlock(&aic->lock);
 	}
 
 	put_io_context(RQ_IOC(rq));

-- 
Jens Axboe


^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: system hang on latest git
  2008-01-29 21:46     ` Olof Johansson
@ 2008-01-29 21:38       ` Jens Axboe
  2008-01-29 21:56         ` Olof Johansson
  0 siblings, 1 reply; 6+ messages in thread
From: Jens Axboe @ 2008-01-29 21:38 UTC (permalink / raw)
  To: Olof Johansson; +Cc: Luck, Tony, LKML, linux-ia64

On Tue, Jan 29 2008, Olof Johansson wrote:
> On Tue, Jan 29, 2008 at 09:16:48PM +0100, Jens Axboe wrote:
> 
> > Actually, can you try this? It has a known race but nothing to worry
> > about, and it removes ioc->lock from irq context.
> 
> I just tried this myself, since I saw hangs within seconds of running
> 'aiostress' from autotest on this morning's kernel as well (g0ba6c33).
> 
> It didn't help. My config is 2 cores (powerpc), built with
> arch/powerpc/configs/pasemi_defconfig. Hardware is sata disk on marvell
> 7042.

Please try this.

diff --git a/block/as-iosched.c b/block/as-iosched.c
index b201d16..9603684 100644
--- a/block/as-iosched.c
+++ b/block/as-iosched.c
@@ -1275,9 +1275,13 @@ static void as_merged_requests(struct request_queue *q, struct request *req,
 			 * Don't copy here but swap, because when anext is
 			 * removed below, it must contain the unused context
 			 */
-			double_spin_lock(&rioc->lock, &nioc->lock, rioc < nioc);
-			swap_io_context(&rioc, &nioc);
-			double_spin_unlock(&rioc->lock, &nioc->lock, rioc < nioc);
+			if (rioc != nioc) {
+				double_spin_lock(&rioc->lock, &nioc->lock,
+								rioc < nioc);
+				swap_io_context(&rioc, &nioc);
+				double_spin_unlock(&rioc->lock, &nioc->lock,
+								rioc < nioc);
+			}
 		}
 	}
 

> Stacktraces:
> 
> 0:mon> t           
> [link register   ] c000000000221984 .as_merged_requests+0x164/0x1e0
> [c00000007e8ff750] c00000007e8ff800 (unreliable)
> [c00000007e8ff800] c000000000212f20 .elv_merge_requests+0x50/0xb0
> [c00000007e8ff890] c000000000218c98 .attempt_merge+0x318/0x3d0
> [c00000007e8ff940] c00000000021af98 .__make_request+0x2f8/0x720
> [c00000007e8ffa20] c000000000216e6c .generic_make_request+0x22c/0x2f0
> [c00000007e8ffad0] c000000000216fd8 .submit_bio+0xa8/0x150
> [c00000007e8ffb90] c0000000000efa7c .submit_bh+0x15c/0x1e0
> [c00000007e8ffc20] c000000000145d5c .journal_do_submit_data+0x6c/0xb0
> [c00000007e8ffcc0] c000000000147460 .journal_commit_transaction+0x1640/0x16e0
> [c00000007e8ffe10] c00000000014c238 .kjournald+0x108/0x2f0
> [c00000007e8fff00] c000000000067ca8 .kthread+0xc8/0xe0
> [c00000007e8fff90] c0000000000237bc .kernel_thread+0x4c/0x68
> 0:mon> c1
> 1:mon> t
> [link register   ] c00000000033c20c .scsi_device_unbusy+0x8c/0x120
> [c00000007e10b6f0] c00000000033c1a8 .scsi_device_unbusy+0x28/0x120 (unreliable)
> [c00000007e10b780] c000000000333078 .scsi_finish_command+0x38/0x130
> [c00000007e10b810] c00000000033c478 .scsi_softirq_done+0xd8/0x1a0
> [c00000007e10b8b0] c00000000021a780 .blk_done_softirq+0xb0/0xe0
> [c00000007e10b940] c000000000051e58 .__do_softirq+0xe8/0x1d0
> [c00000007e10ba00] c00000000000ac64 .do_softirq+0x64/0xa0
> [c00000007e10ba80] c000000000051b84 .irq_exit+0xc4/0xd0
> [c00000007e10bb00] c00000000000b3c0 .do_IRQ+0xe0/0x120
> [c00000007e10bb80] c00000000000405c hardware_interrupt_entry+0x18/0x3c
> --- Exception: 501 (Hardware Interrupt) at c000000000010564 .cpu_idle+0xb4/0x160
> [c00000007e10be70] c000000000010524 .cpu_idle+0x74/0x160 (unreliable)
> [c00000007e10bf00] c000000000540e6c
> [c00000007e10bf90] c000000000007364 .start_secondary_prolog+0xc/0x10
> 
> 
> scsi_device_busy is sitting on:
> 
>         spin_lock(sdev->request_queue->queue_lock);
> 
> and as_merged_requests on the second lock at:
> 
> 	double_spin_lock(&rioc->lock, &nioc->lock, rioc < nioc);
> 
> 

The latter is the problem, but since the queue lock is also held before
doing the double_spin_lock(), we get into even more trouble. The locking
hierarchy is fine, it's always queue lock -> io context locks, so the
above patch should be all that is needed to fix this.

My initial analysis was wrong, that's all :/

-- 
Jens Axboe


^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: system hang on latest git
  2008-01-29 20:16   ` Jens Axboe
@ 2008-01-29 21:46     ` Olof Johansson
  2008-01-29 21:38       ` Jens Axboe
  0 siblings, 1 reply; 6+ messages in thread
From: Olof Johansson @ 2008-01-29 21:46 UTC (permalink / raw)
  To: Jens Axboe; +Cc: Luck, Tony, LKML, linux-ia64

On Tue, Jan 29, 2008 at 09:16:48PM +0100, Jens Axboe wrote:

> Actually, can you try this? It has a known race but nothing to worry
> about, and it removes ioc->lock from irq context.

I just tried this myself, since I saw hangs within seconds of running
'aiostress' from autotest on this morning's kernel as well (g0ba6c33).

It didn't help. My config is 2 cores (powerpc), built with
arch/powerpc/configs/pasemi_defconfig. Hardware is sata disk on marvell
7042.

Stacktraces:

0:mon> t           
[link register   ] c000000000221984 .as_merged_requests+0x164/0x1e0
[c00000007e8ff750] c00000007e8ff800 (unreliable)
[c00000007e8ff800] c000000000212f20 .elv_merge_requests+0x50/0xb0
[c00000007e8ff890] c000000000218c98 .attempt_merge+0x318/0x3d0
[c00000007e8ff940] c00000000021af98 .__make_request+0x2f8/0x720
[c00000007e8ffa20] c000000000216e6c .generic_make_request+0x22c/0x2f0
[c00000007e8ffad0] c000000000216fd8 .submit_bio+0xa8/0x150
[c00000007e8ffb90] c0000000000efa7c .submit_bh+0x15c/0x1e0
[c00000007e8ffc20] c000000000145d5c .journal_do_submit_data+0x6c/0xb0
[c00000007e8ffcc0] c000000000147460 .journal_commit_transaction+0x1640/0x16e0
[c00000007e8ffe10] c00000000014c238 .kjournald+0x108/0x2f0
[c00000007e8fff00] c000000000067ca8 .kthread+0xc8/0xe0
[c00000007e8fff90] c0000000000237bc .kernel_thread+0x4c/0x68
0:mon> c1
1:mon> t
[link register   ] c00000000033c20c .scsi_device_unbusy+0x8c/0x120
[c00000007e10b6f0] c00000000033c1a8 .scsi_device_unbusy+0x28/0x120 (unreliable)
[c00000007e10b780] c000000000333078 .scsi_finish_command+0x38/0x130
[c00000007e10b810] c00000000033c478 .scsi_softirq_done+0xd8/0x1a0
[c00000007e10b8b0] c00000000021a780 .blk_done_softirq+0xb0/0xe0
[c00000007e10b940] c000000000051e58 .__do_softirq+0xe8/0x1d0
[c00000007e10ba00] c00000000000ac64 .do_softirq+0x64/0xa0
[c00000007e10ba80] c000000000051b84 .irq_exit+0xc4/0xd0
[c00000007e10bb00] c00000000000b3c0 .do_IRQ+0xe0/0x120
[c00000007e10bb80] c00000000000405c hardware_interrupt_entry+0x18/0x3c
--- Exception: 501 (Hardware Interrupt) at c000000000010564 .cpu_idle+0xb4/0x160
[c00000007e10be70] c000000000010524 .cpu_idle+0x74/0x160 (unreliable)
[c00000007e10bf00] c000000000540e6c
[c00000007e10bf90] c000000000007364 .start_secondary_prolog+0xc/0x10


scsi_device_busy is sitting on:

        spin_lock(sdev->request_queue->queue_lock);

and as_merged_requests on the second lock at:

	double_spin_lock(&rioc->lock, &nioc->lock, rioc < nioc);


-Olof

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: system hang on latest git
  2008-01-29 21:38       ` Jens Axboe
@ 2008-01-29 21:56         ` Olof Johansson
  0 siblings, 0 replies; 6+ messages in thread
From: Olof Johansson @ 2008-01-29 21:56 UTC (permalink / raw)
  To: Jens Axboe; +Cc: Luck, Tony, LKML, linux-ia64

On Tue, Jan 29, 2008 at 10:38:52PM +0100, Jens Axboe wrote:
> Please try this.

[...]

> The latter is the problem, but since the queue lock is also held before
> doing the double_spin_lock(), we get into even more trouble. The locking
> hierarchy is fine, it's always queue lock -> io context locks, so the
> above patch should be all that is needed to fix this.
> 
> My initial analysis was wrong, that's all :/

Yep, looks much better now.

Tested-by: Olof Johansson <olof@lixom.net>


-Olof

^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2008-01-29 21:44 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2008-01-29 19:15 system hang on latest git Luck, Tony
2008-01-29 20:11 ` Jens Axboe
2008-01-29 20:16   ` Jens Axboe
2008-01-29 21:46     ` Olof Johansson
2008-01-29 21:38       ` Jens Axboe
2008-01-29 21:56         ` Olof Johansson

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).