LKML Archive on lore.kernel.org
help / color / mirror / Atom feed
* 2.6.24 Page Allocation Failure
@ 2008-01-31 15:49 AndrewL733
  2008-02-01 14:13 ` AndrewL733
  2008-02-01 18:53 ` Mel Gorman
  0 siblings, 2 replies; 3+ messages in thread
From: AndrewL733 @ 2008-01-31 15:49 UTC (permalink / raw)
  To: linux-kernel

My setup is Mandriva 2007 64-bit:

IntelS5000PSLSATA motherboard
Single Dual Core 3 Ghz Xeon
2 GB ECC RAM
Myricom 10 GbE Network Adapter (using out of kernel 1.4.0 driver 
compiled with "MYRI10GE_ALLOC_ORDER=2")


I have two identical systems doing data transfer via Myricom cards 
connected directly to one another (in other words, not going through a 
switch).

No problems hammering Myricom with 350 MB/sec NFS traffic while running 
2.6.20.15 and 2.6.22.9 (kernel.org "plain vanilla" kernels I compiled 
and run, also with updated Myricom drivers). However, I just compiled 
and installed 2.6.24 and get the following errors as soon as I begin 
doing NFS I/O over the Myricom card. Maybe I missed something in the 
kernel configuration? Throughput when running 2.6.24 is about 40 percent 
lower than when running 2.6.22.9-- surely due to these errors.

Jan 29 22:13:36 master kernel: nfsd: page allocation failure. order:2, 
mode:0x4020
Jan 29 22:13:36 master kernel: Pid: 6586, comm: nfsd Not tainted 2.6.24 #1
Jan 29 22:13:36 master kernel:
Jan 29 22:13:36 master kernel: Call Trace:
Jan 29 22:13:36 master kernel:  <IRQ>  [__alloc_pages+822/912] 
__alloc_pages+0x336/0x390
Jan 29 22:13:36 master kernel:  <IRQ>  [<ffffffff80279d66>] 
__alloc_pages+0x336/0x390
Jan 29 22:13:36 master kernel:  [ip_local_deliver+37/112] 
ip_local_deliver+0x25/0x70
Jan 29 22:13:36 master kernel:  [<ffffffff80417bb5>] 
ip_local_deliver+0x25/0x70
Jan 29 22:13:36 master kernel:  [_end+130301422/2130457992] 
:myri10ge:myri10ge_alloc_rx_pages+0x156/0x270
Jan 29 22:13:36 master kernel:  [<ffffffff88280866>] 
:myri10ge:myri10ge_alloc_rx_pages+0x156/0x270
Jan 29 22:13:36 master kernel:  [_end+130325174/2130457992] 
:myri10ge:myri10ge_poll+0x57e/0xae0
Jan 29 22:13:36 master kernel:  [<ffffffff8828652e>] 
:myri10ge:myri10ge_poll+0x57e/0xae0
Jan 29 22:13:36 master kernel:  [_spin_lock_bh+9/32] _spin_lock_bh+0x9/0x20
Jan 29 22:13:36 master kernel:  [<ffffffff80466819>] _spin_lock_bh+0x9/0x20
Jan 29 22:13:36 master kernel:  [_end+129719928/2130457992] 
:sunrpc:svc_sock_enqueue+0x80/0x340
Jan 29 22:13:36 master kernel:  [<ffffffff881f28f0>] 
:sunrpc:svc_sock_enqueue+0x80/0x340
Jan 29 22:13:36 master kernel:  [net_rx_action+153/448] 
net_rx_action+0x99/0x1c0
Jan 29 22:13:36 master kernel:  [<ffffffff803f6729>] 
net_rx_action+0x99/0x1c0
Jan 29 22:13:36 master kernel:  [__do_softirq+105/224] 
__do_softirq+0x69/0xe0
Jan 29 22:13:36 master kernel:  [<ffffffff80241db9>] __do_softirq+0x69/0xe0
Jan 29 22:13:36 master kernel:  [call_softirq+28/48] call_softirq+0x1c/0x30
Jan 29 22:13:36 master kernel:  [<ffffffff8020cd9c>] call_softirq+0x1c/0x30
Jan 29 22:13:36 master kernel:  <EOI>  [do_softirq+53/144] 
do_softirq+0x35/0x90
Jan 29 22:13:36 master kernel:  <EOI>  [<ffffffff8020eef5>] 
do_softirq+0x35/0x90
Jan 29 22:13:36 master kernel:  [local_bh_enable+91/160] 
local_bh_enable+0x5b/0xa0
Jan 29 22:13:36 master kernel:  [<ffffffff80241c6b>] 
local_bh_enable+0x5b/0xa0
Jan 29 22:13:36 master kernel:  [_end+129726255/2130457992] 
:sunrpc:svc_udp_recvfrom+0x407/0x430
Jan 29 22:13:36 master kernel:  [<ffffffff881f41a7>] 
:sunrpc:svc_udp_recvfrom+0x407/0x430
Jan 29 22:13:36 master kernel:  [_end+129730321/2130457992] 
:sunrpc:svc_recv+0x2a9/0x4e0
Jan 29 22:13:36 master kernel:  [<ffffffff881f5189>] 
:sunrpc:svc_recv+0x2a9/0x4e0
Jan 29 22:13:36 master kernel:  [default_wake_function+0/16] 
default_wake_function+0x0/0x10
Jan 29 22:13:36 master kernel:  [<ffffffff80233460>] 
default_wake_function+0x0/0x10
Jan 29 22:13:36 master kernel:  [__down_read+18/177] __down_read+0x12/0xb1
Jan 29 22:13:36 master kernel:  [<ffffffff80466242>] __down_read+0x12/0xb1
Jan 29 22:13:36 master kernel:  [_end+130772360/2130457992] 
:nfsd:nfsd+0x0/0x2e0
an 29 22:13:36 master kernel:  [<ffffffff882f3800>] :nfsd:nfsd+0x0/0x2e0
Jan 29 22:13:36 master kernel:  [_end+130772586/2130457992] 
:nfsd:nfsd+0xe2/0x2e0
Jan 29 22:13:36 master kernel:  [<ffffffff882f38e2>] :nfsd:nfsd+0xe2/0x2e0
Jan 29 22:13:36 master kernel:  [child_rip+10/18] child_rip+0xa/0x12
Jan 29 22:13:36 master kernel:  [<ffffffff8020ca28>] child_rip+0xa/0x12
Jan 29 22:13:36 master kernel:  [_end+130772360/2130457992] 
:nfsd:nfsd+0x0/0x2e0
Jan 29 22:13:36 master kernel:  [<ffffffff882f3800>] :nfsd:nfsd+0x0/0x2e0
Jan 29 22:13:36 master kernel:  [_end+130772360/2130457992] 
:nfsd:nfsd+0x0/0x2e0
Jan 29 22:13:36 master kernel:  [<ffffffff882f3800>] :nfsd:nfsd+0x0/0x2e0
Jan 29 22:13:36 master kernel:  [child_rip+0/18] child_rip+0x0/0x12
Jan 29 22:13:36 master kernel:  [<ffffffff8020ca1e>] child_rip+0x0/0x12
Jan 29 22:13:36 master kernel:
Jan 29 22:13:36 master kernel: Mem-info:
Jan 29 22:13:36 master kernel: DMA per-cpu:
Jan 29 22:13:36 master kernel: CPU    0: Hot: hi:    0, btch:   1 usd:   
0   Cold: hi:    0, btch:   1 usd:   0
Jan 29 22:13:36 master kernel: CPU    1: Hot: hi:    0, btch:   1 usd:   
0   Cold: hi:    0, btch:   1 usd:   0
Jan 29 22:13:36 master kernel: DMA32 per-cpu:
Jan 29 22:13:36 master kernel: CPU    0: Hot: hi:  186, btch:  31 usd: 
150   Cold: hi:   62, btch:  15 usd:  62
Jan 29 22:13:36 master kernel: CPU    1: Hot: hi:  186, btch:  31 usd: 
105   Cold: hi:   62, btch:  15 usd:  52
Jan 29 22:13:36 master kernel: Active:41027 inactive:437314 dirty:29708 
writeback:0 unstable:0
Jan 29 22:13:36 master kernel:  free:3892 slab:19405 mapped:11793 
pagetables:1512 bounce:0
Jan 29 22:13:36 master kernel: DMA free:8008kB min:28kB low:32kB 
high:40kB active:0kB inactive:2384kB present:11100kB pages_scanned:0 
all_unreclaimable? no
Jan 29 22:13:36 master kernel: lowmem_reserve[]: 0 1998 1998 1998
Jan 29 22:13:36 master kernel: DMA32 free:7560kB min:5704kB low:7128kB 
high:8556kB active:164108kB inactive:1746872kB present:2046800kB 
pages_scanned:32 all_unreclaimab
le? no
Jan 29 22:13:36 master kernel: lowmem_reserve[]: 0 0 0 0
Jan 29 22:13:36 master kernel: DMA: 4*4kB 5*8kB 6*16kB 6*32kB 4*64kB 
2*128kB 4*256kB 4*512kB 0*1024kB 0*2048kB 1*4096kB = 8024kB
Jan 29 22:13:36 master kernel: DMA32: 1365*4kB 11*8kB 38*16kB 18*32kB 
0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 7756kB
Jan 29 22:13:36 master kernel: Swap cache: add 43, delete 43, find 0/0, 
race 0+0
Jan 29 22:13:36 master kernel: Free swap  = 4088332kB
Jan 29 22:13:36 master kernel: Total swap = 4088500kB
Jan 29 22:13:36 master kernel: Free swap:       4088332kB
Jan 29 22:13:36 master kernel: 523264 pages of RAM
Jan 29 22:13:36 master kernel: 9510 reserved pages
Jan 29 22:13:36 master kernel: 492885 pages shared
Jan 29 22:13:36 master kernel: 0 pages swap cached
Jan 29 22:13:44 master nmbd[3438]: [2008/01/29 22:13:44, 0] 
nmbd/nmbd_packets.c:process_browse_packet(1061)
Jan 29 22:13:44 master nmbd[3438]:   process_browse_packet: Discarding 
datagram from IP 192.168.1.240. Source name master<00> is one of our names !
Jan 29 22:13:44 master nmbd[3438]: [2008/01/29 22:13:44, 0] 
nmbd/nmbd_packets.c:process_browse_packet(1061)
Jan 29 22:13:44 master nmbd[3438]:   process_browse_packet: Discarding 
datagram from IP 192.168.1.237. Source name master<00> is one of our names !
Jan 29 22:13:44 master kernel: nfsd: page allocation failure. order:2, 
mode:0x4020
Jan 29 22:13:44 master kernel: Pid: 6591, comm: nfsd Not tainted 2.6.24 #1
Jan 29 22:13:44 master kernel:
Jan 29 22:13:44 master kernel: Call Trace:
Jan 29 22:13:44 master kernel:  <IRQ>  [__alloc_pages+822/912] 
__alloc_pages+0x336/0x390
Jan 29 22:13:44 master kernel:  <IRQ>  [<ffffffff80279d66>] 
__alloc_pages+0x336/0x390
Jan 29 22:13:44 master kernel:  [_end+130301422/2130457992] 
:myri10ge:myri10ge_alloc_rx_pages+0x156/0x270
Jan 29 22:13:44 master kernel:  [<ffffffff88280866>] 
:myri10ge:myri10ge_alloc_rx_pages+0x156/0x270
Jan 29 22:13:44 master kernel:  [_end+130325174/2130457992] 
:myri10ge:myri10ge_poll+0x57e/0xae0
Jan 29 22:13:44 master kernel:  [<ffffffff8828652e>] 
:myri10ge:myri10ge_poll+0x57e/0xae0
Jan 29 22:13:44 master kernel:  [_spin_lock_bh+9/32] _spin_lock_bh+0x9/0x20
Jan 29 22:13:44 master kernel:  [<ffffffff80466819>] _spin_lock_bh+0x9/0x20
Jan 29 22:13:44 master kernel:  [_end+129719928/2130457992] 
:sunrpc:svc_sock_enqueue+0x80/0x340
Jan 29 22:13:44 master kernel:  [<ffffffff881f28f0>] 
:sunrpc:svc_sock_enqueue+0x80/0x340
Jan 29 22:13:44 master kernel:  [net_rx_action+153/448] 
net_rx_action+0x99/0x1c0
Jan 29 22:13:44 master kernel:  [<ffffffff803f6729>] 
net_rx_action+0x99/0x1c0
Jan 29 22:13:44 master kernel:  [__do_softirq+105/224] 
__do_softirq+0x69/0xe0
Jan 29 22:13:44 master kernel:  [<ffffffff80241db9>] __do_softirq+0x69/0xe0
Jan 29 22:13:44 master kernel:  [call_softirq+28/48] call_softirq+0x1c/0x30
Jan 29 22:13:44 master kernel:  [<ffffffff8020cd9c>] call_softirq+0x1c/0x30
Jan 29 22:13:44 master kernel:  [do_softirq+53/144] do_softirq+0x35/0x90
Jan 29 22:13:44 master kernel:  [<ffffffff8020eef5>] do_softirq+0x35/0x90
Jan 29 22:13:44 master kernel:  [do_IRQ+128/256] do_IRQ+0x80/0x100
Jan 29 22:13:44 master kernel:  [<ffffffff8020f170>] do_IRQ+0x80/0x100
Jan 29 22:13:44 master kernel:  [ret_from_intr+0/10] ret_from_intr+0x0/0xa
Jan 29 22:13:44 master kernel:  [<ffffffff8020c121>] ret_from_intr+0x0/0xa
Jan 29 22:13:44 master kernel:  <EOI>  [__copy_user_nocache+35/336] 
__copy_user_nocache+0x23/0x150
Jan 29 22:13:44 master kernel:  <EOI>  [<ffffffff803233b3>] 
__copy_user_nocache+0x23/0x150
Jan 29 22:13:44 master kernel:  [__iovec_copy_from_user_inatomic+67/128] 
__iovec_copy_from_user_inatomic+0x43/0x80
Jan 29 22:13:44 master kernel:  [<ffffffff80273ba3>] 
__iovec_copy_from_user_inatomic+0x43/0x80
Jan 29 22:13:44 master kernel:  [iov_iter_copy_from_user_atomic+163/176] 
iov_iter_copy_from_user_atomic+0xa3/0xb0
Jan 29 22:13:44 master kernel:  [<ffffffff80273d03>] 
iov_iter_copy_from_user_atomic+0xa3/0xb0
Jan 29 22:13:44 master kernel:  [generic_file_buffered_write+383/1728] 
generic_file_buffered_write+0x17f/0x6c0
Jan 29 22:13:44 master kernel:  [<ffffffff802756ff>] 
generic_file_buffered_write+0x17f/0x6c0
Jan 29 22:13:44 master kernel:  [__alloc_skb+133/336] __alloc_skb+0x85/0x150
Jan 29 22:13:44 master kernel:  [<ffffffff803eea15>] __alloc_skb+0x85/0x150
Jan 29 22:13:44 master kernel:  [_end+129525959/2130457992] 
:xfs:xfs_write+0x67f/0x8f0
Jan 29 22:13:44 master kernel:  [<ffffffff881c333f>] 
:xfs:xfs_write+0x67f/0x8f0
Jan 29 22:13:44 master kernel:  [_end+129343578/2130457992] 
:xfs:xfs_iget+0x142/0x160
Jan 29 22:13:44 master kernel:  [<ffffffff88196ad2>] 
:xfs:xfs_iget+0x142/0x160
Jan 29 22:13:44 master kernel:  [_end+129509416/2130457992] 
:xfs:xfs_file_aio_write+0x0/0x50
Jan 29 22:13:44 master kernel:  [<ffffffff881bf2a0>] 
:xfs:xfs_file_aio_write+0x0/0x50
Jan 29 22:13:44 master kernel:  [do_sync_readv_writev+203/272] 
do_sync_readv_writev+0xcb/0x110
Jan 29 22:13:44 master kernel:  [<ffffffff8029a46b>] 
do_sync_readv_writev+0xcb/0x110
Jan 29 22:13:44 master kernel:  [autoremove_wake_function+0/48] 
autoremove_wake_function+0x0/0x30
Jan 29 22:13:44 master kernel:  [<ffffffff80251e10>] 
autoremove_wake_function+0x0/0x30
Jan 29 22:13:44 master kernel:  [_end+130243723/2130457992] 
:exportfs:exportfs_decode_fh+0x93/0x278
Jan 29 22:13:44 master kernel:  [<ffffffff88272703>] 
:exportfs:exportfs_decode_fh+0x93/0x278
Jan 29 22:13:44 master kernel:  [<ffffffff88272703>] 
:exportfs:exportfs_decode_fh+0x93/0x278
Jan 29 22:13:44 master kernel:  [_end+130783832/2130457992] 
:nfsd:nfsd_acceptable+0x0/0x100
Jan 29 22:13:44 master kernel:  [<ffffffff882f64d0>] 
:nfsd:nfsd_acceptable+0x0/0x100
Jan 29 22:13:44 master kernel:  [_end+129748673/2130457992] 
:sunrpc:cache_check+0x49/0x490
Jan 29 22:13:44 master kernel:  [<ffffffff881f9939>] 
:sunrpc:cache_check+0x49/0x490
Jan 29 22:13:44 master kernel:  [rw_copy_check_uvector+157/304] 
rw_copy_check_uvector+0x9d/0x130
Jan 29 22:13:44 master kernel:  [<ffffffff8029a2cd>] 
rw_copy_check_uvector+0x9d/0x130
Jan 29 22:13:44 master kernel:  [do_readv_writev+226/528] 
do_readv_writev+0xe2/0x210
Jan 29 22:13:44 master kernel:  [<ffffffff8029abb2>] 
do_readv_writev+0xe2/0x210
Jan 29 22:13:44 master kernel:  [_end+130813642/2130457992] 
:nfsd:nfsd_setuser+0x82/0x210
Jan 29 22:13:44 master kernel:  [<ffffffff882fd942>] 
:nfsd:nfsd_setuser+0x82/0x210
Jan 29 22:13:44 master kernel:  [_end+130782532/2130457992] 
:nfsd:nfsd_setuser_and_check_port+0x5c/0x70
Jan 29 22:13:44 master kernel:  [<ffffffff882f5fbc>] 
:nfsd:nfsd_setuser_and_check_port+0x5c/0x70
Jan 29 22:13:44 master kernel:  [_end+130788665/2130457992] 
:nfsd:nfsd_vfs_write+0xf1/0x3a0
Jan 29 22:13:44 master kernel:  [<ffffffff882f77b1>] 
:nfsd:nfsd_vfs_write+0xf1/0x3a0
Jan 29 22:13:44 master kernel:  [__dentry_open+408/528] 
__dentry_open+0x198/0x210
Jan 29 22:13:44 master kernel:  [<ffffffff80298ab8>] 
__dentry_open+0x198/0x210
Jan 29 22:13:44 master kernel:  [_end+130790708/2130457992] 
:nfsd:nfsd_open+0x12c/0x1e0
Jan 29 22:13:44 master kernel:  [<ffffffff882f7fac>] 
:nfsd:nfsd_open+0x12c/0x1e0
Jan 29 22:13:44 master kernel:  [_end+130791566/2130457992] 
:nfsd:nfsd_write+0x106/0x110
Jan 29 22:13:44 master kernel:  [<ffffffff882f8306>] 
:nfsd:nfsd_write+0x106/0x110
Jan 29 22:13:44 master kernel:  [_end+130825459/2130457992] 
:nfsd:nfsd3_proc_write+0x10b/0x160
Jan 29 22:13:44 master kernel:  [<ffffffff8830076b>] 
:nfsd:nfsd3_proc_write+0x10b/0x160
Jan 29 22:13:44 master kernel:  [_end+130770937/2130457992] 
:nfsd:nfsd_dispatch+0xb1/0x220
Jan 29 22:13:44 master kernel:  [<ffffffff882f3271>] 
:nfsd:nfsd_dispatch+0xb1/0x220
Jan 29 22:13:44 master kernel:  [_end+129716706/2130457992] 
:sunrpc:svc_process+0x47a/0x780
Jan 29 22:13:44 master kernel:  [<ffffffff881f1c5a>] 
:sunrpc:svc_process+0x47a/0x780
Jan 29 22:13:44 master kernel:  [__down_read+18/177] __down_read+0x12/0xb1
Jan 29 22:13:44 master kernel:  [<ffffffff80466242>] __down_read+0x12/0xb1
Jan 29 22:13:44 master kernel:  [_end+130772360/2130457992] 
:nfsd:nfsd+0x0/0x2e0
Jan 29 22:13:44 master kernel:  [<ffffffff882f3800>] :nfsd:nfsd+0x0/0x2e0
Jan 29 22:13:44 master kernel:  [_end+130772771/2130457992] 
:nfsd:nfsd+0x19b/0x2e0
Jan 29 22:13:44 master kernel:  [<ffffffff882f399b>] :nfsd:nfsd+0x19b/0x2e0
Jan 29 22:13:44 master kernel:  [child_rip+10/18] child_rip+0xa/0x12
Jan 29 22:13:44 master kernel:  [<ffffffff8020ca28>] child_rip+0xa/0x12
Jan 29 22:13:44 master kernel:  [_end+130772360/2130457992] 
:nfsd:nfsd+0x0/0x2e0
Jan 29 22:13:44 master kernel:  [<ffffffff882f3800>] :nfsd:nfsd+0x0/0x2e0
Jan 29 22:13:44 master kernel:  [_end+130772360/2130457992] 
:nfsd:nfsd+0x0/0x2e0
Jan 29 22:13:44 master kernel:  [<ffffffff882f3800>] :nfsd:nfsd+0x0/0x2e0
Jan 29 22:13:44 master kernel:  [child_rip+0/18] child_rip+0x0/0x12
Jan 29 22:13:44 master kernel:  [<ffffffff8020ca1e>] child_rip+0x0/0x12
Jan 29 22:13:44 master kernel:
Jan 29 22:13:44 master kernel: Mem-info:
Jan 29 22:13:44 master kernel: DMA per-cpu:
Jan 29 22:13:44 master kernel: CPU    0: Hot: hi:    0, btch:   1 usd:   
0   Cold: hi:    0, btch:   1 usd:   0
Jan 29 22:13:44 master kernel: CPU    1: Hot: hi:    0, btch:   1 usd:   
0   Cold: hi:    0, btch:   1 usd:   0
Jan 29 22:13:44 master kernel: DMA32 per-cpu:
Jan 29 22:13:44 master kernel: CPU    0: Hot: hi:  186, btch:  31 usd:  
55   Cold: hi:   62, btch:  15 usd:  60
Jan 29 22:13:44 master kernel: CPU    1: Hot: hi:  186, btch:  31 usd: 
183   Cold: hi:   62, btch:  15 usd:  48
Jan 29 22:13:44 master kernel: Active:40766 inactive:437591 dirty:29984 
writeback:1045 unstable:0
Jan 29 22:13:44 master kernel:  free:4000 slab:18952 mapped:11793 
pagetables:1512 bounce:0
Jan 29 22:13:44 master kernel: DMA free:8008kB min:28kB low:32kB 
high:40kB active:0kB inactive:1676kB present:11100kB pages_scanned:0 
all_unreclaimable? no
Jan 29 22:13:44 master kernel: lowmem_reserve[]: 0 1998 1998 1998
Jan 29 22:13:44 master kernel: DMA32 free:8076kB min:5704kB low:7128kB 
high:8556kB active:163064kB inactive:1748688kB present:2046800kB 
pages_scanned:0 all_unreclaimabl
e? no
Jan 29 22:13:44 master kernel: lowmem_reserve[]: 0 0 0 0
Jan 29 22:13:44 master kernel: DMA: 6*4kB 6*8kB 13*16kB 6*32kB 4*64kB 
1*128kB 12*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 8024kB
Jan 29 22:13:44 master kernel: DMA32: 1160*4kB 117*8kB 82*16kB 19*32kB 
2*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 8136kB
Jan 29 22:13:44 master kernel: Swap cache: add 45, delete 44, find 0/0, 
race 0+0
Jan 29 22:13:44 master kernel: Free swap  = 4088324kB
Jan 29 22:13:44 master kernel: Total swap = 4088500kB
Jan 29 22:13:44 master kernel: Free swap:       4088324kB
Jan 29 22:13:44 master kernel: 523264 pages of RAM
Jan 29 22:13:44 master kernel: 9510 reserved pages
Jan 29 22:13:44 master kernel: 494203 pages shared
Jan 29 22:13:44 master kernel: 1 pages swap cached
Jan 29 22:13:45 master kernel: nfsd: page allocation failure. order:2, 
mode:0x4020
Jan 29 22:13:45 master kernel: Pid: 6595, comm: nfsd Not tainted 2.6.24 #1
Jan 29 22:13:45 master kernel:
Jan 29 22:13:45 master kernel: Call Trace:
Jan 29 22:13:45 master kernel:  <IRQ>  [__alloc_pages+822/912] 
__alloc_pages+0x336/0x390
Jan 29 22:13:45 master kernel:  <IRQ>  [<ffffffff80279d66>] 
__alloc_pages+0x336/0x390
Jan 29 22:13:45 master kernel:  [ip_local_deliver_finish+129/512] 
ip_local_deliver_finish+0x81/0x200
Jan 29 22:13:45 master kernel:  [<ffffffff80417611>] 
ip_local_deliver_finish+0x81/0x200
Jan 29 22:13:45 master kernel:  [_end+130301422/2130457992] 
:myri10ge:myri10ge_alloc_rx_pages+0x156/0x270
Jan 29 22:13:45 master kernel:  [<ffffffff88280866>] 
:myri10ge:myri10ge_alloc_rx_pages+0x156/0x270
Jan 29 22:13:45 master kernel:  [_end+130325174/2130457992] 
:myri10ge:myri10ge_poll+0x57e/0xae0
Jan 29 22:13:45 master kernel:  [<ffffffff8828652e>] 
:myri10ge:myri10ge_poll+0x57e/0xae0
Jan 29 22:13:45 master kernel:  [_spin_lock_bh+9/32] _spin_lock_bh+0x9/0x20
Jan 29 22:13:45 master kernel:  [<ffffffff80466819>] _spin_lock_bh+0x9/0x20
Jan 29 22:13:45 master kernel:  [_end+129719928/2130457992] 
:sunrpc:svc_sock_enqueue+0x80/0x340
Jan 29 22:13:45 master kernel:  [<ffffffff881f28f0>] 
:sunrpc:svc_sock_enqueue+0x80/0x340
Jan 29 22:13:45 master kernel:  [net_rx_action+153/448] 
net_rx_action+0x99/0x1c0
Jan 29 22:13:45 master kernel:  [<ffffffff803f6729>] 
net_rx_action+0x99/0x1c0
Jan 29 22:13:45 master kernel:  [__do_softirq+105/224] 
__do_softirq+0x69/0xe0
Jan 29 22:13:45 master kernel:  [<ffffffff80241db9>] __do_softirq+0x69/0xe0
Jan 29 22:13:45 master kernel:  [call_softirq+28/48] call_softirq+0x1c/0x30
Jan 29 22:13:45 master kernel:  [<ffffffff8020cd9c>] call_softirq+0x1c/0x30
Jan 29 22:13:45 master kernel:  [do_softirq+53/144] do_softirq+0x35/0x90
Jan 29 22:13:45 master kernel:  [<ffffffff8020eef5>] do_softirq+0x35/0x90
Jan 29 22:13:45 master kernel:  [do_IRQ+128/256] do_IRQ+0x80/0x100
Jan 29 22:13:45 master kernel:  [<ffffffff8020f170>] do_IRQ+0x80/0x100
Jan 29 22:13:45 master kernel:  [ret_from_intr+0/10] ret_from_intr+0x0/0xa
Jan 29 22:13:45 master kernel:  [<ffffffff8020c121>] ret_from_intr+0x0/0xa
Jan 29 22:13:45 master kernel:  <EOI>  [block_write_begin+21/224] 
block_write_begin+0x15/0xe0
Jan 29 22:13:45 master kernel:  <EOI>  [<ffffffff802c03b5>] 
block_write_begin+0x15/0xe0
Jan 29 22:13:45 master kernel:  [_end+129490730/2130457992] 
:xfs:xfs_vm_write_begin+0x22/0x30
Jan 29 22:13:45 master kernel:  [<ffffffff881ba9a2>] 
:xfs:xfs_vm_write_begin+0x22/0x30
Jan 29 22:13:45 master kernel:  [_end+129492696/2130457992] 
:xfs:xfs_get_blocks+0x0/0x10
Jan 29 22:13:45 master kernel:  [<ffffffff881bb150>] 
:xfs:xfs_get_blocks+0x0/0x10
Jan 29 22:13:45 master kernel:  [generic_file_buffered_write+328/1728] 
generic_file_buffered_write+0x148/0x6c0
Jan 29 22:13:45 master kernel:  [<ffffffff802756c8>] 
generic_file_buffered_write+0x148/0x6c0
Jan 29 22:13:45 master kernel:  [__alloc_skb+133/336] __alloc_skb+0x85/0x150
Jan 29 22:13:45 master kernel:  [<ffffffff803eea15>] __alloc_skb+0x85/0x150
Jan 29 22:13:45 master kernel:  [_end+129525959/2130457992] 
:xfs:xfs_write+0x67f/0x8f0
Jan 29 22:13:45 master kernel:  [<ffffffff881c333f>] 
:xfs:xfs_write+0x67f/0x8f0
Jan 29 22:13:45 master kernel:  [_end+129343578/2130457992] 
:xfs:xfs_iget+0x142/0x160
Jan 29 22:13:45 master kernel:  [<ffffffff88196ad2>] 
:xfs:xfs_iget+0x142/0x160
Jan 29 22:13:45 master kernel:  [_end+129509416/2130457992] 
:xfs:xfs_file_aio_write+0x0/0x50
Jan 29 22:13:45 master kernel:  [<ffffffff881bf2a0>] 
:xfs:xfs_file_aio_write+0x0/0x50
Jan 29 22:13:45 master kernel:  [do_sync_readv_writev+203/272] 
do_sync_readv_writev+0xcb/0x110
Jan 29 22:13:45 master kernel:  [<ffffffff8029a46b>] 
do_sync_readv_writev+0xcb/0x110
Jan 29 22:13:45 master kernel:  [autoremove_wake_function+0/48] 
autoremove_wake_function+0x0/0x30
Jan 29 22:13:45 master kernel:  [<ffffffff80251e10>] 
autoremove_wake_function+0x0/0x30
Jan 29 22:13:45 master kernel:  [_end+130243723/2130457992] 
:exportfs:exportfs_decode_fh+0x93/0x278
Jan 29 22:13:45 master kernel:  [<ffffffff88272703>] 
:exportfs:exportfs_decode_fh+0x93/0x278
Jan 29 22:13:45 master kernel:  [_end+130783832/2130457992] 
:nfsd:nfsd_acceptable+0x0/0x100
Jan 29 22:13:45 master kernel:  [<ffffffff882f64d0>] 
:nfsd:nfsd_acceptable+0x0/0x100
Jan 29 22:13:45 master kernel:  [_end+129748673/2130457992] 
:sunrpc:cache_check+0x49/0x490
Jan 29 22:13:45 master kernel:  [<ffffffff881f9939>] 
:sunrpc:cache_check+0x49/0x490
Jan 29 22:13:45 master kernel:  [rw_copy_check_uvector+157/304] 
rw_copy_check_uvector+0x9d/0x130
Jan 29 22:13:45 master kernel:  [<ffffffff8029a2cd>] 
rw_copy_check_uvector+0x9d/0x130
Jan 29 22:13:45 master kernel:  [do_readv_writev+226/528] 
do_readv_writev+0xe2/0x210
Jan 29 22:13:45 master kernel:  [<ffffffff8029abb2>] 
do_readv_writev+0xe2/0x210
Jan 29 22:13:45 master kernel:  [_end+130813642/2130457992] 
:nfsd:nfsd_setuser+0x82/0x210
Jan 29 22:13:45 master kernel:  [<ffffffff882fd942>] 
:nfsd:nfsd_setuser+0x82/0x210
Jan 29 22:13:45 master kernel:  [_end+130782532/2130457992] 
:nfsd:nfsd_setuser_and_check_port+0x5c/0x70
Jan 29 22:13:45 master kernel:  [<ffffffff882f5fbc>] 
:nfsd:nfsd_setuser_and_check_port+0x5c/0x70
Jan 29 22:13:45 master kernel:  [_end+130788665/2130457992] 
:nfsd:nfsd_vfs_write+0xf1/0x3a0
Jan 29 22:13:45 master kernel:  [<ffffffff882f77b1>] 
:nfsd:nfsd_vfs_write+0xf1/0x3a0
Jan 29 22:13:45 master kernel:  [__dentry_open+408/528] 
__dentry_open+0x198/0x210
Jan 29 22:13:45 master kernel:  [<ffffffff80298ab8>] 
__dentry_open+0x198/0x210
Jan 29 22:13:45 master kernel:  [_end+130790708/2130457992] 
:nfsd:nfsd_open+0x12c/0x1e0
Jan 29 22:13:45 master kernel:  [<ffffffff882f7fac>] 
:nfsd:nfsd_open+0x12c/0x1e0
Jan 29 22:13:45 master kernel:  [_end+130791566/2130457992] 
:nfsd:nfsd_write+0x106/0x110
Jan 29 22:13:45 master kernel:  [<ffffffff882f8306>] 
:nfsd:nfsd_write+0x106/0x110
Jan 29 22:13:45 master kernel:  [_end+130825459/2130457992] 
:nfsd:nfsd3_proc_write+0x10b/0x160
Jan 29 22:13:45 master kernel:  [<ffffffff8830076b>] 
:nfsd:nfsd3_proc_write+0x10b/0x160
Jan 29 22:13:45 master kernel:  [_end+130770937/2130457992] 
:nfsd:nfsd_dispatch+0xb1/0x220
Jan 29 22:13:45 master kernel:  [<ffffffff882f3271>] 
:nfsd:nfsd_dispatch+0xb1/0x220
Jan 29 22:13:45 master kernel:  [_end+129716706/2130457992] 
:sunrpc:svc_process+0x47a/0x780
Jan 29 22:13:45 master kernel:  [<ffffffff881f1c5a>] 
:sunrpc:svc_process+0x47a/0x780
Jan 29 22:13:45 master kernel:  [__down_read+18/177] __down_read+0x12/0xb1
Jan 29 22:13:45 master kernel:  [<ffffffff80466242>] __down_read+0x12/0xb1
Jan 29 22:13:45 master kernel:  [_end+130772360/2130457992] 
:nfsd:nfsd+0x0/0x2e0
Jan 29 22:13:45 master kernel:  [<ffffffff882f3800>] :nfsd:nfsd+0x0/0x2e0
Jan 29 22:13:45 master kernel:  [_end+130772771/2130457992] 
:nfsd:nfsd+0x19b/0x2e0
Jan 29 22:13:45 master kernel:  [<ffffffff882f399b>] :nfsd:nfsd+0x19b/0x2e0
Jan 29 22:13:45 master kernel:  [child_rip+10/18] child_rip+0xa/0x12
Jan 29 22:13:45 master kernel:  [<ffffffff8020ca28>] child_rip+0xa/0x12
Jan 29 22:13:45 master kernel:  [_end+130772360/2130457992] 
:nfsd:nfsd+0x0/0x2e0
Jan 29 22:13:45 master kernel:  [<ffffffff882f3800>] :nfsd:nfsd+0x0/0x2e0
Jan 29 22:13:45 master kernel:  [_end+130772360/2130457992] 
:nfsd:nfsd+0x0/0x2e0
Jan 29 22:13:45 master kernel:  [<ffffffff882f3800>] :nfsd:nfsd+0x0/0x2e0
Jan 29 22:13:45 master kernel:  [child_rip+0/18] child_rip+0x0/0x12
Jan 29 22:13:45 master kernel:  [<ffffffff8020ca1e>] child_rip+0x0/0x12
Jan 29 22:13:45 master kernel:
Jan 29 22:13:45 master kernel: Mem-info:
Jan 29 22:13:45 master kernel: DMA per-cpu:
Jan 29 22:13:45 master kernel: CPU    0: Hot: hi:    0, btch:   1 usd:   
0   Cold: hi:    0, btch:   1 usd:   0
Jan 29 22:13:45 master kernel: CPU    1: Hot: hi:    0, btch:   1 usd:   
0   Cold: hi:    0, btch:   1 usd:   0
Jan 29 22:13:45 master kernel: DMA32 per-cpu:
Jan 29 22:13:45 master kernel: CPU    0: Hot: hi:  186, btch:  31 usd: 
156   Cold: hi:   62, btch:  15 usd:  48
Jan 29 22:13:45 master kernel: CPU    1: Hot: hi:  186, btch:  31 usd:  
73   Cold: hi:   62, btch:  15 usd:  56
Jan 29 22:13:45 master kernel: Active:40772 inactive:437908 dirty:24942 
writeback:1051 unstable:0
Jan 29 22:13:45 master kernel:  free:3819 slab:18993 mapped:11793 
pagetables:1512 bounce:0
Jan 29 22:13:45 master kernel: DMA free:8008kB min:28kB low:32kB 
high:40kB active:0kB inactive:400kB present:11100kB pages_scanned:0 
all_unreclaimable? no
Jan 29 22:13:45 master kernel: lowmem_reserve[]: 0 1998 1998 1998
Jan 29 22:13:45 master kernel: DMA32 free:7268kB min:5704kB low:7128kB 
high:8556kB active:163088kB inactive:1751232kB present:2046800kB 
pages_scanned:0 all_unreclaimabl
e? no
Jan 29 22:13:45 master kernel: lowmem_reserve[]: 0 0 0 0
Jan 29 22:13:45 master kernel: DMA: 10*4kB 8*8kB 17*16kB 13*32kB 7*64kB 
5*128kB 8*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 8024kB
Jan 29 22:13:45 master kernel: DMA32: 1555*4kB 0*8kB 1*16kB 1*32kB 
1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 7228kB
Jan 29 22:13:45 master kernel: Swap cache: add 45, delete 44, find 0/0, 
race 0+0
Jan 29 22:13:45 master kernel: Free swap  = 4088324kB
Jan 29 22:13:45 master kernel: Total swap = 4088500kB
Jan 29 22:13:45 master kernel: Free swap:       4088324kB
Jan 29 22:13:45 master kernel: 523264 pages of RAM





^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: 2.6.24 Page Allocation Failure
  2008-01-31 15:49 2.6.24 Page Allocation Failure AndrewL733
@ 2008-02-01 14:13 ` AndrewL733
  2008-02-01 18:53 ` Mel Gorman
  1 sibling, 0 replies; 3+ messages in thread
From: AndrewL733 @ 2008-02-01 14:13 UTC (permalink / raw)
  To: linux-kernel, Myricom Technical Support

The cause of this problem seems to be compiling the Myricom driver with 
the ALLOC_ORDER=2 option. When I use the in-kernel driver, (1.3.2) or 
recompile the Myricom 1.4.0 driver WITHOUT the option, the problem seems 
to go away even after heavy hammering of the system.

The ALLOC_ORDER=2 compiling option doesn't seem to cause any problem for 
the Myricom 1.4.0 driver in the 2.6.22 kernel but it does cause the 
problem when I run it in 2.6.24. 

Maybe with 2.6.24 more RAM is required to use this option? I will 
contact Myricom to inform them.

Andrew

AndrewL733 wrote:
> My setup is Mandriva 2007 64-bit:
>
> IntelS5000PSLSATA motherboard
> Single Dual Core 3 Ghz Xeon
> 2 GB ECC RAM
> Myricom 10 GbE Network Adapter (using out of kernel 1.4.0 driver 
> compiled with "MYRI10GE_ALLOC_ORDER=2")
>
>
> I have two identical systems doing data transfer via Myricom cards 
> connected directly to one another (in other words, not going through a 
> switch).
>
> No problems hammering Myricom with 350 MB/sec NFS traffic while 
> running 2.6.20.15 and 2.6.22.9 (kernel.org "plain vanilla" kernels I 
> compiled and run, also with updated Myricom drivers). However, I just 
> compiled and installed 2.6.24 and get the following errors as soon as 
> I begin doing NFS I/O over the Myricom card. Maybe I missed something 
> in the kernel configuration? Throughput when running 2.6.24 is about 
> 40 percent lower than when running 2.6.22.9-- surely due to these errors.
>
> Jan 29 22:13:36 master kernel: nfsd: page allocation failure. order:2, 
> mode:0x4020
> Jan 29 22:13:36 master kernel: Pid: 6586, comm: nfsd Not tainted 
> 2.6.24 #1
> Jan 29 22:13:36 master kernel:
> Jan 29 22:13:36 master kernel: Call Trace:
> Jan 29 22:13:36 master kernel:  <IRQ>  [__alloc_pages+822/912] 
> __alloc_pages+0x336/0x390
> Jan 29 22:13:36 master kernel:  <IRQ>  [<ffffffff80279d66>] 
> __alloc_pages+0x336/0x390
> Jan 29 22:13:36 master kernel:  [ip_local_deliver+37/112] 
> ip_local_deliver+0x25/0x70
> Jan 29 22:13:36 master kernel:  [<ffffffff80417bb5>] 
> ip_local_deliver+0x25/0x70
> Jan 29 22:13:36 master kernel:  [_end+130301422/2130457992] 
> :myri10ge:myri10ge_alloc_rx_pages+0x156/0x270
> Jan 29 22:13:36 master kernel:  [<ffffffff88280866>] 
> :myri10ge:myri10ge_alloc_rx_pages+0x156/0x270
> Jan 29 22:13:36 master kernel:  [_end+130325174/2130457992] 
> :myri10ge:myri10ge_poll+0x57e/0xae0
> Jan 29 22:13:36 master kernel:  [<ffffffff8828652e>] 
> :myri10ge:myri10ge_poll+0x57e/0xae0
> Jan 29 22:13:36 master kernel:  [_spin_lock_bh+9/32] 
> _spin_lock_bh+0x9/0x20
> Jan 29 22:13:36 master kernel:  [<ffffffff80466819>] 
> _spin_lock_bh+0x9/0x20
> Jan 29 22:13:36 master kernel:  [_end+129719928/2130457992] 
> :sunrpc:svc_sock_enqueue+0x80/0x340
> Jan 29 22:13:36 master kernel:  [<ffffffff881f28f0>] 
> :sunrpc:svc_sock_enqueue+0x80/0x340
> Jan 29 22:13:36 master kernel:  [net_rx_action+153/448] 
> net_rx_action+0x99/0x1c0
> Jan 29 22:13:36 master kernel:  [<ffffffff803f6729>] 
> net_rx_action+0x99/0x1c0
> Jan 29 22:13:36 master kernel:  [__do_softirq+105/224] 
> __do_softirq+0x69/0xe0
> Jan 29 22:13:36 master kernel:  [<ffffffff80241db9>] 
> __do_softirq+0x69/0xe0
> Jan 29 22:13:36 master kernel:  [call_softirq+28/48] 
> call_softirq+0x1c/0x30
> Jan 29 22:13:36 master kernel:  [<ffffffff8020cd9c>] 
> call_softirq+0x1c/0x30
> Jan 29 22:13:36 master kernel:  <EOI>  [do_softirq+53/144] 
> do_softirq+0x35/0x90
> Jan 29 22:13:36 master kernel:  <EOI>  [<ffffffff8020eef5>] 
> do_softirq+0x35/0x90
> Jan 29 22:13:36 master kernel:  [local_bh_enable+91/160] 
> local_bh_enable+0x5b/0xa0
> Jan 29 22:13:36 master kernel:  [<ffffffff80241c6b>] 
> local_bh_enable+0x5b/0xa0
> Jan 29 22:13:36 master kernel:  [_end+129726255/2130457992] 
> :sunrpc:svc_udp_recvfrom+0x407/0x430
> Jan 29 22:13:36 master kernel:  [<ffffffff881f41a7>] 
> :sunrpc:svc_udp_recvfrom+0x407/0x430
> Jan 29 22:13:36 master kernel:  [_end+129730321/2130457992] 
> :sunrpc:svc_recv+0x2a9/0x4e0
> Jan 29 22:13:36 master kernel:  [<ffffffff881f5189>] 
> :sunrpc:svc_recv+0x2a9/0x4e0
> Jan 29 22:13:36 master kernel:  [default_wake_function+0/16] 
> default_wake_function+0x0/0x10
> Jan 29 22:13:36 master kernel:  [<ffffffff80233460>] 
> default_wake_function+0x0/0x10
> Jan 29 22:13:36 master kernel:  [__down_read+18/177] 
> __down_read+0x12/0xb1
> Jan 29 22:13:36 master kernel:  [<ffffffff80466242>] 
> __down_read+0x12/0xb1
> Jan 29 22:13:36 master kernel:  [_end+130772360/2130457992] 
> :nfsd:nfsd+0x0/0x2e0
> an 29 22:13:36 master kernel:  [<ffffffff882f3800>] :nfsd:nfsd+0x0/0x2e0
> Jan 29 22:13:36 master kernel:  [_end+130772586/2130457992] 
> :nfsd:nfsd+0xe2/0x2e0
> Jan 29 22:13:36 master kernel:  [<ffffffff882f38e2>] 
> :nfsd:nfsd+0xe2/0x2e0
> Jan 29 22:13:36 master kernel:  [child_rip+10/18] child_rip+0xa/0x12
> Jan 29 22:13:36 master kernel:  [<ffffffff8020ca28>] child_rip+0xa/0x12
> Jan 29 22:13:36 master kernel:  [_end+130772360/2130457992] 
> :nfsd:nfsd+0x0/0x2e0
> Jan 29 22:13:36 master kernel:  [<ffffffff882f3800>] :nfsd:nfsd+0x0/0x2e0
> Jan 29 22:13:36 master kernel:  [_end+130772360/2130457992] 
> :nfsd:nfsd+0x0/0x2e0
> Jan 29 22:13:36 master kernel:  [<ffffffff882f3800>] :nfsd:nfsd+0x0/0x2e0
> Jan 29 22:13:36 master kernel:  [child_rip+0/18] child_rip+0x0/0x12
> Jan 29 22:13:36 master kernel:  [<ffffffff8020ca1e>] child_rip+0x0/0x12
> Jan 29 22:13:36 master kernel:
> Jan 29 22:13:36 master kernel: Mem-info:
> Jan 29 22:13:36 master kernel: DMA per-cpu:
> Jan 29 22:13:36 master kernel: CPU    0: Hot: hi:    0, btch:   1 
> usd:   0   Cold: hi:    0, btch:   1 usd:   0
> Jan 29 22:13:36 master kernel: CPU    1: Hot: hi:    0, btch:   1 
> usd:   0   Cold: hi:    0, btch:   1 usd:   0
> Jan 29 22:13:36 master kernel: DMA32 per-cpu:
> Jan 29 22:13:36 master kernel: CPU    0: Hot: hi:  186, btch:  31 usd: 
> 150   Cold: hi:   62, btch:  15 usd:  62
> Jan 29 22:13:36 master kernel: CPU    1: Hot: hi:  186, btch:  31 usd: 
> 105   Cold: hi:   62, btch:  15 usd:  52
> Jan 29 22:13:36 master kernel: Active:41027 inactive:437314 
> dirty:29708 writeback:0 unstable:0
> Jan 29 22:13:36 master kernel:  free:3892 slab:19405 mapped:11793 
> pagetables:1512 bounce:0
> Jan 29 22:13:36 master kernel: DMA free:8008kB min:28kB low:32kB 
> high:40kB active:0kB inactive:2384kB present:11100kB pages_scanned:0 
> all_unreclaimable? no
> Jan 29 22:13:36 master kernel: lowmem_reserve[]: 0 1998 1998 1998
> Jan 29 22:13:36 master kernel: DMA32 free:7560kB min:5704kB low:7128kB 
> high:8556kB active:164108kB inactive:1746872kB present:2046800kB 
> pages_scanned:32 all_unreclaimab
> le? no
> Jan 29 22:13:36 master kernel: lowmem_reserve[]: 0 0 0 0
> Jan 29 22:13:36 master kernel: DMA: 4*4kB 5*8kB 6*16kB 6*32kB 4*64kB 
> 2*128kB 4*256kB 4*512kB 0*1024kB 0*2048kB 1*4096kB = 8024kB
> Jan 29 22:13:36 master kernel: DMA32: 1365*4kB 11*8kB 38*16kB 18*32kB 
> 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 7756kB
> Jan 29 22:13:36 master kernel: Swap cache: add 43, delete 43, find 
> 0/0, race 0+0
> Jan 29 22:13:36 master kernel: Free swap  = 4088332kB
> Jan 29 22:13:36 master kernel: Total swap = 4088500kB
> Jan 29 22:13:36 master kernel: Free swap:       4088332kB
> Jan 29 22:13:36 master kernel: 523264 pages of RAM
> Jan 29 22:13:36 master kernel: 9510 reserved pages
> Jan 29 22:13:36 master kernel: 492885 pages shared
> Jan 29 22:13:36 master kernel: 0 pages swap cached
> Jan 29 22:13:44 master nmbd[3438]: [2008/01/29 22:13:44, 0] 
> nmbd/nmbd_packets.c:process_browse_packet(1061)
> Jan 29 22:13:44 master nmbd[3438]:   process_browse_packet: Discarding 
> datagram from IP 192.168.1.240. Source name master<00> is one of our 
> names !
> Jan 29 22:13:44 master nmbd[3438]: [2008/01/29 22:13:44, 0] 
> nmbd/nmbd_packets.c:process_browse_packet(1061)
> Jan 29 22:13:44 master nmbd[3438]:   process_browse_packet: Discarding 
> datagram from IP 192.168.1.237. Source name master<00> is one of our 
> names !
>


^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: 2.6.24 Page Allocation Failure
  2008-01-31 15:49 2.6.24 Page Allocation Failure AndrewL733
  2008-02-01 14:13 ` AndrewL733
@ 2008-02-01 18:53 ` Mel Gorman
  1 sibling, 0 replies; 3+ messages in thread
From: Mel Gorman @ 2008-02-01 18:53 UTC (permalink / raw)
  To: AndrewL733; +Cc: linux-kernel

Hi Andrew

On (31/01/08 10:49), AndrewL733 didst pronounce:
> My setup is Mandriva 2007 64-bit:
> 
> IntelS5000PSLSATA motherboard
> Single Dual Core 3 Ghz Xeon
> 2 GB ECC RAM
> Myricom 10 GbE Network Adapter (using out of kernel 1.4.0 driver 
> compiled with "MYRI10GE_ALLOC_ORDER=2")
> 
> 
> I have two identical systems doing data transfer via Myricom cards 
> connected directly to one another (in other words, not going through a 
> switch).
> 
> No problems hammering Myricom with 350 MB/sec NFS traffic while running 
> 2.6.20.15 and 2.6.22.9 (kernel.org "plain vanilla" kernels I compiled 
> and run, also with updated Myricom drivers). However, I just compiled 
> and installed 2.6.24 and get the following errors as soon as I begin 
> doing NFS I/O over the Myricom card. Maybe I missed something in the 
> kernel configuration? Throughput when running 2.6.24 is about 40 percent 
> lower than when running 2.6.22.9-- surely due to these errors.
> 

It's possible. Is there a stream of these errors coming through constantly?

> Jan 29 22:13:36 master kernel: nfsd: page allocation failure. order:2, mode:0x4020

So, that is a ZONE_NORMAL high-order atomic allocation (GFP_COMPOUND as well
which is a little surprise but should make no difference). Not exactly
recommended but there are a few drivers that do it and I guess yours is one.

> Jan 29 22:13:36 master kernel: Pid: 6586, comm: nfsd Not tainted 2.6.24 #1
> Jan 29 22:13:36 master kernel:
> Jan 29 22:13:36 master kernel: Call Trace:
> Jan 29 22:13:36 master kernel:  <IRQ>  [__alloc_pages+822/912] __alloc_pages+0x336/0x390
> Jan 29 22:13:36 master kernel:  <IRQ>  [<ffffffff80279d66>]    __alloc_pages+0x336/0x390
> Jan 29 22:13:36 master kernel:  [ip_local_deliver+37/112]      ip_local_deliver+0x25/0x70
> Jan 29 22:13:36 master kernel:  [<ffffffff80417bb5>]           ip_local_deliver+0x25/0x70
> Jan 29 22:13:36 master kernel:  [_end+130301422/2130457992]    :myri10ge:myri10ge_alloc_rx_pages+0x156/0x270
> Jan 29 22:13:36 master kernel:  [<ffffffff88280866>]           :myri10ge:myri10ge_alloc_rx_pages+0x156/0x270
> Jan 29 22:13:36 master kernel:  [_end+130325174/2130457992]    :myri10ge:myri10ge_poll+0x57e/0xae0
> Jan 29 22:13:36 master kernel:  [<ffffffff8828652e>]           :myri10ge:myri10ge_poll+0x57e/0xae0

Coming from the driver in question so no suprise there
> <SNIP>
> Jan 29 22:13:36 master kernel: Mem-info:
> Jan 29 22:13:36 master kernel: DMA per-cpu:
> Jan 29 22:13:36 master kernel: CPU    0: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
> Jan 29 22:13:36 master kernel: CPU    1: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
> Jan 29 22:13:36 master kernel: DMA32 per-cpu:
> Jan 29 22:13:36 master kernel: CPU    0: Hot: hi:  186, btch:  31 usd: 150   Cold: hi:   62, btch:  15 usd:  62
> Jan 29 22:13:36 master kernel: CPU    1: Hot: hi:  186, btch:  31 usd: 105   Cold: hi:   62, btch:  15 usd:  52
> Jan 29 22:13:36 master kernel: Active:41027 inactive:437314 dirty:29708 writeback:0 unstable:0
> Jan 29 22:13:36 master kernel:  free:3892 slab:19405 mapped:11793 pagetables:1512 bounce:0
> Jan 29 22:13:36 master kernel: DMA free:8008kB min:28kB low:32kB high:40kB active:0kB inactive:2384kB present:11100kB pages_scanned:0 all_unreclaimable? no
> Jan 29 22:13:36 master kernel: lowmem_reserve[]: 0 1998 1998 1998
> Jan 29 22:13:36 master kernel: DMA32 free:7560kB min:5704kB low:7128kB high:8556kB active:164108kB inactive:1746872kB present:2046800kB pages_scanned:32 all_unreclaimable? no
> Jan 29 22:13:36 master kernel: lowmem_reserve[]: 0 0 0 0

No apparent memory pressure there. We are not even below the low
watermark for order-0 allocations and the lowmem_reserve is not a factor
for DMA32 allocations.

> Jan 29 22:13:36 master kernel: DMA: 4*4kB 5*8kB 6*16kB 6*32kB 4*64kB 2*128kB 4*256kB 4*512kB 0*1024kB 0*2048kB 1*4096kB = 8024kB
> Jan 29 22:13:36 master kernel: DMA32: 1365*4kB 11*8kB 38*16kB 18*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 7756kB

There are sufficient numbers of order-2 and higher pages free there. It is
similar for the subsequent failures. This does not appear to be a fragmentation
problem at least so lets see what watermarks look like just for zone->low_pages
which should be relatively relaxed.

Zone watermarks in pages;
Low:                    1782
Free:                   1890
classzone_reserve:         0

Adjusted Low calculation
        low = 1782 - 4 + 1 == 1779
                ALLOC_HIGH for __GFP_HIGH so
        low -= (low / 2) == 890
                ALLOC_HARDER for !__GFP_WAIT so
        low -= (low / 4) == 668

Free pages at each order
DMA32: 1365*4kB 11*8kB 38*16kB 18*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 7756kB
Watermark check
        order 0: is free_pages(1890) < low(668) + class_rsrv(0) -- no, ok
        order 1: is free_pages(525)  < low(334)                 -- no, ok
        order 2: is free_pages(503)  < low(167)                 -- no. ok

Err, so it looks like we are not hitting the watermarks either. This does
not make sense, the allocation should not have failed. I have to be making
a mistake with the zone watermark calculation, can anyone spot it?

Is this definetly 2.6.24 and otherwise unpatched?

> Jan 29 22:13:36 master kernel: Swap cache: add 43, delete 43, find 0/0, race 0+0
> Jan 29 22:13:36 master kernel: Free swap  = 4088332kB
> Jan 29 22:13:36 master kernel: Total swap = 4088500kB
> Jan 29 22:13:36 master kernel: Free swap:       4088332kB
> Jan 29 22:13:36 master kernel: 523264 pages of RAM
> Jan 29 22:13:36 master kernel: 9510 reserved pages
> Jan 29 22:13:36 master kernel: 492885 pages shared
> Jan 29 22:13:36 master kernel: 0 pages swap cached
> Jan 29 22:13:44 master nmbd[3438]: [2008/01/29 22:13:44, 0] 
> <SNIP>
 
Few questions to try pinning down what is going wrong.

1. Is the stream of errors constant once they start?
2. If you increase /proc/sys/vm/min_free_kbytes to 16384 or 64738, do
   the allocation errors still occur? If the errors no longer occur,
   is the performance restored?
3. If you compile with MYRI10GE_ALLOC_ORDER=0 on the old and new
   kernels, do the allocations errors occur? If not, how does the throughput
   compare between kernel versions?
   (I am checking for changes in NFS/IO/writeback that causes a writeout
   performance regression. Such a slowdown may cause memory pressure
   later and then these allocation errors when watermarks are hit)
4. If you apply the patch below, do the allocation errors still occur?
   (The patch disables fragmentation avoidance. I don't think fragmentation
   is the problem here because ample memory is free and watermarks look
   ok)
5. Any chance you could test 2.6.23 in case the problem also happens there?

Thanks

diff -rup -X /usr/src/patchset-0.6/bin//dontdiff linux-2.6.24-clean/include/linux/mmzone.h linux-2.6.24-disable-antifrag/include/linux/mmzone.h
--- linux-2.6.24-clean/include/linux/mmzone.h	2008-01-24 22:58:37.000000000 +0000
+++ linux-2.6.24-disable-antifrag/include/linux/mmzone.h	2008-02-01 17:23:39.000000000 +0000
@@ -45,14 +45,11 @@
 	for (order = 0; order < MAX_ORDER; order++) \
 		for (type = 0; type < MIGRATE_TYPES; type++)
 
-extern int page_group_by_mobility_disabled;
+#define page_group_by_mobility_disabled (0)
 
 static inline int get_pageblock_migratetype(struct page *page)
 {
-	if (unlikely(page_group_by_mobility_disabled))
-		return MIGRATE_UNMOVABLE;
-
-	return get_pageblock_flags_group(page, PB_migrate, PB_migrate_end);
+	return MIGRATE_UNMOVABLE;
 }
 
 struct free_area {
diff -rup -X /usr/src/patchset-0.6/bin//dontdiff linux-2.6.24-clean/mm/page_alloc.c linux-2.6.24-disable-antifrag/mm/page_alloc.c
--- linux-2.6.24-clean/mm/page_alloc.c	2008-01-24 22:58:37.000000000 +0000
+++ linux-2.6.24-disable-antifrag/mm/page_alloc.c	2008-02-01 17:23:39.000000000 +0000
@@ -164,12 +164,9 @@ int nr_node_ids __read_mostly = MAX_NUMN
 EXPORT_SYMBOL(nr_node_ids);
 #endif
 
-int page_group_by_mobility_disabled __read_mostly;
-
 static void set_pageblock_migratetype(struct page *page, int migratetype)
 {
-	set_pageblock_flags_group(page, (unsigned long)migratetype,
-					PB_migrate, PB_migrate_end);
+	return;
 }
 
 #ifdef CONFIG_DEBUG_VM
@@ -2354,17 +2351,6 @@ void build_all_zonelists(void)
 		/* cpuset refresh routine should be here */
 	}
 	vm_total_pages = nr_free_pagecache_pages();
-	/*
-	 * Disable grouping by mobility if the number of pages in the
-	 * system is too low to allow the mechanism to work. It would be
-	 * more accurate, but expensive to check per-zone. This check is
-	 * made on memory-hotadd so a system can start with mobility
-	 * disabled and enable it later
-	 */
-	if (vm_total_pages < (pageblock_nr_pages * MIGRATE_TYPES))
-		page_group_by_mobility_disabled = 1;
-	else
-		page_group_by_mobility_disabled = 0;
 
 	printk("Built %i zonelists in %s order, mobility grouping %s.  "
 		"Total pages: %ld\n",
-- 
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab

^ permalink raw reply	[flat|nested] 3+ messages in thread

end of thread, other threads:[~2008-02-01 18:54 UTC | newest]

Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2008-01-31 15:49 2.6.24 Page Allocation Failure AndrewL733
2008-02-01 14:13 ` AndrewL733
2008-02-01 18:53 ` Mel Gorman

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).