Re: 2.6.6 is crashing repeatedly

Re: 2.6.6 is crashing repeatedly

[linux]

I already reported (I thought) this message:

May 19 02:23:47: Unable to handle kernel paging request at virtual address efd78000
May 19 02:23:47:  printing eip:
May 19 02:23:47: c01a999b
May 19 02:23:47: *pde = 004a7067
May 19 02:23:47: *pte = 2fd78000
May 19 02:23:47: Oops: 0002 [#1]
May 19 02:23:47: DEBUG_PAGEALLOC
May 19 02:23:47: CPU:    0
May 19 02:23:47: EIP:    0060:[<c01a999b>]    Not tainted
May 19 02:23:47: EFLAGS: 00010246   (2.6.6) 
May 19 02:23:47: EIP is at encode_entry+0x4b/0x530
May 19 02:23:47: eax: 00000000   ebx: 00000000   ecx: 00000644   edx: f3532df8
May 19 02:23:47: esi: efd78000   edi: e4a19644   ebp: 00000654   esp: f14b7b98
May 19 02:23:47: ds: 007b   es: 007b   ss: 0068
May 19 02:23:47: Process nfsd (pid: 1029, threadinfo=f14b6000 task=f1532a80)
May 19 02:23:47: Stack: 00000008 0000002f f5217080 00000027 00000000 f5217084 f3532df8 00000006 
May 19 02:23:47:        e4a1964c 0000001c 02000001 04000900 0019a291 0022e7a8 14343bfc 0022e76b 
May 19 02:23:47:        14343b61 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
May 19 02:23:47: Call Trace:
May 19 02:23:47:  [<c016af1f>] ext3_bread+0x1f/0x90
May 19 02:23:47:  [<c0168505>] ext3_readdir+0x305/0x490
May 19 02:23:47:  [<c011dd18>] in_group_p+0x38/0x70
May 19 02:23:47:  [<c01a9ed0>] nfs3svc_encode_entry_plus+0x0/0x50
May 19 02:23:47:  [<c019d930>] fh_verify+0x280/0x550
May 19 02:23:47:  [<c019d5d0>] nfsd_acceptable+0x0/0xe0
May 19 02:23:47:  [<c013da8c>] open_private_file+0x1c/0x90
May 19 02:23:47:  [<c014ba2d>] vfs_readdir+0x7d/0x90
May 19 02:23:47:  [<c01a11f9>] nfsd_readdir+0x79/0xc0
May 19 02:23:47:  [<c01a6b8a>] nfsd3_proc_readdirplus+0xda/0x1d0
May 19 02:23:47:  [<c01a9ed0>] nfs3svc_encode_entry_plus+0x0/0x50
May 19 02:23:47:  [<c01a8e90>] nfs3svc_decode_readdirplusargs+0x0/0x180
May 19 02:23:47:  [<c019bce6>] nfsd_dispatch+0xb6/0x1a0
May 19 02:23:47:  [<c03139fd>] svc_authenticate+0x4d/0x80
May 19 02:23:47:  [<c03112d7>] svc_process+0x487/0x5f0
May 19 02:23:47:  [<c0103b6c>] common_interrupt+0x18/0x20
May 19 02:23:47:  [<c019bad9>] nfsd+0x179/0x2d0
May 19 02:23:47:  [<c019b960>] nfsd+0x0/0x2d0
May 19 02:23:47:  [<c0101fbd>] kernel_thread_helper+0x5/0x18
May 19 02:23:47: 
May 19 02:23:47: Code: 89 06 89 c8 0f c8 89 46 04 81 7c 24 1c 00 01 00 00 b8 ff 00 

Which later turned into a reboot-requiring cascade of these:

May 19 06:29:57:  <4>qmail-rspawn: page allocation failure. order:0, mode:0x20
May 19 06:29:57: Call Trace:
May 19 06:29:57:  [<c0127395>] __alloc_pages+0x2d5/0x330
May 19 06:29:57:  [<c012740f>] __get_free_pages+0x1f/0x40
May 19 06:29:57:  [<c012a961>] cache_grow+0x91/0x270
May 19 06:29:57:  [<c012ae3e>] cache_alloc_refill+0x2fe/0x400
May 19 06:29:57:  [<c01261a7>] mempool_alloc+0x67/0x110
May 19 06:29:57:  [<c012b520>] kmem_cache_alloc+0x1a0/0x1b0
May 19 06:29:57:  [<c020dad8>] submit_bio+0x58/0xf0
May 19 06:29:57:  [<c01d1490>] radix_tree_node_alloc+0x10/0x50
May 19 06:29:57:  [<c01d16f9>] radix_tree_insert+0xe9/0x110
May 19 06:29:57:  [<c0138a25>] __add_to_swap_cache+0x55/0xc0
May 19 06:29:57:  [<c0138bbe>] add_to_swap+0x4e/0xb0
May 19 06:29:57:  [<c012deef>] shrink_list+0x3af/0x3e0
May 19 06:29:57:  [<c012e065>] shrink_cache+0x145/0x300
May 19 06:29:57:  [<c012e7a9>] shrink_caches+0x79/0x80
May 19 06:29:57:  [<c012e851>] try_to_free_pages+0xa1/0x180
May 19 06:29:57:  [<c012728c>] __alloc_pages+0x1cc/0x330
May 19 06:29:57:  [<c014515e>] <4>qmail-rspawn: page allocation failure. order:0, mode:0x20
May 19 06:29:57: Call Trace:
May 19 06:29:57:  [<c0127395>] __alloc_pages+0x2d5/0x330
May 19 06:29:57:  [<c012740f>] __get_free_pages+0x1f/0x40
May 19 06:29:57:  [<c012a961>] cache_grow+0x91/0x270
May 19 06:29:57:  [<c012ae3e>] cache_alloc_refill+0x2fe/0x400
May 19 06:29:57:  [<c012b520>] kmem_cache_alloc+0x1a0/0x1b0
May 19 06:29:57:  [<c029dce2>] dst_alloc+0x22/0x90
May 19 06:29:57:  [<c02b7cee>] ip_route_input_slow+0x47e/0x800
May 19 06:29:57:  [<c0253a38>] lba_28_rw_disk+0x88/0x90
May 19 06:29:57:  [<c02dc0e8>] arp_process+0x1e8/0x4c0
May 19 06:29:57:  [<c0303c79>] packet_rcv+0x189/0x2f0
May 19 06:29:57:  [<c029a2b0>] netif_receive_skb+0x150/0x180
May 19 06:29:57:  [<c023f3ac>] tulip_poll+0x44c/0x660
May 19 06:29:57:  [<c029a44c>] net_rx_action+0x6c/0xf0
May 19 06:29:57:  [<c0115b09>] __do_softirq+0x79/0x80
May 19 06:29:57:  [<c0115b37>] do_softirq+0x27/0x30
May 19 06:29:57:  [<c0105525>] do_IRQ+0xd5/0x110
May 19 06:29:57:  [<c0103b6c>] common_interrupt+0x18/0x20
May 19 06:29:57:  [<c012007b>] param_get_long+0x1b/0x30
May 19 06:29:57:  [<c0122ebc>] kallsyms_lookup+0xdc/0x180
May 19 06:29:57:  [<c014515e>] copy_strings+0x1be/0x1f0
May 19 06:29:57:  [<c014515e>] copy_strings+0x1be/0x1f0
May 19 06:29:57:  [<c0122f9a>] __print_symbol+0x3a/0x170
May 19 06:29:57:  [<c01ef150>] complement_pos+0x10/0x160
May 19 06:29:57:  [<c01ef150>] complement_pos+0x10/0x160
May 19 06:29:57:  [<c014515e>] copy_strings+0x1be/0x1f0
May 19 06:29:57:  [<c0103e81>] show_trace+0xb1/0xc0
May 19 06:29:57:  [<c014515e>] copy_strings+0x1be/0x1f0
May 19 06:29:57:  [<c0103f33>] dump_stack+0x13/0x20
May 19 06:29:57:  [<c0127395>] __alloc_pages+0x2d5/0x330
May 19 06:29:57:  [<c012740f>] __get_free_pages+0x1f/0x40
May 19 06:29:57:  [<c012a961>] cache_grow+0x91/0x270
May 19 06:29:57:  [<c012ae3e>] cache_alloc_refill+0x2fe/0x400
May 19 06:29:57:  [<c01261a7>] mempool_alloc+0x67/0x110
May 19 06:29:57:  [<c012b520>] kmem_cache_alloc+0x1a0/0x1b0
May 19 06:29:57:  [<c020dad8>] submit_bio+0x58/0xf0
May 19 06:29:57:  [<c01d1490>] radix_tree_node_alloc+0x10/0x50
May 19 06:29:57:  [<c01d16f9>] radix_tree_insert+0xe9/0x110
May 19 06:29:57:  [<c0138a25>] __add_to_swap_cache+0x55/0xc0
May 19 06:29:57:  [<c0138bbe>] add_to_swap+0x4e/0xb0
May 19 06:29:57:  [<c012deef>] shrink_list+0x3af/0x3e0
May 19 06:29:57:  [<c012e065>] shrink_cache+0x145/0x300
May 19 06:29:57:  [<c012e7a9>] shrink_caches+0x79/0x80
May 19 06:29:57:  [<c012e851>] try_to_free_pages+0xa1/0x180
May 19 06:29:57:  [<c012728c>] __alloc_pages+0x1cc/0x330
May 19 06:29:57:  [<c014515e>] copy_strings+0x1be/0x1f0
May 19 06:29:57:  [<c01451b0>] copy_strings_kernel+0x20/0x30
May 19 06:29:57:  [<c0146125>] do_execve+0x125/0x200
May 19 06:29:57:  [<c0102775>] sys_execve+0x35/0x70
May 19 06:29:57:  [<c01039ff>] syscall_call+0x7/0xb
May 19 06:29:57: 

(Similar repetitions ad nauseam; write if you want them all.)


The memory map is pretty full, as it's a 1GB machine run with a regular
3+1 split.  /proc/iomem is as follows:

00000000-0009e7ff : System RAM
0009e800-0009ffff : reserved
000a0000-000bffff : Video RAM area
000c0000-000cb3ff : Video ROM
000cc000-000cc7ff : Adapter ROM
000d0000-000d1fff : Adapter ROM
000f0000-000fffff : System ROM
00100000-3fffbfff : System RAM
  00100000-0031b833 : Kernel code
  0031b834-003f5ebf : Kernel data
3fffc000-3fffefff : ACPI Tables
3ffff000-3fffffff : ACPI Non-volatile Storage
40000000-40000fff : 0000:00:09.0
  40000000-40000fff : yenta_socket
40001000-40001fff : 0000:00:09.1
  40001000-40001fff : yenta_socket
40400000-407fffff : PCI CardBus #03
40800000-40bfffff : PCI CardBus #03
40c00000-40ffffff : PCI CardBus #07
41000000-413fffff : PCI CardBus #07
de800000-e07fffff : PCI Bus #02
  de800000-de8003ff : 0000:02:07.0
    de800000-de8003ff : tulip
  df000000-df0003ff : 0000:02:06.0
    df000000-df0003ff : tulip
  df800000-df8003ff : 0000:02:05.0
    df800000-df8003ff : tulip
  e0000000-e00003ff : 0000:02:04.0
    e0000000-e00003ff : tulip
e0800000-e0803fff : 0000:00:0c.0
e1000000-e1003fff : 0000:00:0b.0
e1800000-e1800fff : 0000:00:0a.0
e2000000-e3cfffff : PCI Bus #01
  e2000000-e2ffffff : 0000:01:00.0
e3d00000-e3dfffff : PCI Bus #02
e3f00000-e5ffffff : PCI Bus #01
  e4000000-e5ffffff : 0000:01:00.0
e6000000-e7ffffff : 0000:00:00.0
ffff0000-ffffffff : reserved

Re: 2.6.6 is crashing repeatedly

[Andrew Morton]

linux@horizon.com wrote:
>
> I already reported (I thought) this message:
> 
>  May 19 02:23:47: Unable to handle kernel paging request at virtual address efd78000
>  May 19 02:23:47:  printing eip:
>  May 19 02:23:47: c01a999b
>  May 19 02:23:47: *pde = 004a7067
>  May 19 02:23:47: *pte = 2fd78000
>  May 19 02:23:47: Oops: 0002 [#1]
>  May 19 02:23:47: DEBUG_PAGEALLOC
>  May 19 02:23:47: CPU:    0
>  May 19 02:23:47: EIP:    0060:[<c01a999b>]    Not tainted
>  May 19 02:23:47: EFLAGS: 00010246   (2.6.6) 
>  May 19 02:23:47: EIP is at encode_entry+0x4b/0x530
>  May 19 02:23:47: eax: 00000000   ebx: 00000000   ecx: 00000644   edx: f3532df8
>  May 19 02:23:47: esi: efd78000   edi: e4a19644   ebp: 00000654   esp: f14b7b98
>  May 19 02:23:47: ds: 007b   es: 007b   ss: 0068
>  May 19 02:23:47: Process nfsd (pid: 1029, threadinfo=f14b6000 task=f1532a80)
>  May 19 02:23:47: Stack: 00000008 0000002f f5217080 00000027 00000000 f5217084 f3532df8 00000006 
>  May 19 02:23:47:        e4a1964c 0000001c 02000001 04000900 0019a291 0022e7a8 14343bfc 0022e76b 
>  May 19 02:23:47:        14343b61 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
>  May 19 02:23:47: Call Trace:
>  May 19 02:23:47:  [<c016af1f>] ext3_bread+0x1f/0x90
>  May 19 02:23:47:  [<c0168505>] ext3_readdir+0x305/0x490
>  May 19 02:23:47:  [<c011dd18>] in_group_p+0x38/0x70
>  May 19 02:23:47:  [<c01a9ed0>] nfs3svc_encode_entry_plus+0x0/0x50
>  May 19 02:23:47:  [<c019d930>] fh_verify+0x280/0x550
>  May 19 02:23:47:  [<c019d5d0>] nfsd_acceptable+0x0/0xe0
>  May 19 02:23:47:  [<c013da8c>] open_private_file+0x1c/0x90
>  May 19 02:23:47:  [<c014ba2d>] vfs_readdir+0x7d/0x90
>  May 19 02:23:47:  [<c01a11f9>] nfsd_readdir+0x79/0xc0
>  May 19 02:23:47:  [<c01a6b8a>] nfsd3_proc_readdirplus+0xda/0x1d0
>  May 19 02:23:47:  [<c01a9ed0>] nfs3svc_encode_entry_plus+0x0/0x50
>  May 19 02:23:47:  [<c01a8e90>] nfs3svc_decode_readdirplusargs+0x0/0x180
>  May 19 02:23:47:  [<c019bce6>] nfsd_dispatch+0xb6/0x1a0
>  May 19 02:23:47:  [<c03139fd>] svc_authenticate+0x4d/0x80
>  May 19 02:23:47:  [<c03112d7>] svc_process+0x487/0x5f0
>  May 19 02:23:47:  [<c0103b6c>] common_interrupt+0x18/0x20
>  May 19 02:23:47:  [<c019bad9>] nfsd+0x179/0x2d0
>  May 19 02:23:47:  [<c019b960>] nfsd+0x0/0x2d0
>  May 19 02:23:47:  [<c0101fbd>] kernel_thread_helper+0x5/0x18
>  May 19 02:23:47: 
>  May 19 02:23:47: Code: 89 06 89 c8 0f c8 89 46 04 81 7c 24 1c 00 01 00 00 b8 ff 00 

Is it repeatable?  If so, does it go away if you disable
CONFIG_DEBUG_PAGEALLOC?

>  Which later turned into a reboot-requiring cascade of these:
> 
>  May 19 06:29:57:  <4>qmail-rspawn: page allocation failure. order:0, mode:0x20
>  May 19 06:29:57: Call Trace:
>  May 19 06:29:57:  [<c0127395>] __alloc_pages+0x2d5/0x330
>  May 19 06:29:57:  [<c012740f>] __get_free_pages+0x1f/0x40
>  May 19 06:29:57:  [<c012a961>] cache_grow+0x91/0x270
>  May 19 06:29:57:  [<c012ae3e>] cache_alloc_refill+0x2fe/0x400
>  May 19 06:29:57:  [<c01261a7>] mempool_alloc+0x67/0x110

hm, you seem to be totally out of memory in the swapout path and in the
tulip Rx ISR.  Both codepaths handle that OK as it's not uncommon.  This is
probably unrelated to the oops, but it's not good.

Re: 2.6.6 is crashing repeatedly

[Neil Brown]

On Wednesday May 19, akpm@osdl.org wrote:
> linux@horizon.com wrote:
> >
> > I already reported (I thought) this message:
> > 
> >  May 19 02:23:47: Unable to handle kernel paging request at virtual address efd78000

> >  May 19 02:23:47: EIP is at encode_entry+0x4b/0x530
> >  May 19 02:23:47: eax: 00000000   ebx: 00000000   ecx: 00000644   edx: f3532df8
> >  May 19 02:23:47: esi: efd78000   edi: e4a19644   ebp: 00000654   esp: f14b7b98

> >  May 19 02:23:47: 
> >  May 19 02:23:47: Code: 89 06 89 c8 0f c8 89 46 04 81 7c 24 1c 00 01 00 00 b8 ff 00 
> 

cd->offset is 0xefd78000, which is the start of a, presumably unused, page.

Yes... this patch should fix it.

Thanks for the report.

NeilBrown

===============================================================
Fix NFSD oops in readdir.

If a single readdir entry needs to be split over two pages in
the reply, we first encode it into a new page, and then copy the
bits into place.  When we do this relocation, we have to modify
the "offset" pointer to be either in the first or the second page,
as appropriate.

If the pointer should be at the start of the second page, it is currently
put past the end of the first page.

Note that as the offset and whole response is known to be 4byte-aligned,
the offset pointer will never be split over two pages.


 ----------- Diffstat output ------------
 ./fs/nfsd/nfs3xdr.c |    2 +-
 1 files changed, 1 insertion(+), 1 deletion(-)

diff ./fs/nfsd/nfs3xdr.c~current~ ./fs/nfsd/nfs3xdr.c
--- ./fs/nfsd/nfs3xdr.c~current~	2004-05-20 15:10:15.000000000 +1000
+++ ./fs/nfsd/nfs3xdr.c	2004-05-20 15:22:23.000000000 +1000
@@ -936,7 +936,7 @@ encode_entry(struct readdir_cd *ccd, con
 			memmove(tmp, (caddr_t)tmp+len1, len2);
 
 			/* update offset */
-			if (((cd->offset - tmp) << 2) <= len1)
+			if (((cd->offset - tmp) << 2) < len1)
 				cd->offset = p + (cd->offset - tmp);
 			else
 				cd->offset -= len1 >> 2;

Re: 2.6.6 is crashing repeatedly

[linux]

> Yes... this patch should fix it.
>
> Thanks for the report.

Thanks even more for the fix!

This is indeed playing NFS server to a herd of Suns.

Compiling now.  I'll report again in a week or two if the crashes
stop.  (Sooner if they don't, of course.)

Re: 2.6.6 is crashing repeatedly

[linux]

Even with neilb's patch, I just got an nfs oops:
Unable to handle kernel paging request at virtual address f2590000
 printing eip:
c01a99a1
*pde = 004b1067
*pte = 32590000
Oops: 0002 [#1]
DEBUG_PAGEALLOC
CPU:    0
EIP:    0060:[<c01a99a1>]    Not tainted
EFLAGS: 00010246   (2.6.6) 
EIP is at encode_entry+0x51/0x530
eax: cc010000   ebx: 00000000   ecx: 000001cc   edx: f32dcdf8
esi: f258fffc   edi: d4ec21cc   ebp: 000001e0   esp: ebfd9b98
ds: 007b   es: 007b   ss: 0068
Process nfsd (pid: 1059, threadinfo=ebfd8000 task=ec000a80)
Stack: 00000006 0000002f e2c63080 00000027 00000000 e2c63080 f32dcdf8 0000000a 
       d4ec21d4 0000001c 02000001 04000900 002bed40 0019a454 00249005 0019a43b 
       00248f88 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
Call Trace:
 [<c016af69>] ext3_bread+0x69/0x90
 [<c0168505>] ext3_readdir+0x305/0x490
 [<c011dba7>] set_current_groups+0x37/0x40
 [<c01a9ed0>] nfs3svc_encode_entry_plus+0x0/0x50
 [<c019d930>] fh_verify+0x280/0x550
 [<c019d5d0>] nfsd_acceptable+0x0/0xe0
 [<c013da8c>] open_private_file+0x1c/0x90
 [<c014ba2d>] vfs_readdir+0x7d/0x90
 [<c01a11f9>] nfsd_readdir+0x79/0xc0
 [<c01a6b8a>] nfsd3_proc_readdirplus+0xda/0x1d0
 [<c01a9ed0>] nfs3svc_encode_entry_plus+0x0/0x50
 [<c01a8e90>] nfs3svc_decode_readdirplusargs+0x0/0x180
 [<c019bce6>] nfsd_dispatch+0xb6/0x1a0
 [<c03139fd>] svc_authenticate+0x4d/0x80
 [<c03112d7>] svc_process+0x487/0x5f0
 [<c019bad9>] nfsd+0x179/0x2d0
 [<c019b960>] nfsd+0x0/0x2d0
 [<c0101fbd>] kernel_thread_helper+0x5/0x18

Code: 89 46 04 81 7c 24 1c 00 01 00 00 b8 ff 00 00 00 8b 94 24 24 

This is causing NFS client hangs.  Bugger; time to reboot.

Re: 2.6.6 is crashing repeatedly

[Neil Brown]

On  May 27, linux@horizon.com wrote:
> Even with neilb's patch, I just got an nfs oops:

As Olaf Kirch just said on nfs@lists.sourceforge.net:

> Hi Neil,
> 
> you recently posted a patch that should fix readdir encoding in
> nfsd. You say there
> 
> 	Note that as the offset and whole response is known to be
> 	4byte-aligned, the offset pointer will never be split over
> 	two pages.
> 
> This is not true. The dirent offset is a 64bit quantity, so it's quite
> possible it will be split across the page boundary. I'm working on a
> patch...

And that is exactly the problem you have hit:

> Unable to handle kernel paging request at virtual address f2590000
>  printing eip:
> c01a99a1
> *pde = 004b1067
> *pte = 32590000
> Oops: 0002 [#1]
> DEBUG_PAGEALLOC
> CPU:    0
> EIP:    0060:[<c01a99a1>]    Not tainted
> EFLAGS: 00010246   (2.6.6) 
> EIP is at encode_entry+0x51/0x530
> eax: cc010000   ebx: 00000000   ecx: 000001cc   edx: f32dcdf8
> esi: f258fffc   edi: d4ec21cc   ebp: 000001e0   esp: ebfd9b98

Note that "esi" is pointing to 4 bytes from the end of a page, and you
are getting a bad reference at the start of the next page.
The code is storing a 64bit value here, and it doesn't fit.

Stay tuned....

NeilBrown

Re: 2.6.6 is crashing repeatedly

[Olaf Kirch]

[-- Attachment #1: Type: text/plain, Size: 383 bytes --]

> > This is not true. The dirent offset is a 64bit quantity, so it's quite
> > possible it will be split across the page boundary. I'm working on a
> > patch...

Here's a patch that hopefully fixes this problem. Please give it
a try and let me know.

Neil, does this look okay to you?

Cheers,
Olaf
-- 
Olaf Kirch     |  The Hardware Gods hate me.
okir@suse.de   |
---------------+ 

[-- Attachment #2: nfsd-encode-dirent3 --]
[-- Type: text/plain, Size: 2757 bytes --]

Index: linux-2.6.5/fs/nfsd/nfs3proc.c
===================================================================
--- linux-2.6.5.orig/fs/nfsd/nfs3proc.c	2004-05-27 12:22:43.000000000 +0200
+++ linux-2.6.5/fs/nfsd/nfs3proc.c	2004-05-27 12:34:40.000000000 +0200
@@ -437,7 +437,6 @@
 	resp->buflen = count;
 	resp->common.err = nfs_ok;
 	resp->buffer = argp->buffer;
-	resp->offset = NULL;
 	resp->rqstp = rqstp;
 	nfserr = nfsd_readdir(rqstp, &resp->fh, (loff_t*) &argp->cookie, 
 					&resp->common, nfs3svc_encode_entry);
Index: linux-2.6.5/fs/nfsd/nfs3xdr.c
===================================================================
--- linux-2.6.5.orig/fs/nfsd/nfs3xdr.c	2004-05-27 12:22:43.000000000 +0200
+++ linux-2.6.5/fs/nfsd/nfs3xdr.c	2004-05-27 12:39:09.000000000 +0200
@@ -887,8 +887,18 @@
 	int		elen;		/* estimated entry length in words */
 	int		num_entry_words = 0;	/* actual number of words */
 
-	if (cd->offset)
-		xdr_encode_hyper(cd->offset, (u64) offset);
+	if (cd->offset) {
+		u64 offset64 = offset;
+
+		if (unlikely(cd->offset1)) {
+			/* we ended up with offset on a page boundary */
+			*cd->offset = htonl(offset64 >> 32);
+			*cd->offset1 = htonl(offset64 & 0xffffffff);
+			cd->offset1 = NULL;
+		} else {
+			xdr_encode_hyper(cd->offset, (u64) offset);
+		}
+	}
 
 	/*
 	dprintk("encode_entry(%.*s @%ld%s)\n",
@@ -969,17 +979,32 @@
 			/* update offset */
 			cd->offset = cd->buffer + (cd->offset - tmp);
 		} else {
+			unsigned int offset_r = (cd->offset - tmp) << 2;
+
+			/* update pointer to offset location.
+			 * This is a 64bit quantity, so we need to
+			 * deal with 3 cases:
+			 *  -	entirely in first page
+			 *  -	entirely in second page
+			 *  -	4 bytes in each page
+			 */
+			if (offset_r + 8 <= len1) {
+				cd->offset = p + (cd->offset - tmp);
+			} else if (offset_r >= len1) {
+				cd->offset -= len1 >> 2;
+			} else {
+				/* sitting on the fence */
+				BUG_ON(offset_r != len1 - 4);
+				cd->offset = p + (cd->offset - tmp);
+				cd->offset1 = tmp;
+			}
+
 			len2 = (num_entry_words << 2) - len1;
 
 			/* move from temp page to current and next pages */
 			memmove(p, tmp, len1);
 			memmove(tmp, (caddr_t)tmp+len1, len2);
 
-			/* update offset */
-			if (((cd->offset - tmp) << 2) <= len1)
-				cd->offset = p + (cd->offset - tmp);
-			else
-				cd->offset -= len1 >> 2;
 			p = tmp + (len2 >> 2);
 		}
 	}
Index: linux-2.6.5/include/linux/nfsd/xdr3.h
===================================================================
--- linux-2.6.5.orig/include/linux/nfsd/xdr3.h	2004-05-27 12:22:43.000000000 +0200
+++ linux-2.6.5/include/linux/nfsd/xdr3.h	2004-05-27 12:28:20.000000000 +0200
@@ -183,6 +183,7 @@
 	u32 *			buffer;
 	int			buflen;
 	u32 *			offset;
+	u32 *			offset1;
 	struct svc_rqst *	rqstp;
 
 };

Re: 2.6.6 is crashing repeatedly

[Neil Brown]

On Thursday May 27, okir@suse.de wrote:
> > > This is not true. The dirent offset is a 64bit quantity, so it's quite
> > > possible it will be split across the page boundary. I'm working on a
> > > patch...
> 
> Here's a patch that hopefully fixes this problem. Please give it
> a try and let me know.
> 
> Neil, does this look okay to you?

Yes, it looks fine.

I'll rediff against 2.6.7-rc1-mm1 (there is a small conflict) and send
it to Andrew.

Thanks,
NeilBrown

Re: 2.6.6 is crashing repeatedly

[John Heil]

On Fri, 14 May 2004, Andrew Morton wrote:

> Date: Fri, 14 May 2004 23:28:42 -0700
> From: Andrew Morton <akpm@osdl.org>
> To: linux@horizon.com
> Cc: linux-kernel@vger.kernel.org
> Subject: Re: 2.6.6 is crashing repeatedly
>
> linux@horizon.com wrote:
> >
> >  I have now captured a kernel crash.  Everything after iput in the second crash
> >  was hand-coped, and may suffer from transcription errors, but it was done
> >  quite carefully.
> >
> >  System has ECC memory and has been very stable, with uptimes in excess of
> >  1 year when kernel upgrades were infrequent (2.5 development).
> >
> >  Stock 2.6.6 kernel, config as posted before.
> >
> >  Unable to handle kernel NULL pointer dereference at virtual address 00000004
> >   printing eip:
> >  c012a392
> >  *pde = 00000000
> >  Oops: 0002 [#1]
> >  CPU:    0
> >  EIP:    0060:[<c012a392>]    Not tainted
> >  EFLAGS: 00010012   (2.6.6)
> >  EIP is at free_block+0x52/0xd0
> >  eax: 00000000   ebx: e9a3f000   ecx: e9a3f200   edx: df654000
> >  esi: f7f8a560   edi: 00000016   ebp: f7f8a56c   esp: f7d89dec
> >  ds: 007b   es: 007b   ss: 0068
> >  Process kswapd0 (pid: 8, threadinfo=f7d88000 task=f7d8eb50)
> >  Stack: f7f8a57c 0000001b c17fd784 c17fd784 f7f8a560 dc3cdac0 0000001b c012a449
> >         f7fe73dc c17fd774 c17fd774 00000296 dc3cdac0 c037a304 c012a61a dc3cdb40
> >         f7d89e5c 0000003d c014f385 dc3cdb40 c014f5c3 dc19c0c8 dc19c0c0 00000080
> >  Call Trace:
> >   [<c012a449>] cache_flusharray+0x39/0xc0
> >   [<c012a61a>] kmem_cache_free+0x3a/0x50
> >   [<c014f385>] destroy_inode+0x35/0x40
> >   [<c014f5c3>] dispose_list+0x43/0x70
> >   [<c014f87e>] prune_icache+0xae/0x1b0
> >   [<c014f995>] shrink_icache_memory+0x15/0x20
>
> Drat, random memory corruption.
>

Interesting...

FWIW, a data point: I've been chasing memory corruption in 2.6.5-mm2.
(Although I suspect it pre-dates that level.)

The first 2K of a DMA page is overlayed w what looks like code
(even disassembles to something almost meaningful, looking like
interrupt handling code ie w several iret's. Althought I've yet to
find where exactly it's from.) I obtain dma mem from the page
via dma_pool_alloc, having created the pool w dma_pool_create.
The hi 2k of the page w the overlay, remains in tact.

I've been running w CONFIG_SLAB_DEBUG  And CONFIG_DEBUG_PAGEALLOC
usually to no avail. However I do sometimes find freed memory
0xA7 poison words occasionally involved. I temporarily commented out
the dma_pool_free that would release the memory and I still get the
2K overlay.

johnh

> Can you enable CONFIG_SLAB_DEBUG?  And CONFIG_DEBUG_PAGEALLOC too, although
> beware that the latter is a bit costly in terms of CPU cycles and memory
> usage.
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

-
-----------------------------------------------------------------
John Heil
South Coast Software
Custom systems software for UNIX and IBM MVS mainframes
1-714-774-6952
johnhscs@sc-software.com
http://www.sc-software.com
-----------------------------------------------------------------

Re: 2.6.6 is crashing repeatedly

[linux]

> Can you enable CONFIG_SLAB_DEBUG?  And CONFIG_DEBUG_PAGEALLOC too, although
> beware that the latter is a bit costly in terms of CPU cycles and memory
> usage.

Just rebooted with
CONFIG_DEBUG_KERNEL=y
CONFIG_DEBUG_STACKOVERFLOW=y
CONFIG_DEBUG_SLAB=y
CONFIG_DEBUG_PAGEALLOC=y

More news when something happens.

Thanks for your instant diagnosis!

Re: 2.6.6 is crashing repeatedly

[Andrew Morton]

linux@horizon.com wrote:
>
>  I have now captured a kernel crash.  Everything after iput in the second crash
>  was hand-coped, and may suffer from transcription errors, but it was done
>  quite carefully.
> 
>  System has ECC memory and has been very stable, with uptimes in excess of
>  1 year when kernel upgrades were infrequent (2.5 development).
> 
>  Stock 2.6.6 kernel, config as posted before.
> 
>  Unable to handle kernel NULL pointer dereference at virtual address 00000004
>   printing eip:
>  c012a392
>  *pde = 00000000
>  Oops: 0002 [#1]
>  CPU:    0
>  EIP:    0060:[<c012a392>]    Not tainted
>  EFLAGS: 00010012   (2.6.6) 
>  EIP is at free_block+0x52/0xd0
>  eax: 00000000   ebx: e9a3f000   ecx: e9a3f200   edx: df654000
>  esi: f7f8a560   edi: 00000016   ebp: f7f8a56c   esp: f7d89dec
>  ds: 007b   es: 007b   ss: 0068
>  Process kswapd0 (pid: 8, threadinfo=f7d88000 task=f7d8eb50)
>  Stack: f7f8a57c 0000001b c17fd784 c17fd784 f7f8a560 dc3cdac0 0000001b c012a449 
>         f7fe73dc c17fd774 c17fd774 00000296 dc3cdac0 c037a304 c012a61a dc3cdb40 
>         f7d89e5c 0000003d c014f385 dc3cdb40 c014f5c3 dc19c0c8 dc19c0c0 00000080 
>  Call Trace:
>   [<c012a449>] cache_flusharray+0x39/0xc0
>   [<c012a61a>] kmem_cache_free+0x3a/0x50
>   [<c014f385>] destroy_inode+0x35/0x40
>   [<c014f5c3>] dispose_list+0x43/0x70
>   [<c014f87e>] prune_icache+0xae/0x1b0
>   [<c014f995>] shrink_icache_memory+0x15/0x20

Drat, random memory corruption.

Can you enable CONFIG_SLAB_DEBUG?  And CONFIG_DEBUG_PAGEALLOC too, although
beware that the latter is a bit costly in terms of CPU cycles and memory
usage.

Re: 2.6.6 is crashing repeatedly

[linux]

[All this time and I still havitually send e-mail to vger.rutgers.edu...]

Sorry for the delay in getting this.

I have now captured a kernel crash.  Everything after iput in the second crash
was hand-coped, and may suffer from transcription errors, but it was done
quite carefully.

System has ECC memory and has been very stable, with uptimes in excess of
1 year when kernel upgrades were infrequent (2.5 development).

Stock 2.6.6 kernel, config as posted before.

Unable to handle kernel NULL pointer dereference at virtual address 00000004
 printing eip:
c012a392
*pde = 00000000
Oops: 0002 [#1]
CPU:    0
EIP:    0060:[<c012a392>]    Not tainted
EFLAGS: 00010012   (2.6.6) 
EIP is at free_block+0x52/0xd0
eax: 00000000   ebx: e9a3f000   ecx: e9a3f200   edx: df654000
esi: f7f8a560   edi: 00000016   ebp: f7f8a56c   esp: f7d89dec
ds: 007b   es: 007b   ss: 0068
Process kswapd0 (pid: 8, threadinfo=f7d88000 task=f7d8eb50)
Stack: f7f8a57c 0000001b c17fd784 c17fd784 f7f8a560 dc3cdac0 0000001b c012a449 
       f7fe73dc c17fd774 c17fd774 00000296 dc3cdac0 c037a304 c012a61a dc3cdb40 
       f7d89e5c 0000003d c014f385 dc3cdb40 c014f5c3 dc19c0c8 dc19c0c0 00000080 
Call Trace:
 [<c012a449>] cache_flusharray+0x39/0xc0
 [<c012a61a>] kmem_cache_free+0x3a/0x50
 [<c014f385>] destroy_inode+0x35/0x40
 [<c014f5c3>] dispose_list+0x43/0x70
 [<c014f87e>] prune_icache+0xae/0x1b0
 [<c014f995>] shrink_icache_memory+0x15/0x20
 [<c012bfa9>] shrink_slab+0x129/0x180
 [<c012d0b3>] balance_pgdat+0x1f3/0x260
 [<c012d20e>] kswapd+0xee/0xf0
 [<c0110e30>] autoremove_wake_function+0x0/0x40
 [<c0110e30>] autoremove_wake_function+0x0/0x40
 [<c012d120>] kswapd+0x0/0xf0
 [<c0101fbd>] kernel_thread_helper+0x5/0x18

Code: 89 50 04 89 02 8b 43 0c 31 d2 c7 03 00 01 10 00 c7 43 04 00 
 <1>Unable to handle kernel paging request at virtual address 00200204
 printing eip:
c012a38d
*pde = 00000000
Oops: 0000 [#2]
CPU:    0
EIP:    0060:[<c012a38d>]    Not tainted
EFLAGS: 00010012   (2.6.6) 
EIP is at free_block+0x4d/0xd0
eax: 00383380   ebx: 00200200   ecx: dc19cac0   edx: c1000000
esi: f7f8a560   edi: 00000000   ebp: f7f8a56c   esp: f42f5f20
ds: 007b   es: 007b   ss: 0068
Process qmail-clean (pid: 1027, threadinfo=f42f4000 task=f433c6f0)
Stack: f7f8a57c 0000001b c17fd784 c17fd784 f7f8a560 da337200 0000001b c012a449 
       f7fe73dc c17fd774 c17fd774 00000296 da337200 f42f5f80 c012a61a da337280 
       00000000 da337280 c014f385 da337280 c01501ac f6e4f000 c0147f44 ec9fc8b8 
Call Trace:
 [<c012a449>] cache_flusharray+0x39/0xc0
 [<c012a61a>] kmem_cache_free+0x3a/0x50
 [<c014f385>] destroy_inode+0x35/0x40
 [<c01501ac>] iput+0x4c/0x60
 [<c0147f44>] sys_unlink+0xf4/0x120
 [<c013b538>] sys_read+0x38/0x60
 [<c01039ff>] syscall_call+0x7/0xb

Code: 8b 53 04 8b 03 89 50 04 89 02 86 43 0c 31 d2 c7 03 00 01 10
 <1>Unable to handle kernel paging request at virtual address 00200204

<cascading errors not copied, ending with>

Kernel panic: Fatal exception in interrupt

2.6.6 is crashing repeatedly

[linux]

I just upgraded a server from 2.6.3 (which was stable) to 2.6.6 on
Tuesday night.  In the last 3 days, it has crashed twice, both
times around 02:45.

Basically:
Wednesday morning at 02:45 (less than 4 hours after kernel ugrade) - crash.
	Person in the morning forgot to reboot to the old kernel, so we
	left 2.6.6. running some more for a while.
Thursday morning - no crash.  Interesting.
Friday morning: kaboom.  Thus this report.

The only thing I can find in the crontab is the multi-system Amanda backup
process kicks off nightly at 02:00, and for those who don't know, it
starts with a fairly long phase of computing the size of compressed level-N
backups on each of the associated file systems and various values of N
before deciding which combination will fit on the backup tape and starting the
backup in earnest.

Notably the directory where Amanda stages the network backup images has been
empty both times.

I'm sory I don't have the oops logs or any other meaningful error-tracing
info, but the morning people have been rebooting the server as fast as
possible to get work done.  I'll go run some backups by hand to see if
I can trigger the problem at a more convenient time and capture enough
data to debug it.

I just thought I'd at least mention it in case anyone else is seeing
instability (the association with the kernel upgrade is just too strong)
to bring some reports to the front.  I haven't seen any other reports, but
it's hard to imageine I'm the only one who's managed to trigger it.

2.6.6. has been stable on my home machine where I test new Debian packages
and kernels before upgrading the servers.  In particular, I've been
testing regparm=y for quite a while before enabling it.

Some info:

=== grep ^CONFIG .config
CONFIG_X86=y
CONFIG_MMU=y
CONFIG_UID16=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_EXPERIMENTAL=y
CONFIG_CLEAN_COMPILE=y
CONFIG_STANDALONE=y
CONFIG_BROKEN_ON_SMP=y
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_SYSCTL=y
CONFIG_LOG_BUF_SHIFT=15
CONFIG_HOTPLUG=y
CONFIG_IKCONFIG=y
CONFIG_KALLSYMS=y
CONFIG_FUTEX=y
CONFIG_EPOLL=y
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_AS=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=y
CONFIG_X86_PC=y
CONFIG_MPENTIUMII=y
CONFIG_X86_CMPXCHG=y
CONFIG_X86_XADD=y
CONFIG_X86_L1_CACHE_SHIFT=5
CONFIG_RWSEM_XCHGADD_ALGORITHM=y
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_INVLPG=y
CONFIG_X86_BSWAP=y
CONFIG_X86_POPAD_OK=y
CONFIG_X86_GOOD_APIC=y
CONFIG_X86_INTEL_USERCOPY=y
CONFIG_X86_USE_PPRO_CHECKSUM=y
CONFIG_X86_UP_APIC=y
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_TSC=y
CONFIG_X86_MCE=y
CONFIG_X86_MSR=y
CONFIG_NOHIGHMEM=y
CONFIG_MTRR=y
CONFIG_REGPARM=y
CONFIG_ACPI_BOOT=y
CONFIG_PCI=y
CONFIG_PCI_GOANY=y
CONFIG_PCI_BIOS=y
CONFIG_PCI_DIRECT=y
CONFIG_PCI_MMCONFIG=y
CONFIG_PCI_NAMES=y
CONFIG_ISA=y
CONFIG_PCMCIA=y
CONFIG_PCMCIA_DEBUG=y
CONFIG_YENTA=y
CONFIG_CARDBUS=y
CONFIG_PCMCIA_PROBE=y
CONFIG_BINFMT_ELF=y
CONFIG_PARPORT=y
CONFIG_PARPORT_PC=y
CONFIG_PARPORT_PC_CML1=y
CONFIG_PARPORT_PC_FIFO=y
CONFIG_PARPORT_PC_SUPERIO=y
CONFIG_PARPORT_1284=y
CONFIG_BLK_DEV_FD=y
CONFIG_BLK_DEV_LOOP=y
CONFIG_BLK_DEV_CRYPTOLOOP=y
CONFIG_IDE=y
CONFIG_BLK_DEV_IDE=y
CONFIG_BLK_DEV_IDEDISK=y
CONFIG_IDEDISK_MULTI_MODE=y
CONFIG_IDE_TASKFILE_IO=y
CONFIG_IDE_GENERIC=y
CONFIG_BLK_DEV_IDEPCI=y
CONFIG_IDEPCI_SHARE_IRQ=y
CONFIG_BLK_DEV_GENERIC=y
CONFIG_BLK_DEV_IDEDMA_PCI=y
CONFIG_IDEDMA_PCI_AUTO=y
CONFIG_BLK_DEV_ADMA=y
CONFIG_BLK_DEV_PIIX=y
CONFIG_BLK_DEV_PDC202XX_NEW=y
CONFIG_BLK_DEV_IDEDMA=y
CONFIG_IDEDMA_AUTO=y
CONFIG_SCSI=y
CONFIG_SCSI_PROC_FS=y
CONFIG_BLK_DEV_SD=y
CONFIG_CHR_DEV_ST=y
CONFIG_BLK_DEV_SR=y
CONFIG_CHR_DEV_SG=y
CONFIG_SCSI_CONSTANTS=y
CONFIG_SCSI_LOGGING=y
CONFIG_SCSI_BUSLOGIC=y
CONFIG_SCSI_OMIT_FLASHPOINT=y
CONFIG_SCSI_QLA2XXX=y
CONFIG_MD=y
CONFIG_BLK_DEV_MD=y
CONFIG_MD_RAID0=y
CONFIG_MD_RAID1=y
CONFIG_NET=y
CONFIG_PACKET=y
CONFIG_PACKET_MMAP=y
CONFIG_NETLINK_DEV=y
CONFIG_UNIX=y
CONFIG_NET_KEY=y
CONFIG_INET=y
CONFIG_IP_ADVANCED_ROUTER=y
CONFIG_IP_ROUTE_VERBOSE=y
CONFIG_NET_IPIP=y
CONFIG_SYN_COOKIES=y
CONFIG_INET_AH=y
CONFIG_INET_ESP=y
CONFIG_INET_IPCOMP=y
CONFIG_NETFILTER=y
CONFIG_IP_NF_CONNTRACK=y
CONFIG_IP_NF_FTP=y
CONFIG_IP_NF_IRC=y
CONFIG_IP_NF_AMANDA=y
CONFIG_IP_NF_QUEUE=y
CONFIG_IP_NF_IPTABLES=y
CONFIG_IP_NF_MATCH_LIMIT=y
CONFIG_IP_NF_MATCH_MARK=y
CONFIG_IP_NF_MATCH_RECENT=y
CONFIG_IP_NF_MATCH_ECN=y
CONFIG_IP_NF_MATCH_HELPER=y
CONFIG_IP_NF_MATCH_STATE=y
CONFIG_IP_NF_MATCH_CONNTRACK=y
CONFIG_IP_NF_FILTER=y
CONFIG_IP_NF_TARGET_REJECT=y
CONFIG_IP_NF_NAT=y
CONFIG_IP_NF_NAT_NEEDED=y
CONFIG_IP_NF_TARGET_MASQUERADE=y
CONFIG_IP_NF_TARGET_REDIRECT=y
CONFIG_IP_NF_NAT_IRC=y
CONFIG_IP_NF_NAT_FTP=y
CONFIG_IP_NF_NAT_AMANDA=y
CONFIG_IP_NF_MANGLE=y
CONFIG_IP_NF_TARGET_TOS=y
CONFIG_IP_NF_TARGET_ECN=y
CONFIG_IP_NF_TARGET_MARK=y
CONFIG_IP_NF_TARGET_LOG=y
CONFIG_IP_NF_TARGET_ULOG=y
CONFIG_IP_NF_TARGET_NOTRACK=y
CONFIG_IP_NF_RAW=y
CONFIG_XFRM=y
CONFIG_XFRM_USER=y
CONFIG_NET_SCHED=y
CONFIG_NET_SCH_CBQ=y
CONFIG_NET_SCH_HTB=y
CONFIG_NET_SCH_HFSC=y
CONFIG_NET_SCH_RED=y
CONFIG_NET_SCH_TBF=y
CONFIG_NET_SCH_DELAY=y
CONFIG_NET_SCH_INGRESS=y
CONFIG_NET_CLS=y
CONFIG_NET_CLS_TCINDEX=y
CONFIG_NET_CLS_ROUTE4=y
CONFIG_NET_CLS_ROUTE=y
CONFIG_NET_CLS_FW=y
CONFIG_NET_CLS_U32=y
CONFIG_NETDEVICES=y
CONFIG_DUMMY=y
CONFIG_NET_ETHERNET=y
CONFIG_MII=y
CONFIG_NET_TULIP=y
CONFIG_TULIP=y
CONFIG_TULIP_MWI=y
CONFIG_TULIP_MMIO=y
CONFIG_TULIP_NAPI=y
CONFIG_TULIP_NAPI_HW_MITIGATION=y
CONFIG_NET_RADIO=y
CONFIG_PCMCIA_WAVELAN=y
CONFIG_PCMCIA_NETWAVE=y
CONFIG_PCMCIA_RAYCS=y
CONFIG_AIRO=y
CONFIG_HERMES=y
CONFIG_PCMCIA_HERMES=y
CONFIG_AIRO_CS=y
CONFIG_NET_WIRELESS=y
CONFIG_NET_PCMCIA=y
CONFIG_PPP=y
CONFIG_PPP_FILTER=y
CONFIG_PPP_ASYNC=y
CONFIG_PPP_DEFLATE=y
CONFIG_PPP_BSDCOMP=y
CONFIG_INPUT=y
CONFIG_INPUT_MOUSEDEV=y
CONFIG_INPUT_MOUSEDEV_PSAUX=y
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
CONFIG_SOUND_GAMEPORT=y
CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
CONFIG_INPUT_KEYBOARD=y
CONFIG_KEYBOARD_ATKBD=y
CONFIG_INPUT_MOUSE=y
CONFIG_MOUSE_PS2=y
CONFIG_VT=y
CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y
CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_NR_UARTS=4
CONFIG_SERIAL_CORE=y
CONFIG_UNIX98_PTYS=y
CONFIG_PRINTER=y
CONFIG_AGP=y
CONFIG_AGP_INTEL=y
CONFIG_VIDEO_SELECT=y
CONFIG_VGA_CONSOLE=y
CONFIG_DUMMY_CONSOLE=y
CONFIG_EXT2_FS=y
CONFIG_EXT3_FS=y
CONFIG_JBD=y
CONFIG_ISO9660_FS=y
CONFIG_JOLIET=y
CONFIG_FAT_FS=y
CONFIG_MSDOS_FS=y
CONFIG_VFAT_FS=y
CONFIG_PROC_FS=y
CONFIG_PROC_KCORE=y
CONFIG_SYSFS=y
CONFIG_TMPFS=y
CONFIG_RAMFS=y
CONFIG_NFS_FS=y
CONFIG_NFS_V3=y
CONFIG_NFSD=y
CONFIG_NFSD_V3=y
CONFIG_NFSD_TCP=y
CONFIG_LOCKD=y
CONFIG_LOCKD_V4=y
CONFIG_EXPORTFS=y
CONFIG_SUNRPC=y
CONFIG_SMB_FS=y
CONFIG_MSDOS_PARTITION=y
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="cp437"
CONFIG_NLS_CODEPAGE_437=y
CONFIG_NLS_ISO8859_1=y
CONFIG_NLS_ISO8859_15=y
CONFIG_DEBUG_KERNEL=y
CONFIG_EARLY_PRINTK=y
CONFIG_MAGIC_SYSRQ=y
CONFIG_X86_FIND_SMP_CONFIG=y
CONFIG_X86_MPPARSE=y
CONFIG_CRYPTO=y
CONFIG_CRYPTO_HMAC=y
CONFIG_CRYPTO_MD5=y
CONFIG_CRYPTO_SHA1=y
CONFIG_CRYPTO_DES=y
CONFIG_CRYPTO_TWOFISH=y
CONFIG_CRYPTO_AES=y
CONFIG_CRYPTO_DEFLATE=y
CONFIG_CRC32=y
CONFIG_ZLIB_INFLATE=y
CONFIG_ZLIB_DEFLATE=y
CONFIG_X86_BIOS_REBOOT=y
CONFIG_X86_STD_RESOURCES=y
CONFIG_PC=y

Notes about the above:
- There was some thought about adding an 802.11 card to the machine so it
  could play firewall, but that's never been done.  The PCMICA modules and
  cardbus adapater are unused.
- Likewise, IPSEC is planned, but not currently used at all.
- Backup device is a SCSI tape drive.
- Machine does a lot of firewalling and file serving.
- RAID-0 and RAID-1 are used a lot.  Main storage space is RAID-10
  (2-way RAID-0 over 2-way RAID-1).  All swap space is 2-way RAID-1.
- Console is not used much, although it has an X session left up on it
  sometimes.

=== /proc/cpuinfo
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 6
model name	: Celeron (Mendocino)
stepping	: 5
cpu MHz		: 467.915
cache size	: 128 KB
fdiv_bug	: no
hlt_bug		: no
f00f_bug	: no
coma_bug	: no
fpu		: yes
fpu_exception	: yes
cpuid level	: 2
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 mmx fxsr
bogomips	: 923.64

=== lspci
0000:00:00.0 Host bridge: Intel Corp. 440BX/ZX/DX - 82443BX/ZX/DX Host bridge (rev 03)
0000:00:01.0 PCI bridge: Intel Corp. 440BX/ZX/DX - 82443BX/ZX/DX AGP bridge (rev 03)
0000:00:04.0 ISA bridge: Intel Corp. 82371AB/EB/MB PIIX4 ISA (rev 02)
0000:00:04.1 IDE interface: Intel Corp. 82371AB/EB/MB PIIX4 IDE (rev 01)
0000:00:04.2 USB Controller: Intel Corp. 82371AB/EB/MB PIIX4 USB (rev 01)
0000:00:04.3 Bridge: Intel Corp. 82371AB/EB/MB PIIX4 ACPI (rev 02)
0000:00:09.0 CardBus bridge: Ricoh Co Ltd RL5c476 II (rev 80)
0000:00:09.1 CardBus bridge: Ricoh Co Ltd RL5c476 II (rev 80)
0000:00:0a.0 SCSI storage controller: BusLogic BT-946C (BA80C30) [MultiMaster 10] (rev 08)
0000:00:0b.0 Unknown mass storage controller: Promise Technology, Inc. 20268 (rev 01)
0000:00:0c.0 Unknown mass storage controller: Promise Technology, Inc. 20268 (rev 01)
0000:00:0d.0 PCI bridge: Digital Equipment Corporation DECchip 21152 (rev 03)
0000:01:00.0 VGA compatible controller: nVidia Corporation NV6 [Vanta/Vanta LT] (rev 15)
0000:02:04.0 Ethernet controller: Digital Equipment Corporation DECchip 21142/43 (rev 41)
0000:02:05.0 Ethernet controller: Digital Equipment Corporation DECchip 21142/43 (rev 41)
0000:02:06.0 Ethernet controller: Digital Equipment Corporation DECchip 21142/43 (rev 41)
0000:02:07.0 Ethernet controller: Digital Equipment Corporation DECchip 21142/43 (rev 41)

=== lspci -vn
0000:00:00.0 Class 0600: 8086:7190 (rev 03)
	Flags: bus master, medium devsel, latency 64
	Memory at e6000000 (32-bit, prefetchable)
	Capabilities: [a0] AGP version 1.0

0000:00:01.0 Class 0604: 8086:7191 (rev 03)
	Flags: bus master, 66MHz, medium devsel, latency 64
	Bus: primary=00, secondary=01, subordinate=01, sec-latency=64
	Memory behind bridge: e2000000-e3cfffff
	Prefetchable memory behind bridge: e3f00000-e5ffffff

0000:00:04.0 Class 0601: 8086:7110 (rev 02)
	Flags: bus master, medium devsel, latency 0

0000:00:04.1 Class 0101: 8086:7111 (rev 01) (prog-if 80 [Master])
	Flags: bus master, medium devsel, latency 48
	I/O ports at d800 [size=16]

0000:00:04.2 Class 0c03: 8086:7112 (rev 01)
	Flags: bus master, medium devsel, latency 48, IRQ 9
	I/O ports at d400 [size=32]

0000:00:04.3 Class 0680: 8086:7113 (rev 02)
	Flags: medium devsel, IRQ 9

0000:00:09.0 Class 0607: 1180:0476 (rev 80)
	Subsystem: 14ef:0202
	Flags: bus master, medium devsel, latency 168, IRQ 9
	Memory at 40000000 (32-bit, non-prefetchable)
	Bus: primary=00, secondary=03, subordinate=06, sec-latency=176
	Memory window 0: 40400000-407ff000 (prefetchable)
	Memory window 1: 40800000-40bff000
	I/O window 0: 00004000-000040ff
	I/O window 1: 00004400-000044ff
	16-bit legacy interface ports at 0001

0000:00:09.1 Class 0607: 1180:0476 (rev 80)
	Subsystem: 14ef:0202
	Flags: bus master, medium devsel, latency 168, IRQ 11
	Memory at 40001000 (32-bit, non-prefetchable)
	Bus: primary=00, secondary=07, subordinate=0a, sec-latency=176
	Memory window 0: 40c00000-40fff000 (prefetchable)
	Memory window 1: 41000000-413ff000
	I/O window 0: 00004800-000048ff
	I/O window 1: 00004c00-00004cff
	16-bit legacy interface ports at 0001

0000:00:0a.0 Class 0100: 104b:1040 (rev 08)
	Subsystem: 104b:1040
	Flags: bus master, fast devsel, latency 48, IRQ 9
	I/O ports at d000
	Memory at e1800000 (32-bit, non-prefetchable) [size=4K]

0000:00:0b.0 Class 0180: 105a:4d68 (rev 01) (prog-if 85)
	Subsystem: 105a:4d68
	Flags: bus master, 66MHz, slow devsel, latency 48, IRQ 10
	I/O ports at b800
	I/O ports at b400 [size=4]
	I/O ports at b000 [size=8]
	I/O ports at a800 [size=4]
	I/O ports at a400 [size=16]
	Memory at e1000000 (32-bit, non-prefetchable) [size=16K]
	Capabilities: [60] Power Management version 1

0000:00:0c.0 Class 0180: 105a:4d68 (rev 01) (prog-if 85)
	Subsystem: 105a:4d68
	Flags: bus master, 66MHz, slow devsel, latency 48, IRQ 11
	I/O ports at a000
	I/O ports at 9800 [size=4]
	I/O ports at 9400 [size=8]
	I/O ports at 9000 [size=4]
	I/O ports at 8800 [size=16]
	Memory at e0800000 (32-bit, non-prefetchable) [size=16K]
	Capabilities: [60] Power Management version 1

0000:00:0d.0 Class 0604: 1011:0024 (rev 03)
	Flags: bus master, medium devsel, latency 48
	Bus: primary=00, secondary=02, subordinate=02, sec-latency=48
	I/O behind bridge: 00006000-00007fff
	Memory behind bridge: de800000-e07fffff
	Prefetchable memory behind bridge: 00000000e3d00000-00000000e3d00000
	Expansion ROM at 00006000 [disabled] [size=8K]
	Capabilities: [dc] Power Management version 1

0000:01:00.0 Class 0300: 10de:002c (rev 15)
	Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 11
	Memory at e2000000 (32-bit, non-prefetchable) [size=e3ff0000]
	Memory at e4000000 (32-bit, prefetchable) [size=32M]
	Expansion ROM at 00010000 [disabled]
	Capabilities: [60] Power Management version 1
	Capabilities: [44] AGP version 2.0

0000:02:04.0 Class 0200: 1011:0019 (rev 41)
	Subsystem: 1186:1112
	Flags: bus master, medium devsel, latency 48, IRQ 9
	I/O ports at 7800
	Memory at e0000000 (32-bit, non-prefetchable) [size=1K]

0000:02:05.0 Class 0200: 1011:0019 (rev 41)
	Subsystem: 1186:1112
	Flags: bus master, medium devsel, latency 48, IRQ 11
	I/O ports at 7400
	Memory at df800000 (32-bit, non-prefetchable) [size=1K]

0000:02:06.0 Class 0200: 1011:0019 (rev 41)
	Subsystem: 1186:1112
	Flags: bus master, medium devsel, latency 48, IRQ 10
	I/O ports at 7000
	Memory at df000000 (32-bit, non-prefetchable) [size=1K]

0000:02:07.0 Class 0200: 1011:0019 (rev 41)
	Subsystem: 1186:1112
	Flags: bus master, medium devsel, latency 48, IRQ 9
	I/O ports at 6800
	Memory at de800000 (32-bit, non-prefetchable) [size=1K]