LKML Archive on lore.kernel.org
help / color / mirror / Atom feed
* Re: Re: [Oops]  i386 mm/slab.c (cache_flusharray)
@ 2003-11-25 17:30 pinotj
  2003-11-25 22:51 ` Linus Torvalds
  0 siblings, 1 reply; 32+ messages in thread
From: pinotj @ 2003-11-25 17:30 UTC (permalink / raw)
  To: manfred; +Cc: torvalds, akpm, linux-kernel

Here are the results for test10 about the oops in slab.c
When I say `compilation` with no explanation, it means compilation of 2.6.0-test10 when using this same kernel, with my .config file (vmlinuz is 2.5M).

1.  2.6.0-test10 vanilla + PREEMPT_CONFIG=y + patch printk
Kernel oops during compilation, as for 2.6.0-test9 : at around 80% of the task.

---
slab: double free detected in cache 'buffer_head', objp cc2dbb58, objnr 42, slabp cc2db000, s_mem cc2db180, bufctl ffffffff.
------------[ cut here ]------------
kernel BUG at mm/slab.c:1956!
invalid operand: 0000 [#1]
CPU:    0
EIP:    0060:[free_block+357/784]    Not tainted
EIP:    0060:[<c015ad55>]    Not tainted
EFLAGS: 00010092
EIP is at free_block+0x165/0x310
eax: 00000080   ebx: 00000000   ecx: c0697854   edx: c05714f8
esi: cc2db000   edi: cc2db018   ebp: cf821c68   esp: cf821c34
ds: 007b   es: 007b   ss: 0068
Process kswapd0 (pid: 8, threadinfo=cf820000 task=cf849960)
Stack: c0504f40 c0505b3d cc2dbb58 0000002a cc2db000 cc2db180 ffffffff 0000002a
       cc2dbb58 0000000d cffdef08 c9e93180 00000010 cf821ca0 c015afda cffed800
       cffdef08 00000010 00000282 c11c89a0 00000000 00000001 cffee730 00000010
 Call Trace:
 [cache_flusharray+218/688] cache_flusharray+0xda/0x2b0
 [<c015afda>] cache_flusharray+0xda/0x2b0
 [kmem_cache_free+429/912] kmem_cache_free+0x1ad/0x390
---

full log can be found here: http://cercle-daejeon.homelinux.org/oops-full.txt
Cleaned oops (ksymoops):    http://cercle-daejeon.homelinux.org/oops.txt
Config of the kernel :      http://cercle-daejeon.homelinux.org/config.txt

NB: I don't get any oops if I compile the kernel with default settings (vmlinuz around 1.5M)

2. 2.6.0-test10 vanilla + PREEMPT_CONFIG=n + patch printk
Argh, oops at the speed of light in the beginning of compilation. Too fast to catch something in the logs.
This invalidates the first idea of bad effect of PREEMPT, it's exactly the contrary in this case.
Second try, compilation is OK, 1 times, 2 times and failed during the third time.
Again, no logs, but this time I wrote down the printk:

---
slab: double free detected in cache 'bio', objp c888fc28, objnr 42, slabp c888f000, s_mem c888f1000, bufctl ffffffff.
---

This case is not easily reproductible.

3. 2.6.0-test10 vanilla + PREEMPT_CONFIG=y + patch printk + patch magic numbers
The patch solves the problem, I can compile 4 times a kernel and do heavy work in parallele (load average around 1.2 during 2 hours) without any problems.

Conclusion: well, this confirms some facts for 2.6.0-test10:

- oops reproductible if PREEMPT_CONFIG=y each time heavy load.
The limit of load, for my system (AMD tbird 1.2GHz, 256Mo) is somewhere between the compilation of a kernel of 1.5M (default settings) and a kernel of 2.5M (custom settings). Always occurs at about 80% of compilation in this second case.

- oops occurs even if PREEMPT_CONFIG=n, but with no really reproductibility. It needs heavy load too, but it's not enough. System hangs really quickly, no logs.

Finally, I just recall the patches of Manfred used here (printk and magic number):

diff -Nru a/mm/slab.c b/mm/slab.c 2003-11-22 09:00:00 +0900
--- a/mm/slab.c         2003-11-22 08:43:02.189656536 +0900
+++ b/mm/slab.c         2003-11-22 08:45:44.158033600 +0900
@@ -1952,8 +1952,7 @@
                check_slabp(cachep, slabp);
 #if DEBUG
                if (slab_bufctl(slabp)[objnr] != BUFCTL_FREE) {
-                       printk(KERN_ERR "slab: double free detected in cache '%s', objp %p.\n",
-                                               cachep->name, objp);
+                       printk(KERN_ERR "slab: double free detected in cache '%s', objp %p, objnr %d, slabp %p, s_mem %p, bufctl %x.\n", cachep->name, objp, objnr, slabp, slabp->s_mem, slab_bufctl(slabp)[objnr]);
                        BUG();
                }
 #endif

diff -Nru a/mm/slab.c b/mm/slab.c 2003-11-22 09:00:00 +0900
--- a/mm/slab.c         2003-11-22 08:43:02.189656536 +0900
+++ b/mm/slab.c         2003-11-22 08:45:44.158033600 +0900
@@ -153,9 +153,9 @@
  * is less than 512 (PAGE_SIZE<<3), but greater than 256.
  */

-#define BUFCTL_END     0xffffFFFF
-#define BUFCTL_FREE    0xffffFFFE
-#define        SLAB_LIMIT      0xffffFFFD
+#define BUFCTL_END     0xfeffFFFF
+#define BUFCTL_FREE    0xf7ffFFFF
+#define SLAB_LIMIT     0xf0ffFFFD
 typedef unsigned int kmem_bufctl_t;

 /* Max number of objs-per-slab for caches which use off-slab slabs.

Regards,

Jerome Pinot


^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: Re: [Oops]  i386 mm/slab.c (cache_flusharray)
  2003-11-25 17:30 Re: [Oops] i386 mm/slab.c (cache_flusharray) pinotj
@ 2003-11-25 22:51 ` Linus Torvalds
  2003-11-27 18:07   ` Manfred Spraul
  0 siblings, 1 reply; 32+ messages in thread
From: Linus Torvalds @ 2003-11-25 22:51 UTC (permalink / raw)
  To: pinotj; +Cc: manfred, akpm, linux-kernel



On Tue, 25 Nov 2003 pinotj@club-internet.fr wrote:
>
> 3. 2.6.0-test10 vanilla + PREEMPT_CONFIG=y + patch printk + patch magic numbers
> The patch solves the problem, I can compile 4 times a kernel and do heavy work in parallele (load average around 1.2 during 2 hours) without any problems.

Those magic numbers don't make any sense. In particular, SLAB_LIMIT is
clearly bogus both in the original version and in the "magic number
patch". The only place that uses SLAB_LIMIT is the code that decides how
many entries fit in one slab, and quite frankly, it makes no _sense_ to
have a SLAB_LIMIT that is big enough to be unsigned.

"SLAB_LIMIT" should be something in the few hundreds, maybe.

Manfred?  What is the logic behind those nonsensical numbers?

		Linus

^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: [Oops]  i386 mm/slab.c (cache_flusharray)
  2003-11-25 22:51 ` Linus Torvalds
@ 2003-11-27 18:07   ` Manfred Spraul
  0 siblings, 0 replies; 32+ messages in thread
From: Manfred Spraul @ 2003-11-27 18:07 UTC (permalink / raw)
  To: Linus Torvalds; +Cc: pinotj, akpm, linux-kernel

Linus Torvalds wrote:

>On Tue, 25 Nov 2003 pinotj@club-internet.fr wrote:
>  
>
>>3. 2.6.0-test10 vanilla + PREEMPT_CONFIG=y + patch printk + patch magic numbers
>>The patch solves the problem, I can compile 4 times a kernel and do heavy work in parallele (load average around 1.2 during 2 hours) without any problems.
>>    
>>
>
>Those magic numbers don't make any sense. In particular, SLAB_LIMIT is
>clearly bogus both in the original version and in the "magic number
>patch". The only place that uses SLAB_LIMIT is the code that decides how
>many entries fit in one slab, and quite frankly, it makes no _sense_ to
>have a SLAB_LIMIT that is big enough to be unsigned.
>
Object numbers  (kmem_bufctl_t) are unsigned, but some values have a 
special meaning:
"-1" is the magic value for end-of-list.
"-2" is the magic value for object in use.
All other values are valid object numbers. Right now object numbers are 
unsigned int, but initially I considered unsigned char or unsigned 
short. And then an explicit SLAB_LIMIT is necessary - with unsigned 
char, the limit would be 253 objects per slab, which could be reached if 
someone creates objects smaller than 16 bytes.

In Jerome's case, the debug checks noticed that the object-in-use 
sentinel was not in the bufctl entry during free, instead there was a 
"-1". There are several sources for the "-1": My initial guess was 
either a bug in slab, or a bad memory cell (only one bit difference). 
Thus I sent him a patch that changes multiple bits. Result: It remained 
a single bit change, i.e it's proven that slab doesn't write BUFCTL_END 
into the wrong slot.

--
    Manfred


^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: [Oops]  i386 mm/slab.c (cache_flusharray)
  2003-12-12 19:00       ` Christoph Hellwig
@ 2003-12-12 20:07         ` Manfred Spraul
  0 siblings, 0 replies; 32+ messages in thread
From: Manfred Spraul @ 2003-12-12 20:07 UTC (permalink / raw)
  To: Christoph Hellwig
  Cc: Nathan Scott, pinotj, torvalds, neilb, akpm, linux-kernel

Christoph Hellwig wrote:

>Looks like whe're better of fixing mm/slab.c
>
Maybe I'm blind, but I don't see how kmem_cache_alloc can return NULL 
with __GFP_NOFAIL:
- kmem_cache_alloc calls __cache_alloc.
- cache_alloc only return 0 if cache_alloc_refill returns 0.
- cache_alloc_refill only return 0 if cache_grow returns 0.
- cache_grow only return 0 if
    * SLAB_NO_GROW is set in flags.
    * get_free_pages(flags) fails.
    * kmem_cache_alloc(flags&SLAB_LEVEL_MASK) fails.
       SLAB_LEVEL_MASK contains __GFP_NOFAIL.

--
    Manfred



^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: [Oops]  i386 mm/slab.c (cache_flusharray)
  2003-12-09 23:58     ` Nathan Scott
@ 2003-12-12 19:00       ` Christoph Hellwig
  2003-12-12 20:07         ` Manfred Spraul
  0 siblings, 1 reply; 32+ messages in thread
From: Christoph Hellwig @ 2003-12-12 19:00 UTC (permalink / raw)
  To: Nathan Scott; +Cc: pinotj, torvalds, neilb, manfred, akpm, linux-kernel

On Wed, Dec 10, 2003 at 10:58:32AM +1100, Nathan Scott wrote:
> > > [ Christoph, is this failure expected?  I think you/Steve made
> > > some changes there to use __GFP_NOFAIL and assume it wont fail?
> > > (in 2.4 we do memory allocations differently to better handle
> > > failures, but that code was removed...) ]
> > 
> > It looks like the slab allocator doesn't like __GFP_NOFAIL, we'll
> > probably have to revert the XFS memory allocation wrappers to the
> > 2.4 versions.
> > 
> 
> OK, thanks - I'll look into it.

This starts to look really fishy:

hch@bird:/repo/repo/linux-2.5/fs/jbd$ egrep -r NOFAIL *
journal.c:      new_bh = alloc_buffer_head(GFP_NOFS|__GFP_NOFAIL);
journal.c:      return kmalloc(size, flags | (retry ? __GFP_NOFAIL :
0));

both of these end up in the slab layer, just like XFS - except
that __cache_alloc still may fail.  Looks like whe're better of
fixing mm/slab.c


^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: [Oops]  i386 mm/slab.c (cache_flusharray)
  2003-12-09  7:21   ` Christoph Hellwig
@ 2003-12-09 23:58     ` Nathan Scott
  2003-12-12 19:00       ` Christoph Hellwig
  0 siblings, 1 reply; 32+ messages in thread
From: Nathan Scott @ 2003-12-09 23:58 UTC (permalink / raw)
  To: Christoph Hellwig, pinotj, torvalds, neilb, manfred, akpm, linux-kernel

On Tue, Dec 09, 2003 at 08:21:32AM +0100, Christoph Hellwig wrote:
> On Tue, Dec 09, 2003 at 01:03:22PM +1100, Nathan Scott wrote:
> > [ Christoph, is this failure expected?  I think you/Steve made
> > some changes there to use __GFP_NOFAIL and assume it wont fail?
> > (in 2.4 we do memory allocations differently to better handle
> > failures, but that code was removed...) ]
> 
> It looks like the slab allocator doesn't like __GFP_NOFAIL, we'll
> probably have to revert the XFS memory allocation wrappers to the
> 2.4 versions.
> 

OK, thanks - I'll look into it.

cheers.

-- 
Nathan

^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: [Oops]  i386 mm/slab.c (cache_flusharray)
  2003-12-09  2:03 ` Nathan Scott
@ 2003-12-09  7:21   ` Christoph Hellwig
  2003-12-09 23:58     ` Nathan Scott
  0 siblings, 1 reply; 32+ messages in thread
From: Christoph Hellwig @ 2003-12-09  7:21 UTC (permalink / raw)
  To: Nathan Scott; +Cc: pinotj, torvalds, hch, neilb, manfred, akpm, linux-kernel

On Tue, Dec 09, 2003 at 01:03:22PM +1100, Nathan Scott wrote:
> [ Christoph, is this failure expected?  I think you/Steve made
> some changes there to use __GFP_NOFAIL and assume it wont fail?
> (in 2.4 we do memory allocations differently to better handle
> failures, but that code was removed...) ]

It looks like the slab allocator doesn't like __GFP_NOFAIL, we'll
probably have to revert the XFS memory allocation wrappers to the
2.4 versions.


^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: [Oops]  i386 mm/slab.c (cache_flusharray)
  2003-12-09  0:57 pinotj
@ 2003-12-09  2:03 ` Nathan Scott
  2003-12-09  7:21   ` Christoph Hellwig
  0 siblings, 1 reply; 32+ messages in thread
From: Nathan Scott @ 2003-12-09  2:03 UTC (permalink / raw)
  To: pinotj; +Cc: torvalds, hch, neilb, manfred, akpm, linux-kernel

On Tue, Dec 09, 2003 at 01:57:35AM +0100, pinotj@club-internet.fr wrote:
> Results about testing on test11 this week-end.

thanks.

>   ---
>   ld: page allocation failure. order:0, mode:0x8d0
>   Unable to handle kernel NULL pointer dereference at virtual address 00000074
>   c01d4cbd
>   *pde = 00000000
>   Oops: 0002 [#1]
>   CPU:    0
>   EIP:    0060:[_xfs_trans_alloc+149/160]    Not tainted
>   EIP:    0060:[<c01d4cbd>]    Not tainted
>   ---

Ah, yes, I know what this is (and can reproduce it) and I don't
have a fix as yet.  This is an unrelated problem - basically,
we are doing kmem_cache_alloc with __GFP_NOFAIL set within XFS,
but the allocations are failing and returning NULL (but we are
assuming they will not).

You (and I) are hitting this more frequently now because Linus'
patch bypasses the slab allocator and is more expensive in terms
of memory used.  However, I have heard of one person who hit this
"in the wild" too, so its something we will need to address.

[ Christoph, is this failure expected?  I think you/Steve made
some changes there to use __GFP_NOFAIL and assume it wont fail?
(in 2.4 we do memory allocations differently to better handle
failures, but that code was removed...) ]

> 
>  B. With Ext3 (and without XFS)
> 
>   1. no patch
>   same as I.A.1
>   2. patch-xfs & patch-slab
>   Compilations looked good but I got a lot of errors in my logs:
> 
>   ---
>   kernel: ld: page allocation failure. order:0, mode:0x50
>   last message repeated 31 times
>   klogd: page allocation failure. order:0, mode:0x50
>   last message repeated 63 times
>   kswapd0: page allocation failure. order:0, mode:0x50
>   ENOMEM in journal_alloc_journal_head, retrying.
>   ion failure. order:0, mode:0x50
>   kswapd0: page allocation failure. order:0, mode:0x50
>   last message repeated 291 times

I expect that this is a similar issue to the above, but from
ext2/3's point of view.

> 
> Tests in I. confirm that it's not an XFS-only problem but seems to
> affect page allocation for fs in general.
> I hope these oops will be clearer to you. I still have no problem with test9.

FWIW and IIRC, we didn't push any XFS changes (nothing major
anyway, that I could foresee causing this) on to Linus in
between -test9 and -test11.

cheers.

-- 
Nathan

^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: [Oops]  i386 mm/slab.c (cache_flusharray)
  2003-12-05  9:34         ` Nathan Scott
@ 2003-12-05 14:22           ` Christoph Hellwig
  0 siblings, 0 replies; 32+ messages in thread
From: Christoph Hellwig @ 2003-12-05 14:22 UTC (permalink / raw)
  To: Nathan Scott; +Cc: Linus Torvalds, pinotj, manfred, akpm, linux-kernel

On Fri, Dec 05, 2003 at 08:34:25PM +1100, Nathan Scott wrote:
> You might be mixing up pb_pages and pb_addr there?  pb_addr is
> always a pointer.  We need to distinguish whether it was slab
> alloc'd or whether it points into page cache pages, so we know
> whether to page_cache_release the pages or kfree the pointer
> when we're done with the pagebuf.
> 
> The pb_page_array works just as you describe, with a prealloc'd
> array of page pointers, and pb_pages either points to the array
> of to a larger kmalloc'd array as necessary.

Indeed.  I'm not remembering the code as good as I hoped to :)

Sorry for the noise.


^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: [Oops]  i386 mm/slab.c (cache_flusharray)
  2003-12-05  7:14       ` Christoph Hellwig
@ 2003-12-05  9:34         ` Nathan Scott
  2003-12-05 14:22           ` Christoph Hellwig
  0 siblings, 1 reply; 32+ messages in thread
From: Nathan Scott @ 2003-12-05  9:34 UTC (permalink / raw)
  To: Christoph Hellwig, Linus Torvalds, pinotj, manfred, akpm, linux-kernel

On Fri, Dec 05, 2003 at 07:14:55AM +0000, Christoph Hellwig wrote:
> On Fri, Dec 05, 2003 at 08:21:10AM +1100, Nathan Scott wrote:
> > Yeah, thats pretty silly stuff - and should be fairly easy to
> > fix by using a pagebuf flag to differentiate the two.  Will do.
> 
> IMHO a flags is wrong here.  Just maek pb_addr always a pointer and
> for the case it's the preallocated array make it point pb_page_array
> or something like that.  Then check whether pb_addr is pointing to the
> preallocated array.

You might be mixing up pb_pages and pb_addr there?  pb_addr is
always a pointer.  We need to distinguish whether it was slab
alloc'd or whether it points into page cache pages, so we know
whether to page_cache_release the pages or kfree the pointer
when we're done with the pagebuf.

The pb_page_array works just as you describe, with a prealloc'd
array of page pointers, and pb_pages either points to the array
of to a larger kmalloc'd array as necessary.

cheers.

-- 
Nathan

^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: [Oops]  i386 mm/slab.c (cache_flusharray)
  2003-12-04 21:21     ` Nathan Scott
@ 2003-12-05  7:14       ` Christoph Hellwig
  2003-12-05  9:34         ` Nathan Scott
  0 siblings, 1 reply; 32+ messages in thread
From: Christoph Hellwig @ 2003-12-05  7:14 UTC (permalink / raw)
  To: Nathan Scott; +Cc: Linus Torvalds, pinotj, manfred, akpm, linux-kernel

On Fri, Dec 05, 2003 at 08:21:10AM +1100, Nathan Scott wrote:
> Yeah, thats pretty silly stuff - and should be fairly easy to
> fix by using a pagebuf flag to differentiate the two.  Will do.

IMHO a flags is wrong here.  Just maek pb_addr always a pointer and
for the case it's the preallocated array make it point pb_page_array
or something like that.  Then check whether pb_addr is pointing to the
preallocated array.

^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: [Oops]  i386 mm/slab.c (cache_flusharray)
  2003-12-05  3:00     ` Nathan Scott
@ 2003-12-05  6:40       ` Linus Torvalds
  0 siblings, 0 replies; 32+ messages in thread
From: Linus Torvalds @ 2003-12-05  6:40 UTC (permalink / raw)
  To: Nathan Scott; +Cc: Neil Brown, pinotj, manfred, akpm, linux-kernel



On Fri, 5 Dec 2003, Nathan Scott wrote:
>
> This patch removes that code, fixes a small memory leak that was
> lurking in there too, and adds the missing-bio_put-on-error case
> that Neil found in pagebuf.

Ok, so we've got the RAID5 issue explained, does anybody have any idea
about what's wrong with the non-RAID5 cases? We've got at least one report
of page corruption on RAID0, and one on a plain disk. Both of which looked
XFS-related - or at least shared that in their configs.

Jerome - can you test Nathan's patch together with my "avoid the
complicated slab logic"? The slab avoidance thing got ext3 stable for you,
now with Nathan's patch hopefully XFS will be stable too.

Which still doesn't _explain_ anything, but it would be interesting to see
if that removes your problems. It would definitely point to a slab issue,
but the big question is why it's not more common if so. Compiler bug?
Other strange trigger?

			Linus

^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: [Oops]  i386 mm/slab.c (cache_flusharray)
  2003-12-04 19:09   ` Linus Torvalds
  2003-12-04 21:21     ` Nathan Scott
@ 2003-12-05  3:00     ` Nathan Scott
  2003-12-05  6:40       ` Linus Torvalds
  1 sibling, 1 reply; 32+ messages in thread
From: Nathan Scott @ 2003-12-05  3:00 UTC (permalink / raw)
  To: Linus Torvalds, Neil Brown; +Cc: pinotj, manfred, akpm, linux-kernel

On Thu, Dec 04, 2003 at 11:09:29AM -0800, Linus Torvalds wrote:
> ...
> So the oops it found was apparently triggered by the debugging changes,
> not necessarily by a real bug.
> 
> Ugh, that XFS code is _broken_. Instead of keeping track of how it got the
> memory, it totally forgets where the memory came from, and then it later
> asks "oh, btw, how the hell did I allocate this?".
> 

This patch removes that code, fixes a small memory leak that was
lurking in there too, and adds the missing-bio_put-on-error case
that Neil found in pagebuf.

Neil, with this & Linus' 2 patches (and CONFIG_SLAB_DEBUG off ;)
I now have what looks like a 100% reproducible test case for the
handle_stripe already-freed-bio panic.  This doesn't tickle the
raid5.c BUG_ON you sent me but its exactly the same spot as last
time (i.e. handle_stripe+0xda6), every time.

# raidstart /dev/md0
# mkfs.xfs -f /dev/md0
# mount /dev/md0
# umount /dev/md0
# mount /dev/md0

On my (quad p3) test machine, this second mount panics every time.

cheers.

-- 
Nathan


--- fs/xfs/pagebuf/page_buf.h.orig	2003-12-05 13:47:12.275589232 +1100
+++ fs/xfs/pagebuf/page_buf.h	2003-12-05 13:43:30.898243704 +1100
@@ -123,12 +123,13 @@
 	_PBF_PRIVATE_BH = (1 << 17), /* do not use public buffer heads	   */
 	_PBF_ALL_PAGES_MAPPED = (1 << 18), /* all pages in range mapped	   */
 	_PBF_ADDR_ALLOCATED = (1 << 19), /* pb_addr space was allocated	   */
-	_PBF_MEM_ALLOCATED = (1 << 20), /* pb_mem+underlying pages alloc'd */
+	_PBF_MEM_ALLOCATED = (1 << 20), /* underlying pages are allocated  */
+	_PBF_MEM_SLAB = (1 << 21), /* underlying pages are slab allocated  */
 
-	PBF_FORCEIO = (1 << 21),
-	PBF_FLUSH = (1 << 22),	/* flush disk write cache		   */
-	PBF_READ_AHEAD = (1 << 23),
-	PBF_RUN_QUEUES = (1 << 24), /* run block device task queue	   */
+	PBF_FORCEIO = (1 << 22),
+	PBF_FLUSH = (1 << 23),	/* flush disk write cache		   */
+	PBF_READ_AHEAD = (1 << 24),
+	PBF_RUN_QUEUES = (1 << 25), /* run block device task queue	   */
 
 } page_buf_flags_t;
 
--- fs/xfs/pagebuf/page_buf.c.orig	2003-12-05 13:47:06.888408208 +1100
+++ fs/xfs/pagebuf/page_buf.c	2003-12-05 13:43:30.888245224 +1100
@@ -343,9 +343,6 @@
 			page_cache_release(page);
 		}
 	}
-
-	if (pb->pb_pages != pb->pb_page_array)
-		kfree(pb->pb_pages);
 }
 
 /*
@@ -384,20 +381,17 @@
 		if (pb->pb_flags & _PBF_MEM_ALLOCATED) {
 			if (pb->pb_pages) {
 				/* release the pages in the address list */
-				if (pb->pb_pages[0] &&
-				    PageSlab(pb->pb_pages[0])) {
-					/*
-					 * This came from the slab
-					 * allocator free it as such
-					 */
+				if ((pb->pb_pages[0]) &&
+				    (pb->pb_flags & _PBF_MEM_SLAB)) {
 					kfree(pb->pb_addr);
 				} else {
 					_pagebuf_freepages(pb);
 				}
-
+				if (pb->pb_pages != pb->pb_page_array)
+					kfree(pb->pb_pages);
 				pb->pb_pages = NULL;
 			}
-			pb->pb_flags &= ~_PBF_MEM_ALLOCATED;
+			pb->pb_flags &= ~(_PBF_MEM_ALLOCATED | _PBF_MEM_SLAB);
 		}
 	}
 
@@ -944,7 +938,7 @@
 		return NULL;
 	}
 	/* otherwise pagebuf_free just ignores it */
-	pb->pb_flags |= _PBF_MEM_ALLOCATED;
+	pb->pb_flags |= (_PBF_MEM_ALLOCATED | _PBF_MEM_SLAB);
 	PB_CLEAR_OWNER(pb);
 	up(&pb->pb_sema);	/* Return unlocked pagebuf */
 
@@ -1412,6 +1406,7 @@
 		if (size)
 			goto next_chunk;
 	} else {
+		bio_put(bio);
 		pagebuf_ioerror(pb, EIO);
 	}
 

^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: [Oops]  i386 mm/slab.c (cache_flusharray)
  2003-12-04 18:49 ` Linus Torvalds
  2003-12-04 19:09   ` Linus Torvalds
  2003-12-04 19:19   ` Manfred Spraul
@ 2003-12-04 21:26   ` Nathan Scott
  2 siblings, 0 replies; 32+ messages in thread
From: Nathan Scott @ 2003-12-04 21:26 UTC (permalink / raw)
  To: Linus Torvalds; +Cc: pinotj, manfred, akpm, linux-kernel

On Thu, Dec 04, 2003 at 10:49:07AM -0800, Linus Torvalds wrote:
> 
> Nathan - did you see the two debug patches I sent out that caught this?
> 
> One adds checks to "atomic_dec_and_test()" to verify that the count never
> goes negative. The other basically disables all the slab code, and
> replaces them with straight page allocations, and that together with
> CONFIG_DEBUG_PAGEALLOC helps find bad behaviour (touching allocations
> after they are free'd etc).
> 

Yep - I'll remove that PageSlab use in there, and start testing
with these too.

thanks.

-- 
Nathan

^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: [Oops]  i386 mm/slab.c (cache_flusharray)
  2003-12-04 19:09   ` Linus Torvalds
@ 2003-12-04 21:21     ` Nathan Scott
  2003-12-05  7:14       ` Christoph Hellwig
  2003-12-05  3:00     ` Nathan Scott
  1 sibling, 1 reply; 32+ messages in thread
From: Nathan Scott @ 2003-12-04 21:21 UTC (permalink / raw)
  To: Linus Torvalds; +Cc: pinotj, manfred, akpm, linux-kernel

On Thu, Dec 04, 2003 at 11:09:29AM -0800, Linus Torvalds wrote:
> 
> 
> On Thu, 4 Dec 2003, Linus Torvalds wrote:
> > > ---
> > > kernel BUG at include/linux/mm.h:267!
> >
> > YEAH! That's "put_page_testzero()", and either the BUG_ON() or the
> > atomic_dec_and_test() noticing bad things.
> 
> Oh, damn. Looking closer, it appears that it's actually XFS being a bit
> too intimate with slab knowledge: the code does
> 
>                         if (pb->pb_pages) {
>                                 /* release the pages in the address list */
>                                 if (pb->pb_pages[0] &&
>                                     PageSlab(pb->pb_pages[0])) {
>                                         /*
>                                          * This came from the slab
>                                          * allocator free it as such
>                                          */
>                                         kfree(pb->pb_addr);
>                                 } else {
>                                         _pagebuf_freepages(pb);
>                                 }
> 
> and that code gets really confused by the fact that I'm bypassing the slab
> logic (and thus the PageSlab flag never gets set).
> 
> So the oops it found was apparently triggered by the debugging changes,
> not necessarily by a real bug.
> 
> Ugh, that XFS code is _broken_. Instead of keeping track of how it got the
> memory, it totally forgets where the memory came from, and then it later
> asks "oh, btw, how the hell did I allocate this?".

Yeah, thats pretty silly stuff - and should be fairly easy to
fix by using a pagebuf flag to differentiate the two.  Will do.

thanks.

-- 
Nathan

^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: [Oops]  i386 mm/slab.c (cache_flusharray)
  2003-12-04 18:49 ` Linus Torvalds
  2003-12-04 19:09   ` Linus Torvalds
@ 2003-12-04 19:19   ` Manfred Spraul
  2003-12-04 21:26   ` Nathan Scott
  2 siblings, 0 replies; 32+ messages in thread
From: Manfred Spraul @ 2003-12-04 19:19 UTC (permalink / raw)
  To: Linus Torvalds; +Cc: pinotj, nathans, akpm, linux-kernel

[-- Attachment #1: Type: text/plain, Size: 1224 bytes --]

Linus Torvalds wrote:

>Manfred, any ideas? What's different between 2.6.x and 2.4.x in slab?
>
>But it may also be that the bug is in some slab user - since my slab-
>translates-to-page-alloc hack always calls the slab constructor function
>on every allocation, and the destructor gets called immediately after the
>free, my debug version might hide some usage bugs.
>  
>
The changes between 2.4 and 2.6 are huge, for both debug and non-debug. 
Slab with debugging enabled now calls the destructors/constructor on 
every alloc. If page debugging is enabled, then all objects larger than 
128 bytes get their own page and are unmapped after kmem_cache_free(). 
The bio structure is smaller than 128 bytes - that probably explains why 
slab didn't catch the oopses that were mentioned in the other thread.
Perhaps something like the attached patch could help to trigger the 
oops: It increase the size of the bio structures, then they are handled 
by slab debugging.
If it oopses, then call ptrinfo() from the trap handler - it prints the 
name of the cache and the caller of the last slab operation. And hexdump 
the object (after ptrinfo mapped it), it contains a backtrace from the 
kmem_cache_free call.

--
    Manfred

[-- Attachment #2: patch-bio --]
[-- Type: text/plain, Size: 690 bytes --]

--- 2.6/fs/bio.c	2003-10-25 20:43:54.000000000 +0200
+++ build-2.6/fs/bio.c	2003-12-04 20:13:52.000000000 +0100
@@ -798,7 +798,7 @@
 
 		size = bp->nr_vecs * sizeof(struct bio_vec);
 
-		bp->slab = kmem_cache_create(bp->name, size, 0,
+		bp->slab = kmem_cache_create(bp->name, max(128,size), 0,
 						SLAB_HWCACHE_ALIGN, NULL, NULL);
 		if (!bp->slab)
 			panic("biovec: can't init slab cache\n");
@@ -815,7 +815,7 @@
 
 static int __init init_bio(void)
 {
-	bio_slab = kmem_cache_create("bio", sizeof(struct bio), 0,
+	bio_slab = kmem_cache_create("bio", max(128U,sizeof(struct bio)), 0,
 					SLAB_HWCACHE_ALIGN, NULL, NULL);
 	if (!bio_slab)
 		panic("bio: can't create slab cache\n");

^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: [Oops]  i386 mm/slab.c (cache_flusharray)
  2003-12-04 18:49 ` Linus Torvalds
@ 2003-12-04 19:09   ` Linus Torvalds
  2003-12-04 21:21     ` Nathan Scott
  2003-12-05  3:00     ` Nathan Scott
  2003-12-04 19:19   ` Manfred Spraul
  2003-12-04 21:26   ` Nathan Scott
  2 siblings, 2 replies; 32+ messages in thread
From: Linus Torvalds @ 2003-12-04 19:09 UTC (permalink / raw)
  To: pinotj; +Cc: nathans, manfred, akpm, linux-kernel



On Thu, 4 Dec 2003, Linus Torvalds wrote:
> > ---
> > kernel BUG at include/linux/mm.h:267!
>
> YEAH! That's "put_page_testzero()", and either the BUG_ON() or the
> atomic_dec_and_test() noticing bad things.

Oh, damn. Looking closer, it appears that it's actually XFS being a bit
too intimate with slab knowledge: the code does

                        if (pb->pb_pages) {
                                /* release the pages in the address list */
                                if (pb->pb_pages[0] &&
                                    PageSlab(pb->pb_pages[0])) {
                                        /*
                                         * This came from the slab
                                         * allocator free it as such
                                         */
                                        kfree(pb->pb_addr);
                                } else {
                                        _pagebuf_freepages(pb);
                                }

and that code gets really confused by the fact that I'm bypassing the slab
logic (and thus the PageSlab flag never gets set).

So the oops it found was apparently triggered by the debugging changes,
not necessarily by a real bug.

Ugh, that XFS code is _broken_. Instead of keeping track of how it got the
memory, it totally forgets where the memory came from, and then it later
asks "oh, btw, how the hell did I allocate this?".

			Linus

^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: [Oops]  i386 mm/slab.c (cache_flusharray)
  2003-12-04 18:27 pinotj
@ 2003-12-04 18:49 ` Linus Torvalds
  2003-12-04 19:09   ` Linus Torvalds
                     ` (2 more replies)
  0 siblings, 3 replies; 32+ messages in thread
From: Linus Torvalds @ 2003-12-04 18:49 UTC (permalink / raw)
  To: pinotj; +Cc: nathans, manfred, akpm, linux-kernel



On Thu, 4 Dec 2003 pinotj@club-internet.fr wrote:
>
> OK, I tried again the patch on "small kernel" test11 with
> CONFIG_DEBUG_PAGEALLOC only. Here are the first results. I will do more
> tests later because it seems weird. This time I have very different
> behavior for XFS and Ext3.

Ok, interesting indeed.

> I got an oops at boot time, when system try to mount root filesystem (XFS).
>
> But when I tried on a root Ext3, I got no problem at all. I even
> compiled 2 kernel straight without any problems. It seems to be the
> first time a test11 works flawless on my system.

All right. That may or may not mean that the bug is actually in mm/slab.c,
rather than anywhere else. It doesn't explain why _you_ hit it and few
others do, but it's still an interesting fact.

Manfred, any ideas? What's different between 2.6.x and 2.4.x in slab?

But it may also be that the bug is in some slab user - since my slab-
translates-to-page-alloc hack always calls the slab constructor function
on every allocation, and the destructor gets called immediately after the
free, my debug version might hide some usage bugs.

> The XFS oops (I was too lazy to write down all the backtraces, all about
> xfs_* things, I can send if needed):
>
> ---
> kernel BUG at include/linux/mm.h:267!

YEAH! That's "put_page_testzero()", and either the BUG_ON() or the
atomic_dec_and_test() noticing bad things.

So that's the "test for count going negative" bug, and it seems to have
found somebody doing a double free on a page. Which is _exactly_ what we
expected from the XFS problems, and would explain the "struct page"
corruption that people report.

> Call Trace:
>  [<c01aaf90>] pagebuf_free+0x24/0x30
>  [<c019669b>] xlog_find_verify_cycle+0x18b/0x1e0
> Code: 0f 0b 0b 01 d6 cb 1f c0 eb 8e ff 73 54 eb b3 90 53 e8 0e 11

It would be good to get the full backtrace, though.

Nathan - did you see the two debug patches I sent out that caught this?

One adds checks to "atomic_dec_and_test()" to verify that the count never
goes negative. The other basically disables all the slab code, and
replaces them with straight page allocations, and that together with
CONFIG_DEBUG_PAGEALLOC helps find bad behaviour (touching allocations
after they are free'd etc).

		Linus

^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: [Oops]  i386 mm/slab.c (cache_flusharray)
@ 2003-12-04 18:27 pinotj
  2003-12-04 18:49 ` Linus Torvalds
  0 siblings, 1 reply; 32+ messages in thread
From: pinotj @ 2003-12-04 18:27 UTC (permalink / raw)
  To: torvalds, nathans; +Cc: manfred, akpm, linux-kernel

OK, I tried again the patch on "small kernel" test11 with CONFIG_DEBUG_PAGEALLOC only. Here are the first results. I will do more tests later because it seems weird. This time I have very different behavior for XFS and Ext3.

I got an oops at boot time, when system try to mount root filesystem (XFS).

But when I tried on a root Ext3, I got no problem at all. I even compiled 2 kernel straight without any problems. It seems to be the first time a test11 works flawless on my system.

Of course, XFS and Ext3 were respectively enabled in the kernels.

I hope I didn't make any mistake, I will confirm tomorrow.


Jerome Pinot

The XFS oops (I was too lazy to write down all the backtraces, all about xfs_* things, I can send if needed):

---
kernel BUG at include/linux/mm.h:267!
invalid operand: 0000 [#1]
CPU:    0
EIP:    0060:[<c01aa5ac>]    Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010246
eax: 00000000   ebx: cef3d000   ecx: cef3c000   edx: c1254e28
esi: 00310020   edi: 00000001   ebp: c12f3b18   esp: c12f3b0c
ds: 007b   es: 007b   ss: 0068
Stack: 00000100 00000000 ceee0000 c12f3b28 c01aaf90 c0253d40 cef3d000 c12f3b7c
       c019669b cef3d000 00000100 00000000 00000100 00000000 00000a02 00000000
       00000100 00000000 cef3d000 00000100 00000000 00000a02 00000000 00000802
Call Trace:
 [<c01aaf90>] pagebuf_free+0x24/0x30
 [<c019669b>] xlog_find_verify_cycle+0x18b/0x1e0
Code: 0f 0b 0b 01 d6 cb 1f c0 eb 8e ff 73 54 eb b3 90 53 e8 0e 11


>>EIP; c01aa5ac <_pagebuf_free_object+108/134>   <=====

>>ebx; cef3d000 <_end+ece0e4c/3fda1e4c>
>>ecx; cef3c000 <_end+ecdfe4c/3fda1e4c>
>>edx; c1254e28 <_end+ff8c74/3fda1e4c>
>>ebp; c12f3b18 <_end+1097964/3fda1e4c>
>>esp; c12f3b0c <_end+1097958/3fda1e4c>

Trace; c01aaf90 <pagebuf_free+24/30>
Trace; c019669b <xlog_find_verify_cycle+18b/1e0>

Code;  c01aa5ac <_pagebuf_free_object+108/134>
00000000 <_EIP>:
Code;  c01aa5ac <_pagebuf_free_object+108/134>   <=====
   0:   0f 0b                     ud2a      <=====
Code;  c01aa5ae <_pagebuf_free_object+10a/134>
   2:   0b 01                     or     (%ecx),%eax
Code;  c01aa5b0 <_pagebuf_free_object+10c/134>
   4:   d6                        (bad)
Code;  c01aa5b1 <_pagebuf_free_object+10d/134>
   5:   cb                        lret
Code;  c01aa5b2 <_pagebuf_free_object+10e/134>
   6:   1f                        pop    %ds
Code;  c01aa5b3 <_pagebuf_free_object+10f/134>
   7:   c0 eb 8e                  shr    $0x8e,%bl
Code;  c01aa5b6 <_pagebuf_free_object+112/134>
   a:   ff 73 54                  pushl  0x54(%ebx)
Code;  c01aa5b9 <_pagebuf_free_object+115/134>
   d:   eb b3                     jmp    ffffffc2 <_EIP+0xffffffc2>
Code;  c01aa5bb <_pagebuf_free_object+117/134>
   f:   90                        nop
Code;  c01aa5bc <_pagebuf_free_object+118/134>
  10:   53                        push   %ebx
Code;  c01aa5bd <_pagebuf_free_object+119/134>
  11:   e8 0e 11 00 00            call   1124 <_EIP+0x1124>

 <0>Kernel panic: Attempted to kill the idle task!
---


^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: [Oops]  i386 mm/slab.c (cache_flusharray)
  2003-12-03 23:06 pinotj
@ 2003-12-03 23:26 ` Linus Torvalds
  0 siblings, 0 replies; 32+ messages in thread
From: Linus Torvalds @ 2003-12-03 23:26 UTC (permalink / raw)
  To: pinotj; +Cc: nathans, manfred, akpm, linux-kernel




On Thu, 4 Dec 2003 pinotj@club-internet.fr wrote:
>
> [slab.c patch from Linus]
>
> I tried the patch on the same small config (XFS and
> CONFIG_DEBUG_PAGEALLOC enabled) and I got oops at the beginning of boot
> sequence. I spent some times to write this down but I'm not so sure it's
> a good news. Just say me it's not a hw problem...

Sorry - it's not a hw problem, and in fact it's a problem with my slab
debugging patch: please don't use CONFIG_DEBUG_SLAB together with my
crappy patch. My patch wants _only_ CONFIG_DEBUG_PAGEALLOC instead of
using the SLAB debugger.

So please turn the slab debugger off, and depend on CONFIG_DEBUG_PAGEALLOC
entirely and try again,

		Linus

^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: [Oops]  i386 mm/slab.c (cache_flusharray)
@ 2003-12-03 23:06 pinotj
  2003-12-03 23:26 ` Linus Torvalds
  0 siblings, 1 reply; 32+ messages in thread
From: pinotj @ 2003-12-03 23:06 UTC (permalink / raw)
  To: torvalds, nathans; +Cc: manfred, akpm, linux-kernel

I tried the "small kernel" test11 with Ext3 support instead of XFS by booting on my slack partition.

I confirm to have the same problem without XFS, oops as usual when I try to compile a kernel:

---
slab: double free detected in cache 'buffer_head' objp c605733c, objnr 9, slabp c6057000, s_mem c6057120, bufctl 12
---

[slab.c patch from Linus]

I tried the patch on the same small config (XFS and CONFIG_DEBUG_PAGEALLOC enabled) and I got oops at the beginning of boot sequence. I spent some times to write this down but I'm not so sure it's a good news. Just say me it's not a hw problem...

---
kernel BUG at mm/slab.c:1655!
invalid operand: 0000 [#1]
CPU:    0
EIP:    0060:[<c013bf35>]    Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010046
eax: c0243c00   ebx: 0027f448   ecx: 00000000   edx: cffb5000
esi: cffb6000   edi: cffb5000   ebp: c027bf70   esp: c027bf60
ds: 007b   es: 007b   ss: 0068
Stack: 00000286 00000000 cffb6000 cffb5000 c027bf94 c013c1a5 cffb5000 cffb6044
       cffb6000 cffb5000 cffb6000 cffb6000 c029be80 c027bfb0 c013c413 cffb6000
       00000008 00000004 00000000 000000d0 c027bfec c0283d8a cffb6000 00000090
Call Trace:
 [<c013c1a5>] do_tune_cpucache+0x1cd/0x3fc
 [<c013c413>] enable_cpucache+0x3f/0x64
 [<c0283d8a>] kmem_cache_init+0x1ae/0x280
 [<c0105000>] _stext+0x0/0x20
 [<c027c5af>] start_kernel+0xb3/0x138
Code: 0f 0b 77 06 75 2f 24 c0 8b 15 c4 bf 29 c0 eb af 8d 76 00 57


>>EIP; c013bf35 <kfree+81/b0>   <=====

>>eax; c0243c00 <bpp4tab+5f40/13436>
>>edx; cffb5000 <_end+fd0c990/3fd55990>
>>esi; cffb6000 <_end+fd0d990/3fd55990>
>>edi; cffb5000 <_end+fd0c990/3fd55990>
>>ebp; c027bf70 <init_thread_union+1f70/2000>
>>esp; c027bf60 <init_thread_union+1f60/2000>

Trace; c013c1a5 <do_tune_cpucache+1cd/3fc>
Trace; c013c413 <enable_cpucache+3f/64>
Trace; c0283d8a <kmem_cache_init+1ae/280>
Trace; c0105000 <_stext+0/0>
Trace; c027c5af <start_kernel+b3/138>

Code;  c013bf35 <kfree+81/b0>
00000000 <_EIP>:
Code;  c013bf35 <kfree+81/b0>   <=====
   0:   0f 0b                     ud2a      <=====
Code;  c013bf37 <kfree+83/b0>
   2:   77 06                     ja     a <_EIP+0xa>
Code;  c013bf39 <kfree+85/b0>
   4:   75 2f                     jne    35 <_EIP+0x35>
Code;  c013bf3b <kfree+87/b0>
   6:   24 c0                     and    $0xc0,%al
Code;  c013bf3d <kfree+89/b0>
   8:   8b 15 c4 bf 29 c0         mov    0xc029bfc4,%edx
Code;  c013bf43 <kfree+8f/b0>
   e:   eb af                     jmp    ffffffbf <_EIP+0xffffffbf>
Code;  c013bf45 <kfree+91/b0>
  10:   8d 76 00                  lea    0x0(%esi),%esi
Code;  c013bf48 <kfree+94/b0>
  13:   57                        push   %edi

 <0>Kernel panic: Attempted to kill the idle task!
---

Jerome


^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: [Oops]  i386 mm/slab.c (cache_flusharray)
  2003-12-02 18:05       ` Mike Fedyk
@ 2003-12-02 20:05         ` Nathan Scott
  0 siblings, 0 replies; 32+ messages in thread
From: Nathan Scott @ 2003-12-02 20:05 UTC (permalink / raw)
  To: Kernel Mailing List

On Tue, Dec 02, 2003 at 10:05:40AM -0800, Mike Fedyk wrote:
> On Tue, Dec 02, 2003 at 05:44:18PM +1100, Nathan Scott wrote:
> > I'm not seeing anything to suggest random slab corruption, and I'm
> > so far unable to trip things up as easily as you're able to Jerome.
> > Do you have just a very small amount of memory perhaps?  I can try
> > running while very low on memory, but thats the only other obvious
> > thing I can think of atm.
> 
> How about XFS on DM on RAID?

I didn't see references to those in Jeromes original description,
so haven't been testing those in this context.  But I need to do
more testing on those subsystems anyway, so, will do.

cheers.

-- 
Nathan

^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: [Oops]  i386 mm/slab.c (cache_flusharray)
  2003-12-02  6:44     ` Nathan Scott
@ 2003-12-02 18:05       ` Mike Fedyk
  2003-12-02 20:05         ` Nathan Scott
  0 siblings, 1 reply; 32+ messages in thread
From: Mike Fedyk @ 2003-12-02 18:05 UTC (permalink / raw)
  To: Nathan Scott
  Cc: Linus Torvalds, pinotj, manfred, Andrew Morton, Kernel Mailing List

On Tue, Dec 02, 2003 at 05:44:18PM +1100, Nathan Scott wrote:
> I'm not seeing anything to suggest random slab corruption, and I'm
> so far unable to trip things up as easily as you're able to Jerome.
> Do you have just a very small amount of memory perhaps?  I can try
> running while very low on memory, but thats the only other obvious
> thing I can think of atm.

How about XFS on DM on RAID?

^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: [Oops]  i386 mm/slab.c (cache_flusharray)
  2003-12-02  1:37   ` Nathan Scott
@ 2003-12-02  6:44     ` Nathan Scott
  2003-12-02 18:05       ` Mike Fedyk
  0 siblings, 1 reply; 32+ messages in thread
From: Nathan Scott @ 2003-12-02  6:44 UTC (permalink / raw)
  To: Linus Torvalds, pinotj; +Cc: manfred, Andrew Morton, Kernel Mailing List

Hi there,

On Tue, Dec 02, 2003 at 12:37:16PM +1100, Nathan Scott wrote:
> On Mon, Dec 01, 2003 at 04:36:33PM -0800, Linus Torvalds wrote:
> > 
> > I assume it's not an option to try another filesystem on this setup, but
> > it's entirely possible that the 2.6.x buffer-head removal has impacted XFS
> > negatively - although I'm a bit surprised at how easily you seem to show
> > problems, since XFS actually has active maintenance.
> > 
> > Nathan - I don't know if you follow linux-kernel, but Jerome Pinot has
> 
> Yep, although I try to filter out "noise" and have inadvertently
> missed this discussion so far.
> 
> > been having bad slab problems for some time now. Do normal XFS users
> > compile with slab debugging turned on?
> 
> Hmm - I know I do - my nightly QA testing runs with this set.
> Let me dig through the archives and catch up a bit on this issue;
> I'll get back to you.

OK, I've run XFS through hours and hours of very heavy stress now,
using a variety of different tests, and have tried different mount
and mkfs options as well.  And with a few kernel compiles thrown in
in the background for good measure.  Either we have quite different
hardware configs, compilers, etc; or this is something else.  This
was done with preempt enabled too (which I usually test without).

I'm not seeing anything to suggest random slab corruption, and I'm
so far unable to trip things up as easily as you're able to Jerome.
Do you have just a very small amount of memory perhaps?  I can try
running while very low on memory, but thats the only other obvious
thing I can think of atm.

cheers.

-- 
Nathan

^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: [Oops]  i386 mm/slab.c (cache_flusharray)
  2003-12-02  0:36 ` Linus Torvalds
@ 2003-12-02  1:37   ` Nathan Scott
  2003-12-02  6:44     ` Nathan Scott
  0 siblings, 1 reply; 32+ messages in thread
From: Nathan Scott @ 2003-12-02  1:37 UTC (permalink / raw)
  To: Linus Torvalds; +Cc: pinotj, manfred, Andrew Morton, Kernel Mailing List

On Mon, Dec 01, 2003 at 04:36:33PM -0800, Linus Torvalds wrote:
> 
> I assume it's not an option to try another filesystem on this setup, but
> it's entirely possible that the 2.6.x buffer-head removal has impacted XFS
> negatively - although I'm a bit surprised at how easily you seem to show
> problems, since XFS actually has active maintenance.
> 
> Nathan - I don't know if you follow linux-kernel, but Jerome Pinot has

Yep, although I try to filter out "noise" and have inadvertently
missed this discussion so far.

> been having bad slab problems for some time now. Do normal XFS users
> compile with slab debugging turned on?

Hmm - I know I do - my nightly QA testing runs with this set.
Let me dig through the archives and catch up a bit on this issue;
I'll get back to you.

thanks.

-- 
Nathan

^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: [Oops]  i386 mm/slab.c (cache_flusharray)
  2003-11-27 18:42 pinotj
  2003-11-27 18:55 ` Manfred Spraul
@ 2003-12-02  1:03 ` Mike Fedyk
  1 sibling, 0 replies; 32+ messages in thread
From: Mike Fedyk @ 2003-12-02  1:03 UTC (permalink / raw)
  To: pinotj; +Cc: manfred, torvalds, akpm, linux-kernel

On Thu, Nov 27, 2003 at 07:42:39PM +0100, pinotj@club-internet.fr wrote:
> first, some news
> 
> 2.6.0-test11 makes same oops during second compilation of kernel. The vanilla kernel with PREEMPT always oops the same way. No matter, it's always reproductible.
> 
> 2.6.0-test11 + Manfred's patch doesn't hang but I found a slab error in the logs that occured during a compilation. (I didn't find this for -test10, I was lucky ?)
> 
> So, there is no more way for my system to run a kernel > -test9 without problem.

Can you try some of the test9-bkN kernels and see where the problem starts?

^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: [Oops]  i386 mm/slab.c (cache_flusharray)
  2003-11-27 18:42 pinotj
@ 2003-11-27 18:55 ` Manfred Spraul
  2003-12-02  1:03 ` Mike Fedyk
  1 sibling, 0 replies; 32+ messages in thread
From: Manfred Spraul @ 2003-11-27 18:55 UTC (permalink / raw)
  To: pinotj; +Cc: torvalds, akpm, linux-kernel

pinotj@club-internet.fr wrote:

>Thanks for your explanation.
>Should I try with L1 and/or L2 cache disable on my computer (I don't know if it's safe) ?
>I trust my hardware but it's better to get some facts.
>
No, it wouldn't help. Something in the kernel randomly corrupts memory. 
I'm certain that it's not slab. I'm also fairly certain that it's not 
the hardware - IBM guys reproduced corruptions on both ppc64 and i386 
systems (bugzilla 1097 and 1497). The corrupted object is the slab 
structure or the bufctl entries - data near the beginning of a page. But 
I have no idea how to pinpoint it.

--
    Manfred


^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: [Oops]  i386 mm/slab.c (cache_flusharray)
  2003-11-22  7:47 Re: " pinotj
@ 2003-11-22 10:55 ` Manfred Spraul
  0 siblings, 0 replies; 32+ messages in thread
From: Manfred Spraul @ 2003-11-22 10:55 UTC (permalink / raw)
  To: pinotj; +Cc: torvalds, akpm, linux-kernel

pinotj@club-internet.fr wrote:

>c6fd7870: redzone 1: 0x170fc2a5, redzone 2: 0x160fc2a5.
>
Single bit error: redzone 2 must be 0x170fc2a5.

>---
>System looks OK, I tried a second compilation just after and this time I got an oops:
>---
>slab: double free detected in cache 'buffer_head', objp cc3f9798, objnr 26, slabp cc3f9000, s_mem cc3f9180 bufctl f7ffffff.
>  
>
Good.

+#define BUFCTL_END	0xfeffFFFF
+#define BUFCTL_FREE	0xf7ffFFFE
+#define	SLAB_LIMIT	0xf0ffFFFD

f7ffffff is not a valid value, slab never writes that into a bufctl. 
Someone did a ++ or "|= 1", or a hw bug.
I think the Athlon cpus have ECC for the L2 cache - could you check in 
the bios that ECC checking is enabled?

--
    Manfred


^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: [Oops]  i386 mm/slab.c (cache_flusharray)
  2003-11-21 18:12 pinotj
@ 2003-11-21 18:58 ` Manfred Spraul
  0 siblings, 0 replies; 32+ messages in thread
From: Manfred Spraul @ 2003-11-21 18:58 UTC (permalink / raw)
  To: pinotj; +Cc: akpm, linux-kernel

[-- Attachment #1: Type: text/plain, Size: 1016 bytes --]

pinotj@club-internet.fr wrote:

>0. Increase verbosity of the printk (thanks to Manfred):
>(compilation of kernel)
>slab: double free detected in cache 'buffer_head', objp c4c8e3d8, objnr 10,
>slabp c4c8e000, s_mem c4c8e180, bufctl ffffffff.
>(compilation of firebird)
>slab: double free detected in cache 'pte_chain', objp c18a6600, objnr 10,
>slabp c18a6000, s_mem c18a6100, bufctl ffffffff.
>  
>
The correct value for the bufctl would be 0xfffffffe - a single bit is 
wrong, but OTHO 0xffffffff is also a valid value.
Two different caches are affected.
The addresses of the corrupted variable differ, the offset into the page 
is identical. I think that rules out bad memory. That leaves a bug in 
slab.c, a bad bit in the L1/L2 cache, or random pointer scribbling.
Jerome, could you try the attached patch? It changes the magic constants 
that are used by slab.c. And please pay attention to the objnr: Is it 
always objnr 10, slabp xxxxx000, or do you see other values as well?

--
    Manfred

--
    Manfred

[-- Attachment #2: x --]
[-- Type: text/plain, Size: 496 bytes --]

--- 2.6/mm/slab.c	2003-11-18 18:18:20.000000000 +0100
+++ build-2.6/mm/slab.c	2003-11-21 19:50:02.000000000 +0100
@@ -153,9 +153,9 @@
  * is less than 512 (PAGE_SIZE<<3), but greater than 256.
  */
 
-#define BUFCTL_END	0xffffFFFF
-#define BUFCTL_FREE	0xffffFFFE
-#define	SLAB_LIMIT	0xffffFFFD
+#define BUFCTL_END	0xfeffFFFF
+#define BUFCTL_FREE	0xf7ffFFFE
+#define	SLAB_LIMIT	0xf0ffFFFD
 typedef unsigned int kmem_bufctl_t;
 
 /* Max number of objs-per-slab for caches which use off-slab slabs.

^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: [Oops]  i386 mm/slab.c (cache_flusharray)
  2003-11-20  1:50 pinotj
@ 2003-11-20  2:09 ` Andrew Morton
  0 siblings, 0 replies; 32+ messages in thread
From: Andrew Morton @ 2003-11-20  2:09 UTC (permalink / raw)
  To: pinotj; +Cc: linux-kernel

pinotj@club-internet.fr wrote:
>
> kernel BUG at mm/slab.c:1957!
>  ---
> 
>  >Don't know, sorry.
> 
>  Is there any thing I can do to help figure out where does the problem comes from ? 

Well it's interesting that it is repeatable.

First thing to do is to eliminate hardware failures:

1: Is the oops always the same, or does the machine crash in other ways,
   with different backtraces?

2: Try running memtest86 on that machine for 12 hours or more.

3: Can the problem be reproduced on other machines?

4: try a different compiler version.

Thanks.

^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: [Oops]  i386 mm/slab.c (cache_flusharray)
  2003-11-19 18:19 pinotj
@ 2003-11-20  1:07 ` Andrew Morton
  0 siblings, 0 replies; 32+ messages in thread
From: Andrew Morton @ 2003-11-20  1:07 UTC (permalink / raw)
  To: pinotj; +Cc: linux-kernel

pinotj@club-internet.fr wrote:
>
> kernel BUG at mm/slab.c:1957!
> invalid operand: 0000 [#1]
> CPU:    0
> EIP:    0060:[free_block+336/752]    Not tainted
> EIP:    0060:[<c015ad40>]    Not tainted
> Using defaults from ksymoops -t elf32-i386 -a i386
> EFLAGS: 00010096
> eax: 00000045   ebx: 00000006   ecx: c0693854   edx: c056e4f8
> esi: cd09a000   edi: cd09a018   ebp: cf821c68   esp: cf821c3c
> ds: 007b   es: 007b   ss: 0068
> Stack: c0502240 c0502e1d cd09af18 c0652a00 00000001 0000003a cd09af18 0000000f
>        cffdef08 c4bcd180 00000010 cf821ca0 c015afba cffed800 cffdef08 00000010
>        00000282 c1161ca0 00000000 00000001 cffee730 00000010 00010c00 c4bcd180
> Call Trace:
>  [<c015afba>] cache_flusharray+0xda/0x2b0
>  [<c015b7ad>] kmem_cache_free+0x1ad/0x3a0
>  [<c018158c>] free_buffer_head+0x2c/0x60
>  [<c018158c>] free_buffer_head+0x2c/0x60

urgh, there are several reports of this and it's always the buffer_head
slab.  The code in there is trivial so perhaps it's just that the large
number of buffer_heads makes them a fat target.

You should have also seen the message "slab: double free detected in cache
'buffer_head', objp 0xNNNNNNNN".

Don't know, sorry.

^ permalink raw reply	[flat|nested] 32+ messages in thread

* [Oops]  i386 mm/slab.c (cache_flusharray)
@ 2003-11-19 18:19 pinotj
  2003-11-20  1:07 ` Andrew Morton
  0 siblings, 1 reply; 32+ messages in thread
From: pinotj @ 2003-11-19 18:19 UTC (permalink / raw)
  To: linux-kernel

Here is an Oops from the kernel 2.6.0-test9 + cset-20031115_0206.txt.gz (means all current patches till 03/11/14 [PATCH] PPC32: cancel syscall restart on signal delivery ).

Seems to come from the cache_flusharray function of mm/slab.c
It occurs in general during heavy load (compilation of kernel or xfree86...), is quite reproductible but not automatic.
Same Oops on different distro/gcc (slack 9.1 et lfs 5.0).
Arch is i386 (AMD athlon-tbird 1.2GHz)

I never had this problem with 2.6.0-test8 and before.

This oops is very annoying, I got it around 5 times a day. I'm sorry but I don't have the skill to patch this. If someone can help :-) Please CC me in answer, I'm not on the lkml.

Regards,

Jerome Pinot

---
ksymoops 2.4.9 on i686 2.6.0-test9.  Options used
     -V (default)
     -K (specified)
     -l /proc/modules (default)
     -o /mnt/lfs/lib/modules/2.6.0-test9 (specified)
     -m /mnt/lfs/boot/System.map (specified)

No modules in ksyms, skipping objects
No ksyms, skipping lsmod
kernel BUG at mm/slab.c:1957!
invalid operand: 0000 [#1]
CPU:    0
EIP:    0060:[free_block+336/752]    Not tainted
EIP:    0060:[<c015ad40>]    Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010096
eax: 00000045   ebx: 00000006   ecx: c0693854   edx: c056e4f8
esi: cd09a000   edi: cd09a018   ebp: cf821c68   esp: cf821c3c
ds: 007b   es: 007b   ss: 0068
Stack: c0502240 c0502e1d cd09af18 c0652a00 00000001 0000003a cd09af18 0000000f
       cffdef08 c4bcd180 00000010 cf821ca0 c015afba cffed800 cffdef08 00000010
       00000282 c1161ca0 00000000 00000001 cffee730 00000010 00010c00 c4bcd180
Call Trace:
 [<c015afba>] cache_flusharray+0xda/0x2b0
 [<c015b7ad>] kmem_cache_free+0x1ad/0x3a0
 [<c018158c>] free_buffer_head+0x2c/0x60
 [<c018158c>] free_buffer_head+0x2c/0x60
 [<c0181353>] try_to_free_buffers+0x143/0x220
 [<c0294bab>] linvfs_release_page+0x7b/0x80
 [<c017f243>] try_to_release_page+0x53/0x60
 [<c015fec7>] shrink_list+0x8b7/0xac0
 [<c010cadc>] common_interrupt+0x18/0x20
 [<c015e489>] __pagevec_release+0x29/0x40
 [<c01602d5>] shrink_cache+0x205/0x610
 [<c0161228>] shrink_zone+0x78/0xa0
 [<c01615fa>] balance_pgdat+0x17a/0x200
 [<c0161759>] kswapd+0xd9/0xf0
 [<c01274f0>] autoremove_wake_function+0x0/0x50
 [<c010c046>] ret_from_fork+0x6/0x14
 [<c01274f0>] autoremove_wake_function+0x0/0x50
 [<c0161680>] kswapd+0x0/0xf0
 [<c01092a9>] kernel_thread_helper+0x5/0xc
Code: 0f 0b a5 07 82 15 50 c0 8b 46 14 8b 4d e8 31 db 89 04 8f 89


>>EIP; c015ad40 <free_block+150/2f0>   <=====

>>ecx; c0693854 <per_cpu__runqueues+4b4/940>
>>edx; c056e4f8 <log_wait+18/20>
>>esi; cd09a000 <__crc_unregister_sound_dsp+164f0/974e1>
>>edi; cd09a018 <__crc_unregister_sound_dsp+16508/974e1>
>>ebp; cf821c68 <__crc_scsi_register_driver+6c7b9/1f2ec0>
>>esp; cf821c3c <__crc_scsi_register_driver+6c78d/1f2ec0>

Trace; c015afba <cache_flusharray+da/2b0>
Trace; c015b7ad <kmem_cache_free+1ad/3a0>
Trace; c018158c <free_buffer_head+2c/60>
Trace; c018158c <free_buffer_head+2c/60>
Trace; c0181353 <try_to_free_buffers+143/220>
Trace; c0294bab <linvfs_release_page+7b/80>
Trace; c017f243 <try_to_release_page+53/60>
Trace; c015fec7 <shrink_list+8b7/ac0>
Trace; c010cadc <common_interrupt+18/20>
Trace; c015e489 <__pagevec_release+29/40>
Trace; c01602d5 <shrink_cache+205/610>
Trace; c0161228 <shrink_zone+78/a0>
Trace; c01615fa <balance_pgdat+17a/200>
Trace; c0161759 <kswapd+d9/f0>
Trace; c01274f0 <autoremove_wake_function+0/50>
Trace; c010c046 <ret_from_fork+6/14>
Trace; c01274f0 <autoremove_wake_function+0/50>
Trace; c0161680 <kswapd+0/f0>
Trace; c01092a9 <kernel_thread_helper+5/c>

Code;  c015ad40 <free_block+150/2f0>
00000000 <_EIP>:
Code;  c015ad40 <free_block+150/2f0>   <=====
   0:   0f 0b                     ud2a      <=====
Code;  c015ad42 <free_block+152/2f0>
   2:   a5                        movsl  %ds:(%esi),%es:(%edi)
Code;  c015ad43 <free_block+153/2f0>
   3:   07                        pop    %es
Code;  c015ad44 <free_block+154/2f0>
   4:   82                        (bad)
Code;  c015ad45 <free_block+155/2f0>
   5:   15 50 c0 8b 46            adc    $0x468bc050,%eax
Code;  c015ad4a <free_block+15a/2f0>
   a:   14 8b                     adc    $0x8b,%al
Code;  c015ad4c <free_block+15c/2f0>
   c:   4d                        dec    %ebp
Code;  c015ad4d <free_block+15d/2f0>
   d:   e8 31 db 89 04            call   489db43 <_EIP+0x489db43>
Code;  c015ad52 <free_block+162/2f0>
  12:   8f 89 00 00 00 00         popl   0x0(%ecx)

Unable to handle kernel paging request at virtual address 00100104
c015ac3c
*pde = 00000000
Oops: 0002 [#2]
CPU:    0
EIP:    0060:[free_block+76/752]    Not tainted
EIP:    0060:[<c015ac3c>]    Not tainted
EFLAGS: 00010017
eax: 00100100   ebx: 00000000   ecx: cd09a810   edx: 00200200
esi: cd09a000   edi: c7430018   ebp: c558bafc   esp: c558bad0
ds: 007b   es: 007b   ss: 0068
Stack: c558badc 00000086 80010c00 ccf77000 cffdc080 0000001a cd09a810 0000000b 
       cffdef08 cd09a180 00000010 c558bb34 c015afba cffed800 cffdef08 00000010
       00000286 c11d0c68 00000000 00000001 cffee730 00000010 00010c00 cd09a180 
Call Trace:
 [<c015afba>] cache_flusharray+0xda/0x2b0
 [<c015b7ad>] kmem_cache_free+0x1ad/0x3a0
 [<c018158c>] free_buffer_head+0x2c/0x60
 [<c018158c>] free_buffer_head+0x2c/0x60
 [<c0181353>] try_to_free_buffers+0x143/0x220
 [<c0294bab>] linvfs_release_page+0x7b/0x80
 [<c017f243>] try_to_release_page+0x53/0x60
 [<c015fec7>] shrink_list+0x8b7/0xac0
 [<c0123eef>] schedule+0x36f/0x8a0
 [<c015e489>] __pagevec_release+0x29/0x40
 [<c01602d5>] shrink_cache+0x205/0x610
 [<c0127515>] autoremove_wake_function+0x25/0x50
 [<c0161228>] shrink_zone+0x78/0xa0
 [<c01612ee>] shrink_caches+0x9e/0xc0
 [<c01613b7>] try_to_free_pages+0xa7/0x170
 [<c015541a>] __alloc_pages+0x1fa/0x380
 [<c0165c17>] do_anonymous_page+0xc7/0x520
 [<c017a899>] do_sync_write+0x89/0xc0
 [<c01668f2>] handle_mm_fault+0x132/0x310
 [<c0121940>] do_page_fault+0x350/0x56a
 [<c01698de>] do_brk+0x14e/0x230
 [<c016790e>] sys_brk+0xee/0x120
 [<c01215f0>] do_page_fault+0x0/0x56a
 [<c010cb79>] error_code+0x2d/0x38
Code: 89 50 04 89 02 c7 46 04 00 02 20 00 c7 06 00 01 10 00 89 c8


>>EIP; c015ac3c <free_block+4c/2f0>   <=====

>>eax; 00100100 <__crc_prepare_to_wait_exclusive+ce3e5/1424fd>
>>ecx; cd09a810 <__crc_unregister_sound_dsp+16d00/974e1>
>>edx; 00200200 <__crc___user_walk+3d8ad/1cb584>
>>esi; cd09a000 <__crc_unregister_sound_dsp+164f0/974e1>
>>edi; c7430018 <__crc_inet_getsockopt+60fc6/20c3c2>
>>ebp; c558bafc <__crc_fat_write_inode+336fc/668510>
>>esp; c558bad0 <__crc_fat_write_inode+336d0/668510>

Trace; c015afba <cache_flusharray+da/2b0>
Trace; c015b7ad <kmem_cache_free+1ad/3a0>
Trace; c018158c <free_buffer_head+2c/60>
Trace; c018158c <free_buffer_head+2c/60>
Trace; c0181353 <try_to_free_buffers+143/220>
Trace; c0294bab <linvfs_release_page+7b/80>
Trace; c017f243 <try_to_release_page+53/60>
Trace; c015fec7 <shrink_list+8b7/ac0>
Trace; c0123eef <schedule+36f/8a0>
Trace; c015e489 <__pagevec_release+29/40>
Trace; c01602d5 <shrink_cache+205/610>
Trace; c0127515 <autoremove_wake_function+25/50>
Trace; c0161228 <shrink_zone+78/a0>
Trace; c01612ee <shrink_caches+9e/c0>
Trace; c01613b7 <try_to_free_pages+a7/170>
Trace; c015541a <__alloc_pages+1fa/380>
Trace; c0165c17 <do_anonymous_page+c7/520>
Trace; c017a899 <do_sync_write+89/c0>
Trace; c01668f2 <handle_mm_fault+132/310>
Trace; c0121940 <do_page_fault+350/56a>
Trace; c01698de <do_brk+14e/230>
Trace; c016790e <sys_brk+ee/120>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c010cb79 <error_code+2d/38>

Code;  c015ac3c <free_block+4c/2f0>
00000000 <_EIP>:
Code;  c015ac3c <free_block+4c/2f0>   <=====
   0:   89 50 04                  mov    %edx,0x4(%eax)   <=====
Code;  c015ac3f <free_block+4f/2f0>
   3:   89 02                     mov    %eax,(%edx)
Code;  c015ac41 <free_block+51/2f0>
   5:   c7 46 04 00 02 20 00      movl   $0x200200,0x4(%esi)
Code;  c015ac48 <free_block+58/2f0>
   c:   c7 06 00 01 10 00         movl   $0x100100,(%esi)
Code;  c015ac4e <free_block+5e/2f0>
  12:   89 c8                     mov    %ecx,%eax

Call Trace:
 [<c0124411>] schedule+0x891/0x8a0
 [<c0163ae1>] unmap_page_range+0x41/0x70
 [<c0163d24>] unmap_vmas+0x214/0x330
 [<c0169adb>] exit_mmap+0xcb/0x2b0
 [<c0127935>] mmput+0xb5/0x150
 [<c012e062>] do_exit+0x1b2/0x960
 [<c01215f0>] do_page_fault+0x0/0x56a
 [<c01215f0>] do_page_fault+0x0/0x56a
 [<c010d367>] die+0x227/0x230
 [<c01217d6>] do_page_fault+0x1e6/0x56a
 [<c0122d1e>] recalc_task_prio+0x8e/0x1b0
 [<c0122f9a>] try_to_wake_up+0x15a/0x290
 [<c01215f0>] do_page_fault+0x0/0x56a
 [<c010cb79>] error_code+0x2d/0x38
 [<c015ac3c>] free_block+0x4c/0x2f0
 [<c015afba>] cache_flusharray+0xda/0x2b0
 [<c015b7ad>] kmem_cache_free+0x1ad/0x3a0
 [<c018158c>] free_buffer_head+0x2c/0x60
 [<c018158c>] free_buffer_head+0x2c/0x60
 [<c0181353>] try_to_free_buffers+0x143/0x220
 [<c0294bab>] linvfs_release_page+0x7b/0x80
 [<c017f243>] try_to_release_page+0x53/0x60
 [<c015fec7>] shrink_list+0x8b7/0xac0
 [<c0123eef>] schedule+0x36f/0x8a0
 [<c015e489>] __pagevec_release+0x29/0x40
 [<c01602d5>] shrink_cache+0x205/0x610
 [<c0127515>] autoremove_wake_function+0x25/0x50
 [<c0161228>] shrink_zone+0x78/0xa0
 [<c01612ee>] shrink_caches+0x9e/0xc0
 [<c01613b7>] try_to_free_pages+0xa7/0x170
 [<c015541a>] __alloc_pages+0x1fa/0x380
 [<c0165c17>] do_anonymous_page+0xc7/0x520
 [<c017a899>] do_sync_write+0x89/0xc0
 [<c01668f2>] handle_mm_fault+0x132/0x310
 [<c0121940>] do_page_fault+0x350/0x56a
 [<c01698de>] do_brk+0x14e/0x230
 [<c016790e>] sys_brk+0xee/0x120
 [<c01215f0>] do_page_fault+0x0/0x56a
 [<c010cb79>] error_code+0x2d/0x38
Call Trace:
 [<c0124411>] schedule+0x891/0x8a0
 [<c0163ae1>] unmap_page_range+0x41/0x70
 [<c0163d24>] unmap_vmas+0x214/0x330
 [<c0169adb>] exit_mmap+0xcb/0x2b0
 [<c0127935>] mmput+0xb5/0x150
 [<c012e062>] do_exit+0x1b2/0x960
 [<c01215f0>] do_page_fault+0x0/0x56a
 [<c01215f0>] do_page_fault+0x0/0x56a
 [<c010d367>] die+0x227/0x230
 [<c01217d6>] do_page_fault+0x1e6/0x56a
 [<c0122d1e>] recalc_task_prio+0x8e/0x1b0
 [<c0122f9a>] try_to_wake_up+0x15a/0x290
 [<c01215f0>] do_page_fault+0x0/0x56a
 [<c010cb79>] error_code+0x2d/0x38
 [<c015ac3c>] free_block+0x4c/0x2f0
 [<c015afba>] cache_flusharray+0xda/0x2b0
 [<c015b7ad>] kmem_cache_free+0x1ad/0x3a0
 [<c018158c>] free_buffer_head+0x2c/0x60
 [<c018158c>] free_buffer_head+0x2c/0x60
 [<c0181353>] try_to_free_buffers+0x143/0x220
 [<c0294bab>] linvfs_release_page+0x7b/0x80
 [<c017f243>] try_to_release_page+0x53/0x60
 [<c015fec7>] shrink_list+0x8b7/0xac0
 [<c0123eef>] schedule+0x36f/0x8a0
 [<c015e489>] __pagevec_release+0x29/0x40
 [<c01602d5>] shrink_cache+0x205/0x610
 [<c0127515>] autoremove_wake_function+0x25/0x50
 [<c0161228>] shrink_zone+0x78/0xa0
 [<c01612ee>] shrink_caches+0x9e/0xc0
 [<c01613b7>] try_to_free_pages+0xa7/0x170
 [<c015541a>] __alloc_pages+0x1fa/0x380
 [<c0165c17>] do_anonymous_page+0xc7/0x520
 [<c017a899>] do_sync_write+0x89/0xc0
 [<c01668f2>] handle_mm_fault+0x132/0x310
 [<c0121940>] do_page_fault+0x350/0x56a
 [<c01698de>] do_brk+0x14e/0x230
 [<c016790e>] sys_brk+0xee/0x120
 [<c01215f0>] do_page_fault+0x0/0x56a
 [<c010cb79>] error_code+0x2d/0x38
Call Trace:
 [<c0124411>] schedule+0x891/0x8a0
 [<c0163ae1>] unmap_page_range+0x41/0x70
 [<c0163d24>] unmap_vmas+0x214/0x330
 [<c0169adb>] exit_mmap+0xcb/0x2b0
 [<c0127935>] mmput+0xb5/0x150
 [<c012e062>] do_exit+0x1b2/0x960
 [<c01215f0>] do_page_fault+0x0/0x56a
 [<c01215f0>] do_page_fault+0x0/0x56a
 [<c010d367>] die+0x227/0x230
 [<c01217d6>] do_page_fault+0x1e6/0x56a
 [<c0122d1e>] recalc_task_prio+0x8e/0x1b0
 [<c0122f9a>] try_to_wake_up+0x15a/0x290
 [<c01215f0>] do_page_fault+0x0/0x56a
 [<c010cb79>] error_code+0x2d/0x38
 [<c015ac3c>] free_block+0x4c/0x2f0
 [<c015afba>] cache_flusharray+0xda/0x2b0
 [<c015b7ad>] kmem_cache_free+0x1ad/0x3a0
 [<c018158c>] free_buffer_head+0x2c/0x60
 [<c018158c>] free_buffer_head+0x2c/0x60
 [<c0181353>] try_to_free_buffers+0x143/0x220
 [<c0294bab>] linvfs_release_page+0x7b/0x80
 [<c017f243>] try_to_release_page+0x53/0x60
 [<c015fec7>] shrink_list+0x8b7/0xac0
 [<c0123eef>] schedule+0x36f/0x8a0
 [<c015e489>] __pagevec_release+0x29/0x40
 [<c01602d5>] shrink_cache+0x205/0x610
 [<c0127515>] autoremove_wake_function+0x25/0x50
 [<c0161228>] shrink_zone+0x78/0xa0
 [<c01612ee>] shrink_caches+0x9e/0xc0
 [<c01613b7>] try_to_free_pages+0xa7/0x170
 [<c015541a>] __alloc_pages+0x1fa/0x380
 [<c0165c17>] do_anonymous_page+0xc7/0x520
 [<c017a899>] do_sync_write+0x89/0xc0
 [<c01668f2>] handle_mm_fault+0x132/0x310
 [<c0121940>] do_page_fault+0x350/0x56a
 [<c01698de>] do_brk+0x14e/0x230
 [<c016790e>] sys_brk+0xee/0x120
 [<c01215f0>] do_page_fault+0x0/0x56a
 [<c010cb79>] error_code+0x2d/0x38
Call Trace:
 [<c0126b1b>] __might_sleep+0xab/0xd0
 [<c01677b9>] remove_shared_vm_struct+0x39/0xa0
 [<c0169be1>] exit_mmap+0x1d1/0x2b0
 [<c0127935>] mmput+0xb5/0x150
 [<c012e062>] do_exit+0x1b2/0x960
 [<c01215f0>] do_page_fault+0x0/0x56a
 [<c01215f0>] do_page_fault+0x0/0x56a
 [<c010d367>] die+0x227/0x230
 [<c01217d6>] do_page_fault+0x1e6/0x56a
 [<c0122d1e>] recalc_task_prio+0x8e/0x1b0
 [<c0122f9a>] try_to_wake_up+0x15a/0x290
 [<c01215f0>] do_page_fault+0x0/0x56a
 [<c010cb79>] error_code+0x2d/0x38
 [<c015ac3c>] free_block+0x4c/0x2f0
 [<c015b7ad>] kmem_cache_free+0x1ad/0x3a0
 [<c018158c>] free_buffer_head+0x2c/0x60
 [<c018158c>] free_buffer_head+0x2c/0x60
 [<c0181353>] try_to_free_buffers+0x143/0x220
 [<c0294bab>] linvfs_release_page+0x7b/0x80
 [<c017f243>] try_to_release_page+0x53/0x60
 [<c015fec7>] shrink_list+0x8b7/0xac0
 [<c0123eef>] schedule+0x36f/0x8a0
 [<c015e489>] __pagevec_release+0x29/0x40
 [<c01602d5>] shrink_cache+0x205/0x610
 [<c0127515>] autoremove_wake_function+0x25/0x50
 [<c0161228>] shrink_zone+0x78/0xa0
 [<c01612ee>] shrink_caches+0x9e/0xc0
 [<c01613b7>] try_to_free_pages+0xa7/0x170
 [<c015541a>] __alloc_pages+0x1fa/0x380
 [<c0165c17>] do_anonymous_page+0xc7/0x520
 [<c017a899>] do_sync_write+0x89/0xc0
 [<c01668f2>] handle_mm_fault+0x132/0x310
 [<c0121940>] do_page_fault+0x350/0x56a
 [<c01698de>] do_brk+0x14e/0x230
 [<c016790e>] sys_brk+0xee/0x120
 [<c01215f0>] do_page_fault+0x0/0x56a
 [<c010cb79>] error_code+0x2d/0x38
Warning (Oops_read): Code line not seen, dumping what data is available


Trace; c0124411 <schedule+891/8a0>
Trace; c0163ae1 <unmap_page_range+41/70>
Trace; c0163d24 <unmap_vmas+214/330>
Trace; c0169adb <exit_mmap+cb/2b0>
Trace; c0127935 <mmput+b5/150>
Trace; c012e062 <do_exit+1b2/960>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c010d367 <die+227/230>
Trace; c01217d6 <do_page_fault+1e6/56a>
Trace; c0122d1e <recalc_task_prio+8e/1b0>
Trace; c0122f9a <try_to_wake_up+15a/290>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c010cb79 <error_code+2d/38>
Trace; c015ac3c <free_block+4c/2f0>
Trace; c015afba <cache_flusharray+da/2b0>
Trace; c015b7ad <kmem_cache_free+1ad/3a0>
Trace; c018158c <free_buffer_head+2c/60>
Trace; c018158c <free_buffer_head+2c/60>
Trace; c0181353 <try_to_free_buffers+143/220>
Trace; c0294bab <linvfs_release_page+7b/80>
Trace; c017f243 <try_to_release_page+53/60>
Trace; c015fec7 <shrink_list+8b7/ac0>
Trace; c0123eef <schedule+36f/8a0>
Trace; c015e489 <__pagevec_release+29/40>
Trace; c01602d5 <shrink_cache+205/610>
Trace; c0127515 <autoremove_wake_function+25/50>
Trace; c0161228 <shrink_zone+78/a0>
Trace; c01612ee <shrink_caches+9e/c0>
Trace; c01613b7 <try_to_free_pages+a7/170>
Trace; c015541a <__alloc_pages+1fa/380>
Trace; c0165c17 <do_anonymous_page+c7/520>
Trace; c017a899 <do_sync_write+89/c0>
Trace; c01668f2 <handle_mm_fault+132/310>
Trace; c0121940 <do_page_fault+350/56a>
Trace; c01698de <do_brk+14e/230>
Trace; c016790e <sys_brk+ee/120>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c010cb79 <error_code+2d/38>
Trace; c0124411 <schedule+891/8a0>
Trace; c0163ae1 <unmap_page_range+41/70>
Trace; c0163d24 <unmap_vmas+214/330>
Trace; c0169adb <exit_mmap+cb/2b0>
Trace; c0127935 <mmput+b5/150>
Trace; c012e062 <do_exit+1b2/960>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c010d367 <die+227/230>
Trace; c01217d6 <do_page_fault+1e6/56a>
Trace; c0122d1e <recalc_task_prio+8e/1b0>
Trace; c0122f9a <try_to_wake_up+15a/290>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c010cb79 <error_code+2d/38>
Trace; c015ac3c <free_block+4c/2f0>
Trace; c015afba <cache_flusharray+da/2b0>
Trace; c015b7ad <kmem_cache_free+1ad/3a0>
Trace; c018158c <free_buffer_head+2c/60>
Trace; c018158c <free_buffer_head+2c/60>
Trace; c0181353 <try_to_free_buffers+143/220>
Trace; c0294bab <linvfs_release_page+7b/80>
Trace; c017f243 <try_to_release_page+53/60>
Trace; c015fec7 <shrink_list+8b7/ac0>
Trace; c0123eef <schedule+36f/8a0>
Trace; c015e489 <__pagevec_release+29/40>
Trace; c01602d5 <shrink_cache+205/610>
Trace; c0127515 <autoremove_wake_function+25/50>
Trace; c0161228 <shrink_zone+78/a0>
Trace; c01612ee <shrink_caches+9e/c0>
Trace; c01613b7 <try_to_free_pages+a7/170>
Trace; c015541a <__alloc_pages+1fa/380>
Trace; c0165c17 <do_anonymous_page+c7/520>
Trace; c017a899 <do_sync_write+89/c0>
Trace; c01668f2 <handle_mm_fault+132/310>
Trace; c0121940 <do_page_fault+350/56a>
Trace; c01698de <do_brk+14e/230>
Trace; c016790e <sys_brk+ee/120>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c010cb79 <error_code+2d/38>
Trace; c0124411 <schedule+891/8a0>
Trace; c0163ae1 <unmap_page_range+41/70>
Trace; c0163d24 <unmap_vmas+214/330>
Trace; c0169adb <exit_mmap+cb/2b0>
Trace; c0127935 <mmput+b5/150>
Trace; c012e062 <do_exit+1b2/960>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c010d367 <die+227/230>
Trace; c01217d6 <do_page_fault+1e6/56a>
Trace; c0122d1e <recalc_task_prio+8e/1b0>
Trace; c0122f9a <try_to_wake_up+15a/290>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c010cb79 <error_code+2d/38>
Trace; c015ac3c <free_block+4c/2f0>
Trace; c015afba <cache_flusharray+da/2b0>
Trace; c015b7ad <kmem_cache_free+1ad/3a0>
Trace; c018158c <free_buffer_head+2c/60>
Trace; c018158c <free_buffer_head+2c/60>
Trace; c0181353 <try_to_free_buffers+143/220>
Trace; c0294bab <linvfs_release_page+7b/80>
Trace; c017f243 <try_to_release_page+53/60>
Trace; c015fec7 <shrink_list+8b7/ac0>
Trace; c0123eef <schedule+36f/8a0>
Trace; c015e489 <__pagevec_release+29/40>
Trace; c01602d5 <shrink_cache+205/610>
Trace; c0127515 <autoremove_wake_function+25/50>
Trace; c0161228 <shrink_zone+78/a0>
Trace; c01612ee <shrink_caches+9e/c0>
Trace; c01613b7 <try_to_free_pages+a7/170>
Trace; c015541a <__alloc_pages+1fa/380>
Trace; c0165c17 <do_anonymous_page+c7/520>
Trace; c017a899 <do_sync_write+89/c0>
Trace; c01668f2 <handle_mm_fault+132/310>
Trace; c0121940 <do_page_fault+350/56a>
Trace; c01698de <do_brk+14e/230>
Trace; c016790e <sys_brk+ee/120>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c010cb79 <error_code+2d/38>
Trace; c0126b1b <__might_sleep+ab/d0>
Trace; c01677b9 <remove_shared_vm_struct+39/a0>
Trace; c0169be1 <exit_mmap+1d1/2b0>
Trace; c0127935 <mmput+b5/150>
Trace; c012e062 <do_exit+1b2/960>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c010d367 <die+227/230>
Trace; c01217d6 <do_page_fault+1e6/56a>
Trace; c0122d1e <recalc_task_prio+8e/1b0>
Trace; c0122f9a <try_to_wake_up+15a/290>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c010cb79 <error_code+2d/38>
Trace; c015ac3c <free_block+4c/2f0>
Trace; c015b7ad <kmem_cache_free+1ad/3a0>
Trace; c018158c <free_buffer_head+2c/60>
Trace; c018158c <free_buffer_head+2c/60>
Trace; c0181353 <try_to_free_buffers+143/220>
Trace; c0294bab <linvfs_release_page+7b/80>
Trace; c017f243 <try_to_release_page+53/60>
Trace; c015fec7 <shrink_list+8b7/ac0>
Trace; c0123eef <schedule+36f/8a0>
Trace; c015e489 <__pagevec_release+29/40>
Trace; c01602d5 <shrink_cache+205/610>
Trace; c0127515 <autoremove_wake_function+25/50>
Trace; c0161228 <shrink_zone+78/a0>
Trace; c01612ee <shrink_caches+9e/c0>
Trace; c01613b7 <try_to_free_pages+a7/170>
Trace; c015541a <__alloc_pages+1fa/380>
Trace; c0165c17 <do_anonymous_page+c7/520>
Trace; c017a899 <do_sync_write+89/c0>
Trace; c01668f2 <handle_mm_fault+132/310>
Trace; c0121940 <do_page_fault+350/56a>
Trace; c01698de <do_brk+14e/230>
Trace; c016790e <sys_brk+ee/120>
Trace; c01215f0 <do_page_fault+0/56a>
Trace; c010cb79 <error_code+2d/38>


^ permalink raw reply	[flat|nested] 32+ messages in thread

end of thread, other threads:[~2003-12-12 20:08 UTC | newest]

Thread overview: 32+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2003-11-25 17:30 Re: [Oops] i386 mm/slab.c (cache_flusharray) pinotj
2003-11-25 22:51 ` Linus Torvalds
2003-11-27 18:07   ` Manfred Spraul
  -- strict thread matches above, loose matches on Subject: below --
2003-12-09  0:57 pinotj
2003-12-09  2:03 ` Nathan Scott
2003-12-09  7:21   ` Christoph Hellwig
2003-12-09 23:58     ` Nathan Scott
2003-12-12 19:00       ` Christoph Hellwig
2003-12-12 20:07         ` Manfred Spraul
2003-12-04 18:27 pinotj
2003-12-04 18:49 ` Linus Torvalds
2003-12-04 19:09   ` Linus Torvalds
2003-12-04 21:21     ` Nathan Scott
2003-12-05  7:14       ` Christoph Hellwig
2003-12-05  9:34         ` Nathan Scott
2003-12-05 14:22           ` Christoph Hellwig
2003-12-05  3:00     ` Nathan Scott
2003-12-05  6:40       ` Linus Torvalds
2003-12-04 19:19   ` Manfred Spraul
2003-12-04 21:26   ` Nathan Scott
2003-12-03 23:06 pinotj
2003-12-03 23:26 ` Linus Torvalds
2003-11-29 17:41 pinotj
2003-12-02  0:36 ` Linus Torvalds
2003-12-02  1:37   ` Nathan Scott
2003-12-02  6:44     ` Nathan Scott
2003-12-02 18:05       ` Mike Fedyk
2003-12-02 20:05         ` Nathan Scott
2003-11-27 18:42 pinotj
2003-11-27 18:55 ` Manfred Spraul
2003-12-02  1:03 ` Mike Fedyk
2003-11-22  7:47 Re: " pinotj
2003-11-22 10:55 ` Manfred Spraul
2003-11-21 18:12 pinotj
2003-11-21 18:58 ` Manfred Spraul
2003-11-20  1:50 pinotj
2003-11-20  2:09 ` Andrew Morton
2003-11-19 18:19 pinotj
2003-11-20  1:07 ` Andrew Morton

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).