[2.6.17.4] slabinfo.buffer_head increases

[2.6.17.4] slabinfo.buffer_head increases

[Guennadi Liakhovetski]

Hi

I am obsering a steadily increasing buffer_head value in slabinfo under 
2.6.17.4. I searched the net / archives and didn't find anything 
directly relevant. Does anyone have an idea or how shall we debug it?

I first noticed this "feature" on a 2.6.17-rc5 based ARM-system, where if 
I stop some user-space applications, the number stop increasing.

I was also able to reproduce it on a SuSE-9.0 system, where I wasn't able 
to stop this growth even as I stopped all possible services. Then the 
process list looked like this:

  PID TTY      STAT   TIME COMMAND
    1 ?        S      0:00 init [5]  
    2 ?        SWN    0:00 [ksoftirqd/0]
    3 ?        SW     0:00 [watchdog/0]
    4 ?        SW<    0:00 [events/0]
    5 ?        SW<    0:00 [khelper]
    6 ?        SW<    0:00 [kthread]
    8 ?        SW<    0:00 [kblockd/0]
    9 ?        SW<    0:00 [kacpid]
   82 ?        SW<    0:00 [kseriod]
  108 ?        SW     0:00 [pdflush]
  109 ?        SW     0:00 [pdflush]
  110 ?        SW     0:00 [kswapd0]
  111 ?        SW<    0:00 [aio/0]
  721 ?        SW<    0:00 [kpsmoused]
  726 ?        SW<    0:00 [reiserfs/0]
 1560 ?        SW     0:00 [khpsbpkt]
 4015 ?        S      0:00 login -- gl     
 4016 tty3     S      0:00 /sbin/mingetty tty3
 4017 tty4     S      0:00 /sbin/mingetty tty4
 4018 tty5     S      0:00 /sbin/mingetty tty5
 4019 tty6     S      0:00 /sbin/mingetty tty6
 4480 ?        S      0:00 login -- root     
 5051 tty1     R      0:00 -bash
 5330 tty2     S      0:00 -bash
 5601 tty2     S      0:00 sleep 5
 5602 tty1     R      0:00 ps ax

the "sleep 5" comes from the script:

while true; do grep buffer_head /proc/slabinfo; sleep 5; done

Does it look like a memory leak? Here's a fragment of the output:

buffer_head         8110   8112     48   78    1 : tunables  120   60    0 : slabdata    104    104      0
buffer_head         8148   8190     48   78    1 : tunables  120   60    0 : slabdata    105    105      0
buffer_head         8144   8190     48   78    1 : tunables  120   60    0 : slabdata    105    105      0
buffer_head         8166   8190     48   78    1 : tunables  120   60    0 : slabdata    105    105      0
buffer_head         8181   8190     48   78    1 : tunables  120   60    0 : slabdata    105    105      0
buffer_head         8189   8190     48   78    1 : tunables  120   60    0 : slabdata    105    105      0
buffer_head         8226   8268     48   78    1 : tunables  120   60    0 : slabdata    106    106      0
buffer_head         8244   8268     48   78    1 : tunables  120   60    0 : slabdata    106    106      0
buffer_head         8240   8268     48   78    1 : tunables  120   60    0 : slabdata    106    106      0
buffer_head         8260   8268     48   78    1 : tunables  120   60    0 : slabdata    106    106      0
buffer_head         8304   8346     48   78    1 : tunables  120   60    0 : slabdata    107    107      0
buffer_head         8340   8346     48   78    1 : tunables  120   60    0 : slabdata    107    107      0
buffer_head         8332   8346     48   78    1 : tunables  120   60    0 : slabdata    107    107      0
buffer_head         8343   8346     48   78    1 : tunables  120   60    0 : slabdata    107    107      0

Thanks
Guennadi
---------------------------------
Guennadi Liakhovetski, Ph.D.
DSA Daten- und Systemtechnik GmbH
Pascalstr. 28
D-52076 Aachen
Germany

Re: [2.6.17.4] slabinfo.buffer_head increases

[Guennadi Liakhovetski]

On Mon, 10 Jul 2006, Guennadi Liakhovetski wrote:

> I am obsering a steadily increasing buffer_head value in slabinfo under 
> 2.6.17.4. I searched the net / archives and didn't find anything 
> directly relevant. Does anyone have an idea or how shall we debug it?
> 
> I first noticed this "feature" on a 2.6.17-rc5 based ARM-system, where if 
> I stop some user-space applications, the number stop increasing.

Ok, after observing this for a bit longer, it looks like this:

2.6.11.10 (Debian unloaded) - stable
2.6.16.9 (Debian with desktop applications) - grew from 3000 to 6000 then 
	fell back
2.6.17-rc5 (busybox ARM with applications) - grows so far, will watch for 
	a bit longer
2.6.17.4 (SuSE, practically idle) - stable

So, I'll keep it running until tomorrow then report again, unless someone 
can tell how it is supposed to behave and whether there has been a bug and 
it has been fixed?

Thanks
Guennadi
---------------------------------
Guennadi Liakhovetski, Ph.D.
DSA Daten- und Systemtechnik GmbH
Pascalstr. 28
D-52076 Aachen
Germany

Re: [2.6.17.4] slabinfo.buffer_head increases

[Guennadi Liakhovetski]

> On Mon, 10 Jul 2006, Guennadi Liakhovetski wrote:
>
>> I am obsering a steadily increasing buffer_head value in slabinfo under
>> 2.6.17.4. I searched the net / archives and didn't find anything
>> directly relevant. Does anyone have an idea or how shall we debug it?

The problem is still there under 2.6.18-rc2. I narrowed it down to ext3 
journal. To reproduce one just has to mount an ext3 partition and perform 
(write) accesses to it. A loop { touch /mnt/foo; sleep 1; } suffices - 
just let it run for a couple of minutes and monitor buffer_head in 
/proc/slabinfo. If you mount it as ext2 the problem is gone.

Thanks
Guennadi
---------------------------------
Guennadi Liakhovetski, Ph.D.
DSA Daten- und Systemtechnik GmbH
Pascalstr. 28
D-52076 Aachen
Germany

Re: [2.6.17.4] slabinfo.buffer_head increases

[Nick Piggin]

Guennadi Liakhovetski wrote:

>> On Mon, 10 Jul 2006, Guennadi Liakhovetski wrote:
>>
>>> I am obsering a steadily increasing buffer_head value in slabinfo under
>>> 2.6.17.4. I searched the net / archives and didn't find anything
>>> directly relevant. Does anyone have an idea or how shall we debug it?
>>
>
> The problem is still there under 2.6.18-rc2. I narrowed it down to 
> ext3 journal. To reproduce one just has to mount an ext3 partition and 
> perform (write) accesses to it. A loop { touch /mnt/foo; sleep 1; } 
> suffices - just let it run for a couple of minutes and monitor 
> buffer_head in /proc/slabinfo. If you mount it as ext2 the problem is 
> gone.


What data mode is ext3 mounted with?

Is the memory reclaimable? If yes, is it a problem?

--

Send instant messages to your online friends http://au.messenger.yahoo.com 

Re: [2.6.17.4] slabinfo.buffer_head increases

[Guennadi Liakhovetski]

On Tue, 12 Sep 2006, Nick Piggin wrote:

> Guennadi Liakhovetski wrote:
>
>>> On Mon, 10 Jul 2006, Guennadi Liakhovetski wrote:
>>> 
>>>> I am obsering a steadily increasing buffer_head value in slabinfo under
>>>> 2.6.17.4. I searched the net / archives and didn't find anything
>>>> directly relevant. Does anyone have an idea or how shall we debug it?
>>> 
>> 
>> The problem is still there under 2.6.18-rc2. I narrowed it down to ext3 
>> journal. To reproduce one just has to mount an ext3 partition and perform 
>> (write) accesses to it. A loop { touch /mnt/foo; sleep 1; } suffices - just 
>> let it run for a couple of minutes and monitor buffer_head in 
>> /proc/slabinfo. If you mount it as ext2 the problem is gone.
>
>
> What data mode is ext3 mounted with?

Default, i.e., ordered, I guess.

> Is the memory reclaimable? If yes, is it a problem?

Yes, that's why I later wrote that the problem is not real. It was hard to 
see as we had a lot of free RAM on the system, the system was idle apart 
from one script that only did "touch x" periodically with the same "x" 
and the buffer_head slab was growing very steadily. Unlike with ext2 / 
reiserfs. That's why I decided it was not ok. But the memory is 
reclaimable, so, seems like not a problem. Just a bit odd that such a 
"harmless" operation causes a steady growth of buffer_heads...

Thanks
Guennadi
---------------------------------
Guennadi Liakhovetski, Ph.D.
DSA Daten- und Systemtechnik GmbH
Pascalstr. 28
D-52076 Aachen
Germany

Re: [2.6.17.4] slabinfo.buffer_head increases

[Nick Piggin]

Guennadi Liakhovetski wrote:
> On Tue, 12 Sep 2006, Nick Piggin wrote:
> 
>> Guennadi Liakhovetski wrote:
>>
>>>> On Mon, 10 Jul 2006, Guennadi Liakhovetski wrote:
>>>>
>>>>> I am obsering a steadily increasing buffer_head value in slabinfo 
>>>>> under
>>>>> 2.6.17.4. I searched the net / archives and didn't find anything
>>>>> directly relevant. Does anyone have an idea or how shall we debug it?
>>>>
>>>>
>>>
>>> The problem is still there under 2.6.18-rc2. I narrowed it down to 
>>> ext3 journal. To reproduce one just has to mount an ext3 partition 
>>> and perform (write) accesses to it. A loop { touch /mnt/foo; sleep 1; 
>>> } suffices - just let it run for a couple of minutes and monitor 
>>> buffer_head in /proc/slabinfo. If you mount it as ext2 the problem is 
>>> gone.
>>
>>
>>
>> What data mode is ext3 mounted with?
> 
> 
> Default, i.e., ordered, I guess.
> 
>> Is the memory reclaimable? If yes, is it a problem?
> 
> 
> Yes, that's why I later wrote that the problem is not real. It was hard 
> to see as we had a lot of free RAM on the system, the system was idle 
> apart from one script that only did "touch x" periodically with the same 
> "x" and the buffer_head slab was growing very steadily. Unlike with ext2 
> / reiserfs. That's why I decided it was not ok. But the memory is 
> reclaimable, so, seems like not a problem. Just a bit odd that such a 
> "harmless" operation causes a steady growth of buffer_heads...

OK. It is just a quirk in the way that ext3 ordered interacts with page freeing
and reclaim, I think. If it is causing you no performance problems then that's
good. Though it is counter intuitive.

Thanks for the report anyway.

-- 
SUSE Labs, Novell Inc.
Send instant messages to your online friends http://au.messenger.yahoo.com