LKML Archive on lore.kernel.org
help / color / mirror / Atom feed
* Re: [ck] Re: Swap prefetch merge plans
       [not found]   ` <200702092321.20615.kernel@kolivas.org>
@ 2007-02-09 13:13     ` jos poortvliet
  2007-02-09 13:22       ` Con Kolivas
                         ` (2 more replies)
  0 siblings, 3 replies; 12+ messages in thread
From: jos poortvliet @ 2007-02-09 13:13 UTC (permalink / raw)
  To: ck; +Cc: Con Kolivas, Andrew Morton, linux-kernel

[-- Attachment #1: Type: text/plain, Size: 4422 bytes --]

> > > Hold.
> > Why hold?
>
> > It's been shown this patchset really helps desktop users.
>
> Has it?  I don't think I've ever observed any benefits from it and I don't
> think anyone has ever got down and worked out what its drawbacks might be,
> and seen if they can be demonstrated in practice.

Well, I'd say it's disadvantage is that you might use the buffered diskdata in 
memory (which will be replaced with application data from swap) when you come 
back using the computer. But it's far more likely you'll use the app data, 
that's why swap prefetch helps. Anyway, I'm sure you got some response from 
users who did like it. Now I know the desktop isn't the focus of the Linux 
Kernel Developers, and you probably have much to hefty hardware to notice any 
difference. Heck, after my upgrade to 3GB ram, I never noticed swap prefetch 
anymore...

But there ARE ppl using KDE or Gnome on a 256 MB ram system (or even less) and 
they would notice this. I know I do on my 192-mb-ram laptop with Kubuntu... 
It's lovely to do some compiling task, then while you work in vi swap 
prefetch ensures everything is smooth when you try to continue browsing with 
firefox...

> I also have vague memories of some serious-sounding review comments about
> the code from Nick.

Nick's comment, replying to me some time ago:

> > well, the reason i use it is my computer is much more reactive in the
> > morning. linux uses to get very slow after a night of not-doing-much
> > except some 'sleep 5h && blabla' and cron stuff. in the morning it takes
> > a few HOURS to get up and running smoothly. with swap prefetch, it
> > actually feels faster compared to a fresh boot. now you can force swap
> > prefetch to start working, i use it now and then after some heavy taskts
> > which pulled everything to swap.
>
> I have two issues with this argument (not that I'm trying to say it
> couldn't make a difference in your case).
>
> Firstly, swap prefetch actually doesn't handle the midnight updatedb
> pageout problem nicely. It doesn't do any prefetching when the
> pagecache/vfs cache fills memory (which is what would have to happen for
> updatedb to push stuff into swap).

But doesn't it prefetch stuff after updatedb has run and the computer is idle? 
Updatedb here runs at night, and swap prefetch is supposed to get my app data 
back in memory during the time after updatedb pushed it out, right? Maybe Con 
can explain here...

> Secondly, with or without swap prefetch, I think we can do a better job of
> handling these use-once patterns to begin with.
well, i'd love to see you elaborate on this. I mean, if the kernel could be 
made smarter and swap prefetch wouldn't be needed, great...

> > Why keep it in -mm for so long?
>
> I'm waiting to be shown that its benefits exceed its costs.  Sometimes
> that's hard.

Nobody has said anything about costs, indeed. Now afaik, swap prefetch is 
designed to have no/as little as possible costs, so that makes sense. Does it 
have to have some bugs, which have to be adressed, before it can enter? I'm 
sure this can be arranged, right, Con?

Sorry if I sound sarcastic. I'm no hacker myself, and sometimes these 
discussions don't make sense to me. A bit like the Staircase thingy ->

"hi, I've got this piece of code which does the same as that piece, but 
better"
"Why didn't you improve the old code?"
"This is a better design -> half the code but doing a better job"
"Well, it's not tested as much, so it won't go in. Go away!"

There where those comments from Torvalds some time ago in an interview, about 
the kernel community becoming harder to get involved with. As an outsider, it 
sure seems so. I read frustrations everywhere. What about the kevent guy, his 
blog: http://tservice.net.ru/~s0mbre/blog

I stumbled upon it when reading LWN. Seems pretty sad... I don't get the 
technical stuff, but the frustration almost blows up your monitor. Is there 
something fundamentally wrong with the kernel-hackers-culture, or are these 
incidents?

grtz

Jos

-- 
Disclaimer:

Alles wat ik doe denk en zeg is gebaseerd op het wereldbeeld wat ik nu heb. 
Ik ben niet verantwoordelijk voor wijzigingen van de wereld, of het beeld wat 
ik daarvan heb, noch voor de daaruit voortvloeiende gedragingen van mezelf. 
Alles wat ik zeg is aardig bedoeld, tenzij expliciet vermeld.

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [ck] Re: Swap prefetch merge plans
  2007-02-09 13:13     ` [ck] Re: Swap prefetch merge plans jos poortvliet
@ 2007-02-09 13:22       ` Con Kolivas
  2007-02-09 14:00         ` jos poortvliet
  2007-02-09 13:56       ` Con Kolivas
  2007-02-09 20:30       ` Andrew Morton
  2 siblings, 1 reply; 12+ messages in thread
From: Con Kolivas @ 2007-02-09 13:22 UTC (permalink / raw)
  To: jos poortvliet; +Cc: ck, Andrew Morton, linux-kernel

On Saturday 10 February 2007 00:13, jos poortvliet wrote:
> Nobody has said anything about costs, indeed. Now afaik, swap prefetch is
> designed to have no/as little as possible costs, so that makes sense. Does
> it have to have some bugs, which have to be adressed, before it can enter?
> I'm sure this can be arranged, right, Con?
>
> Sorry if I sound sarcastic. I'm no hacker myself, and sometimes these
> discussions don't make sense to me. A bit like the Staircase thingy ->
>
> "hi, I've got this piece of code which does the same as that piece, but
> better"
> "Why didn't you improve the old code?"
> "This is a better design -> half the code but doing a better job"
> "Well, it's not tested as much, so it won't go in. Go away!"
>
> There where those comments from Torvalds some time ago in an interview,
> about the kernel community becoming harder to get involved with. As an
> outsider, it sure seems so. I read frustrations everywhere. What about the
> kevent guy, his blog: http://tservice.net.ru/~s0mbre/blog
>
> I stumbled upon it when reading LWN. Seems pretty sad... I don't get the
> technical stuff, but the frustration almost blows up your monitor. Is there
> something fundamentally wrong with the kernel-hackers-culture, or are these
> incidents?

I greatly appreciate the support. Truly I do. 

But I do not like the direction this argument is going. Please let it go.

-- 
-ck

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [ck] Re: Swap prefetch merge plans
  2007-02-09 13:13     ` [ck] Re: Swap prefetch merge plans jos poortvliet
  2007-02-09 13:22       ` Con Kolivas
@ 2007-02-09 13:56       ` Con Kolivas
  2007-02-09 20:30       ` Andrew Morton
  2 siblings, 0 replies; 12+ messages in thread
From: Con Kolivas @ 2007-02-09 13:56 UTC (permalink / raw)
  To: jos poortvliet; +Cc: ck, Andrew Morton, linux-kernel

On Saturday 10 February 2007 00:13, jos poortvliet wrote:
> > > > Hold.
> > >
> > > Why hold?
> > >
> > > It's been shown this patchset really helps desktop users.
> >
> > Has it?  I don't think I've ever observed any benefits from it and I
> > don't think anyone has ever got down and worked out what its drawbacks
> > might be, and seen if they can be demonstrated in practice.
>
> Well, I'd say it's disadvantage is that you might use the buffered diskdata
> in memory (which will be replaced with application data from swap) when you
> come back using the computer. But it's far more likely you'll use the app
> data, that's why swap prefetch helps

swap prefetch stores the data at the tail end of the lru list which means that 
even if you do want to use that ram for something else, the prefetched pages 
will be immediately dropped. Since the pages are still on backing store 
(swapspace) they will be droppped without any further I/O. 

> . Anyway, I'm sure you got some 
> response from users who did like it. Now I know the desktop isn't the focus
> of the Linux Kernel Developers, and you probably have much to hefty
> hardware to notice any difference. Heck, after my upgrade to 3GB ram, I
> never noticed swap prefetch anymore...

Even on 2GB ram at home, where I regularly move iso images around, compress 
large files with big window compression apps (like rzip), manipulate gimp 
images, print large colour photos in high resolution and so on, I hit swap 
regularly and do notice it being helpful. This is what normal desktop users 
do.

>
> But there ARE ppl using KDE or Gnome on a 256 MB ram system (or even less)
> and they would notice this. I know I do on my 192-mb-ram laptop with
> Kubuntu... It's lovely to do some compiling task, then while you work in vi
> swap prefetch ensures everything is smooth when you try to continue
> browsing with firefox...

Indeed.

>
> > I also have vague memories of some serious-sounding review comments about
> > the code from Nick.
>
> Nick's comment, replying to me some time ago:
> > > well, the reason i use it is my computer is much more reactive in the
> > > morning. linux uses to get very slow after a night of not-doing-much
> > > except some 'sleep 5h && blabla' and cron stuff. in the morning it
> > > takes a few HOURS to get up and running smoothly. with swap prefetch,
> > > it actually feels faster compared to a fresh boot. now you can force
> > > swap prefetch to start working, i use it now and then after some heavy
> > > taskts which pulled everything to swap.
> >
> > I have two issues with this argument (not that I'm trying to say it
> > couldn't make a difference in your case).
> >
> > Firstly, swap prefetch actually doesn't handle the midnight updatedb
> > pageout problem nicely. It doesn't do any prefetching when the
> > pagecache/vfs cache fills memory (which is what would have to happen for
> > updatedb to push stuff into swap).
>
> But doesn't it prefetch stuff after updatedb has run and the computer is
> idle? Updatedb here runs at night, and swap prefetch is supposed to get my
> app data back in memory during the time after updatedb pushed it out,
> right? Maybe Con can explain here...

It can't help the updatedb scenario. Updatedb leaves the ram full and swap 
prefetch wants to cost as little as possible so it will never move anything 
out of ram in preference for the pages it wants to swap back in. The use once 
logic in the vm might if you're lucky help the next time you use the memory, 
but anything you used prior to updatedb is trashed beyond oblivion and this 
has nothing to do with swap prefetch.

> > Secondly, with or without swap prefetch, I think we can do a better job
> > of handling these use-once patterns to begin with.
>
> well, i'd love to see you elaborate on this. I mean, if the kernel could be
> made smarter and swap prefetch wouldn't be needed, great...

They're not mutually exclusive. We will _always_ have a load that causes 
swapping. Whatever the resources available on a pc, an application will come 
along that outstrips it. The nature of normal desktops is they're always 
driven to this level.

> > > Why keep it in -mm for so long?
> >
> > I'm waiting to be shown that its benefits exceed its costs.  Sometimes
> > that's hard.
>
> Nobody has said anything about costs, indeed. Now afaik, swap prefetch is
> designed to have no/as little as possible costs, so that makes sense. Does
> it have to have some bugs, which have to be adressed, before it can enter?
> I'm sure this can be arranged, right, Con?

I'm stuck developing code I'm having trouble proving it helps. Normal users 
find it helps and an artificial testcase shows it helps, but that is not 
enough, since the normal users will be tainted in their opinion, and the 
artificial testcase will always be artificial. My mistake of developing novel 
code in areas that were unquantifiable has been my undoing.  If it's 
unquantifiable, but "obvious" then it has a chance...(like most of what goes 
into mainline). Precious few of the patches that I keep in -ck belong to that 
group though. 

P.S. Sorry about being harsh in the last email Jos, I understood your 
frustration.

-- 
-ck

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [ck] Re: Swap prefetch merge plans
  2007-02-09 13:22       ` Con Kolivas
@ 2007-02-09 14:00         ` jos poortvliet
  0 siblings, 0 replies; 12+ messages in thread
From: jos poortvliet @ 2007-02-09 14:00 UTC (permalink / raw)
  To: Con Kolivas; +Cc: ck, Andrew Morton, linux-kernel

[-- Attachment #1: Type: text/plain, Size: 1660 bytes --]

Op Friday 09 February 2007, schreef Con Kolivas:
> On Saturday 10 February 2007 00:13, jos poortvliet wrote:
> > Nobody has said anything about costs, indeed. Now afaik, swap prefetch is
> > designed to have no/as little as possible costs, so that makes sense.
> > Does it have to have some bugs, which have to be adressed, before it can
> > enter? I'm sure this can be arranged, right, Con?
> >
> > Sorry if I sound sarcastic. I'm no hacker myself, and sometimes these
> > discussions don't make sense to me. A bit like the Staircase thingy ->
> >
> > "hi, I've got this piece of code which does the same as that piece, but
> > better"
> > "Why didn't you improve the old code?"
> > "This is a better design -> half the code but doing a better job"
> > "Well, it's not tested as much, so it won't go in. Go away!"
> >
> > There where those comments from Torvalds some time ago in an interview,
> > about the kernel community becoming harder to get involved with. As an
> > outsider, it sure seems so. I read frustrations everywhere. What about
> > the kevent guy, his blog: http://tservice.net.ru/~s0mbre/blog
> >
> > I stumbled upon it when reading LWN. Seems pretty sad... I don't get the
> > technical stuff, but the frustration almost blows up your monitor. Is
> > there something fundamentally wrong with the kernel-hackers-culture, or
> > are these incidents?
>
> I greatly appreciate the support. Truly I do.
>
> But I do not like the direction this argument is going. Please let it go.

Ok. I appologize to those who might have felt attacked, this wasn't meant 
personally to anybody, more to the community as a whole.

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [ck] Re: Swap prefetch merge plans
  2007-02-09 13:13     ` [ck] Re: Swap prefetch merge plans jos poortvliet
  2007-02-09 13:22       ` Con Kolivas
  2007-02-09 13:56       ` Con Kolivas
@ 2007-02-09 20:30       ` Andrew Morton
  2007-02-09 20:50         ` Con Kolivas
  2007-02-09 23:35         ` [ck] " Chuck Ebbert
  2 siblings, 2 replies; 12+ messages in thread
From: Andrew Morton @ 2007-02-09 20:30 UTC (permalink / raw)
  To: jos poortvliet; +Cc: ck, Con Kolivas, linux-kernel

On Fri, 09 Feb 2007 14:13:03 +0100
jos poortvliet <jos@mijnkamer.nl> wrote:

> Nick's comment, replying to me some time ago:

I think I was thinking of this:

	http://lkml.org/lkml/2006/2/6/509

> Is there 
> something fundamentally wrong with the kernel-hackers-culture, or are these 
> incidents?

Well yes, there are incidents.  We have too many coders and not enough
reviewers, testers and bug-fixers.

If I had two solid days to sit down and carefully review, test,
try-to-exploit and if necessary improve/fix this code then perhaps we'd get
there.  But I'm nowhere vaguely near being able to afford that time and
things are just sitting there.


I have an email sitting in my drafts folder stating that I'll no longer
accept any features unless they've been publically reviewed in detail and
run-time tested by a third party.  The idea being to force people to spend
more time reviewing and testing each other's stuff and less time writing
new stuff.  Maybe on a sufficiently gloomy day I'll actually send it.


^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [ck] Re: Swap prefetch merge plans
  2007-02-09 20:30       ` Andrew Morton
@ 2007-02-09 20:50         ` Con Kolivas
  2007-02-09 22:49           ` Con Kolivas
  2007-02-09 23:35         ` [ck] " Chuck Ebbert
  1 sibling, 1 reply; 12+ messages in thread
From: Con Kolivas @ 2007-02-09 20:50 UTC (permalink / raw)
  To: Andrew Morton; +Cc: jos poortvliet, ck, linux-kernel

On Saturday 10 February 2007 07:30, Andrew Morton wrote:
> On Fri, 09 Feb 2007 14:13:03 +0100
>
> jos poortvliet <jos@mijnkamer.nl> wrote:
> > Nick's comment, replying to me some time ago:
>
> I think I was thinking of this:
>
> 	http://lkml.org/lkml/2006/2/6/509

Fortunately that predates a lot of changes where I did address all those. 
These will seem out of context without looking at that original email so I 
apologise in advance.

buffered_rmqueue and prefetching x86 specific (not into DMA) were dropped

It is NUMA aware

Global cacheline bouncing in page allocation and page reclaim paths I have no 
answer for as I have to tell swap prefetch that the vm is busy somehow and I 
do that by setting precisely one bit in a lockless manner.

The trylocks were dropped.

The other ideas were to :
-extend the prefetching. That's extra features
-knowing for sure when a system is really idle. I've tried hard to do that as 
cheaply as possible.
-putting pages on the lru? well it puts them on the tail
-papering over an issue? As I said, no matter how good the vm is, there will 
always be loads that swap.

-- 
-ck

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Swap prefetch merge plans
  2007-02-09 20:50         ` Con Kolivas
@ 2007-02-09 22:49           ` Con Kolivas
  0 siblings, 0 replies; 12+ messages in thread
From: Con Kolivas @ 2007-02-09 22:49 UTC (permalink / raw)
  To: ck; +Cc: Andrew Morton, linux-kernel

On Saturday 10 February 2007 07:50, Con Kolivas wrote:
> On Saturday 10 February 2007 07:30, Andrew Morton wrote:
> > On Fri, 09 Feb 2007 14:13:03 +0100
> >
> > jos poortvliet <jos@mijnkamer.nl> wrote:
> > > Nick's comment, replying to me some time ago:
> >
> > I think I was thinking of this:
> >
> > 	http://lkml.org/lkml/2006/2/6/509
>
> Fortunately that predates a lot of changes where I did address all those.
> These will seem out of context without looking at that original email so I
> apologise in advance.
>
> buffered_rmqueue and prefetching x86 specific (not into DMA) were dropped
>
> It is NUMA aware
>
> Global cacheline bouncing in page allocation and page reclaim paths I have
> no answer for as I have to tell swap prefetch that the vm is busy somehow
> and I do that by setting precisely one bit in a lockless manner.
>
> The trylocks were dropped.
>
> The other ideas were to :
> -extend the prefetching. That's extra features
> -knowing for sure when a system is really idle. I've tried hard to do that
> as cheaply as possible.
> -putting pages on the lru? well it puts them on the tail
> -papering over an issue? As I said, no matter how good the vm is, there
> will always be loads that swap.

Perhaps I haven't made this clear enough.

Nick has been kind enough to review pretty much all of swap prefetch at every 
turn. He is understandably suspicious about any code that I generate since 
I'm a doctor and not a computer programmer. I have addressed every concern he 
has made along the way and even joked at the end that he "threatened to 
review it over and over and over" which he did not take in jest. 

Swap prefetch has actually had far more review than a lot of code so I'm 
surprised that this is a remaning concern.  It has been reviewed, and I have 
addressed the concerns. It is possible to hold back code forever at the 
suggestion that more review is always required, and basically that's what I 
feel this has become.

If there is one valid concern with swap prefetch, there is a numa=64 scenario 
that Rohit Seth brought to my attention and I gave him a patch to test 2 
months ago. I've pinged him since, but I understand he's busy so has not had 
a chance to test the patch.

-- 
-ck

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [ck] Re: Swap prefetch merge plans
  2007-02-09 20:30       ` Andrew Morton
  2007-02-09 20:50         ` Con Kolivas
@ 2007-02-09 23:35         ` Chuck Ebbert
  2007-02-09 23:54           ` Randy Dunlap
  1 sibling, 1 reply; 12+ messages in thread
From: Chuck Ebbert @ 2007-02-09 23:35 UTC (permalink / raw)
  To: Andrew Morton; +Cc: jos poortvliet, ck, Con Kolivas, linux-kernel

Andrew Morton wrote:
> I have an email sitting in my drafts folder stating that I'll no longer
> accept any features unless they've been publically reviewed in detail and
> run-time tested by a third party.  The idea being to force people to spend
> more time reviewing and testing each other's stuff and less time writing
> new stuff.  Maybe on a sufficiently gloomy day I'll actually send it.
>   
/me sneaks into Andrew's office and sends it out.



^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [ck] Re: Swap prefetch merge plans
  2007-02-09 23:35         ` [ck] " Chuck Ebbert
@ 2007-02-09 23:54           ` Randy Dunlap
  2007-02-10 16:12             ` jos poortvliet
  0 siblings, 1 reply; 12+ messages in thread
From: Randy Dunlap @ 2007-02-09 23:54 UTC (permalink / raw)
  To: Chuck Ebbert; +Cc: Andrew Morton, jos poortvliet, ck, Con Kolivas, linux-kernel

On Fri, 09 Feb 2007 18:35:51 -0500 Chuck Ebbert wrote:

> Andrew Morton wrote:
> > I have an email sitting in my drafts folder stating that I'll no longer
> > accept any features unless they've been publically reviewed in detail and
> > run-time tested by a third party.  The idea being to force people to spend
> > more time reviewing and testing each other's stuff and less time writing
> > new stuff.  Maybe on a sufficiently gloomy day I'll actually send it.
> >   
> /me sneaks into Andrew's office and sends it out.

Thanks.  8)

---
~Randy
*** Remember to use Documentation/SubmitChecklist when testing your code ***

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [ck] Re: Swap prefetch merge plans
  2007-02-09 23:54           ` Randy Dunlap
@ 2007-02-10 16:12             ` jos poortvliet
  0 siblings, 0 replies; 12+ messages in thread
From: jos poortvliet @ 2007-02-10 16:12 UTC (permalink / raw)
  To: Randy Dunlap; +Cc: Chuck Ebbert, Andrew Morton, ck, Con Kolivas, linux-kernel

[-- Attachment #1: Type: text/plain, Size: 1849 bytes --]

Op Saturday 10 February 2007, schreef Randy Dunlap:
> On Fri, 09 Feb 2007 18:35:51 -0500 Chuck Ebbert wrote:
> > Andrew Morton wrote:
> > > I have an email sitting in my drafts folder stating that I'll no longer
> > > accept any features unless they've been publically reviewed in detail
> > > and run-time tested by a third party.  The idea being to force people
> > > to spend more time reviewing and testing each other's stuff and less
> > > time writing new stuff.  Maybe on a sufficiently gloomy day I'll
> > > actually send it.
> >
> > /me sneaks into Andrew's office and sends it out.
>
> Thanks.  8)

Well, is it smart? I mean, it's not stupid of course, but it has its 
disadvantages which Andrew already mentioned. Less time for writing new 
stuff. So it is a tradeoff, most likely the reason why he didn't send it yet. 

What are the sideeffects of this? Does it repel people? Or controversely, does 
it bring new people in, who are asked by others to do a little review? 

Does it lead to better documented and commented code, to help the review 
process? Does it lead to worse reviews? Does it stiffle innovation, or, as 
more ppl work on one patch, does it stimulate it? Will people start to 
cooperate more, or will they argue and fight when someone refuses to review 
patches?

Anyway, you can't know these in advance, and watching what it does is the only 
way to find out. Its Andrew's call, either way.

Days should't be gloomy, but if they do, trying new things is the only way to 
find a solution. The growing amount of people workin on the kernel and the 
increasing demands from the users force the kernel community to evolve, and 
central figures like Andrew and Linus play a big role in this. I think you 
guys have done a great job until now, and I trust you can keep it that way.

Jos

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Swap prefetch merge plans
  2007-02-09 18:09 Andrew Burgess
@ 2007-02-09 20:12 ` Con Kolivas
  0 siblings, 0 replies; 12+ messages in thread
From: Con Kolivas @ 2007-02-09 20:12 UTC (permalink / raw)
  To: Andrew Burgess; +Cc: ck, akpm, linux-kernel

[-- Attachment #1: Type: text/plain, Size: 2061 bytes --]

On Saturday 10 February 2007 05:09, Andrew Burgess wrote:
> >I'm stuck developing code I'm having trouble proving it helps. Normal
> > users find it helps and an artificial testcase shows it helps, but that
> > is not enough, since the normal users will be tainted in their opinion,
> > and the artificial testcase will always be artificial. My mistake of
> > developing novel code in areas that were unquantifiable has been my
> > undoing.
>
> Could you add some statistics gathering to measure
> cumulatively how long processes wait for swapin?  Then you
> could run with and without and maybe say "on average my
> system waits 4 minutes a day without swap prefetch and 2
> minutes with? Or if a simple sum doesn't work, some sort of
> graph? Then anyone could run and measure the benefit.
>
> Apologies if you've already thought of this...

It would depend entirely on the workload / how you use your machine and the 
balance of ram size vs hard drive speed vs swap size. The simple test app I 
used attached below used on a 1GB machine with a fairly modern hard drive 
saved:

Without:
Timed portion 6272175 microseconds
With:
Timed portion 523623 microseconds

This was with 700MB of what would be considered "application data". So if you 
had lots of firefox windows open, openoffice, email client etc open and then 
did something which caused a big swap load (I have this effect when printing 
an full page colour picture at high resolution), then the total time saved 
over clicking those applications back to life after some idle time (so you 
printed and walked away while it was printing) was about 5.5 seconds. 

So the total saved time over a day would depend on how often you hit swap, and 
how often you clicked things back to life. Of course if you never hit swap 
the code does basically nothing.


Note this app is a silly little thing that only worked on 32bit if I recall 
correctly but here it is.

build with 
gcc -o mallocall mallocall.c -W -Wall -lrt

then test without swap prefetch enabled, and then enable it and test again.

-- 
-ck

[-- Attachment #2: mallocall.c --]
[-- Type: text/x-csrc, Size: 2067 bytes --]

#include <stdio.h>
#include <stdarg.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <sys/mman.h>
#include <time.h>

void fatal(const char *format, ...)
{
	va_list ap;

	if (format) {
		va_start(ap, format);
		vfprintf(stderr, format, ap);
		va_end(ap);
	}

	fprintf(stderr, "Fatal error - exiting\n");
	exit(1);
}

size_t get_ram(void)
{
        unsigned long ramsize;
	FILE *meminfo;
        char aux[256];

	if(!(meminfo = fopen("/proc/meminfo", "r")))
		fatal("fopen\n");

	while( !feof(meminfo) && !fscanf(meminfo, "MemTotal: %lu kB", &ramsize) )
		fgets(aux,sizeof(aux),meminfo);
	if (fclose(meminfo) == -1)
		fatal("fclose");
	return ramsize * 1000;
}

unsigned long get_usecs(struct timespec *myts)
{
	if (clock_gettime(CLOCK_REALTIME, myts))
		fatal("clock_gettime");
	return (myts->tv_sec * 1000000 + myts->tv_nsec / 1000 );
}

int main(void)
{
	unsigned long current_time, time_diff;
	struct timespec myts;
	char *buf1, *buf2, *buf3, *buf4;
	size_t size, full_size = get_ram();
	int sleep_seconds = 600;

	size = full_size * 7 / 10;
	printf("Starting first malloc of %d bytes\n", size);
	buf1 = malloc(size);
	if (buf1 == (char *)-1)
		fatal("Failed to malloc 1st buffer\n");
	memset(buf1, 0, size);
	printf("Completed first malloc and starting second malloc of %d bytes\n", full_size);

	buf2 = malloc(full_size);
	if (buf2 == (char *)-1)
		fatal("Failed to malloc 2nd buffer\n");
	memset(buf2, 0, full_size);
	buf4 = malloc(1);
	for (buf3 = buf2 + full_size; buf3 > buf2; buf3--)
		*buf4 = *buf3;
	free(buf2);
	printf("Completed second malloc and free\n");

	printf("Sleeping for %d seconds\n", sleep_seconds);
	sleep(sleep_seconds);

	printf("Important part - starting read of first malloc\n");
	time_diff = current_time = get_usecs(&myts);
	for (buf3 = buf1; buf3 < buf1 + size; buf3++)
		*buf4 = *buf3;
	current_time = get_usecs(&myts);
	free(buf4);
	free(buf1);
	printf("Completed read and freeing of first malloc\n");
	time_diff = current_time - time_diff;
	printf("Timed portion %lu microseconds\n",time_diff);

	return 0;
}

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Swap prefetch merge plans
@ 2007-02-09 18:09 Andrew Burgess
  2007-02-09 20:12 ` Con Kolivas
  0 siblings, 1 reply; 12+ messages in thread
From: Andrew Burgess @ 2007-02-09 18:09 UTC (permalink / raw)
  To: kernel, ck, akpm, linux-kernel

>I'm stuck developing code I'm having trouble proving it helps. Normal users 
>find it helps and an artificial testcase shows it helps, but that is not 
>enough, since the normal users will be tainted in their opinion, and the 
>artificial testcase will always be artificial. My mistake of developing novel 
>code in areas that were unquantifiable has been my undoing.

Could you add some statistics gathering to measure
cumulatively how long processes wait for swapin?  Then you
could run with and without and maybe say "on average my
system waits 4 minutes a day without swap prefetch and 2
minutes with? Or if a simple sum doesn't work, some sort of
graph? Then anyone could run and measure the benefit.

Apologies if you've already thought of this...


^ permalink raw reply	[flat|nested] 12+ messages in thread

end of thread, other threads:[~2007-02-10 16:11 UTC | newest]

Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
     [not found] <200702091038.37143.kernel@kolivas.org>
     [not found] ` <200702092054.10232.ocilent1@gmail.com>
     [not found]   ` <200702092321.20615.kernel@kolivas.org>
2007-02-09 13:13     ` [ck] Re: Swap prefetch merge plans jos poortvliet
2007-02-09 13:22       ` Con Kolivas
2007-02-09 14:00         ` jos poortvliet
2007-02-09 13:56       ` Con Kolivas
2007-02-09 20:30       ` Andrew Morton
2007-02-09 20:50         ` Con Kolivas
2007-02-09 22:49           ` Con Kolivas
2007-02-09 23:35         ` [ck] " Chuck Ebbert
2007-02-09 23:54           ` Randy Dunlap
2007-02-10 16:12             ` jos poortvliet
2007-02-09 18:09 Andrew Burgess
2007-02-09 20:12 ` Con Kolivas

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).