LKML Archive on
help / color / mirror / Atom feed
From: Michael Gerdau <>
To: Ingo Molnar <>
	Linus Torvalds <>,
	Nick Piggin <>,
	Gene Heskett <>,
	Juliusz Chroboczek <>,
	Mike Galbraith <>,
	Peter Williams <>,
	ck list <>,
	Thomas Gleixner <>,
	William Lee Irwin III <>,
	Andrew Morton <>,
	Bill Davidsen <>, Willy Tarreau <>,
	Arjan van de Ven <>
Subject: Re: [REPORT] cfs-v6-rc2 vs sd-0.46 vs 2.6.21-rc7
Date: Thu, 26 Apr 2007 15:22:28 +0200	[thread overview]
Message-ID: <> (raw)
In-Reply-To: <>

[-- Attachment #1: Type: text/plain, Size: 5860 bytes --]

> as a summary: i think your numbers demonstrate it nicely that the 
> shorter 'timeslice length' that both CFS and SD utilizes does not have a 
> measurable negative impact on your workload.

That's my impression as well. In particular I think that for this
type of workload it is pretty much irrelevant which scheduler I'm
using :-)

> To measure the total impact  
> of 'timeslicing' you might want to try the exact same workload with a 
> much higher 'timeslice length' of say 400 msecs, via:
>     echo 400000000 > /proc/sys/kernel/sched_granularity_ns  # on CFS
>     echo 400 > /proc/sys/kernel/rr_interval                 # on SD

I just finished building cfs and sd against 2.6.21 and will try with
these over the next days.

> your existing numbers are a bit hard to analyze because the 3 workloads 
> were started at the same time and they overlapped differently and 
> utilized the system differently.

Right. However the differences w/r to which job finished when, in
particular the change in who "came in" 2. and 3. between sd and cfs
had been consistent with other similar such jobs I ran over the last
couple of days.

In general sd tends to finish all three such jobs at roughly the
same time while cfs clearly "favors" the LTMM-type jobs (which
admittedly involve the least computations). I don't really know
why that is so. However the jobs does some minimal I/O after each
successfully finish run and that might be different w/r to sd and
cfs. Not really knowing what both scheduler do internally I'm in
no position to discuss that with either you or Con :)

> i think the primary number that makes sense to look at (which is perhaps 
> the least sensitive to the 'overlap effect') is the 'combined user times 
> of all 3 workloads' (in order of performance):
> > 2.6.21-rc7:                              20589.423    100.00%
> > 2.6.21-rc7-cfs-v6-rc2 (X @ nice -10):    20613.845     99.88%
> > 2.6.21-rc7-sd046:                        20617.945     99.86%
> > 2.6.21-rc7-cfs-v6-rc2 (X @ nice 0):      20743.564     99.25%
> to me this gives the impression that it's all "within noise".

I think so too. But then I didn't seriously expect any different.

> In  
> particular the two CFS results suggest that there's at least a ~100 
> seconds noise in these results, because the renicing of X should have no 
> impact on the result (the workloads are pure number-crunchers, and all 
> use up the CPUs 100%, correct?),


The differences could as well result from small fluctuations in my
work on the machine during the test (like reading mail, editing sources
and similar stuff).

> another (perhaps less reliable) number is the total wall-clock runtime 
> of all 3 jobs. Provided i did not make any mistakes in my calculations, 
> here are the results:
> > 2.6.21-rc7-sd046:                        10512.611 seconds
> > 2.6.21-rc7-cfs-v6-rc2 (X @ nice -10):    10605.946 seconds
> > 2.6.21-rc7:                              10650.535 seconds
> > 2.6.21-rc7-cfs-v6-rc2 (X @ nice 0):      10788.125 seconds
> (the numbers are lower than the first numbers because this is a 2 CPU 
> system)
> both SD and CFS-nice-10 was faster than mainline, but i'd say this too 
> is noise - especially because this result highly depends on the way the 
> workloads overlap in general, which seems to be different for SD.

I don't think that's noise. As I wrote above IMO this is the one
difference I personally consider significant. While running I could
watch intermediate output of all 3 jobs intermixed on a console.

With sd the jobs were -within reason- head to head while with cfs
and mainline LTMM quickly got ahead and remained there.

As I speculated above this (IMO real) difference in the way sd and
cfs (and mainline) schedule the jobs might be related to the way
the schedulers react on I/O but I don't know.

> system time is interesting too:
> > 2.6.21-rc7:                              35.379
> > 2.6.21-rc7-cfs-v6-rc2 (X @ nice -10):    40.399
> > 2.6.21-rc7-cfs-v6-rc2 (X @ nice 0):      44.239
> > 2.6.21-rc7-sd046:                        45.515

Over the weekend I might try to repeat these tests while doing
nothing else on the machine (i.e. no editing or reading mail).

> combined system+user time:
> > 2.6.21-rc7:                              20624.802
> > 2.6.21-rc7-cfs-v6-rc2 (X @ nice 0):      20658.084
> > 2.6.21-rc7-sd046:                        20663.460
> > 2.6.21-rc7-cfs-v6-rc2 (X @ nice -10):    20783.963
> perhaps it might make more sense to run the workloads serialized, to 
> have better comparabality of the individual workloads. (on a real system 
> you'd naturally want to overlap these workloads to utilize the CPUs, so 
> the numbers you did are very relevant too.)

Yes. In fact the workloads used to be run serialized and only
recently I tried to easily have them run on multicore systems.

The whole set of such jobs consists of 312 tasks and thus the
idea is to get a couple of PS3 and have them run there...
(once I found the time to port the software).

> The vmstat suggested there is occasional idle time in the system - is 
> the workload IO-bound (or memory bound) in those cases?

There is writing out stats after each run (i.e. every 5-8 sec) both
to the PostgreSQL DB as well as to the console.

I could get rid of the console I/O (i.e. make it conditional) and
for testing purpose could switch off the DB if that would make be

 Technosis GmbH, Geschäftsführer: Michael Gerdau, Tobias Dittmar
 Sitz Hamburg; HRB 89145 Amtsgericht Hamburg
 Vote against SPAM - see
 Michael Gerdau       email:
 GPG-keys available on request or at public keyserver

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

  reply	other threads:[~2007-04-26 13:22 UTC|newest]

Thread overview: 7+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2007-04-26 11:12 Michael Gerdau
2007-04-26 12:07 ` Ingo Molnar
2007-04-26 13:22   ` Michael Gerdau [this message]
2007-04-26 13:55     ` Ingo Molnar
2007-04-26 22:59   ` [ck] " Con Kolivas
2007-04-27  5:59     ` Michael Gerdau
2007-04-27  6:52     ` Ingo Molnar

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \
    --subject='Re: [REPORT] cfs-v6-rc2 vs sd-0.46 vs 2.6.21-rc7' \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).