LKML Archive on lore.kernel.org
help / color / mirror / Atom feed
From: Robert Hancock <hancockr@shaw.ca>
To: Olof Johansson <olof@lixom.net>
Cc: linux-kernel@vger.kernel.org,
	Peter Zijlstra <a.p.zijlstra@chello.nl>,
	Ingo Molnar <mingo@elte.hu>
Subject: Re: Scheduler(?) regression from 2.6.22 to 2.6.24 for short-lived threads
Date: Fri, 08 Feb 2008 19:49:49 -0600	[thread overview]
Message-ID: <47AD06BD.7040702@shaw.ca> (raw)
In-Reply-To: <fa.6N2dhyJ1cmBqiuFKgCaYfwduM+0@ifi.uio.no>

Olof Johansson wrote:
> Hi,
> 
> I ended up with a customer benchmark in my lap this week that doesn't
> do well on recent kernels. :(
> 
> After cutting it down to a simple testcase/microbenchmark, it seems like
> recent kernels don't do as well with short-lived threads competing
> with the thread it's cloned off of. The CFS scheduler changes come to
> mind, but I suppose it could be caused by something else as well.
> 
> The pared-down testcase is included below. Reported runtime for the
> testcase has increased almost 3x between 2.6.22 and 2.6.24:
> 
> 2.6.22: 3332 ms
> 2.6.23: 4397 ms
> 2.6.24: 8953 ms
> 2.6.24-git19: 8986 ms
> 
> While running, it'll fork off a bunch of threads, each doing just a little
> work, then busy-waiting on the original thread to finish as well. Yes,
> it's incredibly stupidly coded but that's not my point here.
> 
> During run, (runtime 10s on my 1.5GHz Core2 Duo laptop), vmstat 2  shows:
> 
>  0  0      0 115196 364748 2248396    0    0     0     0  163   89  0  0 100  0
>  2  0      0 115172 364748 2248396    0    0     0     0  270  178 24  0 76  0
>  2  0      0 115172 364748 2248396    0    0     0     0  402  283 52  0 48  0
>  2  0      0 115180 364748 2248396    0    0     0     0  402  281 50  0 50  0
>  2  0      0 115180 364764 2248396    0    0     0    22  403  295 51  0 48  1
>  2  0      0 115056 364764 2248396    0    0     0     0  399  280 52  0 48  0
>  0  0      0 115196 364764 2248396    0    0     0     0  241  141 17  0 83  0
>  0  0      0 115196 364768 2248396    0    0     0     2  155   67  0  0 100  0
>  0  0      0 115196 364768 2248396    0    0     0     0  148   62  0  0 100  0
> 
> I.e. runqueue is 2, but only one cpu is busy. However, this still seems
> true on the kernel that runs the testcase in more reasonable time.
> 
> Also, 'time' reports real and user time roughly the same on all kernels,
> so it's not that the older kernels are better at spreading out the load
> between the two cores (either that or it doesn't account for stuff right).
> 
> I've included the config files, runtime output and vmstat output at
> http://lixom.net/~olof/threadtest/. I see similar behaviour on PPC as
> well as x86, so it's not architecture-specific.
> 
> Testcase below. Yes, I know, there's a bunch of stuff that could be done
> differently and better, but it still doesn't motivate why there's a 3x
> slowdown between kernel versions...

I would say that something coded this bizarrely is really an application 
bug and not something that one could call a kernel regression. Any 
change in how the parent and child threads get scheduled will have a 
huge impact on this test. I bet if you replace that busy wait with a 
pthread_cond_wait or something similar, this problem goes away.

Hopefully it doesn't have to be pointed out that spawning off threads to 
do so little work before terminating is inefficient, a thread pool or 
even just a single thread would likely do a much better job..

       reply	other threads:[~2008-02-09  1:52 UTC|newest]

Thread overview: 23+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
     [not found] <fa.6N2dhyJ1cmBqiuFKgCaYfwduM+0@ifi.uio.no>
2008-02-09  1:49 ` Robert Hancock [this message]
2008-02-09  0:04 Olof Johansson
2008-02-09  0:08 ` Ingo Molnar
2008-02-09  0:32   ` Olof Johansson
2008-02-09  7:58 ` Mike Galbraith
2008-02-09  8:03   ` Willy Tarreau
2008-02-09 10:58     ` Mike Galbraith
2008-02-09 11:40       ` Willy Tarreau
2008-02-09 13:37         ` Mike Galbraith
2008-02-09 16:19           ` Willy Tarreau
2008-02-09 17:33             ` Mike Galbraith
2008-02-10  5:29             ` Olof Johansson
2008-02-10  6:15               ` Willy Tarreau
2008-02-10  7:00                 ` Olof Johansson
2008-02-10  7:58                   ` Willy Tarreau
2008-02-11  8:15                   ` Mike Galbraith
2008-02-11 17:26                     ` Olof Johansson
2008-02-11 19:58                       ` Mike Galbraith
2008-02-11 20:31                         ` Olof Johansson
2008-02-12  9:23                           ` Mike Galbraith
2008-02-13  5:49                             ` Mike Galbraith
2008-02-11 21:45               ` Bill Davidsen
2008-02-12  4:30                 ` Mike Galbraith

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=47AD06BD.7040702@shaw.ca \
    --to=hancockr@shaw.ca \
    --cc=a.p.zijlstra@chello.nl \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mingo@elte.hu \
    --cc=olof@lixom.net \
    --subject='Re: Scheduler(?) regression from 2.6.22 to 2.6.24 for short-lived threads' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).