LKML Archive on lore.kernel.org
help / color / mirror / Atom feed
From: Olof Johansson <olof@lixom.net>
To: linux-kernel@vger.kernel.org
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>, Ingo Molnar <mingo@elte.hu>
Subject: Scheduler(?) regression from 2.6.22 to 2.6.24 for short-lived threads
Date: Fri, 8 Feb 2008 18:04:56 -0600	[thread overview]
Message-ID: <20080209000456.GA21021@lixom.net> (raw)

Hi,

I ended up with a customer benchmark in my lap this week that doesn't
do well on recent kernels. :(

After cutting it down to a simple testcase/microbenchmark, it seems like
recent kernels don't do as well with short-lived threads competing
with the thread it's cloned off of. The CFS scheduler changes come to
mind, but I suppose it could be caused by something else as well.

The pared-down testcase is included below. Reported runtime for the
testcase has increased almost 3x between 2.6.22 and 2.6.24:

2.6.22: 3332 ms
2.6.23: 4397 ms
2.6.24: 8953 ms
2.6.24-git19: 8986 ms

While running, it'll fork off a bunch of threads, each doing just a little
work, then busy-waiting on the original thread to finish as well. Yes,
it's incredibly stupidly coded but that's not my point here.

During run, (runtime 10s on my 1.5GHz Core2 Duo laptop), vmstat 2  shows:

 0  0      0 115196 364748 2248396    0    0     0     0  163   89  0  0 100  0
 2  0      0 115172 364748 2248396    0    0     0     0  270  178 24  0 76  0
 2  0      0 115172 364748 2248396    0    0     0     0  402  283 52  0 48  0
 2  0      0 115180 364748 2248396    0    0     0     0  402  281 50  0 50  0
 2  0      0 115180 364764 2248396    0    0     0    22  403  295 51  0 48  1
 2  0      0 115056 364764 2248396    0    0     0     0  399  280 52  0 48  0
 0  0      0 115196 364764 2248396    0    0     0     0  241  141 17  0 83  0
 0  0      0 115196 364768 2248396    0    0     0     2  155   67  0  0 100  0
 0  0      0 115196 364768 2248396    0    0     0     0  148   62  0  0 100  0

I.e. runqueue is 2, but only one cpu is busy. However, this still seems
true on the kernel that runs the testcase in more reasonable time.

Also, 'time' reports real and user time roughly the same on all kernels,
so it's not that the older kernels are better at spreading out the load
between the two cores (either that or it doesn't account for stuff right).

I've included the config files, runtime output and vmstat output at
http://lixom.net/~olof/threadtest/. I see similar behaviour on PPC as
well as x86, so it's not architecture-specific.

Testcase below. Yes, I know, there's a bunch of stuff that could be done
differently and better, but it still doesn't motivate why there's a 3x
slowdown between kernel versions...


-Olof



#include <stdio.h>
#include <stdlib.h>
#include <sys/time.h>

#ifdef __PPC__
static void atomic_inc(volatile long *a)
{
	asm volatile ("1:\n\
			lwarx  %0,0,%1\n\
			addic  %0,%0,1\n\
			stwcx. %0,0,%1\n\
			bne-  1b" : "=&r" (result) : "r"(a));
}
#else
static void atomic_inc(volatile long *a)
{
	asm volatile ("lock; incl %0" : "+m" (*a));
}
#endif

volatile long stopped;

int thread_func(int cpus)
{
	int j;

	for (j = 0; j < 10000; j++)
		;

	atomic_inc(&stopped);

	/* Busy-wait */
	while (stopped < cpus)
		j++;
}

long usecs(void)
{
	struct timeval tv;
	gettimeofday(&tv, NULL);
	return tv.tv_sec * 1000000 + tv.tv_usec;
}

int main(int argc, char **argv)
{
	pthread_t thread;
	int i;
	long t1, t2;

	t1 = usecs();
	for (i = 0; i < 500; i++) {
		stopped = 0;

		pthread_create(&thread, NULL, thread_func, 2);
		thread_func(2);
		pthread_join(thread, NULL);
	}
	t2 = usecs();

	printf("time %ld ms\n", (t2-t1) / 1000);

	return 0;
}

             reply	other threads:[~2008-02-09  0:03 UTC|newest]

Thread overview: 23+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2008-02-09  0:04 Olof Johansson [this message]
2008-02-09  0:08 ` Ingo Molnar
2008-02-09  0:32   ` Olof Johansson
2008-02-09  7:58 ` Mike Galbraith
2008-02-09  8:03   ` Willy Tarreau
2008-02-09 10:58     ` Mike Galbraith
2008-02-09 11:40       ` Willy Tarreau
2008-02-09 13:37         ` Mike Galbraith
2008-02-09 16:19           ` Willy Tarreau
2008-02-09 17:33             ` Mike Galbraith
2008-02-10  5:29             ` Olof Johansson
2008-02-10  6:15               ` Willy Tarreau
2008-02-10  7:00                 ` Olof Johansson
2008-02-10  7:58                   ` Willy Tarreau
2008-02-11  8:15                   ` Mike Galbraith
2008-02-11 17:26                     ` Olof Johansson
2008-02-11 19:58                       ` Mike Galbraith
2008-02-11 20:31                         ` Olof Johansson
2008-02-12  9:23                           ` Mike Galbraith
2008-02-13  5:49                             ` Mike Galbraith
2008-02-11 21:45               ` Bill Davidsen
2008-02-12  4:30                 ` Mike Galbraith
     [not found] <fa.6N2dhyJ1cmBqiuFKgCaYfwduM+0@ifi.uio.no>
2008-02-09  1:49 ` Robert Hancock

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20080209000456.GA21021@lixom.net \
    --to=olof@lixom.net \
    --cc=a.p.zijlstra@chello.nl \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mingo@elte.hu \
    --subject='Re: Scheduler(?) regression from 2.6.22 to 2.6.24 for short-lived threads' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).