LKML Archive on lore.kernel.org
help / color / mirror / Atom feed
From: Nick Piggin <nickpiggin@yahoo.com.au>
To: Andrea Arcangeli <andrea@suse.de>
Cc: Anton Blanchard <anton@samba.org>, Rik van Riel <riel@redhat.com>,
	Lorenzo Allegrucci <l_allegrucci@yahoo.it>,
	linux-kernel@vger.kernel.org, Ingo Molnar <mingo@elte.hu>,
	Suparna Bhattacharya <suparna@in.ibm.com>,
	Jens Axboe <jens.axboe@oracle.com>
Subject: Re: SMP performance degradation with sysbench
Date: Tue, 13 Mar 2007 23:08:59 +1100	[thread overview]
Message-ID: <45F6945B.2030309@yahoo.com.au> (raw)
In-Reply-To: <20070313114215.GI8992@v2.random>

Andrea Arcangeli wrote:
> On Tue, Mar 13, 2007 at 10:12:19PM +1100, Nick Piggin wrote:
> 
>>They'll be sleeping in futex_wait in the kernel, I think. One thread
>>will hold the critical mutex, some will be off doing their own thing,
>>but importantly there will be many sleeping for the mutex to become
>>available.
> 
> 
> The initial assumption was that there was zero idle time with threads
> = cpus and the idle time showed up only when the number of threads
> increased to the double the number of cpus. If the idle time wouldn't
> increase with the number of threads, nothing would be suspect.

Well I think more threads ~= more probability that this guy is going to
be preempted while holding the mutex?

This might be why FreeBSD works much better, because it looks like MySQL
actually will set RT scheduling for those processes that take critical
resources.

>>However, I tested with a bigger system and actually the idle time
>>comes before we saturate all CPUs. Also, increasing the aggressiveness
>>of the load balancer did not drop idle time at all, so it is not a case
>>of some runqueues idle while others have many threads on them.
> 
> 
> It'd be interesting to see the sysrq+t after the idle time
> increased.
> 
> 
>>I guess googlemalloc (tcmalloc?) isn't suitable for a general purpose
>>glibc allocator. But I wonder if there are other improvements that glibc
>>can do here?
> 
> 
> My wild guess is that they're allocating memory after taking
> futexes. If they do, something like this will happen:
> 
>      taskA		taskB		taskC
>      user lock
> 			mmap_sem lock
>      mmap sem -> schedule
> 					user lock -> schedule
> 
> If taskB wouldn't be there triggering more random trashing over the
> mmap_sem, the lock holder wouldn't wait and task C wouldn't wait too.
> 
> I suspect the real fix is not to allocate memory or to run other
> expensive syscalls that can block inside the futex critical sections...


I would agree that it points to MySQL scalability issues, however the
fact that such large gains come from tcmalloc is still interesting.

-- 
SUSE Labs, Novell Inc.
Send instant messages to your online friends http://au.messenger.yahoo.com 

  parent reply	other threads:[~2007-03-13 12:09 UTC|newest]

Thread overview: 45+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2007-02-25 17:44 Lorenzo Allegrucci
2007-02-25 23:46 ` Rik van Riel
2007-02-26 13:36   ` Nick Piggin
2007-02-26 13:41     ` Nick Piggin
2007-02-26 22:04     ` Pete Harlan
2007-02-26 22:36       ` Dave Jones
2007-02-27  0:32         ` Hiro Yoshioka
2007-02-27  0:43           ` Rik van Riel
2007-02-27  4:03             ` Hiro Yoshioka
2007-02-27  4:31               ` Rik van Riel
2007-02-27  8:14                 ` J.A. Magallón
2007-02-27 14:02                   ` Rik van Riel
2007-02-27 14:56                     ` Paulo Marques
2007-02-27 20:40                       ` Nish Aravamudan
2007-02-28  2:21                       ` Bill Davidsen
2007-02-28  2:52                         ` Nish Aravamudan
2007-03-01  0:20                           ` Nish Aravamudan
2007-02-27 19:05                     ` Lorenzo Allegrucci
2007-03-01 16:57                       ` Lorenzo Allegrucci
2007-02-28  1:27     ` Nish Aravamudan
2007-02-28  2:22       ` Nick Piggin
2007-02-28  2:51         ` Nish Aravamudan
2007-03-12 22:00     ` Anton Blanchard
2007-03-13  5:11       ` Nick Piggin
2007-03-13  9:45         ` Andrea Arcangeli
2007-03-13 10:06           ` Nick Piggin
2007-03-13 10:31             ` Andrea Arcangeli
2007-03-13 10:37               ` Nick Piggin
2007-03-13 10:57                 ` Andrea Arcangeli
2007-03-13 11:12                   ` Nick Piggin
2007-03-13 11:40                     ` Eric Dumazet
2007-03-13 11:56                       ` Nick Piggin
2007-03-13 11:42                     ` Andrea Arcangeli
2007-03-13 12:02                       ` Eric Dumazet
2007-03-13 12:27                         ` Jakub Jelinek
2007-03-13 12:08                       ` Nick Piggin [this message]
2007-03-14 23:33                         ` Siddha, Suresh B
2007-03-20  2:29                           ` Zhang, Yanmin
2007-04-02  2:59                             ` Zhang, Yanmin
2007-03-13  6:00       ` Eric Dumazet
2007-03-14  0:36       ` Nish Aravamudan
2007-03-14  1:00         ` Eric Dumazet
2007-03-14  1:09           ` Nish Aravamudan
     [not found] <fa.V3M3ZgXL+lFlIyhx43YxCU/JFUk@ifi.uio.no>
     [not found] ` <fa.ciL5lzdfskdJHJPgn+UVCHt/9EM@ifi.uio.no>
     [not found]   ` <fa.2ABbHhyCbp3Fx7hSE/Gr0SuzFvw@ifi.uio.no>
     [not found]     ` <fa.oaZk6Aiqd8gyZNsj7+m+w9MibhU@ifi.uio.no>
     [not found]       ` <fa.RjX9Y4ckjRCle5L+uWNdd0snOio@ifi.uio.no>
     [not found]         ` <fa.XocsudxlGplKh0kloTtA0juPwtA@ifi.uio.no>
2007-02-28  0:20           ` Robert Hancock
2007-02-28  1:32             ` Hiro Yoshioka

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=45F6945B.2030309@yahoo.com.au \
    --to=nickpiggin@yahoo.com.au \
    --cc=andrea@suse.de \
    --cc=anton@samba.org \
    --cc=jens.axboe@oracle.com \
    --cc=l_allegrucci@yahoo.it \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mingo@elte.hu \
    --cc=riel@redhat.com \
    --cc=suparna@in.ibm.com \
    --subject='Re: SMP performance degradation with sysbench' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).