LKML Archive on lore.kernel.org
help / color / mirror / Atom feed
From: "Zhang, Yanmin" <yanmin_zhang@linux.intel.com>
To: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>,
	Andrea Arcangeli <andrea@suse.de>,
	Anton Blanchard <anton@samba.org>, Rik van Riel <riel@redhat.com>,
	Lorenzo Allegrucci <l_allegrucci@yahoo.it>,
	linux-kernel@vger.kernel.org, Ingo Molnar <mingo@elte.hu>,
	Suparna Bhattacharya <suparna@in.ibm.com>,
	Jens Axboe <jens.axboe@oracle.com>
Subject: Re: SMP performance degradation with sysbench
Date: Tue, 20 Mar 2007 10:29:08 +0800	[thread overview]
Message-ID: <1174357748.4448.21.camel@ymzhang> (raw)
In-Reply-To: <20070314233301.GK30596@linux-os.sc.intel.com>

On Wed, 2007-03-14 at 16:33 -0700, Siddha, Suresh B wrote:
> On Tue, Mar 13, 2007 at 05:08:59AM -0700, Nick Piggin wrote:
> > I would agree that it points to MySQL scalability issues, however the
> > fact that such large gains come from tcmalloc is still interesting.
> 
> What glibc version are you, Anton and others are using?
> 
> Does that version has this fix included?
> 
> Dynamically size mmap treshold if the program frees mmaped blocks.
> 
> http://sources.redhat.com/cgi-bin/cvsweb.cgi/libc/malloc/malloc.c.diff?r1=1.158&r2=1.159&cvsroot=glibc
> 
Last week, I reproduced it on RHEL4U3 with glibc 2.3.4-2.19. Today, I
installed RHEL5GA and reproduced it again. RHEL5GA uses glibc 2.5-12 which
already includes the dynamically size mmap threshold patch, so this patch doesn’t
resolve the issue.

The problem is really relevant to malloc/free of glibc multi-thread.

My paxville has 16 logical cpu (dual core+HT). I disabled HT by hot
removing the last 8 logical processors.

I captured the schedule status. When sysbench thread=8 (best performance),
there are about 3.4% context switches caused by __down_read/__down_write_nested.
When sysbench thread=10 (best performance), the percentage becomes 11.83%.

I captured the thread status by gdb. When sysbench thread=10, usually 2 threads
are calling mprotect/mmap. When sysbench thread=8, there are no threads calling
mprotect/mmap. Such capture has random behavior, but I tried for many times.

I think the increased percentage of context switch related to
__down_read/__down_write_nested is caused by mprotect/mmap. mprotect/mmap
accesses the semaphore of vm, so there are some contentions on the sema which
make performance down.

The strace shows mysqld often calls mprotect/mmap with the same data length
61440. That’s another evidence. Gdb showed such mprotect is called by
init_io_malloc=>my_malloc=>malloc=>init_malloc=>mprotect. Mmap is caused by
__init_free=>mmap. I checked the source codes of glibc and found the real call
chains are malloc=>init_malloc=>grow_heap=>mprotect and __init_free=>heap_trim=>mmap.

I guess the transaction processing of mysql/sysbench is: mysql accepts a connection
and initiates a block for the connection. After processing a couple of transactions,
sysbench closes the connection. Then, restart the procedure.

So why are there so many mprotect/mmap?

Glibc uses arena to speedup malloc/free at multi-thread environment.
mp.trim_threshold only controls main_arena. In function __init_free,
FASTBIN_CONSOLIDATION_THRE might be helpful, but it’s a fixed value.

The *ROOT CAUSE* is dynamic thresholds don’t apply to non-main arena.

To verify my idea, I created a small patch. When freeing a block, always
check mp_.trim_threshold even though it might not be in main arena. The
patch is just to verify my idea instead of the final solution.

--- glibc-2.5-20061008T1257_bak/malloc/malloc.c	2006-09-08 00:06:02.000000000 +0800
+++ glibc-2.5-20061008T1257/malloc/malloc.c	2007-03-20 07:41:03.000000000 +0800
@@ -4607,10 +4607,13 @@ _int_free(mstate av, Void_t* mem)
       } else {
 	/* Always try heap_trim(), even if the top chunk is not
 	   large, because the corresponding heap might go away.  */
+	if ((unsigned long)(chunksize(av->top)) >=
+	    (unsigned long)(mp_.trim_threshold)) {
 	heap_info *heap = heap_for_ptr(top(av));
 
 	assert(heap->ar_ptr == av);
 	heap_trim(heap, mp_.top_pad);
+	}
       }
     }
 

With the patch, I recompiled glibc and reran sysbench/mysql. The result is good.
When thread number is larger than 8, the tps and response time(avg) are smooth, and
don't drop severely.

Is there anyone being able to test it on AMD machine?

Yanmin

  reply	other threads:[~2007-03-20  2:29 UTC|newest]

Thread overview: 45+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2007-02-25 17:44 Lorenzo Allegrucci
2007-02-25 23:46 ` Rik van Riel
2007-02-26 13:36   ` Nick Piggin
2007-02-26 13:41     ` Nick Piggin
2007-02-26 22:04     ` Pete Harlan
2007-02-26 22:36       ` Dave Jones
2007-02-27  0:32         ` Hiro Yoshioka
2007-02-27  0:43           ` Rik van Riel
2007-02-27  4:03             ` Hiro Yoshioka
2007-02-27  4:31               ` Rik van Riel
2007-02-27  8:14                 ` J.A. Magallón
2007-02-27 14:02                   ` Rik van Riel
2007-02-27 14:56                     ` Paulo Marques
2007-02-27 20:40                       ` Nish Aravamudan
2007-02-28  2:21                       ` Bill Davidsen
2007-02-28  2:52                         ` Nish Aravamudan
2007-03-01  0:20                           ` Nish Aravamudan
2007-02-27 19:05                     ` Lorenzo Allegrucci
2007-03-01 16:57                       ` Lorenzo Allegrucci
2007-02-28  1:27     ` Nish Aravamudan
2007-02-28  2:22       ` Nick Piggin
2007-02-28  2:51         ` Nish Aravamudan
2007-03-12 22:00     ` Anton Blanchard
2007-03-13  5:11       ` Nick Piggin
2007-03-13  9:45         ` Andrea Arcangeli
2007-03-13 10:06           ` Nick Piggin
2007-03-13 10:31             ` Andrea Arcangeli
2007-03-13 10:37               ` Nick Piggin
2007-03-13 10:57                 ` Andrea Arcangeli
2007-03-13 11:12                   ` Nick Piggin
2007-03-13 11:40                     ` Eric Dumazet
2007-03-13 11:56                       ` Nick Piggin
2007-03-13 11:42                     ` Andrea Arcangeli
2007-03-13 12:02                       ` Eric Dumazet
2007-03-13 12:27                         ` Jakub Jelinek
2007-03-13 12:08                       ` Nick Piggin
2007-03-14 23:33                         ` Siddha, Suresh B
2007-03-20  2:29                           ` Zhang, Yanmin [this message]
2007-04-02  2:59                             ` Zhang, Yanmin
2007-03-13  6:00       ` Eric Dumazet
2007-03-14  0:36       ` Nish Aravamudan
2007-03-14  1:00         ` Eric Dumazet
2007-03-14  1:09           ` Nish Aravamudan
     [not found] <fa.V3M3ZgXL+lFlIyhx43YxCU/JFUk@ifi.uio.no>
     [not found] ` <fa.ciL5lzdfskdJHJPgn+UVCHt/9EM@ifi.uio.no>
     [not found]   ` <fa.2ABbHhyCbp3Fx7hSE/Gr0SuzFvw@ifi.uio.no>
     [not found]     ` <fa.oaZk6Aiqd8gyZNsj7+m+w9MibhU@ifi.uio.no>
     [not found]       ` <fa.RjX9Y4ckjRCle5L+uWNdd0snOio@ifi.uio.no>
     [not found]         ` <fa.XocsudxlGplKh0kloTtA0juPwtA@ifi.uio.no>
2007-02-28  0:20           ` Robert Hancock
2007-02-28  1:32             ` Hiro Yoshioka

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1174357748.4448.21.camel@ymzhang \
    --to=yanmin_zhang@linux.intel.com \
    --cc=andrea@suse.de \
    --cc=anton@samba.org \
    --cc=jens.axboe@oracle.com \
    --cc=l_allegrucci@yahoo.it \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mingo@elte.hu \
    --cc=nickpiggin@yahoo.com.au \
    --cc=riel@redhat.com \
    --cc=suparna@in.ibm.com \
    --cc=suresh.b.siddha@intel.com \
    --subject='Re: SMP performance degradation with sysbench' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).