LKML Archive on lore.kernel.org
help / color / mirror / Atom feed
* Performance versus FreeBSD 7.0
@ 2008-02-29 13:38 Stephen Cuppett
  2008-02-29 14:54 ` Diego Calleja
                   ` (2 more replies)
  0 siblings, 3 replies; 14+ messages in thread
From: Stephen Cuppett @ 2008-02-29 13:38 UTC (permalink / raw)
  To: linux-kernel

Not lobbing the first artillery shell by any means, but I saw this
bullet on the FreeBSD release announcement:

# Dramatic improvements in performance and SMP scalability shown by
various database and other benchmarks, in some cases showing peak
performance improvements as high as 350% over FreeBSD 6.X under normal
loads and 1500% at high loads. When compared with the best performing
Linux kernel (2.6.22 or 2.6.24) performance is 15% better. Results are
from benchmarks used to analyze and improve system performance,
results with your specific work load may vary. Some of the changes
that contribute to this improvement are:

    * The 1:1 libthr threading model is now the default.
    * Finer-grained IPC, networking, and scheduler locking.
    * A major focus on optimizing the SMP architecture that was put in
place during the 5.x and 6.x branches.

Some benchmarks show linear scaling up to 8 CPUs. Many workloads see a
significant performance improvement with multicore systems.

The whole thing is available at:

http://www.freebsd.org/releases/7.0R/announce.html

However, their site is pretty scant on details about those benchmarks.
 Was just curious if anybody on this list had any input or links on
the metrics, the hardware, the workloads, how was Linux or FreeBSD
tuned, knew more information, could point me to the stats....

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Performance versus FreeBSD 7.0
  2008-02-29 13:38 Performance versus FreeBSD 7.0 Stephen Cuppett
@ 2008-02-29 14:54 ` Diego Calleja
  2008-03-03 13:04   ` Nick Piggin
  2008-03-03 12:34 ` Andi Kleen
  2008-03-06 17:29 ` debian developer
  2 siblings, 1 reply; 14+ messages in thread
From: Diego Calleja @ 2008-02-29 14:54 UTC (permalink / raw)
  To: Stephen Cuppett; +Cc: linux-kernel

El Fri, 29 Feb 2008 08:38:00 -0500, "Stephen Cuppett" <cuppett@gmail.com> escribió:

> loads and 1500% at high loads. When compared with the best performing
> Linux kernel (2.6.22 or 2.6.24) performance is 15% better. Results are

There has been some performance problems with sysbench performance in linux
which made it slower than freebsd, there were some patches to speed things up,
not sure if they have been merged.

Myself I find fun that they look at Linux as the Leader That Must Be Surpassed.
All the performance highlights of freebsd 7.0 are things that linux already did some
years ago.

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Performance versus FreeBSD 7.0
  2008-02-29 13:38 Performance versus FreeBSD 7.0 Stephen Cuppett
  2008-02-29 14:54 ` Diego Calleja
@ 2008-03-03 12:34 ` Andi Kleen
  2008-03-06 17:29 ` debian developer
  2 siblings, 0 replies; 14+ messages in thread
From: Andi Kleen @ 2008-03-03 12:34 UTC (permalink / raw)
  To: Stephen Cuppett; +Cc: linux-kernel

"Stephen Cuppett" <cuppett@gmail.com> writes:

> Not lobbing the first artillery shell by any means, but I saw this
> bullet on the FreeBSD release announcement:

I posted a comment on that on the LWN thread, but their website 
has eaten it.

Anyways I would be very careful with any benchmarks testing mysql
especially on sysbench on larger systems (like they did with a 8 core
system). I did some similar testing on a slightly larger system (16 core) 
some time ago and found that mysql itself didn't scale at all.
With sysbench which only exercises a single table it would end up basically 
just banging on a single mysql user space lock and spend nearly all
the kernel time in the scheduler over-scheduling like crazy.

The FreeBSD people are not very clear on what they exactly benchmarked
(MySQL has lots of configuration options and engines), but I'll assume
they used the standard MyISAM.

I don't think that's a very useful benchmark to tune for in this
form at least not until MySQL is fixed to actually scale and sysbench
replaced with some a little more sane. It's worse than dbench. 

-Andi

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Performance versus FreeBSD 7.0
  2008-02-29 14:54 ` Diego Calleja
@ 2008-03-03 13:04   ` Nick Piggin
  2008-03-03 19:02     ` Cyrus Massoumi
  0 siblings, 1 reply; 14+ messages in thread
From: Nick Piggin @ 2008-03-03 13:04 UTC (permalink / raw)
  To: Diego Calleja; +Cc: Stephen Cuppett, linux-kernel

On Saturday 01 March 2008 01:54, Diego Calleja wrote:
> El Fri, 29 Feb 2008 08:38:00 -0500, "Stephen Cuppett" <cuppett@gmail.com> 
escribió:
> > loads and 1500% at high loads. When compared with the best performing
> > Linux kernel (2.6.22 or 2.6.24) performance is 15% better. Results are
>
> There has been some performance problems with sysbench performance in linux
> which made it slower than freebsd, there were some patches to speed things
> up, not sure if they have been merged.

There definitely were performance problems with threaded malloc/free
in the Linux kernel and glibc. Fixes have been merged in both packages,
and AFAIK the FreeBSD guys tested with those fixes in place.

I think these were never really run into before in part due to MySQL's
unscalable heap design makes it not scale well on higher numbers of
CPUs anyway, and also made the malloc problems more pronounced (ie.
they added a bit to the contention of the single heap lock, which is
the big killer here).

But it was very nice to be made aware of the problem and be able to
improve it.


> Myself I find fun that they look at Linux as the Leader That Must Be
> Surpassed. All the performance highlights of freebsd 7.0 are things that
> linux already did some years ago.

I don't know very much about FreeBSD nor have verified the results
for myself. But to their credit they seem to have done quite well at
least on the smaller end of the scale.


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Performance versus FreeBSD 7.0
  2008-03-03 13:04   ` Nick Piggin
@ 2008-03-03 19:02     ` Cyrus Massoumi
  2008-03-03 19:24       ` Nick Piggin
  2008-03-03 23:23       ` Andi Kleen
  0 siblings, 2 replies; 14+ messages in thread
From: Cyrus Massoumi @ 2008-03-03 19:02 UTC (permalink / raw)
  To: Nick Piggin
  Cc: Diego Calleja, Stephen Cuppett, linux-kernel, mingo, a.p.zijlstra

Nick Piggin wrote:
> On Saturday 01 March 2008 01:54, Diego Calleja wrote:
>> El Fri, 29 Feb 2008 08:38:00 -0500, "Stephen Cuppett" <cuppett@gmail.com> 
> escribió:
>>> loads and 1500% at high loads. When compared with the best performing
>>> Linux kernel (2.6.22 or 2.6.24) performance is 15% better. Results are
>> There has been some performance problems with sysbench performance in linux
>> which made it slower than freebsd, there were some patches to speed things
>> up, not sure if they have been merged.
> 
> There definitely were performance problems with threaded malloc/free
> in the Linux kernel and glibc. Fixes have been merged in both packages,
> and AFAIK the FreeBSD guys tested with those fixes in place.
> 
> I think these were never really run into before in part due to MySQL's
> unscalable heap design makes it not scale well on higher numbers of
> CPUs anyway, and also made the malloc problems more pronounced (ie.
> they added a bit to the contention of the single heap lock, which is
> the big killer here).

IIRC, going to fine-grained file locking gave them a huge boost in this 
particular benchmark (and maybe others).
As I said on lwn.net Peter Zijlstra posted a patch to break the global 
file list lock about a year ago [1], but I don't think it was ever 
merged. Here [2] are some numbers for the patchset.

[1] http://lkml.org/lkml/2007/1/28/29
[2] http://lkml.org/lkml/2007/1/28/116

> But it was very nice to be made aware of the problem and be able to
> improve it.
> 
> 
>> Myself I find fun that they look at Linux as the Leader That Must Be
>> Surpassed. All the performance highlights of freebsd 7.0 are things that
>> linux already did some years ago.
> 
> I don't know very much about FreeBSD nor have verified the results
> for myself. But to their credit they seem to have done quite well at
> least on the smaller end of the scale.

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Performance versus FreeBSD 7.0
  2008-03-03 19:02     ` Cyrus Massoumi
@ 2008-03-03 19:24       ` Nick Piggin
  2008-03-03 22:47         ` Cyrus Massoumi
  2008-03-03 23:23       ` Andi Kleen
  1 sibling, 1 reply; 14+ messages in thread
From: Nick Piggin @ 2008-03-03 19:24 UTC (permalink / raw)
  To: Cyrus Massoumi
  Cc: Diego Calleja, Stephen Cuppett, linux-kernel, mingo, a.p.zijlstra

On Tuesday 04 March 2008 06:02, Cyrus Massoumi wrote:
> Nick Piggin wrote:
> > On Saturday 01 March 2008 01:54, Diego Calleja wrote:
> >> El Fri, 29 Feb 2008 08:38:00 -0500, "Stephen Cuppett"
> >> <cuppett@gmail.com>
> >
> > escribió:
> >>> loads and 1500% at high loads. When compared with the best performing
> >>> Linux kernel (2.6.22 or 2.6.24) performance is 15% better. Results are
> >>
> >> There has been some performance problems with sysbench performance in
> >> linux which made it slower than freebsd, there were some patches to
> >> speed things up, not sure if they have been merged.
> >
> > There definitely were performance problems with threaded malloc/free
> > in the Linux kernel and glibc. Fixes have been merged in both packages,
> > and AFAIK the FreeBSD guys tested with those fixes in place.
> >
> > I think these were never really run into before in part due to MySQL's
> > unscalable heap design makes it not scale well on higher numbers of
> > CPUs anyway, and also made the malloc problems more pronounced (ie.
> > they added a bit to the contention of the single heap lock, which is
> > the big killer here).
>
> IIRC, going to fine-grained file locking gave them a huge boost in this
> particular benchmark (and maybe others).
> As I said on lwn.net Peter Zijlstra posted a patch to break the global
> file list lock about a year ago [1], but I don't think it was ever
> merged. Here [2] are some numbers for the patchset.
>
> [1] http://lkml.org/lkml/2007/1/28/29
> [2] http://lkml.org/lkml/2007/1/28/116

The mysql workload is not constrained by this lock. The important
part (where freebsd saw their gain) is probably fd lookups, which
are lockless in Linux for a long time.


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Performance versus FreeBSD 7.0
  2008-03-03 19:24       ` Nick Piggin
@ 2008-03-03 22:47         ` Cyrus Massoumi
  0 siblings, 0 replies; 14+ messages in thread
From: Cyrus Massoumi @ 2008-03-03 22:47 UTC (permalink / raw)
  To: Nick Piggin
  Cc: Diego Calleja, Stephen Cuppett, linux-kernel, mingo, a.p.zijlstra

Nick Piggin wrote:
> On Tuesday 04 March 2008 06:02, Cyrus Massoumi wrote:
>> Nick Piggin wrote:
>>> On Saturday 01 March 2008 01:54, Diego Calleja wrote:
>>>> El Fri, 29 Feb 2008 08:38:00 -0500, "Stephen Cuppett"
>>>> <cuppett@gmail.com>
>>> escribió:
>>>>> loads and 1500% at high loads. When compared with the best performing
>>>>> Linux kernel (2.6.22 or 2.6.24) performance is 15% better. Results are
>>>> There has been some performance problems with sysbench performance in
>>>> linux which made it slower than freebsd, there were some patches to
>>>> speed things up, not sure if they have been merged.
>>> There definitely were performance problems with threaded malloc/free
>>> in the Linux kernel and glibc. Fixes have been merged in both packages,
>>> and AFAIK the FreeBSD guys tested with those fixes in place.
>>>
>>> I think these were never really run into before in part due to MySQL's
>>> unscalable heap design makes it not scale well on higher numbers of
>>> CPUs anyway, and also made the malloc problems more pronounced (ie.
>>> they added a bit to the contention of the single heap lock, which is
>>> the big killer here).
>> IIRC, going to fine-grained file locking gave them a huge boost in this
>> particular benchmark (and maybe others).
>> As I said on lwn.net Peter Zijlstra posted a patch to break the global
>> file list lock about a year ago [1], but I don't think it was ever
>> merged. Here [2] are some numbers for the patchset.
>>
>> [1] http://lkml.org/lkml/2007/1/28/29
>> [2] http://lkml.org/lkml/2007/1/28/116
> 
> The mysql workload is not constrained by this lock. The important
> part (where freebsd saw their gain) is probably fd lookups, which
> are lockless in Linux for a long time.

Thanks for pointing this out Nick.


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Performance versus FreeBSD 7.0
  2008-03-03 19:02     ` Cyrus Massoumi
  2008-03-03 19:24       ` Nick Piggin
@ 2008-03-03 23:23       ` Andi Kleen
  2008-03-04  5:51         ` Cyrus Massoumi
  2008-03-06 12:53         ` Nick Piggin
  1 sibling, 2 replies; 14+ messages in thread
From: Andi Kleen @ 2008-03-03 23:23 UTC (permalink / raw)
  To: Cyrus Massoumi
  Cc: Nick Piggin, Diego Calleja, Stephen Cuppett, linux-kernel, mingo,
	a.p.zijlstra

Cyrus Massoumi <cyrusm@gmx.net> writes:
>
> IIRC, going to fine-grained file locking gave them a huge boost in
> this particular benchmark (and maybe others).

What locking? You mean their equivalent of i_mutex? There are 
already file systems on Linux that don't use it significantly
for write.

> As I said on lwn.net Peter Zijlstra posted a patch to break the global
> file list lock about a year ago [1], but I don't think it was ever
> merged. Here [2] are some numbers for the patchset.

The file_list_lock is only for open/close which MySQL is very unlikely
to do in its fast path. You're totally on the wrong path here I think.

-Andi

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Performance versus FreeBSD 7.0
  2008-03-03 23:23       ` Andi Kleen
@ 2008-03-04  5:51         ` Cyrus Massoumi
  2008-03-06 12:53         ` Nick Piggin
  1 sibling, 0 replies; 14+ messages in thread
From: Cyrus Massoumi @ 2008-03-04  5:51 UTC (permalink / raw)
  To: Andi Kleen
  Cc: Nick Piggin, Diego Calleja, Stephen Cuppett, linux-kernel, mingo,
	a.p.zijlstra

Andi Kleen wrote:
> Cyrus Massoumi <cyrusm@gmx.net> writes:
>> IIRC, going to fine-grained file locking gave them a huge boost in
>> this particular benchmark (and maybe others).
> 
> What locking? You mean their equivalent of i_mutex? There are 
> already file systems on Linux that don't use it significantly
> for write.
> 
>> As I said on lwn.net Peter Zijlstra posted a patch to break the global
>> file list lock about a year ago [1], but I don't think it was ever
>> merged. Here [2] are some numbers for the patchset.
> 
> The file_list_lock is only for open/close which MySQL is very unlikely
> to do in its fast path. You're totally on the wrong path here I think.

Yes, Nick already corrected me, sorry for the noise :)


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Performance versus FreeBSD 7.0
  2008-03-03 23:23       ` Andi Kleen
  2008-03-04  5:51         ` Cyrus Massoumi
@ 2008-03-06 12:53         ` Nick Piggin
  2008-03-06 13:12           ` Andi Kleen
  1 sibling, 1 reply; 14+ messages in thread
From: Nick Piggin @ 2008-03-06 12:53 UTC (permalink / raw)
  To: Andi Kleen
  Cc: Cyrus Massoumi, Diego Calleja, Stephen Cuppett, linux-kernel,
	mingo, a.p.zijlstra

On Tuesday 04 March 2008 10:23, Andi Kleen wrote:

> You're totally on the wrong path here I think.

In my testing, I found that Linux is a bit faster and more scalable
than FreeBSD on MySQL sysbench.

http://www.kernel.org/pub/linux/kernel/people/npiggin/sysbench/

I think since the mmap_sem madvise problem was solved, the kernel
didn't really have any further problems with this workload.

It is doing a lot of context switches, no IO, and not a lot of
real work. The DBMS runs into some scalability problems in this
workload, but to be fair it is probably not one that the MySQL
guys care about too much. But it sometimes seems a bit sloppy...
eg. a full 35% of the syscalls the MySQL server makes are failed
sched_setparam calls that are passing in invalid values.

So I don't really consider it a big victory to be slightly faster
here; nobody really noticed until this test that we had the silly
performance bug in our malloc/free paths in glibc and the kernel.


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Performance versus FreeBSD 7.0
  2008-03-06 12:53         ` Nick Piggin
@ 2008-03-06 13:12           ` Andi Kleen
  0 siblings, 0 replies; 14+ messages in thread
From: Andi Kleen @ 2008-03-06 13:12 UTC (permalink / raw)
  To: Nick Piggin
  Cc: Andi Kleen, Cyrus Massoumi, Diego Calleja, Stephen Cuppett,
	linux-kernel, mingo, a.p.zijlstra

> I think since the mmap_sem madvise problem was solved, the kernel
> didn't really have any further problems with this workload.

Good.

> It is doing a lot of context switches, no IO, and not a lot of

Yes sysbench is a pretty poor benchmark i think, especially
on systems with a lot of cores. I ran into this on some of my
own testing too. I don't think it makes too much
sense to tune for it seriously, but of course it doesn't hurt
when Linux does well on it anyways.

-Andi

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Performance versus FreeBSD 7.0
  2008-02-29 13:38 Performance versus FreeBSD 7.0 Stephen Cuppett
  2008-02-29 14:54 ` Diego Calleja
  2008-03-03 12:34 ` Andi Kleen
@ 2008-03-06 17:29 ` debian developer
  2008-03-06 17:34   ` debian developer
  2 siblings, 1 reply; 14+ messages in thread
From: debian developer @ 2008-03-06 17:29 UTC (permalink / raw)
  To: Stephen Cuppett; +Cc: linux-kernel, Ingo Molnar, Linus Torvalds

On Fri, Feb 29, 2008 at 7:08 PM, Stephen Cuppett <cuppett@gmail.com> wrote:
> Not lobbing the first artillery shell by any means, but I saw this
>  bullet on the FreeBSD release announcement:
>
>  # Dramatic improvements in performance and SMP scalability shown by
>  various database and other benchmarks, in some cases showing peak
>  performance improvements as high as 350% over FreeBSD 6.X under normal
>  loads and 1500% at high loads. When compared with the best performing
>  Linux kernel (2.6.22 or 2.6.24) performance is 15% better. Results are
>  from benchmarks used to analyze and improve system performance,
>  results with your specific work load may vary. Some of the changes
>  that contribute to this improvement are:
>
>     * The 1:1 libthr threading model is now the default.
>     * Finer-grained IPC, networking, and scheduler locking.
>     * A major focus on optimizing the SMP architecture that was put in
>  place during the 5.x and 6.x branches.
>
>  Some benchmarks show linear scaling up to 8 CPUs. Many workloads see a
>  significant performance improvement with multicore systems.
>
>  The whole thing is available at:
>
>  http://www.freebsd.org/releases/7.0R/announce.html
>
>  However, their site is pretty scant on details about those benchmarks.
>   Was just curious if anybody on this list had any input or links on
>  the metrics, the hardware, the workloads, how was Linux or FreeBSD
>  tuned, knew more information, could point me to the stats....

The benchmarks are posted here

http://people.freebsd.org/~kris/scaling/7.0%20Preview.pdf

page no 17-20.

A funny quote on CFS from above (page 19)

"The new CFS scheduler in 2.6.23 is "Completely Fair"...to
FreeBSD" :)

good to see some competition at last :D

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Performance versus FreeBSD 7.0
  2008-03-06 17:29 ` debian developer
@ 2008-03-06 17:34   ` debian developer
  0 siblings, 0 replies; 14+ messages in thread
From: debian developer @ 2008-03-06 17:34 UTC (permalink / raw)
  To: Stephen Cuppett; +Cc: linux-kernel, Ingo Molnar, Linus Torvalds

On Thu, Mar 6, 2008 at 10:59 PM, debian developer <debiandev@gmail.com> wrote:

>  http://people.freebsd.org/~kris/scaling/7.0%20Preview.pdf
>
bloody spaces in file names #$%$#..

http://people.freebsd.org/~kris/scaling/7.0 Preview.pdf

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Performance versus FreeBSD 7.0
       [not found] ` <a2eSt-7UK-1@gated-at.bofh.it>
@ 2009-01-25 10:07   ` Lx
  0 siblings, 0 replies; 14+ messages in thread
From: Lx @ 2009-01-25 10:07 UTC (permalink / raw)
  To: Diego Calleja; +Cc: linux-kernel

>
> Myself I find fun that they look at Linux as the Leader That Must Be Surpassed.
> All the performance highlights of freebsd 7.0 are things that linux already did some
> years ago.

GNU : Got Nothing Unique !

So it must be GNU/Linux. What a giant lock and what a benchmark ?


^ permalink raw reply	[flat|nested] 14+ messages in thread

end of thread, other threads:[~2009-01-25 10:15 UTC | newest]

Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2008-02-29 13:38 Performance versus FreeBSD 7.0 Stephen Cuppett
2008-02-29 14:54 ` Diego Calleja
2008-03-03 13:04   ` Nick Piggin
2008-03-03 19:02     ` Cyrus Massoumi
2008-03-03 19:24       ` Nick Piggin
2008-03-03 22:47         ` Cyrus Massoumi
2008-03-03 23:23       ` Andi Kleen
2008-03-04  5:51         ` Cyrus Massoumi
2008-03-06 12:53         ` Nick Piggin
2008-03-06 13:12           ` Andi Kleen
2008-03-03 12:34 ` Andi Kleen
2008-03-06 17:29 ` debian developer
2008-03-06 17:34   ` debian developer
     [not found] <a2dDb-60y-29@gated-at.bofh.it>
     [not found] ` <a2eSt-7UK-1@gated-at.bofh.it>
2009-01-25 10:07   ` Lx

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).