2.6.25-rc1: volanoMark 45% regression

2.6.25-rc1: volanoMark 45% regression

[Zhang, Yanmin]

volanoMark has 45% regression with kernel 2.6.25-rc1 on my both 8-core
stoakley and 16-core Tigerton.

I used bisect to locate below patch.

commit 58e2d4ca581167c2a079f4ee02be2f0bc52e8729
Author: Srivatsa Vaddagiri <vatsa@linux.vnet.ibm.com>
Date:   Fri Jan 25 21:08:00 2008 +0100

    sched: group scheduling, change how cpu load is calculated



hackbench has about 30% regression on 16-core tigerton, but has about 10% improvement
on 8-core stoakley.

In addition, tbench has about 6% regression on my 8-core stoakley and
25% regression on 16-core stoakley. Some other benchmarks, like netperf/aim7
also have some regression. I will verify if they are all related to the
patch.

-yanmin

Re: 2.6.25-rc1: volanoMark 45% regression

[Balbir Singh]

Zhang, Yanmin wrote:
> volanoMark has 45% regression with kernel 2.6.25-rc1 on my both 8-core
> stoakley and 16-core Tigerton.
> 
> I used bisect to locate below patch.
> 
> commit 58e2d4ca581167c2a079f4ee02be2f0bc52e8729
> Author: Srivatsa Vaddagiri <vatsa@linux.vnet.ibm.com>
> Date:   Fri Jan 25 21:08:00 2008 +0100
> 
>     sched: group scheduling, change how cpu load is calculated
> 
> 
> 
> hackbench has about 30% regression on 16-core tigerton, but has about 10% improvement
> on 8-core stoakley.
> 
> In addition, tbench has about 6% regression on my 8-core stoakley and
> 25% regression on 16-core stoakley. Some other benchmarks, like netperf/aim7
> also have some regression. I will verify if they are all related to the
> patch.
> 
> -yanmin

Hi, Yamin,

Thanks for reporting the issue? Any chance we could getthe Oprofile output for
the run? The exact commandline and .config being used would also help.

-- 
	Warm Regards,
	Balbir Singh
	Linux Technology Center
	IBM, ISTL

Re: 2.6.25-rc1: volanoMark 45% regression

[Srivatsa Vaddagiri]

On Wed, Feb 13, 2008 at 03:15:16PM +0530, Balbir Singh wrote:
> Zhang, Yanmin wrote:
> > volanoMark has 45% regression with kernel 2.6.25-rc1 on my both 8-core
> > stoakley and 16-core Tigerton.
> > 
> > I used bisect to locate below patch.
> > 
> > commit 58e2d4ca581167c2a079f4ee02be2f0bc52e8729
> > Author: Srivatsa Vaddagiri <vatsa@linux.vnet.ibm.com>
> > Date:   Fri Jan 25 21:08:00 2008 +0100
> > 
> >     sched: group scheduling, change how cpu load is calculated
> > 
> > 
> > 
> > hackbench has about 30% regression on 16-core tigerton, but has about 10% improvement
> > on 8-core stoakley.
> > 
> > In addition, tbench has about 6% regression on my 8-core stoakley and
> > 25% regression on 16-core stoakley. Some other benchmarks, like netperf/aim7
> > also have some regression. I will verify if they are all related to the
> > patch.
> > 
> > -yanmin
> 
> Hi, Yamin,
> 
> Thanks for reporting the issue? Any chance we could getthe Oprofile output for
> the run? The exact commandline and .config being used would also help.

Yamin,
	I would also like to know against which previous version is this
regression being compared with. Is it 2.6.24? Did you have
CONFIG_FAIR_USER_SCHED enabled in both cases? It would also help to know if you
see the same regression with FAIR_GROUP_SCHED turned off.

-- 
Regards,
vatsa

Re: 2.6.25-rc1: volanoMark 45% regression

[Zhang, Yanmin]

On Wed, 2008-02-13 at 17:37 +0530, Srivatsa Vaddagiri wrote:
> On Wed, Feb 13, 2008 at 03:15:16PM +0530, Balbir Singh wrote:
> > Zhang, Yanmin wrote:
> > > volanoMark has 45% regression with kernel 2.6.25-rc1 on my both 8-core
> > > stoakley and 16-core Tigerton.
> > > 
> > > I used bisect to locate below patch.
> > > 
> > > commit 58e2d4ca581167c2a079f4ee02be2f0bc52e8729
> > > Author: Srivatsa Vaddagiri <vatsa@linux.vnet.ibm.com>
> > > Date:   Fri Jan 25 21:08:00 2008 +0100
> > > 
> > >     sched: group scheduling, change how cpu load is calculated
> > > 
> > > 
> > > 
> > > hackbench has about 30% regression on 16-core tigerton, but has about 10% improvement
> > > on 8-core stoakley.
> > > 
> > > In addition, tbench has about 6% regression on my 8-core stoakley and
> > > 25% regression on 16-core stoakley.
I verified tbench regression is not caused by the same patch. I am digging tbench now.

>  Some other benchmarks, like netperf/aim7
> > > also have some regression. I will verify if they are all related to the
> > > patch.
> > > 
> > > -yanmin
> > 
> > Hi, Yamin,
> > 
> > Thanks for reporting the issue? Any chance we could getthe Oprofile output for
> > the run?
I got oprofile data but it didn't show clear evidence.

When doing volanoMark testing, vmstat showed the good kernel's context switch
is about 1100000, but the bad kernel's context switch is 720000. Good kernel's
idle is about 1%, and bad kernel's idle is about 5%.

>  The exact commandline and .config being used would also help.
I used some scripts to start volanoMark.

Netperf loop UDP-RR-1/512's 10% regression on 16-core tigerton is also related to the patch.
If I set CONFIG_FAIR_GROUP_SCHED=n, there is no the netperf regression. I bind the netserver
process to a core and bind the client to another core in another processor.

It's hard to debug into netperf regression if it's caused by scheduler.

> 
> Yamin,
> 	I would also like to know against which previous version is this
> regression being compared with. Is it 2.6.24?
Yes.

>  Did you have
> CONFIG_FAIR_USER_SCHED enabled in both cases?
Yes.

CONFIG_FAIR_GROUP_SCHED=y
CONFIG_FAIR_USER_SCHED=y

>  It would also help to know if you
> see the same regression with FAIR_GROUP_SCHED turned off.
No regression if CONFIG_FAIR_GROUP_SCHED=n.

-yanmin

Re: 2.6.25-rc1: volanoMark 45% regression

[Zhang, Yanmin]

On Thu, 2008-02-14 at 15:41 +0800, Zhang, Yanmin wrote:
> On Wed, 2008-02-13 at 17:37 +0530, Srivatsa Vaddagiri wrote:
> > On Wed, Feb 13, 2008 at 03:15:16PM +0530, Balbir Singh wrote:
> > > Zhang, Yanmin wrote:
> > > > volanoMark has 45% regression with kernel 2.6.25-rc1 on my both 8-core
> > > > stoakley and 16-core Tigerton.
> > > > 
> > > > I used bisect to locate below patch.
> > > > 
> > > > commit 58e2d4ca581167c2a079f4ee02be2f0bc52e8729
> > > > Author: Srivatsa Vaddagiri <vatsa@linux.vnet.ibm.com>
> > > > Date:   Fri Jan 25 21:08:00 2008 +0100
> > > > 
> > > >     sched: group scheduling, change how cpu load is calculated
> > > > 
> > > > 
> > > > 
> > > > hackbench has about 30% regression on 16-core tigerton, but has about 10% improvement
> > > > on 8-core stoakley.
> > > > 
> > > > In addition, tbench has about 6% regression on my 8-core stoakley and
> > > > 25% regression on 16-core stoakley.
> I verified tbench regression is not caused by the same patch. I am digging tbench now.
> 
> >  Some other benchmarks, like netperf/aim7
> > > > also have some regression. I will verify if they are all related to the
> > > > patch.
> > > > 
> > > > -yanmin
> > > 
> > > Hi, Yamin,
> > > 
> > > Thanks for reporting the issue? Any chance we could getthe Oprofile output for
> > > the run?
> I got oprofile data but it didn't show clear evidence.
> 
> When doing volanoMark testing, vmstat showed the good kernel's context switch
> is about 1100000, but the bad kernel's context switch is 720000. Good kernel's
> idle is about 1%, and bad kernel's idle is about 5%.
> 
> >  The exact commandline and .config being used would also help.
> I used some scripts to start volanoMark.
> 
> Netperf loop UDP-RR-1/512's 10% regression
netperf loopback UDP-RR-1/512 regression is 5% on 8-core stoakley and 16-core tigerton.
On tulsa machine (8 cores+hyperThread), it's 18%.

If set CONFIG_FAIR_GROUP_SCHED=n, part regression of netperf disappears. If I change
dst_entry like what I said in http://marc.info/?l=linux-kernel&m=120305556317006&w=2, plus
CONFIG_FAIR_GROUP_SCHED=n, the result is a little better than 2.6.24.

I have a couple of different machines and sometimes benchmarks might have different
behavior. Sorry for the update.

>  on 16-core tigerton is also related to the patch.
> If I set CONFIG_FAIR_GROUP_SCHED=n, there is no the netperf regression. I bind the netserver
> process to a core and bind the client to another core in another processor.
> 
> It's hard to debug into netperf regression if it's caused by scheduler.
> 
> > 
> > Yamin,
> > 	I would also like to know against which previous version is this
> > regression being compared with. Is it 2.6.24?
> Yes.
> 
> >  Did you have
> > CONFIG_FAIR_USER_SCHED enabled in both cases?
> Yes.
> 
> CONFIG_FAIR_GROUP_SCHED=y
> CONFIG_FAIR_USER_SCHED=y
> 
> >  It would also help to know if you
> > see the same regression with FAIR_GROUP_SCHED turned off.
> No regression if CONFIG_FAIR_GROUP_SCHED=n.

Re: 2.6.25-rc1: volanoMark 45% regression

[Balbir Singh]

Zhang, Yanmin wrote:
>>
>>>  Did you have
>>> CONFIG_FAIR_USER_SCHED enabled in both cases?
>> Yes.
>>
>> CONFIG_FAIR_GROUP_SCHED=y
>> CONFIG_FAIR_USER_SCHED=y
>>

Interesting that you have CONFIG_FAIR_USER_SCHED. Do you have any other users on
the system (are they running anything at the time of the benchmark)? Could you
turn off CONFIG_FAIR_USER_SCHED  and turn on CONFIG_FAIR_GROUP_SCHED. I am going
to run some tests with 2.6.25-rc2 and Volanomark and see how that goes.


-- 
	Warm Regards,
	Balbir Singh
	Linux Technology Center
	IBM, ISTL

Re: 2.6.25-rc1: volanoMark 45% regression

[Zhang, Yanmin]

On Mon, 2008-02-18 at 10:26 +0530, Balbir Singh wrote:
> Zhang, Yanmin wrote:
> >>
> >>>  Did you have
> >>> CONFIG_FAIR_USER_SCHED enabled in both cases?
> >> Yes.
> >>
> >> CONFIG_FAIR_GROUP_SCHED=y
> >> CONFIG_FAIR_USER_SCHED=y
> >>
> 
> Interesting that you have CONFIG_FAIR_USER_SCHED. Do you have any other users on
> the system (are they running anything at the time of the benchmark)?
No. As for volanoMark, I start volanoMark both server and client processes while
starting some system measurement tools such like vmstat/iostat/sar. All the processes
run under the same user.


>  Could you
> turn off CONFIG_FAIR_USER_SCHED  and turn on CONFIG_FAIR_GROUP_SCHED.
I will try.

>  I am going
> to run some tests with 2.6.25-rc2 and Volanomark and see how that goes.
Pls. set /proc/sys/kernel/sched_compat_yield=1 before starting test.

-yanmin

Re: 2.6.25-rc1: volanoMark 45% regression

[Balbir Singh]

Zhang, Yanmin wrote:
> On Mon, 2008-02-18 at 10:26 +0530, Balbir Singh wrote:
>> Zhang, Yanmin wrote:
>>>>>  Did you have
>>>>> CONFIG_FAIR_USER_SCHED enabled in both cases?
>>>> Yes.
>>>>
>>>> CONFIG_FAIR_GROUP_SCHED=y
>>>> CONFIG_FAIR_USER_SCHED=y
>>>>
>> Interesting that you have CONFIG_FAIR_USER_SCHED. Do you have any other users on
>> the system (are they running anything at the time of the benchmark)?
> No. As for volanoMark, I start volanoMark both server and client processes while
> starting some system measurement tools such like vmstat/iostat/sar. All the processes
> run under the same user.
> 

Ok, so the bandwidth is not split between users, which is very good. I assume
the same user is root?

> 
>>  Could you
>> turn off CONFIG_FAIR_USER_SCHED  and turn on CONFIG_FAIR_GROUP_SCHED.
> I will try.
> 
>>  I am going
>> to run some tests with 2.6.25-rc2 and Volanomark and see how that goes.
> Pls. set /proc/sys/kernel/sched_compat_yield=1 before starting test.

Thanks for the tip, I would have definitely missed this one

-- 
	Warm Regards,
	Balbir Singh
	Linux Technology Center
	IBM, ISTL

Re: 2.6.25-rc1: volanoMark 45% regression

[Zhang, Yanmin]

On Mon, 2008-02-18 at 10:52 +0530, Balbir Singh wrote:
> Zhang, Yanmin wrote:
> > On Mon, 2008-02-18 at 10:26 +0530, Balbir Singh wrote:
> >> Zhang, Yanmin wrote:
> >>>>>  Did you have
> >>>>> CONFIG_FAIR_USER_SCHED enabled in both cases?
> >>>> Yes.
> >>>>
> >>>> CONFIG_FAIR_GROUP_SCHED=y
> >>>> CONFIG_FAIR_USER_SCHED=y
> >>>>
> >> Interesting that you have CONFIG_FAIR_USER_SCHED. Do you have any other users on
> >> the system (are they running anything at the time of the benchmark)?
> > No. As for volanoMark, I start volanoMark both server and client processes while
> > starting some system measurement tools such like vmstat/iostat/sar. All the processes
> > run under the same user.
> > 
> 
> Ok, so the bandwidth is not split between users, which is very good. I assume
> the same user is root?
No. I use an ordinary user.

> 
> > 
> >>  Could you
> >> turn off CONFIG_FAIR_USER_SCHED  and turn on CONFIG_FAIR_GROUP_SCHED.
> > I will try.
> > 
> >>  I am going
> >> to run some tests with 2.6.25-rc2 and Volanomark and see how that goes.
> > Pls. set /proc/sys/kernel/sched_compat_yield=1 before starting test.
> 
> Thanks for the tip, I would have definitely missed this one
I use Jrockit JVM and enables hugetlb, but I don't think hugetlb has too much help on
volanoMark, especially for the regression tracking.

-yanmin