LKML Archive on lore.kernel.org
help / color / mirror / Atom feed
From: Ingo Molnar <mingo@kernel.org>
To: changbin.du@intel.com
Cc: yamada.masahiro@socionext.com, michal.lkml@markovi.net,
	tglx@linutronix.de, mingo@redhat.com, akpm@linux-foundation.org,
	x86@kernel.org, lgirdwood@gmail.com, broonie@kernel.org,
	arnd@arndb.de, linux-kbuild@vger.kernel.org,
	linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org
Subject: Re: [PATCH 0/5] kernel hacking: GCC optimization for debug experience (-Og)
Date: Wed, 2 May 2018 09:33:15 +0200	[thread overview]
Message-ID: <20180502073315.sso3aaak45aeuyst@gmail.com> (raw)
In-Reply-To: <1525179614-14571-1-git-send-email-changbin.du@intel.com>


* changbin.du@intel.com <changbin.du@intel.com> wrote:

> Comparison of system performance: a bit drop.
> 
>     w/o CONFIG_DEBUG_EXPERIENCE
>     $ time make -j4
>     real    6m43.619s
>     user    19m5.160s
>     sys     2m20.287s
> 
>     w/ CONFIG_DEBUG_EXPERIENCE
>     $ time make -j4
>     real    6m55.054s
>     user    19m11.129s
>     sys     2m36.345s

Sorry, that's not a proper kbuild performance measurement - there's no noise 
estimation at all.

Below is a description that should produce more reliable numbers.

Thanks,

	Ingo


=========================>

So here's a pretty reliable way to measure kernel build time, which tries to avoid 
the various pitfalls of caching.

First I make sure that cpufreq is set to 'performance':

  for ((cpu=0; cpu<120; cpu++)); do
    G=/sys/devices/system/cpu/cpu$cpu/cpufreq/scaling_governor
    [ -f $G ] && echo performance > $G
  done

[ ... because it can be *really* annoying to discover that an ostensible 
  performance regression was a cpufreq artifact ... again. ;-) ]

Then I copy a kernel tree to /tmp (ramfs) as root:

	cd /tmp
	rm -rf linux
	git clone ~/linux linux
	cd linux
	make defconfig >/dev/null
	
... and then we can build the kernel in such a loop (as root again):

  perf stat --repeat 10 --null --pre			'\
	cp -a kernel ../kernel.copy.$(date +%s);	 \
	rm -rf *;					 \
	git checkout .;					 \
	echo 1 > /proc/sys/vm/drop_caches;		 \
	find ../kernel* -type f | xargs cat >/dev/null;  \
	make -j kernel >/dev/null;			 \
	make clean >/dev/null 2>&1;			 \
	sync						'\
							 \
	make -j16 >/dev/null

( I have tested these by pasting them into a terminal. Adjust the ~/linux source 
  git tree and the '-j16' to your system. )

Notes:

 - the 'pre' script portion is not timed by 'perf stat', only the raw build times

 - we flush all caches via drop_caches and re-establish everything again, but:

 - we also introduce an intentional memory leak by slowly filling up ramfs with 
   copies of 'kernel/', thus continously changing the layout of free memory, 
   cached data such as compiler binaries and the source code hierarchy. (Note 
   that the leak is about 8MB per iteration, so it isn't massive.)

With 10 iterations this is the statistical stability I get this on a big box:

 Performance counter stats for 'make -j128 kernel' (10 runs):

      26.346436425 seconds time elapsed    (+- 0.19%)

... which, despite a high iteration count of 10, is still surprisingly noisy, 
right?

A 0.2% stddev is probably not enough to call a 0.7% regression with good 
confidence, so I had to use *30* iterations to make measurement noise to be about 
an order of magnitude lower than the effect I'm trying to measure:

 Performance counter stats for 'make -j128' (30 runs):

      26.334767571 seconds time elapsed    (+- 0.09% )

i.e. "26.334 +- 0.023" seconds is a number we can have pretty high confidence in, 
on this system.

And just to demonstrate that it's all real, I repeated the whole 30-iteration 
measurement again:

 Performance counter stats for 'make -j128' (30 runs):

      26.311166142 seconds time elapsed    (+- 0.07%)

  parent reply	other threads:[~2018-05-02  7:33 UTC|newest]

Thread overview: 17+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-05-01 13:00 changbin.du
2018-05-01 13:00 ` [PATCH 1/5] x86/mm: surround level4_kernel_pgt with #ifdef CONFIG_X86_5LEVEL...#endif changbin.du
2018-05-01 13:00 ` [PATCH 2/5] regulator: add dummy of_find_regulator_by_node changbin.du
2018-05-01 20:40   ` Mark Brown
2018-05-02  9:24     ` Du, Changbin
2018-05-05  1:44   ` Applied "regulator: add dummy function of_find_regulator_by_node" to the regulator tree Mark Brown
2018-05-01 13:00 ` [PATCH 3/5] kernel hacking: new config NO_AUTO_INLINE to disable compiler atuo-inline optimizations changbin.du
2018-05-01 14:54   ` Steven Rostedt
2018-05-02  9:27     ` Du, Changbin
2018-05-01 13:00 ` [PATCH 4/5] kernel hacking: new config DEBUG_EXPERIENCE to apply GCC -Og optimization changbin.du
2018-05-01 15:25   ` Randy Dunlap
2018-05-02  9:24     ` Du, Changbin
2018-05-02 20:30   ` Arnd Bergmann
2018-05-01 13:00 ` [PATCH 5/5] asm-generic: fix build error in fix_to_virt with CONFIG_DEBUG_EXPERIENCE changbin.du
2018-05-02  7:33 ` Ingo Molnar [this message]
2018-05-02  9:07   ` [PATCH 0/5] kernel hacking: GCC optimization for debug experience (-Og) Du, Changbin
2018-05-02 11:09     ` Ingo Molnar

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20180502073315.sso3aaak45aeuyst@gmail.com \
    --to=mingo@kernel.org \
    --cc=akpm@linux-foundation.org \
    --cc=arnd@arndb.de \
    --cc=broonie@kernel.org \
    --cc=changbin.du@intel.com \
    --cc=lgirdwood@gmail.com \
    --cc=linux-arch@vger.kernel.org \
    --cc=linux-kbuild@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=michal.lkml@markovi.net \
    --cc=mingo@redhat.com \
    --cc=tglx@linutronix.de \
    --cc=x86@kernel.org \
    --cc=yamada.masahiro@socionext.com \
    --subject='Re: [PATCH 0/5] kernel hacking: GCC optimization for debug experience (-Og)' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).