LKML Archive on lore.kernel.org
help / color / mirror / Atom feed
From: Trond Myklebust <trondmy@hammerspace.com>
To: "rong.a.chen@intel.com" <rong.a.chen@intel.com>,
	"zhengjun.xing@linux.intel.com" <zhengjun.xing@linux.intel.com>
Cc: "torvalds@linux-foundation.org" <torvalds@linux-foundation.org>,
	"lkp@01.org" <lkp@01.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Subject: Re: [LKP] [SUNRPC] 0472e47660: fsmark.app_overhead 16.0% regression
Date: Thu, 30 May 2019 19:10:52 +0000	[thread overview]
Message-ID: <9753a9a4a82943f6aacc2bfb0f93efc5f96bcaa5.camel@hammerspace.com> (raw)
In-Reply-To: <d796ac23-d5d6-cdfa-89c8-536e9496b551@linux.intel.com>

On Thu, 2019-05-30 at 15:20 +0800, Xing Zhengjun wrote:
> 
> On 5/30/2019 10:00 AM, Trond Myklebust wrote:
> > Hi Xing,
> > 
> > On Thu, 2019-05-30 at 09:35 +0800, Xing Zhengjun wrote:
> > > Hi Trond,
> > > 
> > > On 5/20/2019 1:54 PM, kernel test robot wrote:
> > > > Greeting,
> > > > 
> > > > FYI, we noticed a 16.0% improvement of fsmark.app_overhead due
> > > > to
> > > > commit:
> > > > 
> > > > 
> > > > commit: 0472e476604998c127f3c80d291113e77c5676ac ("SUNRPC:
> > > > Convert
> > > > socket page send code to use iov_iter()")
> > > > https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git
> > > > master
> > > > 
> > > > in testcase: fsmark
> > > > on test machine: 40 threads Intel(R) Xeon(R) CPU E5-2690 v2 @
> > > > 3.00GHz with 384G memory
> > > > with following parameters:
> > > > 
> > > > 	iterations: 1x
> > > > 	nr_threads: 64t
> > > > 	disk: 1BRD_48G
> > > > 	fs: xfs
> > > > 	fs2: nfsv4
> > > > 	filesize: 4M
> > > > 	test_size: 40G
> > > > 	sync_method: fsyncBeforeClose
> > > > 	cpufreq_governor: performance
> > > > 
> > > > test-description: The fsmark is a file system benchmark to test
> > > > synchronous write workloads, for example, mail servers
> > > > workload.
> > > > test-url: https://sourceforge.net/projects/fsmark/
> > > > 
> > > > 
> > > > 
> > > > Details are as below:
> > > > -------------------------------------------------------------
> > > > ----
> > > > --------------------------------->
> > > > 
> > > > 
> > > > To reproduce:
> > > > 
> > > >           git clone https://github.com/intel/lkp-tests.git
> > > >           cd lkp-tests
> > > >           bin/lkp install job.yaml  # job file is attached in
> > > > this
> > > > email
> > > >           bin/lkp run     job.yaml
> > > > 
> > > > ===============================================================
> > > > ====
> > > > ======================
> > > > compiler/cpufreq_governor/disk/filesize/fs2/fs/iterations/kconf
> > > > ig/n
> > > > r_threads/rootfs/sync_method/tbox_group/test_size/testcase:
> > > >     gcc-7/performance/1BRD_48G/4M/nfsv4/xfs/1x/x86_64-rhel-
> > > > 7.6/64t/debian-x86_64-2018-04-03.cgz/fsyncBeforeClose/lkp-ivb-
> > > > ep01/40G/fsmark
> > > > 
> > > > commit:
> > > >     e791f8e938 ("SUNRPC: Convert xs_send_kvec() to use
> > > > iov_iter_kvec()")
> > > >     0472e47660 ("SUNRPC: Convert socket page send code to use
> > > > iov_iter()")
> > > > 
> > > > e791f8e9380d945e 0472e476604998c127f3c80d291
> > > > ---------------- ---------------------------
> > > >          fail:runs  %reproduction    fail:runs
> > > >              |             |             |
> > > >              :4           50%           2:4     dmesg.WARNING:a
> > > > t#for
> > > > _ip_interrupt_entry/0x
> > > >            %stddev     %change         %stddev
> > > >                \          |                \
> > > >     15118573
> > > > ±  2%     +16.0%   17538083        fsmark.app_overhead
> > > >       510.93           -
> > > > 22.7%     395.12        fsmark.files_per_sec
> > > >        24.90           +22.8%      30.57        fsmark.time.ela
> > > > psed_
> > > > time
> > > >        24.90           +22.8%      30.57        fsmark.time.ela
> > > > psed_
> > > > time.max
> > > >       288.00 ±  2%     -
> > > > 27.8%     208.00        fsmark.time.percent_of_cpu_this_job_got
> > > >        70.03 ±  2%     -
> > > > 11.3%      62.14        fsmark.time.system_time
> > > > 
> > > 
> > > Do you have time to take a look at this regression?
> > 
> >  From your stats, it looks to me as if the problem is increased
> > NUMA
> > overhead. Pretty much everything else appears to be the same or
> > actually performing better than previously. Am I interpreting that
> > correctly?
> The real regression is the throughput(fsmark.files_per_sec) is
> decreased 
> by 22.7%.

Understood, but I'm trying to make sense of why. I'm not able to
reproduce this, so I have to rely on your performance stats to
understand where the 22.7% regression is coming from. As far as I can
see, the only numbers in the stats you published that are showing a
performance regression (other than the fsmark number itself), are the
NUMA numbers. Is that a correct interpretation?

> > If my interpretation above is correct, then I'm not seeing where
> > this
> > patch would be introducing new NUMA regressions. It is just
> > converting
> > from using one method of doing socket I/O to another. Could it
> > perhaps
> > be a memory artefact due to your running the NFS client and server
> > on
> > the same machine?
> > 
> > Apologies for pushing back a little, but I just don't have the
> > hardware available to test NUMA configurations, so I'm relying on
> > external testing for the above kind of scenario.
> > 
> Thanks for looking at this.  If you need more information, please let
> me
> know.
> > Thanks
> >    Trond
> > 
-- 
Trond Myklebust
CTO, Hammerspace Inc
4300 El Camino Real, Suite 105
Los Altos, CA 94022
www.hammer.space



  reply	other threads:[~2019-05-30 19:10 UTC|newest]

Thread overview: 13+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-05-20  5:54 kernel test robot
2019-05-30  1:35 ` [LKP] " Xing Zhengjun
2019-05-30  2:00   ` Trond Myklebust
2019-05-30  7:20     ` Xing Zhengjun
2019-05-30 19:10       ` Trond Myklebust [this message]
2019-05-31  3:27         ` Xing Zhengjun
2019-07-08  8:32           ` Xing Zhengjun
     [not found]             ` <DM5PR13MB1851813BBEA446E25C5001C2B8F60@DM5PR13MB1851.namprd13.prod.outlook.com>
2019-07-09  2:39               ` Xing Zhengjun
2019-07-12  6:42                 ` Xing Zhengjun
2019-07-24  5:17                   ` Xing Zhengjun
2019-08-07  7:56                     ` Xing Zhengjun
2019-08-30  0:43                       ` Xing Zhengjun
2019-09-25  9:00                         ` Xing Zhengjun

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=9753a9a4a82943f6aacc2bfb0f93efc5f96bcaa5.camel@hammerspace.com \
    --to=trondmy@hammerspace.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=lkp@01.org \
    --cc=rong.a.chen@intel.com \
    --cc=torvalds@linux-foundation.org \
    --cc=zhengjun.xing@linux.intel.com \
    --subject='Re: [LKP] [SUNRPC] 0472e47660: fsmark.app_overhead 16.0% regression' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).