Linux-Fsdevel Archive on
help / color / mirror / Atom feed
From: Abdul Anshad Azeez <>
To: "" <>,
	"" <>,
	"" <>,
	"" <>
Cc: "" <>,
	"" <>
Subject: Performance regressions in networking & storage benchmarks in Linux kernel 5.8
Date: Tue, 22 Sep 2020 08:51:05 +0000	[thread overview]
Message-ID: <> (raw)

Part of VMware's performance regression testing for Linux Kernel upstream rele
ases we compared Linux kernel 5.8 against 5.7. Our evaluation revealed perform
ance regressions mostly in networking latency/response-time benchmarks up to 6
0%. Storage throughput & latency benchmarks were also up by 8%.

After performing the bisect between kernel 5.8 and 5.7, we identified the root
 cause behaviour to be an interrupt related change from Thomas Gleixner's "633
260fa143bbed05e65dc557a492667dfdc45bb(x86/irq: Convey vector as argument and n
ot in ptregs)" commit. To confirm this, we backed out the commit from 5.8 & re
ran our tests and found that the performance was similar to 5.7 kernel.

Impacted test cases:

    - Netperf TCP_RR & TCP_CRR - Response time
    - Ping - Response time
    - Memcache - Response time
    - Netperf TCP_STREAM small(8K socket & 256B message)(TCP_NODELAY set) pack
ets - Throughput & CPU utilization(CPU/Gbits)

    - FIO:
        - 4K (rand|seq)_(read|write) local-NVMe MultiVM tests - Throughput & l

From our testing, overall results indicate that above-mentioned commit has int
roduced performance regressions in latency-sensitive workloads for networking.
 For storage, it affected both throughput & latency workloads.

Also, since Linux 5.9-rc4 kernel was released recently, we repeated the same e
xperiments on 5.9-rc4. We observed all regressions were fixed and the performa
nce numbers between 5.7 and 5.9-rc4 were similar.

In order to find the fix commit, we bisected again between 5.8 and 5.9-rc4 and
 identified that regressions were fixed from a commit made by the same author 
Thomas Gleixner, which unbreaks the interrupt affinity settings - "e027fffff79
9cdd70400c5485b1a54f482255985(x86/irq: Unbreak interrupt affinity setting)".

We believe these findings would be useful to the Linux community and wanted to
 document the same.

Abdul Anshad Azeez
Performance Engineering
VMware, Inc.

             reply	other threads:[~2020-09-22  8:51 UTC|newest]

Thread overview: 3+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-09-22  8:51 Abdul Anshad Azeez [this message]
2020-09-22 11:25 ` Thomas Gleixner
2020-09-23  7:09   ` Abdul Anshad Azeez

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \ \ \ \ \ \ \ \ \
    --subject='Re: Performance regressions in networking & storage benchmarks in Linux kernel 5.8' \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).