Linux-Fsdevel Archive on
help / color / mirror / Atom feed
* Performance regressions in networking & storage benchmarks in Linux kernel 5.8
@ 2020-09-22  8:51 Abdul Anshad Azeez
  2020-09-22 11:25 ` Thomas Gleixner
  0 siblings, 1 reply; 3+ messages in thread
From: Abdul Anshad Azeez @ 2020-09-22  8:51 UTC (permalink / raw)
  To: linux-kernel, x86, netdev, linux-fsdevel; +Cc: rostedt, tglx

Part of VMware's performance regression testing for Linux Kernel upstream rele
ases we compared Linux kernel 5.8 against 5.7. Our evaluation revealed perform
ance regressions mostly in networking latency/response-time benchmarks up to 6
0%. Storage throughput & latency benchmarks were also up by 8%.

After performing the bisect between kernel 5.8 and 5.7, we identified the root
 cause behaviour to be an interrupt related change from Thomas Gleixner's "633
260fa143bbed05e65dc557a492667dfdc45bb(x86/irq: Convey vector as argument and n
ot in ptregs)" commit. To confirm this, we backed out the commit from 5.8 & re
ran our tests and found that the performance was similar to 5.7 kernel.

Impacted test cases:

    - Netperf TCP_RR & TCP_CRR - Response time
    - Ping - Response time
    - Memcache - Response time
    - Netperf TCP_STREAM small(8K socket & 256B message)(TCP_NODELAY set) pack
ets - Throughput & CPU utilization(CPU/Gbits)

    - FIO:
        - 4K (rand|seq)_(read|write) local-NVMe MultiVM tests - Throughput & l

From our testing, overall results indicate that above-mentioned commit has int
roduced performance regressions in latency-sensitive workloads for networking.
 For storage, it affected both throughput & latency workloads.

Also, since Linux 5.9-rc4 kernel was released recently, we repeated the same e
xperiments on 5.9-rc4. We observed all regressions were fixed and the performa
nce numbers between 5.7 and 5.9-rc4 were similar.

In order to find the fix commit, we bisected again between 5.8 and 5.9-rc4 and
 identified that regressions were fixed from a commit made by the same author 
Thomas Gleixner, which unbreaks the interrupt affinity settings - "e027fffff79
9cdd70400c5485b1a54f482255985(x86/irq: Unbreak interrupt affinity setting)".

We believe these findings would be useful to the Linux community and wanted to
 document the same.

Abdul Anshad Azeez
Performance Engineering
VMware, Inc.

^ permalink raw reply	[flat|nested] 3+ messages in thread

end of thread, other threads:[~2020-09-23  7:09 UTC | newest]

Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-09-22  8:51 Performance regressions in networking & storage benchmarks in Linux kernel 5.8 Abdul Anshad Azeez
2020-09-22 11:25 ` Thomas Gleixner
2020-09-23  7:09   ` Abdul Anshad Azeez

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).