LKML Archive on lore.kernel.org
help / color / mirror / Atom feed
From: "Srivatsa S. Bhat" <srivatsa@csail.mit.edu>
To: Jan Kara <jack@suse.cz>
Cc: Paolo Valente <paolo.valente@linaro.org>,
	linux-fsdevel@vger.kernel.org,
	linux-block <linux-block@vger.kernel.org>,
	linux-ext4@vger.kernel.org, cgroups@vger.kernel.org,
	kernel list <linux-kernel@vger.kernel.org>,
	Jens Axboe <axboe@kernel.dk>, Jeff Moyer <jmoyer@redhat.com>,
	Theodore Ts'o <tytso@mit.edu>,
	amakhalov@vmware.com, anishs@vmware.com, srivatsab@vmware.com,
	Ulf Hansson <ulf.hansson@linaro.org>,
	Linus Walleij <linus.walleij@linaro.org>,
	Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
	Stable <stable@vger.kernel.org>
Subject: Re: CFQ idling kills I/O performance on ext4 with blkio cgroup controller
Date: Wed, 12 Jun 2019 12:36:53 -0700	[thread overview]
Message-ID: <dd32ed59-a543-fc76-9a9a-2462f0119270@csail.mit.edu> (raw)
In-Reply-To: <20190612130446.GD14578@quack2.suse.cz>


[ Adding Greg to CC ]

On 6/12/19 6:04 AM, Jan Kara wrote:
> On Tue 11-06-19 15:34:48, Srivatsa S. Bhat wrote:
>> On 6/2/19 12:04 AM, Srivatsa S. Bhat wrote:
>>> On 5/30/19 3:45 AM, Paolo Valente wrote:
>>>>
>> [...]
>>>> At any rate, since you pointed out that you are interested in
>>>> out-of-the-box performance, let me complete the context: in case
>>>> low_latency is left set, one gets, in return for this 12% loss,
>>>> a) at least 1000% higher responsiveness, e.g., 1000% lower start-up
>>>> times of applications under load [1];
>>>> b) 500-1000% higher throughput in multi-client server workloads, as I
>>>> already pointed out [2].
>>>>
>>>
>>> I'm very happy that you could solve the problem without having to
>>> compromise on any of the performance characteristics/features of BFQ!
>>>
>>>
>>>> I'm going to prepare complete patches.  In addition, if ok for you,
>>>> I'll report these results on the bug you created.  Then I guess we can
>>>> close it.
>>>>
>>>
>>> Sounds great!
>>>
>>
>> Hi Paolo,
>>
>> Hope you are doing great!
>>
>> I was wondering if you got a chance to post these patches to LKML for
>> review and inclusion... (No hurry, of course!)
>>
>> Also, since your fixes address the performance issues in BFQ, do you
>> have any thoughts on whether they can be adapted to CFQ as well, to
>> benefit the older stable kernels that still support CFQ?
> 
> Since CFQ doesn't exist in current upstream kernel anymore, I seriously
> doubt you'll be able to get any performance improvements for it in the
> stable kernels...
> 

I suspected as much, but that seems unfortunate though. The latest LTS
kernel is based on 4.19, which still supports CFQ. It would have been
great to have a process to address significant issues on older
kernels too.

Greg, do you have any thoughts on this? The context is that both CFQ
and BFQ I/O schedulers have issues that cause I/O throughput to suffer
upto 10x - 30x on certain workloads and system configurations, as
reported in [1].

In this thread, Paolo posted patches to fix BFQ performance on
mainline. However CFQ suffers from the same performance collapse, but
CFQ was removed from the kernel in v5.0. So obviously the usual stable
backporting path won't work here for several reasons:

  1. There won't be a mainline commit to backport from, as CFQ no
     longer exists in mainline.

  2. This is not a security/stability fix, and is likely to involve
     invasive changes.

I was wondering if there was a way to address the performance issues
in CFQ in the older stable kernels (including the latest LTS 4.19),
despite the above constraints, since the performance drop is much too
significant. I guess not, but thought I'd ask :-)

[1]. https://lore.kernel.org/lkml/8d72fcf7-bbb4-2965-1a06-e9fc177a8938@csail.mit.edu/ 


Regards,
Srivatsa
VMware Photon OS

  reply	other threads:[~2019-06-12 19:37 UTC|newest]

Thread overview: 52+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-05-17 22:16 Srivatsa S. Bhat
2019-05-18 18:39 ` Paolo Valente
2019-05-18 19:28   ` Theodore Ts'o
2019-05-20  9:15     ` Jan Kara
2019-05-20 10:45       ` Paolo Valente
2019-05-21 16:48       ` Theodore Ts'o
2019-05-21 18:19         ` Josef Bacik
2019-05-21 19:10           ` Theodore Ts'o
2019-05-20 10:38     ` Paolo Valente
2019-05-21  7:38       ` Andrea Righi
2019-05-18 20:50   ` Srivatsa S. Bhat
2019-05-20 10:19     ` Paolo Valente
2019-05-20 22:45       ` Srivatsa S. Bhat
2019-05-21  6:23         ` Paolo Valente
2019-05-21  7:19           ` Srivatsa S. Bhat
2019-05-21  9:10           ` Jan Kara
2019-05-21 16:31             ` Theodore Ts'o
2019-05-21 11:25       ` Paolo Valente
2019-05-21 13:20         ` Paolo Valente
2019-05-21 16:21           ` Paolo Valente
2019-05-21 17:38             ` Paolo Valente
2019-05-21 22:51               ` Srivatsa S. Bhat
2019-05-22  8:05                 ` Paolo Valente
2019-05-22  9:02                   ` Srivatsa S. Bhat
2019-05-22  9:12                     ` Paolo Valente
2019-05-22 10:02                       ` Srivatsa S. Bhat
2019-05-22  9:09                   ` Paolo Valente
2019-05-22 10:01                     ` Srivatsa S. Bhat
2019-05-22 10:54                       ` Paolo Valente
2019-05-23  2:30                         ` Srivatsa S. Bhat
2019-05-23  9:19                           ` Paolo Valente
2019-05-23 17:22                             ` Paolo Valente
2019-05-23 23:43                               ` Srivatsa S. Bhat
2019-05-24  6:51                                 ` Paolo Valente
2019-05-24  7:56                                   ` Paolo Valente
2019-05-29  1:09                                   ` Srivatsa S. Bhat
2019-05-29  7:41                                     ` Paolo Valente
2019-05-30  8:29                                       ` Srivatsa S. Bhat
2019-05-30 10:45                                         ` Paolo Valente
2019-06-02  7:04                                           ` Srivatsa S. Bhat
2019-06-11 22:34                                             ` Srivatsa S. Bhat
2019-06-12 13:04                                               ` Jan Kara
2019-06-12 19:36                                                 ` Srivatsa S. Bhat [this message]
2019-06-13  6:02                                                   ` Greg Kroah-Hartman
2019-06-13 19:03                                                     ` Srivatsa S. Bhat
2019-06-13  8:20                                                   ` Jan Kara
2019-06-13 19:05                                                     ` Srivatsa S. Bhat
2019-06-13  8:37                                                   ` Jens Axboe
2019-06-13  5:46                                               ` Paolo Valente
2019-06-13 19:13                                                 ` Srivatsa S. Bhat
2019-05-23 23:32                           ` Srivatsa S. Bhat
2019-05-30  8:38                             ` Srivatsa S. Bhat

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=dd32ed59-a543-fc76-9a9a-2462f0119270@csail.mit.edu \
    --to=srivatsa@csail.mit.edu \
    --cc=amakhalov@vmware.com \
    --cc=anishs@vmware.com \
    --cc=axboe@kernel.dk \
    --cc=cgroups@vger.kernel.org \
    --cc=gregkh@linuxfoundation.org \
    --cc=jack@suse.cz \
    --cc=jmoyer@redhat.com \
    --cc=linus.walleij@linaro.org \
    --cc=linux-block@vger.kernel.org \
    --cc=linux-ext4@vger.kernel.org \
    --cc=linux-fsdevel@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=paolo.valente@linaro.org \
    --cc=srivatsab@vmware.com \
    --cc=stable@vger.kernel.org \
    --cc=tytso@mit.edu \
    --cc=ulf.hansson@linaro.org \
    --subject='Re: CFQ idling kills I/O performance on ext4 with blkio cgroup controller' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).