LKML Archive on lore.kernel.org
help / color / mirror / Atom feed
From: Paolo Valente <paolo.valente@linaro.org>
To: "Srivatsa S. Bhat" <srivatsa@csail.mit.edu>
Cc: linux-fsdevel@vger.kernel.org,
	linux-block <linux-block@vger.kernel.org>,
	linux-ext4@vger.kernel.org, cgroups@vger.kernel.org,
	kernel list <linux-kernel@vger.kernel.org>,
	Jens Axboe <axboe@kernel.dk>, Jan Kara <jack@suse.cz>,
	jmoyer@redhat.com, Theodore Ts'o <tytso@mit.edu>,
	amakhalov@vmware.com, anishs@vmware.com, srivatsab@vmware.com
Subject: Re: CFQ idling kills I/O performance on ext4 with blkio cgroup controller
Date: Fri, 24 May 2019 09:56:17 +0200	[thread overview]
Message-ID: <CDB0583F-8852-4BF8-98F9-EAA93838ADD5@linaro.org> (raw)
In-Reply-To: <E270AD92-943E-4529-8158-AB480D6D9DF8@linaro.org>


[-- Attachment #1.1: Type: text/plain, Size: 2262 bytes --]



> Il giorno 24 mag 2019, alle ore 08:51, Paolo Valente <paolo.valente@linaro.org> ha scritto:
> 
> 
> 
>> Il giorno 24 mag 2019, alle ore 01:43, Srivatsa S. Bhat <srivatsa@csail.mit.edu> ha scritto:
>> 
>> On 5/23/19 10:22 AM, Paolo Valente wrote:
>>> 
>>>> Il giorno 23 mag 2019, alle ore 11:19, Paolo Valente <paolo.valente@linaro.org> ha scritto:
>>>> 
>>>>> Il giorno 23 mag 2019, alle ore 04:30, Srivatsa S. Bhat <srivatsa@csail.mit.edu> ha scritto:
>>>>> 
>> [...]
>>>>> Also, I'm very happy to run additional tests or experiments to help
>>>>> track down this issue. So, please don't hesitate to let me know if
>>>>> you'd like me to try anything else or get you additional traces etc. :)
>>>>> 
>>>> 
>>>> Here's to you!  :) I've attached a new small improvement that may
>>>> reduce fluctuations (path to apply on top of the others, of course).
>>>> Unfortunately, I don't expect this change to boost the throughput
>>>> though.
>>>> 
>>>> In contrast, I've thought of a solution that might be rather
>>>> effective: making BFQ aware (heuristically) of trivial
>>>> synchronizations between processes in different groups. This will
>>>> require a little more work and time.
>>>> 
>>> 
>>> Hi Srivatsa,
>>> I'm back :)
>>> 
>>> First, there was a mistake in the last patch I sent you, namely in
>>> 0001-block-bfq-re-sample-req-service-times-when-possible.patch.
>>> Please don't apply that patch at all.
>>> 
>>> I've attached a new series of patches instead.  The first patch in this
>>> series is a fixed version of the faulty patch above (if I'm creating too
>>> much confusion, I'll send you again all patches to apply on top of
>>> mainline).
>>> 
>> 
>> No problem, I got it :)
>> 
>>> This series also implements the more effective idea I told you a few
>>> hours ago.  In my system, the loss is now around only 10%, even with
>>> low_latency on.
>>> 
>> 
>> When trying to run multiple dd tasks simultaneously, I get the kernel
>> panic shown below (mainline is fine, without these patches).
>> 
> 
> Could you please provide me somehow with a list *(bfq_serv_to_charge+0x21) ?
> 

Maybe I've found the cause. Please apply also the two patches attached and retry.

Thanks,
Paolo


[-- Attachment #1.2: fix-patches-for-waker-detection.tgz --]
[-- Type: application/octet-stream, Size: 1228 bytes --]

[-- Attachment #1.3: Type: text/plain, Size: 4543 bytes --]


> Thanks,
> Paolo
> 
>> [  568.232231] BUG: kernel NULL pointer dereference, address: 0000000000000024
>> [  568.232257] #PF: supervisor read access in kernel mode
>> [  568.232273] #PF: error_code(0x0000) - not-present page
>> [  568.232289] PGD 0 P4D 0
>> [  568.232299] Oops: 0000 [#1] SMP PTI
>> [  568.232312] CPU: 0 PID: 1029 Comm: dd Tainted: G            E     5.1.0-io-dbg-4+ #6
>> [  568.232334] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
>> [  568.232388] RIP: 0010:bfq_serv_to_charge+0x21/0x50
>> [  568.232404] Code: ff e8 c3 5e bc ff 0f 1f 00 0f 1f 44 00 00 48 8b 86 20 01 00 00 55 48 89 e5 53 48 89 fb a8 40 75 09 83 be a0 01 00 00 01 76 09 <8b> 43 24 c1 e8 09 5b 5d c3 48 8b 7e 08 e8 5d fd ff ff 84 c0 75 ea
>> [  568.232473] RSP: 0018:ffffa73a42dab750 EFLAGS: 00010002
>> [  568.232489] RAX: 0000000000001052 RBX: 0000000000000000 RCX: ffffa73a42dab7a0
>> [  568.232510] RDX: ffffa73a42dab657 RSI: ffff8b7b6ba2ab70 RDI: 0000000000000000
>> [  568.232530] RBP: ffffa73a42dab758 R08: 0000000000000000 R09: 0000000000000001
>> [  568.232551] R10: 0000000000000000 R11: ffffa73a42dab7a0 R12: ffff8b7b6aed3800
>> [  568.232571] R13: 0000000000000000 R14: 0000000000000000 R15: ffff8b7b6aed3800
>> [  568.232592] FS:  00007fb5b0724540(0000) GS:ffff8b7b6f800000(0000) knlGS:0000000000000000
>> [  568.232615] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [  568.232632] CR2: 0000000000000024 CR3: 00000004266be002 CR4: 00000000001606f0
>> [  568.232690] Call Trace:
>> [  568.232703]  bfq_select_queue+0x781/0x1000
>> [  568.232717]  bfq_dispatch_request+0x1d7/0xd60
>> [  568.232731]  ? bfq_bfqq_handle_idle_busy_switch.isra.36+0x2cd/0xb20
>> [  568.232751]  blk_mq_do_dispatch_sched+0xa8/0xe0
>> [  568.232765]  blk_mq_sched_dispatch_requests+0xe3/0x150
>> [  568.232783]  __blk_mq_run_hw_queue+0x56/0x100
>> [  568.232798]  __blk_mq_delay_run_hw_queue+0x107/0x160
>> [  568.232814]  blk_mq_run_hw_queue+0x75/0x190
>> [  568.232828]  blk_mq_sched_insert_requests+0x7a/0x100
>> [  568.232844]  blk_mq_flush_plug_list+0x1d7/0x280
>> [  568.232859]  blk_flush_plug_list+0xc2/0xe0
>> [  568.232872]  blk_finish_plug+0x2c/0x40
>> [  568.232886]  ext4_writepages+0x592/0xe60
>> [  568.233381]  ? ext4_mark_iloc_dirty+0x52b/0x860
>> [  568.233851]  do_writepages+0x3c/0xd0
>> [  568.234304]  ? ext4_mark_inode_dirty+0x1a0/0x1a0
>> [  568.234748]  ? do_writepages+0x3c/0xd0
>> [  568.235197]  ? __generic_write_end+0x4e/0x80
>> [  568.235644]  __filemap_fdatawrite_range+0xa5/0xe0
>> [  568.236089]  ? __filemap_fdatawrite_range+0xa5/0xe0
>> [  568.236533]  ? ext4_da_write_end+0x13c/0x280
>> [  568.236983]  file_write_and_wait_range+0x5a/0xb0
>> [  568.237407]  ext4_sync_file+0x11e/0x3e0
>> [  568.237819]  vfs_fsync_range+0x48/0x80
>> [  568.238217]  ext4_file_write_iter+0x234/0x3d0
>> [  568.238610]  ? _cond_resched+0x19/0x40
>> [  568.238982]  new_sync_write+0x112/0x190
>> [  568.239347]  __vfs_write+0x29/0x40
>> [  568.239705]  vfs_write+0xb1/0x1a0
>> [  568.240078]  ksys_write+0x89/0xc0
>> [  568.240428]  __x64_sys_write+0x1a/0x20
>> [  568.240771]  do_syscall_64+0x5b/0x140
>> [  568.241115]  entry_SYSCALL_64_after_hwframe+0x49/0xbe
>> [  568.241456] RIP: 0033:0x7fb5b02325f4
>> [  568.241787] Code: 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b3 0f 1f 80 00 00 00 00 48 8d 05 09 11 2d 00 8b 00 85 c0 75 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 f3 c3 66 90 41 54 55 49 89 d4 53 48 89 f5
>> [  568.242842] RSP: 002b:00007ffcb12e2968 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
>> [  568.243220] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fb5b02325f4
>> [  568.243616] RDX: 0000000000000200 RSI: 000055698f2ad000 RDI: 0000000000000001
>> [  568.244026] RBP: 0000000000000200 R08: 0000000000000004 R09: 0000000000000003
>> [  568.244401] R10: 00007fb5b04feca0 R11: 0000000000000246 R12: 000055698f2ad000
>> [  568.244775] R13: 0000000000000000 R14: 0000000000000000 R15: 000055698f2ad000
>> [  568.245154] Modules linked in: xt_MASQUERADE(E) nf_conntrack_netlink(E) nfnetlink(E) xfrm_user(E) xfrm_algo(E) xt_addrtype(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) vmw_vsock_vmci_transport(E) vsock(E) ip6table_filter(E) ip6_tables(E) xt_conntrack(E) iptable_mangle(E) iptable_nat(E) nf_nat(E) iptable_filter
>> [  568.248651] CR2: 0000000000000024
>> [  568.249142] ---[ end trace 0ddd315e0a5bdfba ]---
>> 
>> 
>> Regards,
>> Srivatsa
>> VMware Photon OS


[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

  reply	other threads:[~2019-05-24  7:56 UTC|newest]

Thread overview: 52+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-05-17 22:16 Srivatsa S. Bhat
2019-05-18 18:39 ` Paolo Valente
2019-05-18 19:28   ` Theodore Ts'o
2019-05-20  9:15     ` Jan Kara
2019-05-20 10:45       ` Paolo Valente
2019-05-21 16:48       ` Theodore Ts'o
2019-05-21 18:19         ` Josef Bacik
2019-05-21 19:10           ` Theodore Ts'o
2019-05-20 10:38     ` Paolo Valente
2019-05-21  7:38       ` Andrea Righi
2019-05-18 20:50   ` Srivatsa S. Bhat
2019-05-20 10:19     ` Paolo Valente
2019-05-20 22:45       ` Srivatsa S. Bhat
2019-05-21  6:23         ` Paolo Valente
2019-05-21  7:19           ` Srivatsa S. Bhat
2019-05-21  9:10           ` Jan Kara
2019-05-21 16:31             ` Theodore Ts'o
2019-05-21 11:25       ` Paolo Valente
2019-05-21 13:20         ` Paolo Valente
2019-05-21 16:21           ` Paolo Valente
2019-05-21 17:38             ` Paolo Valente
2019-05-21 22:51               ` Srivatsa S. Bhat
2019-05-22  8:05                 ` Paolo Valente
2019-05-22  9:02                   ` Srivatsa S. Bhat
2019-05-22  9:12                     ` Paolo Valente
2019-05-22 10:02                       ` Srivatsa S. Bhat
2019-05-22  9:09                   ` Paolo Valente
2019-05-22 10:01                     ` Srivatsa S. Bhat
2019-05-22 10:54                       ` Paolo Valente
2019-05-23  2:30                         ` Srivatsa S. Bhat
2019-05-23  9:19                           ` Paolo Valente
2019-05-23 17:22                             ` Paolo Valente
2019-05-23 23:43                               ` Srivatsa S. Bhat
2019-05-24  6:51                                 ` Paolo Valente
2019-05-24  7:56                                   ` Paolo Valente [this message]
2019-05-29  1:09                                   ` Srivatsa S. Bhat
2019-05-29  7:41                                     ` Paolo Valente
2019-05-30  8:29                                       ` Srivatsa S. Bhat
2019-05-30 10:45                                         ` Paolo Valente
2019-06-02  7:04                                           ` Srivatsa S. Bhat
2019-06-11 22:34                                             ` Srivatsa S. Bhat
2019-06-12 13:04                                               ` Jan Kara
2019-06-12 19:36                                                 ` Srivatsa S. Bhat
2019-06-13  6:02                                                   ` Greg Kroah-Hartman
2019-06-13 19:03                                                     ` Srivatsa S. Bhat
2019-06-13  8:20                                                   ` Jan Kara
2019-06-13 19:05                                                     ` Srivatsa S. Bhat
2019-06-13  8:37                                                   ` Jens Axboe
2019-06-13  5:46                                               ` Paolo Valente
2019-06-13 19:13                                                 ` Srivatsa S. Bhat
2019-05-23 23:32                           ` Srivatsa S. Bhat
2019-05-30  8:38                             ` Srivatsa S. Bhat

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=CDB0583F-8852-4BF8-98F9-EAA93838ADD5@linaro.org \
    --to=paolo.valente@linaro.org \
    --cc=amakhalov@vmware.com \
    --cc=anishs@vmware.com \
    --cc=axboe@kernel.dk \
    --cc=cgroups@vger.kernel.org \
    --cc=jack@suse.cz \
    --cc=jmoyer@redhat.com \
    --cc=linux-block@vger.kernel.org \
    --cc=linux-ext4@vger.kernel.org \
    --cc=linux-fsdevel@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=srivatsa@csail.mit.edu \
    --cc=srivatsab@vmware.com \
    --cc=tytso@mit.edu \
    --subject='Re: CFQ idling kills I/O performance on ext4 with blkio cgroup controller' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).