LKML Archive on lore.kernel.org
help / color / mirror / Atom feed
From: Justin TerAvest <teravest@google.com>
To: jaxboe@fusionio.com, vgoyal@redhat.com
Cc: ctalbott@google.com, mrubin@google.com, jmoyer@redhat.com,
	guijanfeng@cn.fujitsu.com, linux-kernel@vger.kernel.org,
	Justin TerAvest <teravest@google.com>
Subject: [PATCH 0/1] Don't update group weights when on service tree.
Date: Thu, 10 Feb 2011 13:08:05 -0800	[thread overview]
Message-ID: <1297372086-16138-1-git-send-email-teravest@google.com> (raw)

With some instrumentation, we can observe that the total_weight for a 
service tree can be badly adjusted; particularly when the weight for a 
group is adjusted without taking it off of the tree. This can be 
reproduced on the HEAD of the linux-2.6-block tree.

We have seen this problem in workloads when total_weight becomes 0 and 
we divide by 0 in cfq_group_slice(), crashing the kernel, but it's
easier to illustrate by adding a BUG_ON and making it signed, like this:

--- a/block/cfq-iosched.c
+++ b/block/cfq-iosched.c
@@ -85,7 +85,7 @@ struct cfq_rb_root {
        struct rb_root rb;
        struct rb_node *left;
        unsigned count;
-       unsigned total_weight;
+       int total_weight;
        u64 min_vdisktime;
 };
 #define CFQ_RB_ROOT    (struct cfq_rb_root) { .rb = RB_ROOT, .left = NULL, \
@@ -903,6 +903,7 @@ cfq_group_service_tree_del(struct cfq_data *cfqd, struct cfq

        cfq_log_cfqg(cfqd, cfqg, "del_from_rr group");
        st->total_weight -= cfqg->weight;
+       BUG_ON(st->total_weight < 0);
        if (!RB_EMPTY_NODE(&cfqg->rb_node))
                cfq_rb_erase(&cfqg->rb_node, st);
        cfqg->saved_workload_slice = 0;

The "fuzzcon" tool adjusts the weight for a task in a manner that 
triggers this bug:
http://google3-2.osuosl.org/?p=tests/blkcgroup.git;a=blob;f=scripts/fuzzcon;h=38cfb9e0c847eb92ceccbac67de2a59b43f241ba;hb=eae5e71ad1116b46a9fd84f5d1b1e4c45aa6bd58


Here's the BUG_ON output when the case is encountered:
 [ 1711.376954] ------------[ cut here ]------------
 [ 1711.377789] kernel BUG at block/cfq-iosched.c:906!
 [ 1711.377789] invalid opcode: 0000 [#1] SMP·
 [ 1711.377789] last sysfs file: /sys/devices/system/node/node0/distance
 [ 1711.377789] CPU 0·
 [ 1711.377789] Modules linked in: tg3 msr cpuid ipv6 genrtc
 [ 1711.377789]·
 [ 1711.377789] Pid: 13702, comm: dd Tainted: G        W 2.6.38-smp-detect
 [ 1711.377789] RIP: 0010:[<ffffffff81210387>]  [<ffffffff81210387>] cfq_group_service_tree_del+0x5b/0x99
 [ 1711.377789] RSP: 0018:ffff8800189c1b68  EFLAGS: 00010086
 [ 1711.377789] RAX: 00000000ffffff9c RBX: ffff880011a6d400 RCX: 000000000000b690
 [ 1711.377789] RDX: 0000000000000000 RSI: ffff880011a6d400 RDI: 0000000000000000
 [ 1711.377789] RBP: ffff8800189c1b78 R08: 0000000000000000 R09: 0000000000000001
 [ 1711.377789] R10: 0000000000000000 R11: 000000000000001b R12: ffff8800065e8000
 [ 1711.377789] R13: ffff880011a6d528 R14: 000000000000001b R15: 000000000000001b
 [ 1711.377789] FS:  0000000000000000(0000) GS:ffff880009c00000(0000) knlGS:0000000000000000
 [ 1711.377789] CS:  0010 DS: 002b ES: 002b CR0: 000000008005003b
 [ 1711.377789] CR2: 00000000f74db000 CR3: 0000000001803000 CR4: 00000000000006f0
 [ 1711.377789] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
 [ 1711.377789] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
 [ 1711.377789] Process dd (pid: 13702, threadinfo ffff8800189c0000, task ffff8800067dafa0)
 [ 1711.377789] Stack:
 [ 1711.377789]  ffff880006596b40 ffff8800065e8000 ffff8800189c1be8 ffffffff81210d2d
 [ 1711.377789]  ffff8800067dafa0 ffff8800060c0418 ffff8800189c1bc8 ffff8800060c0410
 [ 1711.377789]  0000000000000082 ffff8800065e8008 ffff8800189c1bc8 ffff880006596b40
 [ 1711.377789] Call Trace:
 [ 1711.377789]  [<ffffffff81210d2d>] __cfq_slice_expired+0x3e0/0x45d
 [ 1711.377789]  [<ffffffff812111c3>] cfq_exit_cfqq+0x22/0x3f
 [ 1711.377789]  [<ffffffff81211257>] __cfq_exit_single_io_context+0x77/0x84
 [ 1711.377789]  [<ffffffff812112ab>] cfq_exit_single_io_context+0x47/0x62
 [ 1711.377789]  [<ffffffff81211264>] ?  cfq_exit_single_io_context+0x0/0x62
 [ 1711.377789]  [<ffffffff8120fcc1>] call_for_each_cic+0x31/0x3e
 [ 1711.377789]  [<ffffffff8120fce3>] cfq_exit_io_context+0x15/0x17
 [ 1711.377789]  [<ffffffff81207211>] exit_io_context+0x5a/0x6d
 [ 1711.377789]  [<ffffffff810723a2>] do_exit+0x72a/0x756
 [ 1711.377789]  [<ffffffff81072444>] do_group_exit+0x76/0x9e
 [ 1711.377789]  [<ffffffff8107fa9f>] get_signal_to_deliver+0x33c/0x35d
 [ 1711.377789]  [<ffffffff81035f45>] do_signal+0x72/0x699
 [ 1711.377789]  [<ffffffff8111ad8f>] ? fsnotify_modify+0x62/0x6a
 [ 1711.377789]  [<ffffffff81036598>] do_notify_resume+0x2c/0x72
 [ 1711.377789]  [<ffffffff81036d88>] int_signal+0x12/0x17
 [ 1711.377789] Code: 48 85 ff 74 15 48 8d 96 42 01 00 00 31 c0 48 c7 c6 0d 13 72 81 e8 d2 cd eb ff 41 8b 44 24 1c 2b 43 20 85 c0 41 89 44 24 1c 79 04 <0f> 0b eb fe 48 8b 03 48 83 e0 fc 48 39 c3 74 0d 49 8d 74 24 08·
 [ 1711.377789] RIP  [<ffffffff81210387>] cfq_group_service_tree_del+0x5b/0x99
 [ 1711.377789]  RSP <ffff8800189c1b68>
 [ 1711.377789] ---[ end trace 5e95326ab7063969 ]---


Justin TerAvest (1):
  Don't update group weights when on service tree.

 block/cfq-iosched.c |   57 ++++++++++++++++++++++++++++++++++++++------------
 1 files changed, 43 insertions(+), 14 deletions(-)

-- 
1.7.3.1


             reply	other threads:[~2011-02-10 21:08 UTC|newest]

Thread overview: 6+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2011-02-10 21:08 Justin TerAvest [this message]
2011-02-10 21:08 ` [PATCH 1/1] " Justin TerAvest
2011-02-11 18:58   ` Vivek Goyal
2011-02-11 19:56     ` Justin TerAvest
2011-02-11 18:54 ` [PATCH 0/1] " Vivek Goyal
2011-02-12  3:13 ` Gui Jianfeng

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1297372086-16138-1-git-send-email-teravest@google.com \
    --to=teravest@google.com \
    --cc=ctalbott@google.com \
    --cc=guijanfeng@cn.fujitsu.com \
    --cc=jaxboe@fusionio.com \
    --cc=jmoyer@redhat.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mrubin@google.com \
    --cc=vgoyal@redhat.com \
    --subject='Re: [PATCH 0/1] Don'\''t update group weights when on service tree.' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).