From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1757100Ab1BJVIi (ORCPT ); Thu, 10 Feb 2011 16:08:38 -0500 Received: from smtp-out.google.com ([74.125.121.67]:28175 "EHLO smtp-out.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1757084Ab1BJVIg (ORCPT ); Thu, 10 Feb 2011 16:08:36 -0500 DomainKey-Signature: a=rsa-sha1; s=beta; d=google.com; c=nofws; q=dns; h=from:to:cc:subject:date:message-id:x-mailer:mime-version: content-type:content-transfer-encoding; b=ZmsuRgdTNuvQOwQup/c9fpxbHFJIMxWfscfYHeWgii5T8LTZuTNN8GKViedWOn16e mFwHJmZbUYVz2DrrcBKiQ== From: Justin TerAvest To: jaxboe@fusionio.com, vgoyal@redhat.com Cc: ctalbott@google.com, mrubin@google.com, jmoyer@redhat.com, guijanfeng@cn.fujitsu.com, linux-kernel@vger.kernel.org, Justin TerAvest Subject: =?UTF-8?q?=5BPATCH=200/1=5D=20Don=27t=20update=20group=20weights=20when=20on=20service=20tree=2E?= Date: Thu, 10 Feb 2011 13:08:05 -0800 Message-Id: <1297372086-16138-1-git-send-email-teravest@google.com> X-Mailer: git-send-email 1.7.3.1 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org With some instrumentation, we can observe that the total_weight for a service tree can be badly adjusted; particularly when the weight for a group is adjusted without taking it off of the tree. This can be reproduced on the HEAD of the linux-2.6-block tree. We have seen this problem in workloads when total_weight becomes 0 and we divide by 0 in cfq_group_slice(), crashing the kernel, but it's easier to illustrate by adding a BUG_ON and making it signed, like this: --- a/block/cfq-iosched.c +++ b/block/cfq-iosched.c @@ -85,7 +85,7 @@ struct cfq_rb_root { struct rb_root rb; struct rb_node *left; unsigned count; - unsigned total_weight; + int total_weight; u64 min_vdisktime; }; #define CFQ_RB_ROOT (struct cfq_rb_root) { .rb = RB_ROOT, .left = NULL, \ @@ -903,6 +903,7 @@ cfq_group_service_tree_del(struct cfq_data *cfqd, struct cfq cfq_log_cfqg(cfqd, cfqg, "del_from_rr group"); st->total_weight -= cfqg->weight; + BUG_ON(st->total_weight < 0); if (!RB_EMPTY_NODE(&cfqg->rb_node)) cfq_rb_erase(&cfqg->rb_node, st); cfqg->saved_workload_slice = 0; The "fuzzcon" tool adjusts the weight for a task in a manner that triggers this bug: http://google3-2.osuosl.org/?p=tests/blkcgroup.git;a=blob;f=scripts/fuzzcon;h=38cfb9e0c847eb92ceccbac67de2a59b43f241ba;hb=eae5e71ad1116b46a9fd84f5d1b1e4c45aa6bd58 Here's the BUG_ON output when the case is encountered: [ 1711.376954] ------------[ cut here ]------------ [ 1711.377789] kernel BUG at block/cfq-iosched.c:906! [ 1711.377789] invalid opcode: 0000 [#1] SMP· [ 1711.377789] last sysfs file: /sys/devices/system/node/node0/distance [ 1711.377789] CPU 0· [ 1711.377789] Modules linked in: tg3 msr cpuid ipv6 genrtc [ 1711.377789]· [ 1711.377789] Pid: 13702, comm: dd Tainted: G W 2.6.38-smp-detect [ 1711.377789] RIP: 0010:[] [] cfq_group_service_tree_del+0x5b/0x99 [ 1711.377789] RSP: 0018:ffff8800189c1b68 EFLAGS: 00010086 [ 1711.377789] RAX: 00000000ffffff9c RBX: ffff880011a6d400 RCX: 000000000000b690 [ 1711.377789] RDX: 0000000000000000 RSI: ffff880011a6d400 RDI: 0000000000000000 [ 1711.377789] RBP: ffff8800189c1b78 R08: 0000000000000000 R09: 0000000000000001 [ 1711.377789] R10: 0000000000000000 R11: 000000000000001b R12: ffff8800065e8000 [ 1711.377789] R13: ffff880011a6d528 R14: 000000000000001b R15: 000000000000001b [ 1711.377789] FS: 0000000000000000(0000) GS:ffff880009c00000(0000) knlGS:0000000000000000 [ 1711.377789] CS: 0010 DS: 002b ES: 002b CR0: 000000008005003b [ 1711.377789] CR2: 00000000f74db000 CR3: 0000000001803000 CR4: 00000000000006f0 [ 1711.377789] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1711.377789] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 1711.377789] Process dd (pid: 13702, threadinfo ffff8800189c0000, task ffff8800067dafa0) [ 1711.377789] Stack: [ 1711.377789] ffff880006596b40 ffff8800065e8000 ffff8800189c1be8 ffffffff81210d2d [ 1711.377789] ffff8800067dafa0 ffff8800060c0418 ffff8800189c1bc8 ffff8800060c0410 [ 1711.377789] 0000000000000082 ffff8800065e8008 ffff8800189c1bc8 ffff880006596b40 [ 1711.377789] Call Trace: [ 1711.377789] [] __cfq_slice_expired+0x3e0/0x45d [ 1711.377789] [] cfq_exit_cfqq+0x22/0x3f [ 1711.377789] [] __cfq_exit_single_io_context+0x77/0x84 [ 1711.377789] [] cfq_exit_single_io_context+0x47/0x62 [ 1711.377789] [] ? cfq_exit_single_io_context+0x0/0x62 [ 1711.377789] [] call_for_each_cic+0x31/0x3e [ 1711.377789] [] cfq_exit_io_context+0x15/0x17 [ 1711.377789] [] exit_io_context+0x5a/0x6d [ 1711.377789] [] do_exit+0x72a/0x756 [ 1711.377789] [] do_group_exit+0x76/0x9e [ 1711.377789] [] get_signal_to_deliver+0x33c/0x35d [ 1711.377789] [] do_signal+0x72/0x699 [ 1711.377789] [] ? fsnotify_modify+0x62/0x6a [ 1711.377789] [] do_notify_resume+0x2c/0x72 [ 1711.377789] [] int_signal+0x12/0x17 [ 1711.377789] Code: 48 85 ff 74 15 48 8d 96 42 01 00 00 31 c0 48 c7 c6 0d 13 72 81 e8 d2 cd eb ff 41 8b 44 24 1c 2b 43 20 85 c0 41 89 44 24 1c 79 04 <0f> 0b eb fe 48 8b 03 48 83 e0 fc 48 39 c3 74 0d 49 8d 74 24 08· [ 1711.377789] RIP [] cfq_group_service_tree_del+0x5b/0x99 [ 1711.377789] RSP [ 1711.377789] ---[ end trace 5e95326ab7063969 ]--- Justin TerAvest (1): Don't update group weights when on service tree. block/cfq-iosched.c | 57 ++++++++++++++++++++++++++++++++++++++------------ 1 files changed, 43 insertions(+), 14 deletions(-) -- 1.7.3.1