LKML Archive on lore.kernel.org
help / color / mirror / Atom feed
From: Frederic Weisbecker <frederic@kernel.org>
To: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
Cc: linux-kernel@vger.kernel.org, cmetcalf@mellanox.com,
	cl@linux.com, lcapitulino@redhat.com, efault@gmx.de,
	peterz@infradead.org, riel@redhat.com, tglx@linutronix.de,
	kernellwp@gmail.com, mingo@kernel.org
Subject: Re: NO_HZ_FULL and tick running within a reasonable amount of time
Date: Sat, 7 Apr 2018 15:52:13 +0200	[thread overview]
Message-ID: <20180407135211.GB16600@lerouge> (raw)
In-Reply-To: <20180403170858.GO3948@linux.vnet.ibm.com>

On Tue, Apr 03, 2018 at 10:08:58AM -0700, Paul E. McKenney wrote:
> On Tue, Apr 03, 2018 at 01:41:31PM +0200, Frederic Weisbecker wrote:
> > On Mon, Apr 02, 2018 at 03:04:38PM -0700, Paul E. McKenney wrote:
> > > Hello!
> > > 
> > > I am hitting the following on today's mainline under rcutorture, but
> > > only on scenarios built with CONFIG_NO_HZ_FULL=y:
> > > 
> > > ------------------------------------------------------------------------
> > > 
> > > WARNING: CPU: 0 PID: 7 at /home/paulmck/public_git/linux-rcu/kernel/sched/core.c:3124 sched_tick_remote+0x113/0x120
> > > Modules linked in:
> > > CPU: 0 PID: 7 Comm: kworker/u4:0 Not tainted 4.16.0+ #1
> > > Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Ubuntu-1.8.2-1ubuntu1 04/01/2014
> > > Workqueue: events_unbound sched_tick_remote
> > > RIP: 0010:sched_tick_remote+0x113/0x120
> > > RSP: 0018:ffff94d540103e20 EFLAGS: 00010002
> > > RAX: 000000012e9bb357 RBX: ffff8f95dfd21840 RCX: 000000000000001f
> > > RDX: 00000000b2d05e00 RSI: 00000000ffffffff RDI: ffff8f95dfd21858
> > > RBP: ffff94d540103e48 R08: 00000000f6499019 R09: 00000000f6499000
> > > R10: 00000000b163d33b R11: ffffffffa5c8c212 R12: ffff8f95dfd25518
> > > R13: ffff8f95de9e4200 R14: 0000000000003402 R15: ffff8f95dfd21858
> > > FS:  0000000000000000(0000) GS:ffff8f95dfc00000(0000) knlGS:0000000000000000
> > > CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > CR2: 000000000a015b40 CR3: 000000001de14000 CR4: 00000000000006f0
> > > Call Trace:
> > >  process_one_work+0x1d9/0x6a0
> > >  worker_thread+0x42/0x420
> > >  kthread+0xf3/0x130
> > >  ? rescuer_thread+0x340/0x340
> > >  ? kthread_delayed_work_timer_fn+0x80/0x80
> > >  ret_from_fork+0x3a/0x50
> > > Code: ff 48 8b 83 80 0b 00 00 48 85 c0 0f 85 41 ff ff ff e9 45 ff ff ff be ff ff ff ff 4c 89 ff e8 55 44 02 00 85 c0 75 87 0f 0b eb 83 <0f> 0b eb 97 66 0f 1f 84 00 00 00 00 00 41 57 41 56 41 55 41 54
> > > ---[ end trace fbdcbe529a8ae799 ]--
> > > 
> > > ------------------------------------------------------------------------
> > > 
> > > The WARN_ON_ONCE() triggering is this guy:
> > > 
> > > 	delta = rq_clock_task(rq) - curr->se.exec_start;
> > > 	WARN_ON_ONCE(delta > (u64)NSEC_PER_SEC * 3);
> > 
> > Weird. Can you try to print up those values and see how much they drift?
> > 
> >         if (WARN_ON_ONCE(delta > (u64)NSEC_PER_SEC * 3))
> >                printk_once("clock_task: %lld exec_start: %lld\n", rq_clock_task(rq), curr->se.exec_start);
> 
> Here you go!
> 
> 							Thanx, Paul
> 
> ------------------------------------------------------------------------
> 
> WARNING: CPU: 0 PID: 7 at /home/paulmck/public_git/linux-rcu/kernel/sched/core.c:3124 sched_tick_remote+0xdb/0x100
> Modules linked in:
> CPU: 0 PID: 7 Comm: kworker/u4:0 Not tainted 4.16.0+ #1
> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Ubuntu-1.8.2-1ubuntu1 04/01/2014
> Workqueue: events_unbound sched_tick_remote
> RIP: 0010:sched_tick_remote+0xdb/0x100
> RSP: 0018:ffffa2c440103e60 EFLAGS: 00010006
> RAX: 00000000b2d05e00 RBX: ffff96f0dfd20980 RCX: 000000016a8de322
> RDX: 000000000d33a301 RSI: 0000000177c18623 RDI: fffffffeb0bf0e53
> RBP: ffff96f0dfd24328 R08: 0000000000000000 R09: 0000000000000001
> R10: 0000000000000000 R11: 0000000000000000 R12: ffff96f0de9d2640
> R13: 0000000000000000 R14: 0ffff96f0de81700 R15: ffff96f0de96f540
> FS:  0000000000000000(0000) GS:ffff96f0dfc00000(0000) knlGS:0000000000000000
> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 00000000f7ec6ca3 CR3: 000000001e03e000 CR4: 00000000000006f0
> Call Trace:
>  process_one_work+0x139/0x3e0
>  worker_thread+0x42/0x420
>  kthread+0xf3/0x130
>  ? create_worker+0x190/0x190
>  ? kthread_destroy_worker+0x40/0x40
>  ret_from_fork+0x35/0x40
> Code: 89 8b 43 04 85 c0 0f 85 75 ff ff ff 48 8b 83 e0 0a 00 00 48 85 c0 0f 85 65 ff ff ff e9 69 ff ff ff 48 89 df e8 87 fe ff ff eb 8d <0f> 0b 80 3d 7d 57 2b 01 00 75 a8 48 c7 c7 e8 e1 fd a4 c6 05 6d
> ---[ end trace f0c6a1afa55d130d ]---
> clock_task: 6304138787 exec_start: 221487873
> 

That's a 6 second delay, it's huge!

Could it be because you use Qemu and the virtualized CPUs got interrupted for
a long while?

  reply	other threads:[~2018-04-07 13:52 UTC|newest]

Thread overview: 8+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-04-02 22:04 Paul E. McKenney
2018-04-03  9:23 ` Peter Zijlstra
2018-04-03 11:41 ` Frederic Weisbecker
2018-04-03 17:08   ` Paul E. McKenney
2018-04-07 13:52     ` Frederic Weisbecker [this message]
2018-04-23 13:04     ` Frederic Weisbecker
2018-04-23 13:26       ` Paul E. McKenney
2018-04-14  4:07 Jacek Tomaka

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20180407135211.GB16600@lerouge \
    --to=frederic@kernel.org \
    --cc=cl@linux.com \
    --cc=cmetcalf@mellanox.com \
    --cc=efault@gmx.de \
    --cc=kernellwp@gmail.com \
    --cc=lcapitulino@redhat.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mingo@kernel.org \
    --cc=paulmck@linux.vnet.ibm.com \
    --cc=peterz@infradead.org \
    --cc=riel@redhat.com \
    --cc=tglx@linutronix.de \
    --subject='Re: NO_HZ_FULL and tick running within a reasonable amount of time' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).