LKML Archive on lore.kernel.org
help / color / mirror / Atom feed
From: Frederic Weisbecker <frederic@kernel.org>
To: Thomas Gleixner <tglx@linutronix.de>
Cc: "Wan, Kaike" <kaike.wan@intel.com>,
	"Marciniszyn, Mike" <mike.marciniszyn@intel.com>,
	"Dalessandro, Dennis" <dennis.dalessandro@intel.com>,
	"Weiny, Ira" <ira.weiny@intel.com>,
	"Fleck, John" <john.fleck@intel.com>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>,
	Peter Zijlstra <peterz@infradead.org>,
	Anna-Maria Gleixner <anna-maria@linutronix.de>,
	Frederic Weisbecker <fweisbec@gmail.com>,
	Ingo Molnar <mingo@kernel.org>
Subject: Re: [PATCH] tick/sched: Do not mess with an enqueued hrtimer
Date: Wed, 25 Apr 2018 15:22:06 +0200	[thread overview]
Message-ID: <20180425132205.GA12534@lerouge> (raw)
In-Reply-To: <alpine.DEB.2.21.1804242119210.1597@nanos.tec.linutronix.de>

On Tue, Apr 24, 2018 at 09:22:18PM +0200, Thomas Gleixner wrote:
> Kaike reported that in tests rdma hrtimers occasionaly stopped working. He
> did great debugging, which provided enough context to decode the problem.
> 
> CPU 3			     	      	     CPU 2
> 
> idle
> start sched_timer expires = 712171000000
>  queue->next = sched_timer
> 					    start rdmavt timer. expires = 712172915662
> 					    lock(baseof(CPU3))
> tick_nohz_stop_tick()
> tick = 716767000000			    timerqueue_add(tmr)
> 
> hrtimer_set_expires(sched_timer, tick);
>   sched_timer->expires = 716767000000  <---- FAIL
> 					     if (tmr->expires < queue->next->expires)
> hrtimer_start(sched_timer)		          queue->next = tmr;
> lock(baseof(CPU3))
> 					     unlock(baseof(CPU3))
> timerqueue_remove()
> timerqueue_add()
> 
> ts->sched_timer is queued and queue->next is pointing to it, but then
> ts->sched_timer.expires is modified.
> 
> This not only corrupts the ordering of the timerqueue RB tree, it also
> makes CPU2 see the new expiry time of timerqueue->next->expires when
> checking whether timerqueue->next needs to be updated. So CPU2 sees that
> the rdma timer is earlier than timerqueue->next and sets the rdma timer as
> new next.
> 
> Depending on whether it had also seen the new time at RB tree enqueue, it
> might have queued the rdma timer at the wrong place and then after removing
> the sched_timer the RB tree is completely hosed.
> 
> The problem was introduced with a commit which tried to solve inconsistency
> between the hrtimer in the tick_sched data and the underlying hardware
> clockevent. It split out hrtimer_set_expires() to store the new tick time
> in both the NOHZ and the NOHZ + HIGHRES case, but missed the fact that in
> the NOHZ + HIGHRES case the hrtimer might still be queued.
> 
> Use hrtimer_start(timer, tick...) for the NOHZ + HIGHRES case which sets
> timer->expires after canceling the timer and move the hrtimer_set_expires()
> invocation into the NOHZ only code path which is not affected as it merily
> uses the hrtimer as next event storage so code pathes can be shared with
> the NOHZ + HIGHRES case.
> 
> Fixes: d4af6d933ccf ("nohz: Fix spurious warning when hrtimer and clockevent get out of sync")
> Reported-by: "Wan Kaike" <kaike.wan@intel.com>
> Signed-off-by: Thomas Gleixner <tglx@linutronix.de>

Acked-by: Frederic Weisbecker <frederic@kernel.org>

Thanks!

  reply	other threads:[~2018-04-25 13:22 UTC|newest]

Thread overview: 10+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
     [not found] <3F128C9216C9B84BB6ED23EF16290AFB634C8C87@CRSMSX101.amr.corp.intel.com>
2018-04-23 12:53 ` hrtimer (rdmavt RNR timer) was lost Thomas Gleixner
2018-04-23 13:22   ` Wan, Kaike
2018-04-23 13:35     ` Thomas Gleixner
2018-04-24 14:54       ` Thomas Gleixner
2018-04-24 16:48         ` Frederic Weisbecker
2018-04-24 19:22         ` [PATCH] tick/sched: Do not mess with an enqueued hrtimer Thomas Gleixner
2018-04-25 13:22           ` Frederic Weisbecker [this message]
2018-04-25 14:21           ` [tip:timers/urgent] " tip-bot for Thomas Gleixner
2018-04-26 12:56           ` [PATCH] " Wan, Kaike
2018-04-26 12:57           ` [tip:timers/urgent] " tip-bot for Thomas Gleixner

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20180425132205.GA12534@lerouge \
    --to=frederic@kernel.org \
    --cc=anna-maria@linutronix.de \
    --cc=dennis.dalessandro@intel.com \
    --cc=fweisbec@gmail.com \
    --cc=ira.weiny@intel.com \
    --cc=john.fleck@intel.com \
    --cc=kaike.wan@intel.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-rdma@vger.kernel.org \
    --cc=mike.marciniszyn@intel.com \
    --cc=mingo@kernel.org \
    --cc=peterz@infradead.org \
    --cc=tglx@linutronix.de \
    --subject='Re: [PATCH] tick/sched: Do not mess with an enqueued hrtimer' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).