LKML Archive on lore.kernel.org
help / color / mirror / Atom feed
From: Don Zickus <dzickus@redhat.com>
To: Andrew Morton <akpm@linux-foundation.org>
Cc: LKML <linux-kernel@vger.kernel.org>,
	Ulrich Obergfell <uobergfe@redhat.com>,
	Ingo Molnar <mingo@elte.hu>
Subject: Re: [PATCH 6/9] watchdog: implement error handling for failure to set up hardware perf events
Date: Tue, 24 Feb 2015 10:39:25 -0500	[thread overview]
Message-ID: <20150224153925.GI126481@redhat.com> (raw)
In-Reply-To: <20150223131734.61ee63b5f4064e656f0da762@linux-foundation.org>

On Mon, Feb 23, 2015 at 01:17:34PM -0800, Andrew Morton wrote:
> On Thu,  5 Feb 2015 15:40:22 -0500 Don Zickus <dzickus@redhat.com> wrote:
> 
> > From: Ulrich Obergfell <uobergfe@redhat.com>
> > 
> > If watchdog_nmi_enable() fails to set up the hardware perf event
> > of one CPU, the entire hard lockup detector is deemed unreliable.
> > Hence, disable the hard lockup detector and shut down the hardware
> > perf events on all CPUs.
> > 
> > Signed-off-by: Ulrich Obergfell <uobergfe@redhat.com>
> > Signed-off-by: Don Zickus <dzickus@redhat.com>
> > ---
> >  kernel/watchdog.c |   18 ++++++++++++++++++
> >  1 files changed, 18 insertions(+), 0 deletions(-)
> > 
> > diff --git a/kernel/watchdog.c b/kernel/watchdog.c
> > index 26002ed..7ad8949 100644
> > --- a/kernel/watchdog.c
> > +++ b/kernel/watchdog.c
> > @@ -502,6 +502,15 @@ static void watchdog(unsigned int cpu)
> >  	__this_cpu_write(soft_lockup_hrtimer_cnt,
> >  			 __this_cpu_read(hrtimer_interrupts));
> >  	__touch_watchdog();
> > +
> > +	/*
> > +	 * watchdog_nmi_enable() clears the NMI_WATCHDOG_ENABLED bit in the
> > +	 * failure path. Check for failures that can occur asynchronously -
> > +	 * for example, when CPUs are on-lined - and shut down the hardware
> > +	 * perf event on each CPU accordingly.
> > +	 */
> > +	if (!(watchdog_enabled & NMI_WATCHDOG_ENABLED))
> > +		watchdog_nmi_disable(cpu);
> 
> Silently disabling the hardware watchdog.  Wouldn't it be better to
> emit a printk to alert the operator about this event?

We can add something here.  The original hypothetical problem was cpu0 and
1 start the watchdog correctly, but for whatever reason cpu2 fails.  You
would then see output like:

...
NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter.
..
NMI watchdog: disabled (cpu2): hardware events not enabled
..

Asynchronously a few seconds later the above code would disable _all_ the
nmi watchdog across the cpus.

My thought process was the first failure was enough.  But perhaps that
isn't obvious enough that the other cpus would be disabled.  So we can add
something.

> 
> >  
> >  #ifdef CONFIG_HARDLOCKUP_DETECTOR
> > @@ -552,6 +561,15 @@ handle_err:
> >  		goto out_save;
> >  	}
> >  
> > +	/*
> > +	 * Disable the hard lockup detector if _any_ CPU fails to set up
> > +	 * set up the hardware perf event. The watchdog() function checks
> > +	 * the NMI_WATCHDOG_ENABLED bit periodically.
> > +	 */
> > +	smp_mb__before_atomic();
> > +	clear_bit(NMI_WATCHDOG_ENABLED_BIT, &watchdog_enabled);
> > +	smp_mb__after_atomic();
> 
> Please send along a patch which adds comments explaining what these
> barriers are for.
> 
> What are these barriers for? ;)

Sadly, I am not strong with memory barriers, so the bulk of the motivation
for barriers comes from Documentation/atomic_ops.txt when using
clear_bit().

I will add some comment explaining the need to sync up watchdog_enabled
before we clear it.

Cheers,
Don

  reply	other threads:[~2015-02-24 15:39 UTC|newest]

Thread overview: 19+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2015-02-05 20:40 [PATCH 0/9] watchdog: internally split softlockup and hardlockup Don Zickus
2015-02-05 20:40 ` [PATCH 1/9] watchdog: new definitions and variables, initialization Don Zickus
2015-02-05 20:40 ` [PATCH 2/9] watchdog: introduce the proc_watchdog_update() function Don Zickus
2015-02-05 20:40 ` [PATCH 3/9] watchdog: move definition of 'watchdog_proc_mutex' outside of proc_dowatchdog() Don Zickus
2015-02-05 20:40 ` [PATCH 4/9] watchdog: introduce the proc_watchdog_common() function Don Zickus
2015-02-05 20:40 ` [PATCH 5/9] watchdog: introduce separate handlers for parameters in /proc/sys/kernel Don Zickus
2015-02-05 20:40 ` [PATCH 6/9] watchdog: implement error handling for failure to set up hardware perf events Don Zickus
2015-02-23 21:17   ` Andrew Morton
2015-02-24 15:39     ` Don Zickus [this message]
2015-03-02 19:06     ` Don Zickus
2015-02-05 20:40 ` [PATCH 7/9] watchdog: enable the new user interface of the watchdog mechanism Don Zickus
2015-02-23 21:19   ` Andrew Morton
2015-02-24 15:46     ` Don Zickus
2015-03-02 19:05     ` [PATCH 7/9 UPDATE] " Don Zickus
2015-02-05 20:40 ` [PATCH 8/9] watchdog: clean up some function names and arguments Don Zickus
2015-02-05 20:40 ` [PATCH 9/9] watchdog: introduce the hardlockup_detector_disable() function Don Zickus
2015-02-05 23:22 ` [PATCH 0/9] watchdog: internally split softlockup and hardlockup Andrew Morton
2015-02-06 14:12   ` Don Zickus
  -- strict thread matches above, loose matches on Subject: below --
2014-11-04 16:20 Don Zickus
2014-11-04 16:20 ` [PATCH 6/9] watchdog: implement error handling for failure to set up hardware perf events Don Zickus

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20150224153925.GI126481@redhat.com \
    --to=dzickus@redhat.com \
    --cc=akpm@linux-foundation.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mingo@elte.hu \
    --cc=uobergfe@redhat.com \
    --subject='Re: [PATCH 6/9] watchdog: implement error handling for failure to set up hardware perf events' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).