LKML Archive on lore.kernel.org
help / color / mirror / Atom feed
* [PATCH] panic: Add sysctl/cmdline to dump all CPUs backtraces on oops event
@ 2020-03-10 16:37 Guilherme G. Piccoli
  2020-03-10 20:59 ` Randy Dunlap
  0 siblings, 1 reply; 7+ messages in thread
From: Guilherme G. Piccoli @ 2020-03-10 16:37 UTC (permalink / raw)
  To: linux-kernel
  Cc: linux-doc, mcgrof, keescook, yzaikin, tglx, akpm, gpiccoli, kernel

Usually when kernel reach an oops condition, it's a point of no return;
in case not enough debug information is available in the kernel splat,
one of the last resorts would be to collect a kernel crash dump and
analyze it. The problem with this approach is that in order to collect
the dump, a panic is required (to kexec-load the crash kernel). When
in an environment of multiple virtual machines, users may prefer to
try living with the oops, at least until being able to properly
shutdown their VMs / finish their important tasks.

This patch implements a way to collect a bit more debug details when an
oops event is reached, by printing all the CPUs backtraces through the
usage of NMIs (on architectures that support that). The sysctl/kernel
parameter added (and documented) here was called "oops_all_cpu_backtrace"
and when set will (as the name suggests) dump all CPUs backtraces.

Far from ideal, this may be the last option though for users that for
some reason cannot panic on oops. Most of times oopses are clear enough
to indicate the kernel portion that must be investigated, but in virtual
environments it's possible to observe hypervisor/KVM issues that could
lead to oopses shown in other guests CPUs (like virtual APIC crashes).
This patch hence aims to help debug such complex issues without
resorting to kdump.

Signed-off-by: Guilherme G. Piccoli <gpiccoli@canonical.com>
---


As a P.S. note, my choice to put the backtrace dump in the end of
oops_enter() was from previous experience (in which I used this
approach in a kprobes to collect more data on oops), but I'd
gladly accept suggestion in case there's a better place to dump
this. Thanks in advance for the reviews!
Cheers,

Guilherme


 .../admin-guide/kernel-parameters.txt         |  8 +++++++
 Documentation/admin-guide/sysctl/kernel.rst   | 15 +++++++++++++
 include/linux/kernel.h                        |  6 ++++++
 kernel/panic.c                                | 21 +++++++++++++++++++
 kernel/sysctl.c                               | 11 ++++++++++
 5 files changed, 61 insertions(+)

diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index 4c6595b5f6c8..888b1fab3f6e 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -3333,6 +3333,14 @@
 			This will also cause panics on machine check exceptions.
 			Useful together with panic=30 to trigger a reboot.
 
+	oops_all_cpu_backtrace=
+			[KNL] Should kernel generates backtraces on all cpus
+			when oops occurs - this should be a last measure resort
+			in case	a kdump cannot be collected, for example.
+			Defaults to 0 and can be controlled by the sysctl
+			kernel.oops_all_cpu_backtrace.
+			Format: <integer>
+
 	page_alloc.shuffle=
 			[KNL] Boolean flag to control whether the page allocator
 			should randomize its free lists. The randomization may
diff --git a/Documentation/admin-guide/sysctl/kernel.rst b/Documentation/admin-guide/sysctl/kernel.rst
index 218c717c1354..460112c3f656 100644
--- a/Documentation/admin-guide/sysctl/kernel.rst
+++ b/Documentation/admin-guide/sysctl/kernel.rst
@@ -57,6 +57,7 @@ show up in /proc/sys/kernel:
 - msgmnb
 - msgmni
 - nmi_watchdog
+- oops_all_cpu_backtrace
 - osrelease
 - ostype
 - overflowgid
@@ -573,6 +574,20 @@ numa_balancing_scan_size_mb is how many megabytes worth of pages are
 scanned for a given scan.
 
 
+oops_all_cpu_backtrace:
+================
+
+Determines if kernel should NMI all CPUs to dump their backtraces when
+an oops event occurs. It should be used as a last resort in case a panic
+cannot be triggered (to protect VMs running, for example) or kdump can't
+be collected. This file shows up if CONFIG_SMP is enabled.
+
+0: Won't show all CPUs backtraces when an oops is detected.
+This is the default behavior.
+
+1: Will NMI all CPUs and dump their backtraces when an oops is detected.
+
+
 osrelease, ostype & version:
 ============================
 
diff --git a/include/linux/kernel.h b/include/linux/kernel.h
index 0d9db2a14f44..6cd00257b572 100644
--- a/include/linux/kernel.h
+++ b/include/linux/kernel.h
@@ -513,6 +513,12 @@ static inline u32 int_sqrt64(u64 x)
 }
 #endif
 
+#ifdef CONFIG_SMP
+extern unsigned int sysctl_oops_all_cpu_backtrace;
+#else
+#define sysctl_oops_all_cpu_backtrace 0
+#endif /* CONFIG_SMP */
+
 extern void bust_spinlocks(int yes);
 extern int oops_in_progress;		/* If set, an oops, panic(), BUG() or die() is in progress */
 extern int panic_timeout;
diff --git a/kernel/panic.c b/kernel/panic.c
index b69ee9e76cb2..73c340418575 100644
--- a/kernel/panic.c
+++ b/kernel/panic.c
@@ -36,6 +36,24 @@
 #define PANIC_TIMER_STEP 100
 #define PANIC_BLINK_SPD 18
 
+#ifdef CONFIG_SMP
+/*
+ * Should we dump all CPUs backtraces in an oops event?
+ * Defaults to 0, can be changed either via cmdline or sysctl.
+ */
+unsigned int __read_mostly sysctl_oops_all_cpu_backtrace;
+
+static int __init oops_backtrace_setup(char *str)
+{
+	int rc = kstrtouint(str, 0, &sysctl_oops_all_cpu_backtrace);
+
+	if (rc)
+		return rc;
+	return 1;
+}
+__setup("oops_all_cpu_backtrace=", oops_backtrace_setup);
+#endif /* CONFIG_SMP */
+
 int panic_on_oops = CONFIG_PANIC_ON_OOPS_VALUE;
 static unsigned long tainted_mask =
 	IS_ENABLED(CONFIG_GCC_PLUGIN_RANDSTRUCT) ? (1 << TAINT_RANDSTRUCT) : 0;
@@ -515,6 +533,9 @@ void oops_enter(void)
 	/* can't trust the integrity of the kernel anymore: */
 	debug_locks_off();
 	do_oops_enter_exit();
+
+	if (sysctl_oops_all_cpu_backtrace)
+		trigger_all_cpu_backtrace();
 }
 
 /*
diff --git a/kernel/sysctl.c b/kernel/sysctl.c
index 238f268de486..1ac31d9d5b7e 100644
--- a/kernel/sysctl.c
+++ b/kernel/sysctl.c
@@ -813,6 +813,17 @@ static struct ctl_table kern_table[] = {
 		.proc_handler	= proc_dointvec,
 	},
 #endif
+#ifdef CONFIG_SMP
+	{
+		.procname	= "oops_all_cpu_backtrace",
+		.data		= &sysctl_oops_all_cpu_backtrace,
+		.maxlen		= sizeof(int),
+		.mode		= 0644,
+		.proc_handler	= proc_dointvec_minmax,
+		.extra1		= SYSCTL_ZERO,
+		.extra2		= SYSCTL_ONE,
+	},
+#endif /* CONFIG_SMP */
 	{
 		.procname	= "pid_max",
 		.data		= &pid_max,
-- 
2.25.1


^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [PATCH] panic: Add sysctl/cmdline to dump all CPUs backtraces on oops event
  2020-03-10 16:37 [PATCH] panic: Add sysctl/cmdline to dump all CPUs backtraces on oops event Guilherme G. Piccoli
@ 2020-03-10 20:59 ` Randy Dunlap
  2020-03-11  1:26   ` Andrew Morton
  2020-03-14 14:28   ` Matthew Wilcox
  0 siblings, 2 replies; 7+ messages in thread
From: Randy Dunlap @ 2020-03-10 20:59 UTC (permalink / raw)
  To: Guilherme G. Piccoli, linux-kernel
  Cc: linux-doc, mcgrof, keescook, yzaikin, tglx, akpm, kernel

Hi-

On 3/10/20 9:37 AM, Guilherme G. Piccoli wrote:
> 
> Signed-off-by: Guilherme G. Piccoli <gpiccoli@canonical.com>
> ---
> 
> As a P.S. note, my choice to put the backtrace dump in the end of
> oops_enter() was from previous experience (in which I used this
> approach in a kprobes to collect more data on oops), but I'd
> gladly accept suggestion in case there's a better place to dump
> this. Thanks in advance for the reviews!
> Cheers,
> 
> Guilherme
> 
> 
>  .../admin-guide/kernel-parameters.txt         |  8 +++++++
>  Documentation/admin-guide/sysctl/kernel.rst   | 15 +++++++++++++
>  include/linux/kernel.h                        |  6 ++++++
>  kernel/panic.c                                | 21 +++++++++++++++++++
>  kernel/sysctl.c                               | 11 ++++++++++
>  5 files changed, 61 insertions(+)
> 
> diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
> index 4c6595b5f6c8..888b1fab3f6e 100644
> --- a/Documentation/admin-guide/kernel-parameters.txt
> +++ b/Documentation/admin-guide/kernel-parameters.txt
> @@ -3333,6 +3333,14 @@
>  			This will also cause panics on machine check exceptions.
>  			Useful together with panic=30 to trigger a reboot.
>  
> +	oops_all_cpu_backtrace=
> +			[KNL] Should kernel generates backtraces on all cpus

			                    generate backtraces on all CPUs

> +			when oops occurs - this should be a last measure resort
> +			in case	a kdump cannot be collected, for example.
> +			Defaults to 0 and can be controlled by the sysctl
> +			kernel.oops_all_cpu_backtrace.
> +			Format: <integer>
> +
>  	page_alloc.shuffle=
>  			[KNL] Boolean flag to control whether the page allocator
>  			should randomize its free lists. The randomization may
> diff --git a/Documentation/admin-guide/sysctl/kernel.rst b/Documentation/admin-guide/sysctl/kernel.rst
> index 218c717c1354..460112c3f656 100644
> --- a/Documentation/admin-guide/sysctl/kernel.rst
> +++ b/Documentation/admin-guide/sysctl/kernel.rst
> @@ -573,6 +574,20 @@ numa_balancing_scan_size_mb is how many megabytes worth of pages are
>  scanned for a given scan.
>  
>  
> +oops_all_cpu_backtrace:
> +================
> +
> +Determines if kernel should NMI all CPUs to dump their backtraces when

I would much prefer that to be written without using NMI as a verb.

> +an oops event occurs. It should be used as a last resort in case a panic
> +cannot be triggered (to protect VMs running, for example) or kdump can't
> +be collected. This file shows up if CONFIG_SMP is enabled.
> +
> +0: Won't show all CPUs backtraces when an oops is detected.
> +This is the default behavior.
> +
> +1: Will NMI all CPUs and dump their backtraces when an oops is detected.

Same here.

> +
> +
>  osrelease, ostype & version:
>  ============================
>  



Thanks.
-- 
~Randy


^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [PATCH] panic: Add sysctl/cmdline to dump all CPUs backtraces on oops event
  2020-03-10 20:59 ` Randy Dunlap
@ 2020-03-11  1:26   ` Andrew Morton
  2020-03-11 12:46     ` Guilherme G. Piccoli
  2020-03-14 14:28   ` Matthew Wilcox
  1 sibling, 1 reply; 7+ messages in thread
From: Andrew Morton @ 2020-03-11  1:26 UTC (permalink / raw)
  To: Randy Dunlap
  Cc: Guilherme G. Piccoli, linux-kernel, linux-doc, mcgrof, keescook,
	yzaikin, tglx, kernel

On Tue, 10 Mar 2020 13:59:15 -0700 Randy Dunlap <rdunlap@infradead.org> wrote:

> > +oops_all_cpu_backtrace:
> > +================
> > +
> > +Determines if kernel should NMI all CPUs to dump their backtraces when
> 
> I would much prefer that to be written without using NMI as a verb.

"Non maskably interrupt" ;)

I think it's OK.  Concise and the meaning is clear.


Why do we need the kernel boot parameter?  Isn't
/proc/sys/kernel/oops_all_cpu_backtrace sufficient?


^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [PATCH] panic: Add sysctl/cmdline to dump all CPUs backtraces on oops event
  2020-03-11  1:26   ` Andrew Morton
@ 2020-03-11 12:46     ` Guilherme G. Piccoli
  0 siblings, 0 replies; 7+ messages in thread
From: Guilherme G. Piccoli @ 2020-03-11 12:46 UTC (permalink / raw)
  To: Andrew Morton, Randy Dunlap
  Cc: linux-kernel, linux-doc, mcgrof, keescook, yzaikin, tglx, kernel

On 10/03/2020 22:26, Andrew Morton wrote:
> On Tue, 10 Mar 2020 13:59:15 -0700 Randy Dunlap <rdunlap@infradead.org> wrote:
> 
>>> +oops_all_cpu_backtrace:
>>> +================
>>> +
>>> +Determines if kernel should NMI all CPUs to dump their backtraces when
>>
>> I would much prefer that to be written without using NMI as a verb.
> 
> "Non maskably interrupt" ;)
> 
> I think it's OK.  Concise and the meaning is clear.

Hi Andrew, good idea heheh
Thank you and all that reviewed the grammar/wording, certainly I can
change that and resubmit.


> 
> 
> Why do we need the kernel boot parameter?  Isn't
> /proc/sys/kernel/oops_all_cpu_backtrace sufficient?
> 

I kept the kernel parameter as a consistency thing - every sysctl
"*_all_cpubacktrace" has a respective kernel parameter, so I did the
same (and if we get an oops booting a new kernel, this is maybe useful
depending on the point we get the oops). But if it's a problem for you,
I can remove the kernel parameter, your choice.

Cheers,


Guilherme

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [PATCH] panic: Add sysctl/cmdline to dump all CPUs backtraces on oops event
  2020-03-10 20:59 ` Randy Dunlap
  2020-03-11  1:26   ` Andrew Morton
@ 2020-03-14 14:28   ` Matthew Wilcox
  2020-03-14 21:18     ` Randy Dunlap
  1 sibling, 1 reply; 7+ messages in thread
From: Matthew Wilcox @ 2020-03-14 14:28 UTC (permalink / raw)
  To: Randy Dunlap
  Cc: Guilherme G. Piccoli, linux-kernel, linux-doc, mcgrof, keescook,
	yzaikin, tglx, akpm, kernel

On Tue, Mar 10, 2020 at 01:59:15PM -0700, Randy Dunlap wrote:
> > +oops_all_cpu_backtrace:
> > +================
> > +
> > +Determines if kernel should NMI all CPUs to dump their backtraces when
> 
> I would much prefer that to be written without using NMI as a verb.

Concrete suggestion: "If this option is set, the kernel will send an NMI to
all CPUs to dump ..."


^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [PATCH] panic: Add sysctl/cmdline to dump all CPUs backtraces on oops event
  2020-03-14 14:28   ` Matthew Wilcox
@ 2020-03-14 21:18     ` Randy Dunlap
  2020-03-16 13:51       ` Guilherme G. Piccoli
  0 siblings, 1 reply; 7+ messages in thread
From: Randy Dunlap @ 2020-03-14 21:18 UTC (permalink / raw)
  To: Matthew Wilcox
  Cc: Guilherme G. Piccoli, linux-kernel, linux-doc, mcgrof, keescook,
	yzaikin, tglx, akpm, kernel

On 3/14/20 7:28 AM, Matthew Wilcox wrote:
> On Tue, Mar 10, 2020 at 01:59:15PM -0700, Randy Dunlap wrote:
>>> +oops_all_cpu_backtrace:
>>> +================
>>> +
>>> +Determines if kernel should NMI all CPUs to dump their backtraces when
>>
>> I would much prefer that to be written without using NMI as a verb.
> 
> Concrete suggestion: "If this option is set, the kernel will send an NMI to
> all CPUs to dump ..."
> 

Ack.  Thanks for that.

-- 
~Randy


^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [PATCH] panic: Add sysctl/cmdline to dump all CPUs backtraces on oops event
  2020-03-14 21:18     ` Randy Dunlap
@ 2020-03-16 13:51       ` Guilherme G. Piccoli
  0 siblings, 0 replies; 7+ messages in thread
From: Guilherme G. Piccoli @ 2020-03-16 13:51 UTC (permalink / raw)
  To: Randy Dunlap, Matthew Wilcox
  Cc: linux-kernel, linux-doc, mcgrof, keescook, yzaikin, tglx, akpm, kernel

On 14/03/2020 18:18, Randy Dunlap wrote:
> On 3/14/20 7:28 AM, Matthew Wilcox wrote:
>> On Tue, Mar 10, 2020 at 01:59:15PM -0700, Randy Dunlap wrote:
>>>> +oops_all_cpu_backtrace:
>>>> +================
>>>> +
>>>> +Determines if kernel should NMI all CPUs to dump their backtraces when
>>>
>>> I would much prefer that to be written without using NMI as a verb.
>>
>> Concrete suggestion: "If this option is set, the kernel will send an NMI to
>> all CPUs to dump ..."
>>
> 
> Ack.  Thanks for that.
> 

Thanks Randy and Matthew! I'll implement those changes and resend as V2.
If anybody has suggestions of people I should add to CC list, please let
me know.

Thanks again,


Guilherme

^ permalink raw reply	[flat|nested] 7+ messages in thread

end of thread, other threads:[~2020-03-16 13:51 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-03-10 16:37 [PATCH] panic: Add sysctl/cmdline to dump all CPUs backtraces on oops event Guilherme G. Piccoli
2020-03-10 20:59 ` Randy Dunlap
2020-03-11  1:26   ` Andrew Morton
2020-03-11 12:46     ` Guilherme G. Piccoli
2020-03-14 14:28   ` Matthew Wilcox
2020-03-14 21:18     ` Randy Dunlap
2020-03-16 13:51       ` Guilherme G. Piccoli

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).