LKML Archive on
help / color / mirror / Atom feed
To: Philipp Rudo <>,
	Konrad Rzeszutek Wilk <>
Cc: Dave Young <>, Wei Liu <>,
	Tianyu Lan <>,
	"" <>,
	"" <>,
	"" <>,
	Michael Kelley <>,
	HATAYAMA Daisuke <>,
	"Eric W. Biederman" <>,
	Masami Hiramatsu <>,
	Eric DeVolder <>,
	Andrew Morton <>,
Subject: Re: [PATCH] Only allow to set crash_kexec_post_notifiers on boot time
Date: Tue, 29 Sep 2020 15:10:29 -0400	[thread overview]
Message-ID: <> (raw)
In-Reply-To: <20200929153616.3cf68f16@ibm>


On 9/29/20 9:36 AM, Philipp Rudo wrote:
> Hi,
> On Fri, 25 Sep 2020 10:56:25 -0400
> Konrad Rzeszutek Wilk <> wrote:
>> On Fri, Sep 25, 2020 at 11:05:58AM +0800, Dave Young wrote:
>>> Hi,
>>> On 09/24/20 at 01:16pm, wrote:  
>>>> On 9/24/20 12:43 PM, Michael Kelley wrote:  
>>>>> From: Eric W. Biederman <> Sent: Thursday, September 24, 2020 9:26 AM  
>>>>>> Michael Kelley <> writes:
>>>>>>>>> Added Hyper-V people and people who created the param, it is below
>>>>>>>>> commit, I also want to remove it if possible, let's see how people
>>>>>>>>> think, but the least way should be to disable the auto setting in both systemd
>>>>>>>>> and kernel:  
>>>>>>> Hyper-V uses a notifier to inform the host system that a Linux VM has
>>>>>>> panic'ed.  Informing the host is particularly important in a public cloud
>>>>>>> such as Azure so that the cloud software can alert the customer, and can
>>>>>>> track cloud-wide reliability statistics.   Whether a kdump is taken is controlled
>>>>>>> entirely by the customer and how he configures the VM, and we want
>>>>>>> the host to be informed either way.  
>>>>>> Why?
>>>>>> Why does the host care?
>>>>>> Especially if the VM continues executing into a kdump kernel?  
>>>>> The host itself doesn't care.  But the host is a convenient out-of-band
>>>>> channel for recording that a panic has occurred and to collect basic data
>>>>> about the panic.  This out-of-band channel is then used to notify the end
>>>>> customer that his VM has panic'ed.  Sure, the customer should be running
>>>>> his own monitoring software, but customers don't always do what they
>>>>> should.  Equally important, the out-of-band channel allows the cloud
>>>>> infrastructure software to notice trends, such as that the rate of Linux
>>>>> panics has increased, and that perhaps there is a cloud problem that
>>>>> should be investigated.  
>>>> In many cases (especially in cloud environment) your dump device is remote (e.g. iscsi) and kdump sometimes (often?) gets stuck because of connectivity issues (which could be cause of the panic in the first place). So it is quite desirable to inform the infrastructure that the VM is on its way out without waiting for kdump to complete.  
>>> That can probably be done in kdump kernel if it is really needed.  Say
>>> informing host that panic happened and a kdump kernel is runnning.  
>> If kdump kernel gets to that point. Sometimes (sadly) it ends up being
>> misconfigured and it chokes up - and hence having multiple ways to emit
>> the crash information before running kdump kernel is a life-saver.
>>> But I think to set crash_kexec_post_notifiers by default is still bad.   
>> Because of the way it is run today I presume? If there was some
>> safe/unsafe policy that should work right? I would think that the
>> safe ones that work properly all the time are:
>>  - HyperV CRASH_MSRs,
>>  - KVM PVPANIC_[PANIC,CRASHLOAD] push button knob,
>>  - pstore EFI variables
>>  - Dumping in memory,
>> And then some that depend on firmware version (aka BIOS, and vendor) are:
>>  - ACPI ERST,
>> And then the unsafe:
>>  - s390, PowerPC (I don't actually know what they are but that
>>     was Dave's primary motivator).
> that won't work on s390. Let me emphasize that the problems on s390 are not the
> notifiers themselves but the fact that they are called before crash_kexec.
> On s390 we have multiple dump methods besides kdump. We use a panic notifier to
> trigger these dump methods from the panicking kernel. The problem is that these
> dump methods are less powerful than kdump so we only want to use them as
> fallback, i.e. only use them when either kdump wasn't configured or loading of
> the crash kernel failed for whatever reason. That's why (plus historic reasons)
> our notifier stops the machine when it is called and none of the methods is
> configured. Which means that the second crash_kexec is never reached.
> Long story short, the problem on s390 is caused by the two hunks in
> kernel/panic.c:panic from f06e5153f4ae ("kernel/panic.c: add
> "crash_kexec_post_notifiers" option for kdump after panic_notifers").
> Besides the problems on s390 I support Dave and think that setting
> crash_kexec_post_notifiers by default is wrong. We should keep in mind that
> we are in a panic situation. This means that the kernel is in a state where it
> doesn't trust itself anymore. So we should keep the code that is run to the
> bare minimum as we cannot rely on it to work properly.

There is a pending patch to revert notifiers' default in systemd:

If this change goes through then Dave's patch will be unnecessary.


> Thanks
> Philipp
>>>>>> Further like I have mentioned everytime something like this has come up
>>>>>> a call on the kexec on panic code path should be a direct call (That can
>>>>>> be audited) not something hidden in a notifier call chain (which can not).
>>>> We btw already have a direct call from panic() to kmsg_dump() which is indirectly controlled by crash_kexec_post_notifiers, and it would also be preferable to be able to call it before kdump as well.  
>>> Right, that is the same thing we are talking about.
>>> Thanks
>>> Dave
>> _______________________________________________
>> kexec mailing list

  reply	other threads:[~2020-09-29 19:11 UTC|newest]

Thread overview: 20+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-09-18  3:25 Dave Young
2020-09-19  0:47 ` Andrew Morton
2020-09-19  7:26   ` Dave Young
2020-09-21 20:18   ` Konrad Rzeszutek Wilk
2020-09-22  1:45     ` Eric W. Biederman
2020-09-23  2:43       ` Dave Young
2020-09-23 15:48         ` Konrad Rzeszutek Wilk
2020-09-24 16:15           ` Michael Kelley
2020-09-24 16:25             ` Eric W. Biederman
2020-09-24 16:43               ` Michael Kelley
2020-09-24 17:16                 ` boris.ostrovsky
2020-09-25  3:05                   ` Dave Young
2020-09-25 14:56                     ` Konrad Rzeszutek Wilk
2020-09-27  2:51                       ` Dave Young
2020-09-29 13:36                       ` Philipp Rudo
2020-09-29 19:10                         ` boris.ostrovsky [this message]
2020-09-22 10:58     ` Philipp Rudo
2020-09-22 14:50       ` boris.ostrovsky
2020-09-22 17:04         ` Guilherme G. Piccoli
2020-09-23  2:25     ` Dave Young

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \
    --subject='Re: [PATCH] Only allow to set crash_kexec_post_notifiers on boot time' \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).