LKML Archive on lore.kernel.org
help / color / mirror / Atom feed
From: Johannes Erdfelt <johannes@erdfelt.com>
To: Joe Lawrence <joe.lawrence@redhat.com>
Cc: Josh Poimboeuf <jpoimboe@redhat.com>,
	Jessica Yu <jeyu@kernel.org>, Jiri Kosina <jikos@kernel.org>,
	Miroslav Benes <mbenes@suse.cz>,
	Steven Rostedt <rostedt@goodmis.org>,
	Ingo Molnar <mingo@redhat.com>,
	live-patching@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: Re: Oops caused by race between livepatch and ftrace
Date: Mon, 20 May 2019 14:09:05 -0700	[thread overview]
Message-ID: <20190520210905.GC1646@sventech.com> (raw)
In-Reply-To: <90f78070-95ec-ce49-1641-19d061abecf4@redhat.com>

On Mon, May 20, 2019, Joe Lawrence <joe.lawrence@redhat.com> wrote:
> [ fixed jeyu's email address ]

Thank you, the bounce message made it seem like my mail server was
blocked and not that the address didn't exist.

I think MAINTAINERS needs an update since it still has the @redhat.com
address.

> On 5/20/19 3:49 PM, Johannes Erdfelt wrote:
> > [ ... snip ... ]
> > 
> > I have put together a test case that can reproduce the crash using
> > KVM. The tarball includes a minimal kernel and initramfs, along with
> > a script to run qemu and the .config used to build the kernel. By
> > default it will attempt to reproduce by loading multiple livepatches
> > at the same time. Passing 'test=ftrace' to the script will attempt to
> > reproduce by racing with ftrace.
> > 
> > My test setup reproduces the race and oops more reliably by loading
> > multiple livepatches at the same time than with the ftrace method. It's
> > not 100% reproducible, so the test case may need to be run multiple
> > times.
> > 
> > It can be found here (not attached because of its size):
> > http://johannes.erdfelt.com/5.2.0-rc1-a188339ca5-livepatch-race.tar.gz
> 
> Hi Johannes,
> 
> This is cool way to distribute the repro kernel, modules, etc!

This oops was common in our production environment and was particularly
annoying since livepatches would load at boot and early enough to happen
before networking and SSH were started.

Unfortunately it was difficult to reproduce on other hardware (changing
the timing just enough) and our production environment is very
complicated.

I spent more time than I'd like to admit trying to reproduce this fairly
reliably. I knew that I needed to help make it as easy as possible to
reproduce to root cause it and for others to take a look at it as well.

> These two testing scenarios might be interesting to add to our selftests
> suite.  Can you post or add the source(s) to livepatch-test<n>.ko to the
> tarball?

I made the livepatches using kpatch-build and this simple patch:

diff --git a/fs/proc/version.c b/fs/proc/version.c
index 94901e8e700d..6b8a3449f455 100644
--- a/fs/proc/version.c
+++ b/fs/proc/version.c
@@ -12,6 +12,7 @@ static int version_proc_show(struct seq_file *m, void *v)
 		utsname()->sysname,
 		utsname()->release,
 		utsname()->version);
+	seq_printf(m, "example livepatch\n");
 	return 0;
 }

I just created enough livepatches with the same source patch so that I
could reproduce the issue somewhat reliably.

I'll see if I can make something that uses klp directly.

The rest of the userspace in the initramfs is really straight forward
with the only interesting parts being a couple of shell scripts.

JE
 

  reply	other threads:[~2019-05-20 21:09 UTC|newest]

Thread overview: 20+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-05-20 19:49 Johannes Erdfelt
2019-05-20 20:46 ` Joe Lawrence
2019-05-20 21:09   ` Johannes Erdfelt [this message]
2019-05-20 21:19     ` Josh Poimboeuf
2019-05-20 21:39       ` Steven Rostedt
2019-05-21 14:16         ` Josh Poimboeuf
2019-05-21 14:42           ` Steven Rostedt
2019-05-21 16:42             ` Josh Poimboeuf
2019-05-21 16:53               ` Steven Rostedt
2019-05-29 11:17                 ` Jiri Kosina
2019-05-29 12:06                   ` Steven Rostedt
2019-05-29 12:30                     ` Josh Poimboeuf
2019-05-22 13:00               ` Josh Poimboeuf
2019-05-29 17:29               ` Jessica Yu
2019-05-29 17:39                 ` Josh Poimboeuf
2019-05-29 18:34                   ` Jessica Yu
2019-05-20 21:48       ` Johannes Erdfelt
2019-05-20 21:19     ` Joe Lawrence
2019-05-21 19:27       ` Joe Lawrence
2019-05-21 21:00         ` Josh Poimboeuf

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20190520210905.GC1646@sventech.com \
    --to=johannes@erdfelt.com \
    --cc=jeyu@kernel.org \
    --cc=jikos@kernel.org \
    --cc=joe.lawrence@redhat.com \
    --cc=jpoimboe@redhat.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=live-patching@vger.kernel.org \
    --cc=mbenes@suse.cz \
    --cc=mingo@redhat.com \
    --cc=rostedt@goodmis.org \
    --subject='Re: Oops caused by race between livepatch and ftrace' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).