LKML Archive on
help / color / mirror / Atom feed
From: Lukas Bulwahn <>
To: Thorsten Leemhuis <>,
	Paul Albertella <>
Cc: Dmitry Vyukov <>,, LKML <>,
	Greg Kroah-Hartman <>,
	Guillaume Tucker <>,,
	Sasha Levin <>,
	Marco Elver <>,
	syzkaller <>,
	Mara Mihali <>
Subject: Re: finding regressions with syzkaller
Date: Wed, 22 Sep 2021 13:21:18 +0200	[thread overview]
Message-ID: <> (raw)
In-Reply-To: <>

On Wed, Aug 11, 2021 at 1:25 PM Thorsten Leemhuis <> wrote:
> [CCing Lukas]
> Hi Dmitry!
> On 10.08.21 19:08, Dmitry Vyukov wrote:
> > [...]
> > The idea is to generate random test programs (as syzkaller does) and
> > then execute them on 2 different kernels and compare results (so
> > called "differential fuzzing"). This has the potential of finding not
> > just various "crashes" but also logical bugs and regressions.
> Hmmm, interesting concept!
> > The major issue is various false positive differences caused by
> > timings, non-determinism, accumulated state, intentional and
> > semi-intentional changes (e.g. subtle API extensions), etc. We learnt
> > how to deal with some of these to some degree, but feasibility is
> > still an open question.
> Sounds complicated and like a lot of manual work.
> Do you have in mind that Linus and hence many other Kernel developers
> afaics only care about regressions someone actually observed in a
> practice? Like a software or script breaking due to a kernel-side change?
> To quote Linus from
> ```The Linux "no regressions" rule is not about some theoretical
> "the ABI changed". It's about actual observed regressions.
> So if we can improve the ABI without any user program or workflow
> breaking, that's fine.```
> His stance on that afaik has not changed since then.
> Thus after ruling our all false positives syzkaller might find, there
> will always be the follow-up question "well, does anything/anyone
> actually care?". That might be hard to answer and requires yet more
> manual work by some human. Maybe this working hours at least for now are
> better spend in other areas.
> > Since this work is in very early stage, I only have very high-level questions:
> >  - what do you think about feasibility/usefulness of this idea in general?
> TBH I'm a bit sceptical due to the above factors. Don't get me wrong,
> making syzkaller look out for regressions sounds great, but I wonder if
> there are more pressing issues that are worth getting at first.
> Another aspect: CI testing already finds quite a few regressions, but
> those that are harder to catch are afaics often in driver code. And you
> often can't test that without the hardware, which makes me assume that
> syzkaller wouldn't help here (or am I wrong?)
> >  - any suggestions on how to make the tool find more differences/bugs
> > or how to make it more reliable?
> >  - is there a list or pointers to some known past regressions that
> > would be useful to find with such tool? (I've looked at the things
> > reported on the regressions@ list, but it's mostly crashes/not
> > booting, but that's what syzkaller can find already well)
> I first wanted to tell you "look up the reports I compiled in 2017 in
> the LKML archives", but I guess the way better solution is: just grep
> for "regression" in the commit log.
> >  - anybody else we should CC?
> I guess the people from the Elisa project might be interested in this,
> that's why I CCed Lukas.

Thanks, Thorsten. I do follow the syzkaller mailing list, so I have
seen that email before, but I do appreciate your implicit
acknowledgement here :)

... and Dmitry is back from vacation and I guess we will hear more
today at the Testing and Fuzzing MC on this topic.

Further people/lists to CC are: Paul Albertella
<> (already CCed here)

I am personally certainly interested and I think this work gives
companies in the area of building trustable software and systems (see
Paul's area of expertise) a good understanding how reliable and to
which extent the statement "all Linux kernels are backwards
compatible" really holds.

I unfortunately lost the Fuzzing Team (Jouni Högander, Jukka Kaartinen
et al.) previously working with me, and I need to first get back some
budget, build up a new team and hope that we can then also follow this
idea and contribute here as well. (Fingers crossed that I can convince
some others to give me money and work with me on this...)

Looking forward to the presentation at the MC.

Best regards,


      parent reply	other threads:[~2021-09-22 11:21 UTC|newest]

Thread overview: 4+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-08-10 17:08 finding regressions with syzkaller Dmitry Vyukov
2021-08-11 11:25 ` Thorsten Leemhuis
2021-08-12  9:15   ` Dmitry Vyukov
2021-09-22 11:21   ` Lukas Bulwahn [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \ \ \ \ \ \ \ \ \ \ \ \ \ \ \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).