LKML Archive on lore.kernel.org help / color / mirror / Atom feed
From: Thorsten Leemhuis <email@example.com> To: Dmitry Vyukov <firstname.lastname@example.org>, email@example.com Cc: LKML <firstname.lastname@example.org>, Greg Kroah-Hartman <email@example.com>, Guillaume Tucker <firstname.lastname@example.org>, email@example.com, Sasha Levin <firstname.lastname@example.org>, Marco Elver <email@example.com>, syzkaller <firstname.lastname@example.org>, Mara Mihali <email@example.com>, Lukas Bulwahn <firstname.lastname@example.org> Subject: Re: finding regressions with syzkaller Date: Wed, 11 Aug 2021 13:25:54 +0200 [thread overview] Message-ID: <email@example.com> (raw) In-Reply-To: <CACT4Y+YvovgRNC5EFhN_d=jApwSAsWcNj35=FCJf1k867vBqfw@mail.gmail.com> [CCing Lukas] Hi Dmitry! On 10.08.21 19:08, Dmitry Vyukov wrote: > [...] > The idea is to generate random test programs (as syzkaller does) and > then execute them on 2 different kernels and compare results (so > called "differential fuzzing"). This has the potential of finding not > just various "crashes" but also logical bugs and regressions. Hmmm, interesting concept! > The major issue is various false positive differences caused by > timings, non-determinism, accumulated state, intentional and > semi-intentional changes (e.g. subtle API extensions), etc. We learnt > how to deal with some of these to some degree, but feasibility is > still an open question. Sounds complicated and like a lot of manual work. Do you have in mind that Linus and hence many other Kernel developers afaics only care about regressions someone actually observed in a practice? Like a software or script breaking due to a kernel-side change? To quote Linus from https://lore.kernel.org/lkml/CA+55aFx3RswnjmCErk8QhCo0KrCvxZnuES3WALBR1NkPbUZ8qw@mail.gmail.com/ ```The Linux "no regressions" rule is not about some theoretical "the ABI changed". It's about actual observed regressions. So if we can improve the ABI without any user program or workflow breaking, that's fine.``` His stance on that afaik has not changed since then. Thus after ruling our all false positives syzkaller might find, there will always be the follow-up question "well, does anything/anyone actually care?". That might be hard to answer and requires yet more manual work by some human. Maybe this working hours at least for now are better spend in other areas. > Since this work is in very early stage, I only have very high-level questions: > - what do you think about feasibility/usefulness of this idea in general? TBH I'm a bit sceptical due to the above factors. Don't get me wrong, making syzkaller look out for regressions sounds great, but I wonder if there are more pressing issues that are worth getting at first. Another aspect: CI testing already finds quite a few regressions, but those that are harder to catch are afaics often in driver code. And you often can't test that without the hardware, which makes me assume that syzkaller wouldn't help here (or am I wrong?) > - any suggestions on how to make the tool find more differences/bugs > or how to make it more reliable? > - is there a list or pointers to some known past regressions that > would be useful to find with such tool? (I've looked at the things > reported on the regressions@ list, but it's mostly crashes/not > booting, but that's what syzkaller can find already well) I first wanted to tell you "look up the reports I compiled in 2017 in the LKML archives", but I guess the way better solution is: just grep for "regression" in the commit log. > - anybody else we should CC? I guess the people from the Elisa project might be interested in this, that's why I CCed Lukas. Ciao, Thorsten
next prev parent reply other threads:[~2021-08-11 11:46 UTC|newest] Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top 2021-08-10 17:08 finding regressions with syzkaller Dmitry Vyukov 2021-08-11 11:25 ` Thorsten Leemhuis [this message] 2021-08-12 9:15 ` Dmitry Vyukov 2021-09-22 11:21 ` Lukas Bulwahn
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).