LKML Archive on lore.kernel.org
help / color / mirror / Atom feed
From: Tyler Baker <tyler.baker@linaro.org>
To: John Stultz <john.stultz@linaro.org>
Cc: Prarit Bhargava <prarit@redhat.com>,
	lkml <linux-kernel@vger.kernel.org>,
	Shuah Khan <shuahkh@osg.samsung.com>,
	Thomas Gleixner <tglx@linutronix.de>,
	Richard Cochran <richardcochran@gmail.com>
Subject: Re: [PATCH 1/2] kselftests: timers: Make set-timer-lat fail more gracefully for !CAP_WAKE_ALARM
Date: Thu, 26 Mar 2015 10:33:10 -0700	[thread overview]
Message-ID: <CANMBJr723yJe-rENZdGQBW6cjhQr=jW75waXQQcaW-qKcWarEg@mail.gmail.com> (raw)
In-Reply-To: <CALAqxLW65kBPW5knaeGALJnLbGvjRP9Pai6ZiBbePHM1cOppLg@mail.gmail.com>

On 26 March 2015 at 09:29, John Stultz <john.stultz@linaro.org> wrote:
> On Thu, Mar 26, 2015 at 4:31 AM, Prarit Bhargava <prarit@redhat.com> wrote:
>> On 03/25/2015 07:44 PM, John Stultz wrote:
>>> +                     printf("%-22s %s missing CAP_WAKE_ALARM?    : [UNSUPPORTED]\n",
>>> +                                     clockstring(clock_id),
>>> +                                     flags ? "ABSTIME":"RELTIME");
>>
>> Something to think about:  Do you want to write these tests to be more human
>> readable or machine readable?  In theory with awk I guess it doesn't matter too
>> much, however, it is something that we should think about moving forward.
>
> So this came up at ELC in a few discussions. Right now there isn't any
> established output format, but there's some nice and simple
> infrastructure for counting pass/fails.
>
> However, in talking to Tyler, I know he has started looking at how to
> integrate the selftests into our automated infrastructure and was
> interested in how we improve the output parsing for reports. So there
> is interest in improving this, and I'm open to whatever changes might
> be needed (adding extra arguments to the test to put them into "easy
> parse" mode or whatever).

Thanks for looping me in John. My interest in kselftest stems from my
involvement with kernelci.org, a communityservice focused on upstream
kernel validation across multiple architectures. In it's current form,
it is merely build and boot testing boards. However, we are at a point
where we'd like to start running some tests. The automation framework
(LAVA) used to execute these tests essentially uses a regular
expression to parse the test's standard output. This is advantageous
as a test can be written in any language, as long as it produces sane
uniform output.

Ideally, we would like to perform the kernel builds as we do today
along with building all the kseltests present in the tree, and
inserting them into a 'testing' ramdisk for deployment. Once we
successfully boot the platform, we execute all the kselftests, parse
standard out, and report the results. The benefit from this
implementation is that a developer writing a test does have to do
anything 'special' to get his/her test to run once it has been applied
to a upstream tree. I'll explain below some concerns I have about
accomplishing this.

Currently, we have had to write wrappers[1][2] for some kselftests to
be able parse the output. If we can choose/agree on a standard output
format all of this complexity goes away, and then we can dynamically
run kselftests. Integration of new tests will not be needed, as they
all produce output in standard way. I've taken a look at the wiki page
for standardizing output[3] and TAP looks like the good format IMO.

Also, for arch != x86 there are some barriers to overcome to get all
the kselftests cross compiling, which would be nice to have as well.

I realize this may be a good amount of work, so I'd like to help out.
Perhaps working John to convert his timer tests to use TAP output
would be a good starting point?

>
> thanks
> -john

[1] https://git.linaro.org/qa/test-definitions.git/blob/HEAD:/common/scripts/kselftest-runner.sh
[2] https://git.linaro.org/qa/test-definitions.git/blob/HEAD:/common/scripts/kselftest-mqueue.sh
[3] https://kselftest.wiki.kernel.org/standardize_the_test_output

Cheers,

Tyler

  reply	other threads:[~2015-03-26 17:33 UTC|newest]

Thread overview: 20+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2015-03-25 23:44 John Stultz
2015-03-25 23:44 ` [PATCH 2/2] kselftests: timers: Reduce default runtime on inconsistency-check and set-timer-lat John Stultz
2015-03-26 11:32   ` Prarit Bhargava
2015-03-26 16:20     ` John Stultz
2015-03-31 16:01       ` Shuah Khan
2015-03-31 19:47         ` Shuah Khan
2015-03-26 11:31 ` [PATCH 1/2] kselftests: timers: Make set-timer-lat fail more gracefully for !CAP_WAKE_ALARM Prarit Bhargava
2015-03-26 16:29   ` John Stultz
2015-03-26 17:33     ` Tyler Baker [this message]
2015-04-02 10:18       ` Prarit Bhargava
2015-04-02 13:43         ` Shuah Khan
2015-04-02 17:17           ` Tyler Baker
2015-04-02 17:48             ` Shuah Khan
2015-04-02 18:58               ` Tyler Baker
2015-04-02 18:02         ` John Stultz
2015-04-07 14:20           ` Prarit Bhargava
2015-04-08  4:03           ` Michael Ellerman
2015-04-02 10:14     ` Prarit Bhargava
2015-03-31 15:55   ` Shuah Khan
2015-04-02  3:42     ` John Stultz

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CANMBJr723yJe-rENZdGQBW6cjhQr=jW75waXQQcaW-qKcWarEg@mail.gmail.com' \
    --to=tyler.baker@linaro.org \
    --cc=john.stultz@linaro.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=prarit@redhat.com \
    --cc=richardcochran@gmail.com \
    --cc=shuahkh@osg.samsung.com \
    --cc=tglx@linutronix.de \
    --subject='Re: [PATCH 1/2] kselftests: timers: Make set-timer-lat fail more gracefully for '\!'CAP_WAKE_ALARM' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).