LKML Archive on lore.kernel.org
help / color / mirror / Atom feed
From: Prarit Bhargava <prarit@redhat.com>
To: John Stultz <john.stultz@linaro.org>
Cc: Tyler Baker <tyler.baker@linaro.org>,
	lkml <linux-kernel@vger.kernel.org>,
	Shuah Khan <shuahkh@osg.samsung.com>,
	Thomas Gleixner <tglx@linutronix.de>,
	Richard Cochran <richardcochran@gmail.com>
Subject: Re: [PATCH 1/2] kselftests: timers: Make set-timer-lat fail more gracefully for !CAP_WAKE_ALARM
Date: Tue, 07 Apr 2015 10:20:39 -0400	[thread overview]
Message-ID: <5523E7B7.6010106@redhat.com> (raw)
In-Reply-To: <CALAqxLUbhaq4+KTrjj=_FG7xq-yzME44noXhgn1ND0C-6HBdOw@mail.gmail.com>



On 04/02/2015 02:02 PM, John Stultz wrote:
> On Thu, Apr 2, 2015 at 3:18 AM, Prarit Bhargava <prarit@redhat.com> wrote:
>> On 03/26/2015 01:33 PM, Tyler Baker wrote:
>>> I realize this may be a good amount of work, so I'd like to help out.
>>> Perhaps working John to convert his timer tests to use TAP output
>>> would be a good starting point?
>>
>> John, I could probably do that for you.  I'm always willing to give it a shot.
> 
> I took a quick look into it, since I'm definitely interested in
> improving output formatting, but man, TAP is a fairly ugly output
> format if you ask me.
> 
> It only has binary "ok" or "not ok" (why not "fail", or something else
> that's exclusively grep-able, I don't know). So I'm not sure if cases
> where functionality is unsupported should be a pass or fail.
> 
> Most problematically: It seems to want enumeration in the test output
> (so test 2 needs to print: "ok 2 Test complete") which means either
> there needs to be a wrapper that does the TAP output knowing which
> test of N its currently running, or the test number needs to be
> submitted as an runtime argument to the test, and the test then has to
> add it to its output line.
> 
> Anyway, if we do want to go with that format,  I suspect it should be
> something we add to the kselftest pass/fail hooks, rather then to the
> individual tests. Then its just a matter of prefixing normal test
> output with #'s so they can be ignored by the parser.

I've been looking into this and while TAP is useful, I find myself with a few
questions:

1.  I want the output to be both human and machine readable.  I find that "not
okay" vs "okay" is confusing at best for humans to read.  I suppose that is a
minor issue, but it is a real issue when laymen are asked to execute a test.

2.  I think we can get around the TAP numbering, however, maybe a better
approach is to limit tests to doing one, and only one, test?  ie) something like
rtctest.c would have be be broken up into individual tests.

3.  TAP appears to have a BSD-style license?  Not knowing enough about cross
licensing issues like this, is that going to be a problem?

4.  I think John is moving in the right direction by suggesting that we move to
putting this into kselftest.h as hooks, rather than modifying each test.
Another idea is to include a script that allows the running of multiple tests
from the top level tests directory.

Let me muck around with the timer tests and see what I can come up with.  I'm
trying to keep in mind that we're going to be asking people to potentially
modify their tests, as well as what we're going to be telling people to do when
they add new tests, while I make modifications.

P.


> 
> thanks
> -john
> 

  reply	other threads:[~2015-04-07 14:20 UTC|newest]

Thread overview: 20+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2015-03-25 23:44 John Stultz
2015-03-25 23:44 ` [PATCH 2/2] kselftests: timers: Reduce default runtime on inconsistency-check and set-timer-lat John Stultz
2015-03-26 11:32   ` Prarit Bhargava
2015-03-26 16:20     ` John Stultz
2015-03-31 16:01       ` Shuah Khan
2015-03-31 19:47         ` Shuah Khan
2015-03-26 11:31 ` [PATCH 1/2] kselftests: timers: Make set-timer-lat fail more gracefully for !CAP_WAKE_ALARM Prarit Bhargava
2015-03-26 16:29   ` John Stultz
2015-03-26 17:33     ` Tyler Baker
2015-04-02 10:18       ` Prarit Bhargava
2015-04-02 13:43         ` Shuah Khan
2015-04-02 17:17           ` Tyler Baker
2015-04-02 17:48             ` Shuah Khan
2015-04-02 18:58               ` Tyler Baker
2015-04-02 18:02         ` John Stultz
2015-04-07 14:20           ` Prarit Bhargava [this message]
2015-04-08  4:03           ` Michael Ellerman
2015-04-02 10:14     ` Prarit Bhargava
2015-03-31 15:55   ` Shuah Khan
2015-04-02  3:42     ` John Stultz

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=5523E7B7.6010106@redhat.com \
    --to=prarit@redhat.com \
    --cc=john.stultz@linaro.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=richardcochran@gmail.com \
    --cc=shuahkh@osg.samsung.com \
    --cc=tglx@linutronix.de \
    --cc=tyler.baker@linaro.org \
    --subject='Re: [PATCH 1/2] kselftests: timers: Make set-timer-lat fail more gracefully for '\!'CAP_WAKE_ALARM' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).