LKML Archive on lore.kernel.org
help / color / mirror / Atom feed
* QA: Monitor Linux log messages as port of release (candidate) testing
@ 2021-09-07  8:40 Paul Menzel
  2021-09-07 12:53 ` Guenter Roeck
  0 siblings, 1 reply; 6+ messages in thread
From: Paul Menzel @ 2021-09-07  8:40 UTC (permalink / raw)
  To: Guenter Roeck; +Cc: LKML

Dear Guenter,


Thank you for testing release candidates and releases [1]. Is your test 
setup documented somewhere?

If not happening already, could the Linux messages (at least up to log 
level warning) also be monitored? For example, in Linux 5.14, a new 
warning snuck in by cefc7ca462 (ACPI: PRM: implement OperationRegion 
handler for the PlatformRtMechanism subtype) [2], which could have been 
caught early on, and fixed before the release.

The test summaries would then also notify about possible behavior change.


Kind regards,

Paul


[1]: https://lkml.org/lkml/2021/7/11/326
[2]: 
https://lore.kernel.org/linux-acpi/54b6f8cb-4714-587c-b2d0-98134473293d@linux.intel.com/T/#m3f54733714381765c8bb5bfa5b2aa3969407931e

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: QA: Monitor Linux log messages as port of release (candidate) testing
  2021-09-07  8:40 QA: Monitor Linux log messages as port of release (candidate) testing Paul Menzel
@ 2021-09-07 12:53 ` Guenter Roeck
  2021-09-07 13:50   ` Paul Menzel
  0 siblings, 1 reply; 6+ messages in thread
From: Guenter Roeck @ 2021-09-07 12:53 UTC (permalink / raw)
  To: Paul Menzel; +Cc: LKML

On Tue, Sep 07, 2021 at 10:40:31AM +0200, Paul Menzel wrote:
> Dear Guenter,
> 
> 
> Thank you for testing release candidates and releases [1]. Is your test
> setup documented somewhere?
> 
Not really, except its source is available at github:
	https://github.com/groeck/linux-build-test

> If not happening already, could the Linux messages (at least up to log level
> warning) also be monitored? For example, in Linux 5.14, a new warning snuck
> in by cefc7ca462 (ACPI: PRM: implement OperationRegion handler for the
> PlatformRtMechanism subtype) [2], which could have been caught early on, and
> fixed before the release.
> 
> The test summaries would then also notify about possible behavior change.
> 
Logs are available and can be examined at kerneltests.org/builders.
Reports are generated manually, so it would be way too much effort to add
build warnings to those. Besides, logs are way too noisy to be useful in a
summary e-mail.

Also, Geert's build reports already provide build warnings and errors.
The same applies to reports sent by 0-day. Indeed, I do see at least
one 0-day report against commit cefc7ca46235. What would be the point
of adding yet another report of build warnings on top of that ?

Thanks,
Guenter

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: QA: Monitor Linux log messages as port of release (candidate) testing
  2021-09-07 12:53 ` Guenter Roeck
@ 2021-09-07 13:50   ` Paul Menzel
  2021-09-07 14:47     ` Guenter Roeck
  0 siblings, 1 reply; 6+ messages in thread
From: Paul Menzel @ 2021-09-07 13:50 UTC (permalink / raw)
  To: Guenter Roeck; +Cc: LKML

Dear Guenter,


Am 07.09.21 um 14:53 schrieb Guenter Roeck:
> On Tue, Sep 07, 2021 at 10:40:31AM +0200, Paul Menzel wrote:

>> Thank you for testing release candidates and releases [1]. Is your test
>> setup documented somewhere?
>>
> Not really, except its source is available at github:
> 	https://github.com/groeck/linux-build-test

Thank you.

>> If not happening already, could the Linux messages (at least up to log level
>> warning) also be monitored? For example, in Linux 5.14, a new warning snuck
>> in by cefc7ca462 (ACPI: PRM: implement OperationRegion handler for the
>> PlatformRtMechanism subtype) [2], which could have been caught early on, and
>> fixed before the release.
>>
>> The test summaries would then also notify about possible behavior change.
>
> Logs are available and can be examined at kerneltests.org/builders.

Sorry for being blind. Under *qemu-tests*, looking at build #1831 [1], 
clicking on *stdio* [2] under *Steps and Logfiles*, I do not see any 
Linux logs.

> Reports are generated manually, so it would be way too much effort to add
> build warnings to those. Besides, logs are way too noisy to be useful in a
> summary e-mail.

Just to avoid misunderstandings, it’s about the Linux run-time logs.

> Also, Geert's build reports already provide build warnings and errors.
> The same applies to reports sent by 0-day. Indeed, I do see at least
> one 0-day report against commit cefc7ca46235.

How can I find that report?

> What would be the point of adding yet another report of build
> warnings on top of that ?
If the functionality already exists, great. But to be clear, it’s about 
the runtime logs.


Kind regards,

Paul


[1]: https://kerneltests.org/builders/hwmon-x86_64-master/builds/1831
[2]: 
https://kerneltests.org/builders/qemu-x86_64-master/builds/1831/steps/qemubuildcommand/logs/stdio

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: QA: Monitor Linux log messages as port of release (candidate) testing
  2021-09-07 13:50   ` Paul Menzel
@ 2021-09-07 14:47     ` Guenter Roeck
  2021-09-07 15:18       ` Paul Menzel
  0 siblings, 1 reply; 6+ messages in thread
From: Guenter Roeck @ 2021-09-07 14:47 UTC (permalink / raw)
  To: Paul Menzel; +Cc: LKML

Hi Paul,

On Tue, Sep 07, 2021 at 03:50:39PM +0200, Paul Menzel wrote:
> Dear Guenter,
> 
> 
> Am 07.09.21 um 14:53 schrieb Guenter Roeck:
> > On Tue, Sep 07, 2021 at 10:40:31AM +0200, Paul Menzel wrote:
> 
> > > Thank you for testing release candidates and releases [1]. Is your test
> > > setup documented somewhere?
> > > 
> > Not really, except its source is available at github:
> > 	https://github.com/groeck/linux-build-test
> 
> Thank you.
> 
> > > If not happening already, could the Linux messages (at least up to log level
> > > warning) also be monitored? For example, in Linux 5.14, a new warning snuck
> > > in by cefc7ca462 (ACPI: PRM: implement OperationRegion handler for the
> > > PlatformRtMechanism subtype) [2], which could have been caught early on, and
> > > fixed before the release.
> > > 
> > > The test summaries would then also notify about possible behavior change.
> > 
> > Logs are available and can be examined at kerneltests.org/builders.
> 
> Sorry for being blind. Under *qemu-tests*, looking at build #1831 [1],
> clicking on *stdio* [2] under *Steps and Logfiles*, I do not see any Linux
> logs.
> 
> > Reports are generated manually, so it would be way too much effort to add
> > build warnings to those. Besides, logs are way too noisy to be useful in a
> > summary e-mail.
> 
> Just to avoid misunderstandings, it’s about the Linux run-time logs.
> 
Run-time logs are only provided if there are errors or runtime issues
(crashes, warning tracebacks, or test failures).

> > Also, Geert's build reports already provide build warnings and errors.
> > The same applies to reports sent by 0-day. Indeed, I do see at least
> > one 0-day report against commit cefc7ca46235.
> 
> How can I find that report?
> 
I just searched for the SHA.

https://www.spinics.net/lists/linux-acpi/msg101721.html

> > What would be the point of adding yet another report of build
> > warnings on top of that ?
> If the functionality already exists, great. But to be clear, it’s about the
> runtime logs.
> 

If there are (new) runtime issues (crashes, warnings, or other test failures),
I usually analyze, bisect, and report the problem against the patch introducing
the problem unless it was already reported elsewhere.

For example, there is currently a backtrace in arm64 tests:

BUG: sleeping function called from invalid context at kernel/locking/semaphore.c:163

which is due to ACPI code being called from the wrong context.
Another example is the cirular locking problem reported in various
mips tests, from mtd code. Both problems have been fixed in -next,
and the fixes will hopefully be pushed upstream soon.

The qemu tests do not log build warnings. I used to do that, but it
was way too noisy (some builds used to produce hundreds of build
warnings). Also, there is no logging data if there are neither
crashes nor warnings or other test failures, for the same reason.
If you are looking for complete boot logs, I'd suggest looking at
test results from kernelci.org.

Thanks,
Guenter

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: QA: Monitor Linux log messages as port of release (candidate) testing
  2021-09-07 14:47     ` Guenter Roeck
@ 2021-09-07 15:18       ` Paul Menzel
  2021-09-07 15:28         ` Guenter Roeck
  0 siblings, 1 reply; 6+ messages in thread
From: Paul Menzel @ 2021-09-07 15:18 UTC (permalink / raw)
  To: Guenter Roeck; +Cc: LKML

Dear Guenter,


Am 07.09.21 um 16:47 schrieb Guenter Roeck:

> On Tue, Sep 07, 2021 at 03:50:39PM +0200, Paul Menzel wrote:

>> Am 07.09.21 um 14:53 schrieb Guenter Roeck:
>>> On Tue, Sep 07, 2021 at 10:40:31AM +0200, Paul Menzel wrote:
>>
>>>> Thank you for testing release candidates and releases [1]. Is your test
>>>> setup documented somewhere?
>>>>
>>> Not really, except its source is available at github:
>>> 	https://github.com/groeck/linux-build-test
>>
>> Thank you.
>>
>>>> If not happening already, could the Linux messages (at least up to log level
>>>> warning) also be monitored? For example, in Linux 5.14, a new warning snuck
>>>> in by cefc7ca462 (ACPI: PRM: implement OperationRegion handler for the
>>>> PlatformRtMechanism subtype) [2], which could have been caught early on, and
>>>> fixed before the release.
>>>>
>>>> The test summaries would then also notify about possible behavior change.
>>>
>>> Logs are available and can be examined at kerneltests.org/builders.
>>
>> Sorry for being blind. Under *qemu-tests*, looking at build #1831 [1],
>> clicking on *stdio* [2] under *Steps and Logfiles*, I do not see any Linux
>> logs.
>>
>>> Reports are generated manually, so it would be way too much effort to add
>>> build warnings to those. Besides, logs are way too noisy to be useful in a
>>> summary e-mail.
>>
>> Just to avoid misunderstandings, it’s about the Linux run-time logs.
>
> Run-time logs are only provided if there are errors or runtime issues
> (crashes, warning tracebacks, or test failures).

Could this be changed to always publish them? Or is that too demanding 
on storage?

>>> Also, Geert's build reports already provide build warnings and errors.
>>> The same applies to reports sent by 0-day. Indeed, I do see at least
>>> one 0-day report against commit cefc7ca46235.
>>
>> How can I find that report?
>>
> I just searched for the SHA.
> 
> https://www.spinics.net/lists/linux-acpi/msg101721.html

Alright, that is a build time thing. I am looking for runtime things.

>>> What would be the point of adding yet another report of build
>>> warnings on top of that ?
>> If the functionality already exists, great. But to be clear, it’s about the
>> runtime logs.
> 
> If there are (new) runtime issues (crashes, warnings, or other test failures),
> I usually analyze, bisect, and report the problem against the patch introducing
> the problem unless it was already reported elsewhere.
> 
> For example, there is currently a backtrace in arm64 tests:
> 
> BUG: sleeping function called from invalid context at kernel/locking/semaphore.c:163
> 
> which is due to ACPI code being called from the wrong context.
> Another example is the cirular locking problem reported in various
> mips tests, from mtd code. Both problems have been fixed in -next,
> and the fixes will hopefully be pushed upstream soon.
> 
> The qemu tests do not log build warnings. I used to do that, but it
> was way too noisy (some builds used to produce hundreds of build
> warnings). Also, there is no logging data if there are neither
> crashes nor warnings or other test failures, for the same reason.

Thank you for the clarification and your QA work. Sorry about the 
misunderstanding about compile/build logs and runtime logs.

> If you are looking for complete boot logs, I'd suggest looking at
> test results from kernelci.org.

Thank you. I am going to do that then.


Kind regards,

Paul

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: QA: Monitor Linux log messages as port of release (candidate) testing
  2021-09-07 15:18       ` Paul Menzel
@ 2021-09-07 15:28         ` Guenter Roeck
  0 siblings, 0 replies; 6+ messages in thread
From: Guenter Roeck @ 2021-09-07 15:28 UTC (permalink / raw)
  To: Paul Menzel; +Cc: LKML

Hi Paul,

On 9/7/21 8:18 AM, Paul Menzel wrote:
> Dear Guenter,
> 
> 
> Am 07.09.21 um 16:47 schrieb Guenter Roeck:
> 
>> On Tue, Sep 07, 2021 at 03:50:39PM +0200, Paul Menzel wrote:
> 
>>> Am 07.09.21 um 14:53 schrieb Guenter Roeck:
>>>> On Tue, Sep 07, 2021 at 10:40:31AM +0200, Paul Menzel wrote:
>>>
>>>>> Thank you for testing release candidates and releases [1]. Is your test
>>>>> setup documented somewhere?
>>>>>
>>>> Not really, except its source is available at github:
>>>>     https://github.com/groeck/linux-build-test
>>>
>>> Thank you.
>>>
>>>>> If not happening already, could the Linux messages (at least up to log level
>>>>> warning) also be monitored? For example, in Linux 5.14, a new warning snuck
>>>>> in by cefc7ca462 (ACPI: PRM: implement OperationRegion handler for the
>>>>> PlatformRtMechanism subtype) [2], which could have been caught early on, and
>>>>> fixed before the release.
>>>>>
>>>>> The test summaries would then also notify about possible behavior change.
>>>>
>>>> Logs are available and can be examined at kerneltests.org/builders.
>>>
>>> Sorry for being blind. Under *qemu-tests*, looking at build #1831 [1],
>>> clicking on *stdio* [2] under *Steps and Logfiles*, I do not see any Linux
>>> logs.
>>>
>>>> Reports are generated manually, so it would be way too much effort to add
>>>> build warnings to those. Besides, logs are way too noisy to be useful in a
>>>> summary e-mail.
>>>
>>> Just to avoid misunderstandings, it’s about the Linux run-time logs.
>>
>> Run-time logs are only provided if there are errors or runtime issues
>> (crashes, warning tracebacks, or test failures).
> 
> Could this be changed to always publish them? Or is that too demanding on storage?
> 

It isn't a matter of storage, but of noise. It is mostly me looking at the data,
and I really don't want to see logs if nothing interesting is there. With 450+
builds per release I'd drown in noise otherwise. Sure, a large part of that
is a UI issue, but that is where systems like kernelci come in. If I ever
find the time to do it, I might report build and runtime results/logs to kernelci.
But that is a big IF.

>>>> Also, Geert's build reports already provide build warnings and errors.
>>>> The same applies to reports sent by 0-day. Indeed, I do see at least
>>>> one 0-day report against commit cefc7ca46235.
>>>
>>> How can I find that report?
>>>
>> I just searched for the SHA.
>>
>> https://www.spinics.net/lists/linux-acpi/msg101721.html
> 
> Alright, that is a build time thing. I am looking for runtime things.
> 

If there is nothing in my reports, you'd probably not find what you
are looking for (it would be reported if it results in a backtrace).

Guenter

^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2021-09-07 15:28 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-09-07  8:40 QA: Monitor Linux log messages as port of release (candidate) testing Paul Menzel
2021-09-07 12:53 ` Guenter Roeck
2021-09-07 13:50   ` Paul Menzel
2021-09-07 14:47     ` Guenter Roeck
2021-09-07 15:18       ` Paul Menzel
2021-09-07 15:28         ` Guenter Roeck

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).