[PATCH 0/3] A kernel tracing interface

[PATCH 0/3] A kernel tracing interface

[David J. Wilder]

These patches provide a kernel tracing interface called "trace".

(update) Updated documentation and added configuration dependency
on KPROBES for trace sample code.

The motivation for "trace" is to:
- Provide a simple set of tracing primitives that will utilize the high-
  performance and low-overhead of relayfs for passing traces data from
  kernel to user space.
- Provide a common user interface for managing kernel traces.
- Allow for binary as well as ascii trace data.
- Incorporate features from the systemtap runtime that are
  useful to others.

Patches are against 2.6.23-rc8-mm2

Summary of patches:
[patch 1/3]  Trace code and documentation
[patch 2/3]  Relay Reset Consumed
[patch 3/3]  Trace sample

Note: Patch 2/3 provides a service used by patch 1/3
thus must be applied first or at the same time.

Note: The following patches must be applied prior to 3/3.
[patch 3/5] Add samples subdir
	http://lkml.org/lkml/2007/9/28/178

[patch 4/5] Linux Kernel Markers - Samples
	http://lkml.org/lkml/2007/9/28/189

Signed-off-by: David Wilder <dwilder@us.ibm.com>

Re: [PATCH 0/3] A kernel tracing interface

[K. Prasad]

On Fri, Feb 29, 2008 at 11:06:05AM +0530, K. Prasad wrote:
> On Thu, Feb 28, 2008 at 11:40:21AM -0800, David Wilder wrote:
> > Andrew Morton wrote:
> > >On Tue, 26 Feb 2008 14:38:37 -0800 "David J. Wilder" <dwilder@us.ibm.com> 
> > >wrote:
> > >
> > >>These patches provide a kernel tracing interface called "trace".
> > >>
> > >>ChangeLog:
> > >>-Updated to 2.6.25-rc2-mm1
> > >>-Removed sem_watch example, to hard to maintain.
> > >>-Added a new example that demonstrates per-cpu continuous tracing
> > >> of data generated using marker probes.
> > >>-Removed inline from relay patch.
> > >>-Moved examples into /sample directory.
> > >>
> > >>The motivation for "trace" is to:
> > >>- Provide a simple set of tracing primitives that will utilize the high-
> > >>  performance and low-overhead of relayfs for passing traces data from
> > >>  kernel to user space.
> > >>- Provide a common user interface for managing kernel traces.
> > >>- Allow for binary as well as ascii trace data.
> > >>- Incorporate features from the systemtap runtime that are
> > >>  useful to others.
> > >
> > >So... what's the story on this versus lttng?
> > 
> > Trace is simpler to use and a smaller bit of code than lttng.
> > It may not have all the features of lttng but it is good starting point 
> > that can be expanded upon.  I have several users of trace that just 
> > wanted a simple way to get data out of the kernel without the overhead 
> > of lttng trace is working well for them.  Trace also works well with 
> > markers and kprobes.  For example systemtap is is using the basic trace 
> > code to gather data from kprobes and markers.
> > >
> > >Is there some userspace code available for people to test this?  Maybe it's
> > >mentioned and I missed it.
> > 
> > I purposely designed trace so no special user code is required to gather 
> > trace data. Standard user utilities like cat, grep, sort and more is all 
> > that is needed.  I show an example in /samples/trace/fork_trace.c.
> > 
> > However since trace uses relay as its transport any user interfaces 
> > provided by relay can be used to read trace data. (documented in 
> > /Documentation/relay.txt)
> > 
> > >
> > >It'd be interesting to see a writeup of the proposed kernel<->userspace
> > >interfaces.  I see a description of kernel-internal interfaces, but how do
> > >users use it?
> > 
> > I will add some text in the documentation.
> > >
> > >I see it uses things from blktrace.  Can blktrace be switched over to being
> > >a client of this code?
> > 
> > yep.
> >
> Dave,
> 	I might also want to add to the discussion by stating that you
> will find a user for the 'trace' infrastructure in the marker handler 
> code of the proposed markers for futex subsystem. I hope to post the 
> patches soon to the community.
>
Also to let you know that I tested the patches (sent against
2.6.25-rc2-mm1) on a x86 machine and they've been working fine....

Thanks,
K.Prasad

Re: [PATCH 0/3] A kernel tracing interface

[K. Prasad]

On Thu, Feb 28, 2008 at 11:40:21AM -0800, David Wilder wrote:
> Andrew Morton wrote:
> >On Tue, 26 Feb 2008 14:38:37 -0800 "David J. Wilder" <dwilder@us.ibm.com> 
> >wrote:
> >
> >>These patches provide a kernel tracing interface called "trace".
> >>
> >>ChangeLog:
> >>-Updated to 2.6.25-rc2-mm1
> >>-Removed sem_watch example, to hard to maintain.
> >>-Added a new example that demonstrates per-cpu continuous tracing
> >> of data generated using marker probes.
> >>-Removed inline from relay patch.
> >>-Moved examples into /sample directory.
> >>
> >>The motivation for "trace" is to:
> >>- Provide a simple set of tracing primitives that will utilize the high-
> >>  performance and low-overhead of relayfs for passing traces data from
> >>  kernel to user space.
> >>- Provide a common user interface for managing kernel traces.
> >>- Allow for binary as well as ascii trace data.
> >>- Incorporate features from the systemtap runtime that are
> >>  useful to others.
> >
> >So... what's the story on this versus lttng?
> 
> Trace is simpler to use and a smaller bit of code than lttng.
> It may not have all the features of lttng but it is good starting point 
> that can be expanded upon.  I have several users of trace that just 
> wanted a simple way to get data out of the kernel without the overhead 
> of lttng trace is working well for them.  Trace also works well with 
> markers and kprobes.  For example systemtap is is using the basic trace 
> code to gather data from kprobes and markers.
> >
> >Is there some userspace code available for people to test this?  Maybe it's
> >mentioned and I missed it.
> 
> I purposely designed trace so no special user code is required to gather 
> trace data. Standard user utilities like cat, grep, sort and more is all 
> that is needed.  I show an example in /samples/trace/fork_trace.c.
> 
> However since trace uses relay as its transport any user interfaces 
> provided by relay can be used to read trace data. (documented in 
> /Documentation/relay.txt)
> 
> >
> >It'd be interesting to see a writeup of the proposed kernel<->userspace
> >interfaces.  I see a description of kernel-internal interfaces, but how do
> >users use it?
> 
> I will add some text in the documentation.
> >
> >I see it uses things from blktrace.  Can blktrace be switched over to being
> >a client of this code?
> 
> yep.
>
Dave,
	I might also want to add to the discussion by stating that you
will find a user for the 'trace' infrastructure in the marker handler 
code of the proposed markers for futex subsystem. I hope to post the 
patches soon to the community.

Thanks,
K.Prasad
 

Re: [PATCH 0/3] A kernel tracing interface

[David Wilder]

Andrew Morton wrote:
> On Tue, 26 Feb 2008 14:38:37 -0800 "David J. Wilder" <dwilder@us.ibm.com> wrote:
> 
>> These patches provide a kernel tracing interface called "trace".
>>
>> ChangeLog:
>> -Updated to 2.6.25-rc2-mm1
>> -Removed sem_watch example, to hard to maintain.
>> -Added a new example that demonstrates per-cpu continuous tracing
>>  of data generated using marker probes.
>> -Removed inline from relay patch.
>> -Moved examples into /sample directory.
>>
>> The motivation for "trace" is to:
>> - Provide a simple set of tracing primitives that will utilize the high-
>>   performance and low-overhead of relayfs for passing traces data from
>>   kernel to user space.
>> - Provide a common user interface for managing kernel traces.
>> - Allow for binary as well as ascii trace data.
>> - Incorporate features from the systemtap runtime that are
>>   useful to others.
> 
> So... what's the story on this versus lttng?

Trace is simpler to use and a smaller bit of code than lttng.
It may not have all the features of lttng but it is good starting point 
that can be expanded upon.  I have several users of trace that just 
wanted a simple way to get data out of the kernel without the overhead 
of lttng trace is working well for them.  Trace also works well with 
markers and kprobes.  For example systemtap is is using the basic trace 
code to gather data from kprobes and markers.
> 
> Is there some userspace code available for people to test this?  Maybe it's
> mentioned and I missed it.

I purposely designed trace so no special user code is required to gather 
trace data. Standard user utilities like cat, grep, sort and more is all 
that is needed.  I show an example in /samples/trace/fork_trace.c.

However since trace uses relay as its transport any user interfaces 
provided by relay can be used to read trace data. (documented in 
/Documentation/relay.txt)

> 
> It'd be interesting to see a writeup of the proposed kernel<->userspace
> interfaces.  I see a description of kernel-internal interfaces, but how do
> users use it?

I will add some text in the documentation.
> 
> I see it uses things from blktrace.  Can blktrace be switched over to being
> a client of this code?

yep.

> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

Re: [PATCH 0/3] A kernel tracing interface

[Andrew Morton]

On Tue, 26 Feb 2008 14:38:37 -0800 "David J. Wilder" <dwilder@us.ibm.com> wrote:

> These patches provide a kernel tracing interface called "trace".
> 
> ChangeLog:
> -Updated to 2.6.25-rc2-mm1
> -Removed sem_watch example, to hard to maintain.
> -Added a new example that demonstrates per-cpu continuous tracing
>  of data generated using marker probes.
> -Removed inline from relay patch.
> -Moved examples into /sample directory.
> 
> The motivation for "trace" is to:
> - Provide a simple set of tracing primitives that will utilize the high-
>   performance and low-overhead of relayfs for passing traces data from
>   kernel to user space.
> - Provide a common user interface for managing kernel traces.
> - Allow for binary as well as ascii trace data.
> - Incorporate features from the systemtap runtime that are
>   useful to others.

So... what's the story on this versus lttng?

Is there some userspace code available for people to test this?  Maybe it's
mentioned and I missed it.

It'd be interesting to see a writeup of the proposed kernel<->userspace
interfaces.  I see a description of kernel-internal interfaces, but how do
users use it?

I see it uses things from blktrace.  Can blktrace be switched over to being
a client of this code?

[PATCH 0/3] A kernel tracing interface

[David J. Wilder]

These patches provide a kernel tracing interface called "trace".

ChangeLog:
-Updated to 2.6.25-rc2-mm1
-Removed sem_watch example, to hard to maintain.
-Added a new example that demonstrates per-cpu continuous tracing
 of data generated using marker probes.
-Removed inline from relay patch.
-Moved examples into /sample directory.

The motivation for "trace" is to:
- Provide a simple set of tracing primitives that will utilize the high-
  performance and low-overhead of relayfs for passing traces data from
  kernel to user space.
- Provide a common user interface for managing kernel traces.
- Allow for binary as well as ascii trace data.
- Incorporate features from the systemtap runtime that are
  useful to others.

Patches are built against 2.6.25-rc2-mm1.
Tested on: 2.6.25-rc3

Summary of patches:
[patch 1/3]  Relay Reset Consumed
[patch 2/3]  Trace code and documentation
[patch 3/3]  Trace sample

Note: Patch 1/3 provides a service used by patch 2/3
thus must be applied first or at the same time.

Signed-off-by: David Wilder <dwilder@us.ibm.com>

[PATCH 0/3] A kernel tracing interface

[David J. Wilder]

These patches provide a kernel tracing interface called "trace".

ChangeLog:
-Added a new example that demonstrates per-cpu continuous tracing
 of data generated by marker probes.
-Removed inline from relay patch.
-Moved examples into /sample directory.

The motivation for "trace" is to:
- Provide a simple set of tracing primitives that will utilize the high-
  performance and low-overhead of relayfs for passing traces data from
  kernel to user space.
- Provide a common user interface for managing kernel traces.
- Allow for binary as well as ascii trace data.
- Incorporate features from the systemtap runtime that are
  useful to others.

Patches are against 2.6.23-mm1

Summary of patches:
[patch 1/3]  Relay Reset Consumed
[patch 2/3]  Trace code and documentation
[patch 3/3]  Trace samples

Note: Patch 1/3 provides a service used by patch 2/3
thus must be applied first or at the same time.

Note: The sem_watch example requires that  /lib/semaphore-sleepers.c
is patched to add several markers around the semaphore primitives,
see the patch at: http://lkml.org/lkml/2007/11/12/279

Signed-off-by: David Wilder <dwilder@us.ibm.com>

[PATCH 0/3] A kernel tracing interface

[David J. Wilder]

[PATCH 0/3] A kernel tracing interface - (updated)

These patches provide a kernel tracing interface called "trace".

(update) Fixed a notation in the trace documentation as pointed out by
 Randy Dunlap.  Updated references to prerequisite patches.

The motivation for "trace" is to:
- Provide a simple set of tracing primitives that will utilize the high-
  performance and low-overhead of relayfs for passing traces data from
  kernel to user space.
- Provide a common user interface for managing kernel traces.
- Allow for binary as well as ascii trace data.
- Incorporate features from the systemtap runtime that are
  useful to others.

Patches are against 2.6.23-rc8-mm2

Summary of patches:
[patch 1/3]  Trace code and documentation
[patch 2/3]  Relay Reset Consumed
[patch 3/3]  Trace sample

Note: Patch 2/3 provides a service used by patch 1/3
thus must be applied first or at the same time.

Note: The following patches must be applied prior to 3/3.
[patch 3/5] Add samples subdir
	http://lkml.org/lkml/2007/9/28/178

[patch 4/5] Linux Kernel Markers - Samples
	http://lkml.org/lkml/2007/9/28/189

Signed-off-by: David Wilder <dwilder@us.ibm.com>