Latency issues with x86.git

Latency issues with x86.git

[Kevin Winchester]

Hi Ingo,

I have encountered (a handful of times in the past few months) some real
interactivity problems on my system.  Moving the mouse or typing a key
on the keyboard takes around a second to show any response.  Once I
perform a reboot, the problem is gone again.  I am currently running
x86.git mm branch, but I switch between that branch, mainline git, and
mm kernels, so I cannot guarantee on which trees I have or have not seen
the problem.

It seems to have started around the time that CFS came into being, but
it might be something completely unrelated.

When it happened this evening, I ran the cfs-debug-info.sh script to
which you had referred me in another thread.  I'm not sure if this helps
to debug the problem, but it was all I could think to try.  I have
LatencyTOP in my kernel, so I guess I could try the userspace tool as
well the next time.

The output of the script is at:

http://personal.nbnet.nb.ca/kwin/cfs-debug-info-2008.02.13-20.56.38


Unfortunately, I cannot reproduce the problem intentionally - it only
happens once a month or so.

If there is any other info you need, please let me know, or any
suggestions for what to try the next time.

Thanks,

-- 
Kevin Winchester

Re: Latency issues with x86.git

[Ingo Molnar]

* Kevin Winchester <kjwinchester@gmail.com> wrote:

> Hi Ingo,
> 
> I have encountered (a handful of times in the past few months) some 
> real interactivity problems on my system.  Moving the mouse or typing 
> a key on the keyboard takes around a second to show any response.  
> Once I perform a reboot, the problem is gone again.  I am currently 
> running x86.git mm branch, but I switch between that branch, mainline 
> git, and mm kernels, so I cannot guarantee on which trees I have or 
> have not seen the problem.

please try sched-devel.git, which has both the latest scheduler fixes, 
and also the new "ftrace" latency tracing framework that can be used to 
trace various latency problems. You can pick up sched-devel.git via:

   http://people.redhat.com/mingo/sched-devel.git/README

firstly, there's a chance that sched-devel.git solves the problem - in 
that case please report it.

if it doesnt, then you can trace various latencies via these:

 CONFIG_FTRACE=y
 CONFIG_IRQSOFF_TRACER=y
 CONFIG_SCHED_TRACER=y
 CONFIG_CONTEXT_SWITCH_TRACER=y
 CONFIG_DYNAMIC_FTRACE=y

just enable them, boot into the new kernel and mount debugfs:

   mount -t debugfs nodev /debug

and then you can select one of the tracers (that have been enabled in 
the .config) via /debug/tracing/* files. The usage of these files should 
be self-explaining - if any of them wasnt then please let us know and 
we'll make the "first quick glance experience" better :)

the one interesting to you would be the "wakeup" tracer. Enable it, and 
if you echo 0 into /debug/tracing/tracing_max_latency it starts tracking 
the worst-case delay experienced on your system and should start 
reporting them to the syslog.

if you encounter any problems during these steps then please let us know 
- this code is quite fresh so expect some rough edges.

	Ingo