2.6.21-rc2-git3 soft lockup detected on CPU#0 (crash dump kernel)

2.6.21-rc2-git3 soft lockup detected on CPU#0 (crash dump kernel)

[Michal Piotrowski]

Hi,

I hit this bug again
(http://www.ussg.iu.edu/hypermail/linux/kernel/0702.2/1643.html)

BUG: soft lockup detected on CPU#0!
 [<c1604556>] show_trace_log_lvl+0x1a/0x2f
 [<c1604c2c>] show_trace+0x12/0x14
 [<c1604cde>] dump_stack+0x16/0x18
 [<c16426e7>] softlockup_tick+0x97/0xa7
 [<c161de38>] run_local_timers+0x12/0x14
 [<c161e1ce>] update_process_times+0x3e/0x63
 [<c162c44b>] tick_sched_timer+0x5d/0xa1
 [<c1629715>] hrtimer_interrupt+0x11e/0x1b2
 [<c160684e>] timer_interrupt+0xe/0x15
 [<c164299e>] handle_IRQ_event+0x1a/0x46
 [<c1643cfe>] handle_level_irq+0x81/0xcb
 [<c1605dd8>] do_IRQ+0xb4/0xe0

This time it's a crash dump kernel without SMP.

http://www.stardust.webpages.pl/files/tbf/bitis-gabonica/2.6.21-rc2-git3-kdump/kdump-dmesg
http://www.stardust.webpages.pl/files/tbf/bitis-gabonica/2.6.21-rc2-git3-kdump/kdump-config

Regards,
Michal

-- 
Michal K. K. Piotrowski
LTG - Linux Testers Group (PL)
(http://www.stardust.webpages.pl/ltg/)
LTG - Linux Testers Group (EN)
(http://www.stardust.webpages.pl/linux_testers_group_en/)

Re: 2.6.21-rc2-git3 soft lockup detected on CPU#0 (crash dump kernel)

[Thomas Gleixner]

On Sun, 2007-03-04 at 00:31 +0100, Michal Piotrowski wrote:
> Hi,
> 
> I hit this bug again
> (http://www.ussg.iu.edu/hypermail/linux/kernel/0702.2/1643.html)
> 
> BUG: soft lockup detected on CPU#0!
>  [<c1604556>] show_trace_log_lvl+0x1a/0x2f
>  [<c1604c2c>] show_trace+0x12/0x14
>  [<c1604cde>] dump_stack+0x16/0x18
>  [<c16426e7>] softlockup_tick+0x97/0xa7
>  [<c161de38>] run_local_timers+0x12/0x14
>  [<c161e1ce>] update_process_times+0x3e/0x63
>  [<c162c44b>] tick_sched_timer+0x5d/0xa1
>  [<c1629715>] hrtimer_interrupt+0x11e/0x1b2
>  [<c160684e>] timer_interrupt+0xe/0x15
>  [<c164299e>] handle_IRQ_event+0x1a/0x46
>  [<c1643cfe>] handle_level_irq+0x81/0xcb
>  [<c1605dd8>] do_IRQ+0xb4/0xe0
> 
> This time it's a crash dump kernel without SMP.

Is the box stuck, or do you just get those softlockup messages from time
to time ?

	tglx

Re: 2.6.21-rc2-git3 soft lockup detected on CPU#0 (crash dump kernel)

[Michal Piotrowski]

On 04/03/07, Thomas Gleixner <tglx@linutronix.de> wrote:
> On Sun, 2007-03-04 at 00:31 +0100, Michal Piotrowski wrote:
> > Hi,
> >
> > I hit this bug again
> > (http://www.ussg.iu.edu/hypermail/linux/kernel/0702.2/1643.html)
> >
> > BUG: soft lockup detected on CPU#0!
> >  [<c1604556>] show_trace_log_lvl+0x1a/0x2f
> >  [<c1604c2c>] show_trace+0x12/0x14
> >  [<c1604cde>] dump_stack+0x16/0x18
> >  [<c16426e7>] softlockup_tick+0x97/0xa7
> >  [<c161de38>] run_local_timers+0x12/0x14
> >  [<c161e1ce>] update_process_times+0x3e/0x63
> >  [<c162c44b>] tick_sched_timer+0x5d/0xa1
> >  [<c1629715>] hrtimer_interrupt+0x11e/0x1b2
> >  [<c160684e>] timer_interrupt+0xe/0x15
> >  [<c164299e>] handle_IRQ_event+0x1a/0x46
> >  [<c1643cfe>] handle_level_irq+0x81/0xcb
> >  [<c1605dd8>] do_IRQ+0xb4/0xe0
> >
> > This time it's a crash dump kernel without SMP.
>
> Is the box stuck, or do you just get those softlockup messages from time
> to time ?

The box stuck on it.

I can reproduce this on 2.6.21-rc2-git3 and 2.6.20.1.

Regards,
Michal

-- 
Michal K. K. Piotrowski
LTG - Linux Testers Group (PL)
(http://www.stardust.webpages.pl/ltg/)
LTG - Linux Testers Group (EN)
(http://www.stardust.webpages.pl/linux_testers_group_en/)

Re: 2.6.21-rc2-git3 soft lockup detected on CPU#0 (crash dump kernel)

[Thomas Gleixner]

On Sun, 2007-03-04 at 15:36 +0100, Michal Piotrowski wrote:
> > > BUG: soft lockup detected on CPU#0!
> > >  [<c1604556>] show_trace_log_lvl+0x1a/0x2f
> > >  [<c1604c2c>] show_trace+0x12/0x14
> > >  [<c1604cde>] dump_stack+0x16/0x18
> > >  [<c16426e7>] softlockup_tick+0x97/0xa7
> > >  [<c161de38>] run_local_timers+0x12/0x14
> > >  [<c161e1ce>] update_process_times+0x3e/0x63
> > >  [<c162c44b>] tick_sched_timer+0x5d/0xa1
> > >  [<c1629715>] hrtimer_interrupt+0x11e/0x1b2
> > >  [<c160684e>] timer_interrupt+0xe/0x15
> > >  [<c164299e>] handle_IRQ_event+0x1a/0x46
> > >  [<c1643cfe>] handle_level_irq+0x81/0xcb
> > >  [<c1605dd8>] do_IRQ+0xb4/0xe0
> > >
> > > This time it's a crash dump kernel without SMP.
> >
> > Is the box stuck, or do you just get those softlockup messages from time
> > to time ?
> 
> The box stuck on it.
> 
> I can reproduce this on 2.6.21-rc2-git3 and 2.6.20.1.

Ahh. That's something different, i.e. unrelated to highres/dynticks. 
2.6.20.1 does not have them.

Can you please add "initcall_debug" and "ignore_loglevel" to the kernel
command line ?

	tglx

Re: 2.6.21-rc2-git3 soft lockup detected on CPU#0 (crash dump kernel)

[Michal Piotrowski]

On 04/03/07, Thomas Gleixner <tglx@linutronix.de> wrote:
> On Sun, 2007-03-04 at 15:36 +0100, Michal Piotrowski wrote:
> > > > BUG: soft lockup detected on CPU#0!
> > > >  [<c1604556>] show_trace_log_lvl+0x1a/0x2f
> > > >  [<c1604c2c>] show_trace+0x12/0x14
> > > >  [<c1604cde>] dump_stack+0x16/0x18
> > > >  [<c16426e7>] softlockup_tick+0x97/0xa7
> > > >  [<c161de38>] run_local_timers+0x12/0x14
> > > >  [<c161e1ce>] update_process_times+0x3e/0x63
> > > >  [<c162c44b>] tick_sched_timer+0x5d/0xa1
> > > >  [<c1629715>] hrtimer_interrupt+0x11e/0x1b2
> > > >  [<c160684e>] timer_interrupt+0xe/0x15
> > > >  [<c164299e>] handle_IRQ_event+0x1a/0x46
> > > >  [<c1643cfe>] handle_level_irq+0x81/0xcb
> > > >  [<c1605dd8>] do_IRQ+0xb4/0xe0
> > > >
> > > > This time it's a crash dump kernel without SMP.
> > >
> > > Is the box stuck, or do you just get those softlockup messages from time
> > > to time ?
> >
> > The box stuck on it.
> >
> > I can reproduce this on 2.6.21-rc2-git3 and 2.6.20.1.
>
> Ahh. That's something different, i.e. unrelated to highres/dynticks.
> 2.6.20.1 does not have them.
>
> Can you please add "initcall_debug" and "ignore_loglevel" to the kernel
> command line ?

Here is a full serial console log
http://www.stardust.webpages.pl/files/tbf/bitis-gabonica/2.6.21-rc2-git3-kdump/log.txt

Regards,
Michal

-- 
Michal K. K. Piotrowski
LTG - Linux Testers Group (PL)
(http://www.stardust.webpages.pl/ltg/)
LTG - Linux Testers Group (EN)
(http://www.stardust.webpages.pl/linux_testers_group_en/)

Re: 2.6.21-rc2-git3 soft lockup detected on CPU#0 (crash dump kernel)

[Thomas Gleixner]

On Sun, 2007-03-04 at 17:28 +0100, Michal Piotrowski wrote:
> > Can you please add "initcall_debug" and "ignore_loglevel" to the kernel
> > command line ?
> 
> Here is a full serial console log
> http://www.stardust.webpages.pl/files/tbf/bitis-gabonica/2.6.21-rc2-git3-kdump/log.txt

sk98lin: driver has been replaced by the skge driver and is scheduled for removal

Can you try without that one or with the skge driver instead ?

	tglx

Re: 2.6.21-rc2-git3 soft lockup detected on CPU#0 (crash dump kernel)

[Michal Piotrowski]

On 04/03/07, Thomas Gleixner <tglx@linutronix.de> wrote:
> On Sun, 2007-03-04 at 17:28 +0100, Michal Piotrowski wrote:
> > > Can you please add "initcall_debug" and "ignore_loglevel" to the kernel
> > > command line ?
> >
> > Here is a full serial console log
> > http://www.stardust.webpages.pl/files/tbf/bitis-gabonica/2.6.21-rc2-git3-kdump/log.txt
>
> sk98lin: driver has been replaced by the skge driver and is scheduled for removal
>
> Can you try without that one or with the skge driver instead ?
>

Unfortunately the bug still appears.

Regards,
Michal

-- 
Michal K. K. Piotrowski
LTG - Linux Testers Group (PL)
(http://www.stardust.webpages.pl/ltg/)
LTG - Linux Testers Group (EN)
(http://www.stardust.webpages.pl/linux_testers_group_en/)

Re: 2.6.21-rc2-git3 soft lockup detected on CPU#0 (crash dump kernel)

[Thomas Gleixner]

On Sun, 2007-03-04 at 17:47 +0100, Michal Piotrowski wrote:
> > sk98lin: driver has been replaced by the skge driver and is scheduled for removal
> >
> > Can you try without that one or with the skge driver instead ?
> >
> 
> Unfortunately the bug still appears.

Would have been too easy :) 

sysrq-T and sysrq-Q might give use some useful info.

	tglx

Re: 2.6.21-rc2-git3 soft lockup detected on CPU#0 (crash dump kernel)

[Michal Piotrowski]

On 04/03/07, Thomas Gleixner <tglx@linutronix.de> wrote:
> On Sun, 2007-03-04 at 17:47 +0100, Michal Piotrowski wrote:
> > > sk98lin: driver has been replaced by the skge driver and is scheduled for removal
> > >
> > > Can you try without that one or with the skge driver instead ?
> > >
> >
> > Unfortunately the bug still appears.
>
> Would have been too easy :)
>
> sysrq-T and sysrq-Q might give use some useful info.
>

http://www.stardust.webpages.pl/files/tbf/bitis-gabonica/2.6.21-rc2-git3-kdump/log2.txt

Regards,
Michal

-- 
Michal K. K. Piotrowski
LTG - Linux Testers Group (PL)
(http://www.stardust.webpages.pl/ltg/)
LTG - Linux Testers Group (EN)
(http://www.stardust.webpages.pl/linux_testers_group_en/)

Re: 2.6.21-rc2-git3 soft lockup detected on CPU#0 (crash dump kernel)

[Thomas Gleixner]

On Sun, 2007-03-04 at 18:15 +0100, Michal Piotrowski wrote:
>
> http://www.stardust.webpages.pl/files/tbf/bitis-gabonica/2.6.21-rc2-git3-kdump/log2.txt
> 

Kernel command line: .... irqpoll ....
Misrouted IRQ fixup and polling support enabled

It might be related to irqpoll. Not sure how to track that one down.

	tglx


	

Re: 2.6.21-rc2-git3 soft lockup detected on CPU#0 (crash dump kernel)

[Ingo Molnar]

* Michal Piotrowski <michal.k.k.piotrowski@gmail.com> wrote:

> http://www.stardust.webpages.pl/files/tbf/bitis-gabonica/2.6.21-rc2-git3-kdump/log2.txt

weird. Could you do a dump with SysRq-T, SysRq-Q done /twice/ after each 
other, to see how the system progresses while in this lockup? Plus 
please also try to get a register dump via SysRq-P. 

Btw., the task state printout seems busted:

udev_run_devd ? C15B0030  2544   979    853                     (L-TLB)

shouldnt that be 'R' instead of the question mark?

	Ingo

Re: 2.6.21-rc2-git3 soft lockup detected on CPU#0 (crash dump kernel)

[Ingo Molnar]

* Thomas Gleixner <tglx@linutronix.de> wrote:

> > shouldnt that be 'R' instead of the question mark?
> 
> ? are dead ones,  

ok.

> The interesting part is here:
> 
> 1 lock held by udevd/918:
>  #0:  (&mm->mmap_sem){----}, at: [<c161135b>] do_page_fault+0x150/0x50d
> 
> But udevd/918 is nowhere in the sysrq-t output.

also below is a softlockup-debug patch that might be useful, which adds 
a tick/tock output so it spams the console once per second but can show 
the dynamics of lockups (and other delays).

	Ingo

------------------->
Subject: [patch] softlockup: add /proc/sys/kernel/softlockup_print_tick
From: Ingo Molnar <mingo@elte.hu>

/proc/sys/kernel/softlockup_print_tick tunable:

 0: off
 1: print tick/tock messages in softlockup irq and task.
 2: also print stack dump

Signed-off-by: Ingo Molnar <mingo@elte.hu>
---
 Documentation/kernel-parameters.txt |    5 +++++
 include/linux/sched.h               |    2 ++
 kernel/softlockup.c                 |   13 ++++++++++++-
 kernel/sysctl.c                     |   10 ++++++++++
 4 files changed, 29 insertions(+), 1 deletion(-)

Index: linux/Documentation/kernel-parameters.txt
===================================================================
--- linux.orig/Documentation/kernel-parameters.txt
+++ linux/Documentation/kernel-parameters.txt
@@ -1659,6 +1659,11 @@ and is between 256 and 4096 characters. 
 
 	snd-ymfpci=	[HW,ALSA]
 
+	softlockup_print_tick
+			[KNL] Print softlockup tick in the irq and in the
+			watchdog task. If value 2 then print stackdump too.
+			Format: <0/1/2>
+
 	sonycd535=	[HW,CD]
 			Format: <io>[,<irq>]
 
Index: linux/include/linux/sched.h
===================================================================
--- linux.orig/include/linux/sched.h
+++ linux/include/linux/sched.h
@@ -223,6 +223,8 @@ extern void update_process_times(int use
 extern void scheduler_tick(void);
 
 #ifdef CONFIG_DETECT_SOFTLOCKUP
+extern int softlockup_print_tick;
+
 extern void softlockup_tick(void);
 extern void spawn_softlockup_task(void);
 extern void touch_softlockup_watchdog(void);
Index: linux/kernel/softlockup.c
===================================================================
--- linux.orig/kernel/softlockup.c
+++ linux/kernel/softlockup.c
@@ -21,6 +21,7 @@ static DEFINE_PER_CPU(unsigned long, pri
 static DEFINE_PER_CPU(struct task_struct *, watchdog_task);
 
 static int did_panic = 0;
+int softlockup_print_tick = 2;
 
 static int
 softlock_panic(struct notifier_block *this, unsigned long event, void *ptr)
@@ -62,8 +63,15 @@ void softlockup_tick(void)
 	}
 
 	/* Wake up the high-prio watchdog task every second: */
-	if (time_after(jiffies, touch_timestamp + HZ))
+	if (time_after(jiffies, touch_timestamp + HZ)) {
+		if (softlockup_print_tick) {
+			printk("softlockup tick on CPU#%d, %s:%d\n",
+				this_cpu, current->comm, current->pid);
+			if (softlockup_print_tick == 2)
+				dump_stack();
+		}
 		wake_up_process(per_cpu(watchdog_task, this_cpu));
+	}
 
 	/* Warn about unreasonable 10+ seconds delays: */
 	if (time_after(jiffies, touch_timestamp + 10*HZ)) {
@@ -83,6 +91,7 @@ void softlockup_tick(void)
 static int watchdog(void * __bind_cpu)
 {
 	struct sched_param param = { .sched_priority = 99 };
+	int this_cpu = (int)(long)__bind_cpu;
 
 	sched_setscheduler(current, SCHED_FIFO, &param);
 	current->flags |= PF_NOFREEZE;
@@ -96,6 +105,8 @@ static int watchdog(void * __bind_cpu)
 		set_current_state(TASK_INTERRUPTIBLE);
 		touch_softlockup_watchdog();
 		schedule();
+		if (softlockup_print_tick)
+			printk("softlockup tock on CPU#%d\n", this_cpu);
 	}
 
 	return 0;
Index: linux/kernel/sysctl.c
===================================================================
--- linux.orig/kernel/sysctl.c
+++ linux/kernel/sysctl.c
@@ -231,6 +231,16 @@ static ctl_table kern_table[] = {
 		.proc_handler	= &proc_dostring,
 		.strategy	= &sysctl_string,
 	},
+#ifdef CONFIG_DETECT_SOFTLOCKUP
+	{
+		.ctl_name	= CTL_UNNUMBERED,
+		.procname	= "softlockup_print_tick",
+		.data		= &softlockup_print_tick,
+		.maxlen		= sizeof(int),
+		.mode		= 0644,
+		.proc_handler	= &proc_dointvec,
+	},
+#endif
 #ifdef CONFIG_PROC_SYSCTL
 	{
 		.ctl_name	= KERN_TAINTED,

Re: 2.6.21-rc2-git3 soft lockup detected on CPU#0 (crash dump kernel)

[Thomas Gleixner]

On Mon, 2007-03-05 at 09:08 +0100, Ingo Molnar wrote:
> * Michal Piotrowski <michal.k.k.piotrowski@gmail.com> wrote:
> 
> > http://www.stardust.webpages.pl/files/tbf/bitis-gabonica/2.6.21-rc2-git3-kdump/log2.txt
> 
> weird. Could you do a dump with SysRq-T, SysRq-Q done /twice/ after each 
> other, to see how the system progresses while in this lockup? Plus 
> please also try to get a register dump via SysRq-P. 
> 
> Btw., the task state printout seems busted:
> 
> udev_run_devd ? C15B0030  2544   979    853                     (L-TLB)
> 
> shouldnt that be 'R' instead of the question mark?

? are dead ones,  

 [<c16191d9>] do_exit+0x6f9/0x6fd
 [<c161924d>] sys_exit_group+0x0/0x11
 [<c161925c>] sys_exit_group+0xf/0x11
 [<c1603e72>] sysenter_past_esp+0x5f/0x99

The interesting part is here:

1 lock held by udevd/918:
 #0:  (&mm->mmap_sem){----}, at: [<c161135b>] do_page_fault+0x150/0x50d

But udevd/918 is nowhere in the sysrq-t output.

	tglx

Re: 2.6.21-rc2-git3 soft lockup detected on CPU#0 (crash dump kernel)

[Michal Piotrowski]

On 05/03/07, Ingo Molnar <mingo@elte.hu> wrote:
>
> * Thomas Gleixner <tglx@linutronix.de> wrote:
>
> > > shouldnt that be 'R' instead of the question mark?
> >
> > ? are dead ones,
>
> ok.
>
> > The interesting part is here:
> >
> > 1 lock held by udevd/918:
> >  #0:  (&mm->mmap_sem){----}, at: [<c161135b>] do_page_fault+0x150/0x50d
> >
> > But udevd/918 is nowhere in the sysrq-t output.
>
> also below is a softlockup-debug patch that might be useful, which adds
> a tick/tock output so it spams the console once per second but can show
> the dynamics of lockups (and other delays).

I have applied this patch
http://www.stardust.webpages.pl/files/tbf/bitis-gabonica/2.6.21-rc2-git3-kdump/log3.txt

Regards,
Michal

-- 
Michal K. K. Piotrowski
LTG - Linux Testers Group (PL)
(http://www.stardust.webpages.pl/ltg/)
LTG - Linux Testers Group (EN)
(http://www.stardust.webpages.pl/linux_testers_group_en/)

Re: 2.6.21-rc2-git3 soft lockup detected on CPU#0 (crash dump kernel)

[Ingo Molnar]

* Michal Piotrowski <michal.k.k.piotrowski@gmail.com> wrote:

> >also below is a softlockup-debug patch that might be useful, which adds
> >a tick/tock output so it spams the console once per second but can show
> >the dynamics of lockups (and other delays).
> 
> I have applied this patch
> http://www.stardust.webpages.pl/files/tbf/bitis-gabonica/2.6.21-rc2-git3-kdump/log3.txt

thanks. It seems that the 'BUG: soft lockup detected' message does not 
occur in this log though - and that there are bootup messages between 
the two sysrq outputs:

[  221.101833] skge eth1: disabling interface
[  228.740911] audit(1173094442.943:11): user pid=3245 uid=0 
auid=4294967295 subj=system_u:system_r:hwclock_t:s0 msg='changing system 
time: exe="/sbin/hwclock"
(hostname=?, addr=?, terminal=console res=success)'
[  233.148571] md: md1 in immediate safe mode

plus there's trouble with irq 10:

[  132.013194] irq 10: nobody cared (try booting with the "irqpoll" option)
[  132.072771] Disabling IRQ #10

that prompted you to enable irqpoll, right? So am i right that without 
irqpoll there's no lockup?

	Ingo

Re: 2.6.21-rc2-git3 soft lockup detected on CPU#0 (crash dump kernel)

[Michal Piotrowski]

On 05/03/07, Ingo Molnar <mingo@elte.hu> wrote:
>
> * Michal Piotrowski <michal.k.k.piotrowski@gmail.com> wrote:
>
> > >also below is a softlockup-debug patch that might be useful, which adds
> > >a tick/tock output so it spams the console once per second but can show
> > >the dynamics of lockups (and other delays).
> >
> > I have applied this patch
> > http://www.stardust.webpages.pl/files/tbf/bitis-gabonica/2.6.21-rc2-git3-kdump/log3.txt
>
> thanks. It seems that the 'BUG: soft lockup detected' message does not
> occur in this log though - and that there are bootup messages between
> the two sysrq outputs:
>
> [  221.101833] skge eth1: disabling interface
> [  228.740911] audit(1173094442.943:11): user pid=3245 uid=0
> auid=4294967295 subj=system_u:system_r:hwclock_t:s0 msg='changing system
> time: exe="/sbin/hwclock"
> (hostname=?, addr=?, terminal=console res=success)'
> [  233.148571] md: md1 in immediate safe mode
>
> plus there's trouble with irq 10:
>
> [  132.013194] irq 10: nobody cared (try booting with the "irqpoll" option)
> [  132.072771] Disabling IRQ #10
>
> that prompted you to enable irqpoll, right?

No. irqpoll is added by /etc/init.d/kdump script (FC6).

> So am i right that without
> irqpoll there's no lockup?

I will check this.

>
>         Ingo
>

Regards,
Michal

-- 
Michal K. K. Piotrowski
LTG - Linux Testers Group (PL)
(http://www.stardust.webpages.pl/ltg/)
LTG - Linux Testers Group (EN)
(http://www.stardust.webpages.pl/linux_testers_group_en/)

Re: 2.6.21-rc2-git3 soft lockup detected on CPU#0 (crash dump kernel)

[Ingo Molnar]

* Michal Piotrowski <michal.k.k.piotrowski@gmail.com> wrote:

> I can confirm that without irqpoll everything works fine.

i guess this means that irq10 is screaming, and irqpoll does not find 
any registered irq handler for it?

> irqpoll is recommended as a command line option in 
> Documentation/kdump/kdump.txt

hmmm ...

	Ingo

Re: 2.6.21-rc2-git3 soft lockup detected on CPU#0 (crash dump kernel)

[Michal Piotrowski]

On 05/03/07, Michal Piotrowski <michal.k.k.piotrowski@gmail.com> wrote:
> On 05/03/07, Ingo Molnar <mingo@elte.hu> wrote:
> >
> > * Michal Piotrowski <michal.k.k.piotrowski@gmail.com> wrote:
> >
> > > >also below is a softlockup-debug patch that might be useful, which adds
> > > >a tick/tock output so it spams the console once per second but can show
> > > >the dynamics of lockups (and other delays).
> > >
> > > I have applied this patch
> > > http://www.stardust.webpages.pl/files/tbf/bitis-gabonica/2.6.21-rc2-git3-kdump/log3.txt
> >
> > thanks. It seems that the 'BUG: soft lockup detected' message does not
> > occur in this log though - and that there are bootup messages between
> > the two sysrq outputs:
> >
> > [  221.101833] skge eth1: disabling interface
> > [  228.740911] audit(1173094442.943:11): user pid=3245 uid=0
> > auid=4294967295 subj=system_u:system_r:hwclock_t:s0 msg='changing system
> > time: exe="/sbin/hwclock"
> > (hostname=?, addr=?, terminal=console res=success)'
> > [  233.148571] md: md1 in immediate safe mode
> >
> > plus there's trouble with irq 10:
> >
> > [  132.013194] irq 10: nobody cared (try booting with the "irqpoll" option)
> > [  132.072771] Disabling IRQ #10
> >
> > that prompted you to enable irqpoll, right?
>
> No. irqpoll is added by /etc/init.d/kdump script (FC6).
>
> > So am i right that without
> > irqpoll there's no lockup?
>
> I will check this.

I can confirm that without irqpoll everything works fine.
http://www.stardust.webpages.pl/files/tbf/bitis-gabonica/2.6.21-rc2-git3-kdump/log4.txt

irqpoll is recommended as a command line option in Documentation/kdump/kdump.txt

Regards,
Michal

-- 
Michal K. K. Piotrowski
LTG - Linux Testers Group (PL)
(http://www.stardust.webpages.pl/ltg/)
LTG - Linux Testers Group (EN)
(http://www.stardust.webpages.pl/linux_testers_group_en/)