[BUG] 2.6.25-rc2-git4 - Regression Kernel oops while running kernbench and tbench on powerpc

[BUG] 2.6.25-rc2-git4 - Regression Kernel oops while running kernbench and tbench on powerpc

[Kamalesh Babulal]

Hi,

The Kernel oopses is seen while running the kernbench followed by tbench with 2.6.25-rc2-git4 
kernel on powerpc, this oops was reported for the 2.6.24-rc8-mm1 kernel (http://lkml.org/lkml/2008/1/18/71)
and is visible with almost all of the main line ,rc(s) and their git(s) release from then.

This oops is visible in the linux-next-20080220 kernel also.The machine is power4+ box with four cpus and 
has 30 GB RAM.

oops while running kernbench
-----------------------------

 Oops: Kernel access of bad area, sig: 11 [#1]
SMP NR_CPUS=32 NUMA pSeries
Modules linked in:
NIP: 0000000000004570 LR: 000000000ff0135c CTR: 000000000ff012f0
REGS: c000000771a678c0 TRAP: 0300   Not tainted  (2.6.25-rc2-autotest-next-20080220)
MSR: 8000000000001000 <ME>  CR: 28000282  XER: 00000000
DAR: c000000771a67ce0, DSISR: 000000000a000000
TASK = c00000077b9d2c00[22046] 'cc1' THREAD: c000000771a64000 CPU: 3
GPR00: 0000000000004000 c000000771a67b40 0000000000000052 000000000000d032 
GPR04: 0000000000000052 0000000000000400 00000000106823e0 000000000000000e 
GPR08: 0000000000000000 0000000044000288 c000000771a67e30 0000998be2321500 
GPR12: 8000000000001030 c0000000005d7e00 0000000010030000 0000000010030000 
GPR16: 00000000105b0000 00000000105b0000 0000000010440000 00000000105b0000 
GPR20: 00000000105b0000 00000000105b0000 00000000105b0000 00000000105b0000 
GPR24: 00000000105b41e8 0000000000000020 00000000105b0000 0000000000000400 
GPR28: 000000000ffec408 00000000f7ff8000 000000000ffebff4 0000000000000400 
NIP [0000000000004570] 0x4570
LR [000000000ff0135c] 0xff0135c
Call Trace:
[c000000771a67b40] [c00000000062d558] 0xc00000000062d558 (unreliable)
[c000000771a67e08] [00000000f7ff8000] 0xf7ff8000
Instruction dump:
48000000 XXXXXXXX XXXXXXXX XXXXXXXX 41820008 XXXXXXXX XXXXXXXX XXXXXXXX 
48000010 XXXXXXXX XXXXXXXX XXXXXXXX f92101a0 XXXXXXXX XXXXXXXX XXXXXXXX 
---[ end trace 26a7439b76b3cbab ]---

oops while running tbench
--------------------------

Unable to handle kernel paging request for data at address 0xc00000077e2e3ce0
Faulting instruction address: 0x00004570
Oops: Kernel access of bad area, sig: 11 [#2]
SMP NR_CPUS=32 NUMA pSeries
Modules linked in:
NIP: 0000000000004570 LR: c00000000000872c CTR: 0000000000000000
REGS: c00000077e2e38c0 TRAP: 0300   Tainted: G      D   (2.6.25-rc2-autotest-next-20080220)
MSR: 8000000000001000 <ME>  CR: 28022822  XER: 20000000
DAR: c00000077e2e3ce0, DSISR: 000000000a000000
TASK = c00000077cb55800[3900] 'zmd' THREAD: c00000077e2e0000 CPU: 3
GPR00: 0000000000000000 c00000077e2e3b40 c0000000006a29b8 000000000000006e 
GPR04: 000000000fe7bac8 0000000054022888 0000000000004000 000000000fe7bae4 
GPR08: 000000000000d032 0000000044022824 c00000077e2e3e30 0000998be2321500 
GPR12: 8000000000001030 c0000000005d7e00 0000000000000001 0000000000000000 
GPR16: 0000000000000000 00000000f7c16708 00000000f7fa0800 00000000100e0040 
GPR20: 00000000f7ffe018 0000000000000001 00000000f5397550 00000000f5397548 
GPR24: 0000000000000000 0000000000004a19 00000000f7d66474 00000000f53975a8 
GPR28: 0000000000000000 0000000000009433 000000000fe94ff4 00000000f7d66490 
NIP [0000000000004570] 0x4570
LR [c00000000000872c] syscall_exit+0x0/0x40
Call Trace:
[c00000077e2e3b40] [00000496000729b8] 0x496000729b8 (unreliable)
Instruction dump:
48000000 XXXXXXXX XXXXXXXX XXXXXXXX 41820008 XXXXXXXX XXXXXXXX XXXXXXXX 
48000010 XXXXXXXX XXXXXXXX XXXXXXXX f92101a0 XXXXXXXX XXXXXXXX XXXXXXXX 

The machine 
-- 
Thanks & Regards,
Kamalesh Babulal,
Linux Technology Center,
IBM, ISTL.

Re: [BUG] 2.6.25-rc2-git4 - Regression Kernel oops while running kernbench and tbench on powerpc

[Paul Mackerras]

Kamalesh Babulal writes:

> The Kernel oopses is seen while running the kernbench followed by tbench with 2.6.25-rc2-git4 
> kernel on powerpc, this oops was reported for the 2.6.24-rc8-mm1 kernel (http://lkml.org/lkml/2008/1/18/71)
> and is visible with almost all of the main line ,rc(s) and their git(s) release from then.
> 
> This oops is visible in the linux-next-20080220 kernel also.The machine is power4+ box with four cpus and 
> has 30 GB RAM.

Please try to replicate the oops with the patch below applied.  It
doesn't solve the cause of the oops but it should mean the kernel
prints out more useful information about the cause of the oops.

I assume you can replicate the oops easily on this machine - is that
right?

Paul.

diff --git a/arch/powerpc/kernel/head_64.S b/arch/powerpc/kernel/head_64.S
index 11b4f6d..a3ac72a 100644
--- a/arch/powerpc/kernel/head_64.S
+++ b/arch/powerpc/kernel/head_64.S
@@ -621,7 +621,7 @@ END_FW_FTR_SECTION_IFSET(FW_FEATURE_ISERIES)
 	mtlr	r10
 
 	andi.	r10,r12,MSR_RI	/* check for unrecoverable exception */
-	beq-	unrecov_slb
+	beq-	2f
 
 .machine	push
 .machine	"power4"
@@ -643,6 +643,22 @@ END_FW_FTR_SECTION_IFSET(FW_FEATURE_ISERIES)
 	rfid
 	b	.	/* prevent speculative execution */
 
+2:
+#ifdef CONFIG_PPC_ISERIES
+BEGIN_FW_FTR_SECTION
+	b	unrecov_slb
+END_FW_FTR_SECTION_IFSET(FW_FEATURE_ISERIES)
+#endif /* CONFIG_PPC_ISERIES */
+	mfspr	r11,SPRN_SRR0
+	clrrdi	r10,r13,32
+	LOAD_HANDLER(r10,unrecov_slb)
+	mtspr	SPRN_SRR0,r10
+	mfmsr	r10
+	ori	r10,r10,MSR_IR|MSR_DR|MSR_RI
+	mtspr	SPRN_SRR1,r10
+	rfid
+	b	.
+
 unrecov_slb:
 	EXCEPTION_PROLOG_COMMON(0x4100, PACA_EXSLB)
 	DISABLE_INTS

Re: [BUG] 2.6.25-rc2-git4 - Regression Kernel oops while running kernbench and tbench on powerpc

[Kamalesh Babulal]

Paul Mackerras wrote:
> Kamalesh Babulal writes:
> 
>> The Kernel oopses is seen while running the kernbench followed by tbench with 2.6.25-rc2-git4 
>> kernel on powerpc, this oops was reported for the 2.6.24-rc8-mm1 kernel (http://lkml.org/lkml/2008/1/18/71)
>> and is visible with almost all of the main line ,rc(s) and their git(s) release from then.
>>
>> This oops is visible in the linux-next-20080220 kernel also.The machine is power4+ box with four cpus and 
>> has 30 GB RAM.
> 
> Please try to replicate the oops with the patch below applied.  It
> doesn't solve the cause of the oops but it should mean the kernel
> prints out more useful information about the cause of the oops.
> 
> I assume you can replicate the oops easily on this machine - is that
> right?
> 
> Paul.
> 
> diff --git a/arch/powerpc/kernel/head_64.S b/arch/powerpc/kernel/head_64.S
> index 11b4f6d..a3ac72a 100644
> --- a/arch/powerpc/kernel/head_64.S
> +++ b/arch/powerpc/kernel/head_64.S
> @@ -621,7 +621,7 @@ END_FW_FTR_SECTION_IFSET(FW_FEATURE_ISERIES)
>  	mtlr	r10
> 
>  	andi.	r10,r12,MSR_RI	/* check for unrecoverable exception */
> -	beq-	unrecov_slb
> +	beq-	2f
> 
>  .machine	push
>  .machine	"power4"
> @@ -643,6 +643,22 @@ END_FW_FTR_SECTION_IFSET(FW_FEATURE_ISERIES)
>  	rfid
>  	b	.	/* prevent speculative execution */
> 
> +2:
> +#ifdef CONFIG_PPC_ISERIES
> +BEGIN_FW_FTR_SECTION
> +	b	unrecov_slb
> +END_FW_FTR_SECTION_IFSET(FW_FEATURE_ISERIES)
> +#endif /* CONFIG_PPC_ISERIES */
> +	mfspr	r11,SPRN_SRR0
> +	clrrdi	r10,r13,32
> +	LOAD_HANDLER(r10,unrecov_slb)
> +	mtspr	SPRN_SRR0,r10
> +	mfmsr	r10
> +	ori	r10,r10,MSR_IR|MSR_DR|MSR_RI
> +	mtspr	SPRN_SRR1,r10
> +	rfid
> +	b	.
> +
>  unrecov_slb:
>  	EXCEPTION_PROLOG_COMMON(0x4100, PACA_EXSLB)
>  	DISABLE_INTS
Hi Paul,

The kernel oops after applying the patch. Some time it takes more than
one run to reproduce it, it was reproducible in the second run this
time.

 Unrecoverable exception 4100 at c000000000008c8c
Oops: Unrecoverable exception, sig: 6 [#1]
SMP NR_CPUS=128 NUMA pSeries
Modules linked in:
NIP: c000000000008c8c LR: 000000000ff0135c CTR: 000000000ff012f0
REGS: c000000772343bb0 TRAP: 4100   Not tainted  (2.6.25-rc8-autotest)
MSR: 8000000000001030 <ME,IR,DR>  CR: 44044228  XER: 00000000
TASK = c00000077cfa0900[13437] 'cc1' THREAD: c000000772340000 CPU: 2
GPR00: 0000000000004000 c000000772343e30 00000000000000bb 000000000000d032 
GPR04: 00000000000000bb 0000000000000400 000000000000000a 0000000000000002 
GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 
GPR12: 0000000000000000 c000000000734000 0000000000000064 00000000ffe6df08 
GPR16: 00000000105b0000 00000000105b0000 0000000010440000 00000000105b0000 
GPR20: 00000000ffe6e008 00000000105b0000 00000000105b0000 000000000000000a 
GPR24: 000000000ffec408 0000000000000001 00000000ffe6ddca 0000000000000400 
GPR28: 000000000ffec408 00000000f7ff8000 000000000ffebff4 0000000000000400 
NIP [c000000000008c8c] restore+0x8c/0xc0
LR [000000000ff0135c] 0xff0135c
Call Trace:
[c000000772343e30] [c000000000008cd4] do_work+0x14/0x2c (unreliable)
Instruction dump:
7c840078 7c810164 70604000 41820028 60000000 7c4c42e6 e88d01f0 f84d01f0 
7c841050 e84d01e8 7c422214 f84d01e8 <e9a100d8> 7c7b03a6 e84101a0 7c4ff120 

(gdb) l *0xc000000000008cdc
0xc000000000008cdc is at arch/powerpc/kernel/entry_64.S:608.
603             mtmsrd  r10,1
604
605             andi.   r0,r4,_TIF_NEED_RESCHED
606             beq     1f
607             bl      .schedule
608             b       .ret_from_except_lite
609
610     1:      bl      .save_nvgprs
611             li      r3,0
612             addi    r4,r1,STACK_FRAME_OVERHEAD

please let me know if you need more information.
-- 
Thanks & Regards,
Kamalesh Babulal,
Linux Technology Center,
IBM, ISTL.

Re: [BUG] 2.6.25-rc2-git4 - Regression Kernel oops while running kernbench and tbench on powerpc

[Paul Mackerras]

Kamalesh Babulal writes:

> The kernel oops after applying the patch. Some time it takes more than
> one run to reproduce it, it was reproducible in the second run this
> time.
> 
>  Unrecoverable exception 4100 at c000000000008c8c
> Oops: Unrecoverable exception, sig: 6 [#1]
> SMP NR_CPUS=128 NUMA pSeries
> Modules linked in:
> NIP: c000000000008c8c LR: 000000000ff0135c CTR: 000000000ff012f0
> REGS: c000000772343bb0 TRAP: 4100   Not tainted  (2.6.25-rc8-autotest)
> MSR: 8000000000001030 <ME,IR,DR>  CR: 44044228  XER: 00000000
> TASK = c00000077cfa0900[13437] 'cc1' THREAD: c000000772340000 CPU: 2
> GPR00: 0000000000004000 c000000772343e30 00000000000000bb 000000000000d032 
> GPR04: 00000000000000bb 0000000000000400 000000000000000a 0000000000000002 
> GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 
> GPR12: 0000000000000000 c000000000734000 0000000000000064 00000000ffe6df08 
> GPR16: 00000000105b0000 00000000105b0000 0000000010440000 00000000105b0000 
> GPR20: 00000000ffe6e008 00000000105b0000 00000000105b0000 000000000000000a 
> GPR24: 000000000ffec408 0000000000000001 00000000ffe6ddca 0000000000000400 
> GPR28: 000000000ffec408 00000000f7ff8000 000000000ffebff4 0000000000000400 
> NIP [c000000000008c8c] restore+0x8c/0xc0
> LR [000000000ff0135c] 0xff0135c
> Call Trace:
> [c000000772343e30] [c000000000008cd4] do_work+0x14/0x2c (unreliable)
> Instruction dump:
> 7c840078 7c810164 70604000 41820028 60000000 7c4c42e6 e88d01f0 f84d01f0 
> 7c841050 e84d01e8 7c422214 f84d01e8 <e9a100d8> 7c7b03a6 e84101a0 7c4ff120 
> 
> (gdb) l *0xc000000000008cdc
> 0xc000000000008cdc is at arch/powerpc/kernel/entry_64.S:608.
> 603             mtmsrd  r10,1
> 604
> 605             andi.   r0,r4,_TIF_NEED_RESCHED
> 606             beq     1f
> 607             bl      .schedule
> 608             b       .ret_from_except_lite
> 609
> 610     1:      bl      .save_nvgprs
> 611             li      r3,0
> 612             addi    r4,r1,STACK_FRAME_OVERHEAD

The exception happened at c...8c8c but you looked at c...8cdc with
gdb.  What's at c...8c8c?

> please let me know if you need more information.

The .config would be useful, but don't spam everyone on cc with it,
just send it to me privately.

Paul.

Re: [BUG] 2.6.25-rc2-git4 - Regression Kernel oops while running kernbench and tbench on powerpc

[Kamalesh Babulal]

Paul Mackerras wrote:
> Kamalesh Babulal writes:
> 
>> The kernel oops after applying the patch. Some time it takes more than
>> one run to reproduce it, it was reproducible in the second run this
>> time.
>>
>>  Unrecoverable exception 4100 at c000000000008c8c
>> Oops: Unrecoverable exception, sig: 6 [#1]
>> SMP NR_CPUS=128 NUMA pSeries
>> Modules linked in:
>> NIP: c000000000008c8c LR: 000000000ff0135c CTR: 000000000ff012f0
>> REGS: c000000772343bb0 TRAP: 4100   Not tainted  (2.6.25-rc8-autotest)
>> MSR: 8000000000001030 <ME,IR,DR>  CR: 44044228  XER: 00000000
>> TASK = c00000077cfa0900[13437] 'cc1' THREAD: c000000772340000 CPU: 2
>> GPR00: 0000000000004000 c000000772343e30 00000000000000bb 000000000000d032 
>> GPR04: 00000000000000bb 0000000000000400 000000000000000a 0000000000000002 
>> GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 
>> GPR12: 0000000000000000 c000000000734000 0000000000000064 00000000ffe6df08 
>> GPR16: 00000000105b0000 00000000105b0000 0000000010440000 00000000105b0000 
>> GPR20: 00000000ffe6e008 00000000105b0000 00000000105b0000 000000000000000a 
>> GPR24: 000000000ffec408 0000000000000001 00000000ffe6ddca 0000000000000400 
>> GPR28: 000000000ffec408 00000000f7ff8000 000000000ffebff4 0000000000000400 
>> NIP [c000000000008c8c] restore+0x8c/0xc0
>> LR [000000000ff0135c] 0xff0135c
>> Call Trace:
>> [c000000772343e30] [c000000000008cd4] do_work+0x14/0x2c (unreliable)
>> Instruction dump:
>> 7c840078 7c810164 70604000 41820028 60000000 7c4c42e6 e88d01f0 f84d01f0 
>> 7c841050 e84d01e8 7c422214 f84d01e8 <e9a100d8> 7c7b03a6 e84101a0 7c4ff120 
>>
<snip>
> The exception happened at c...8c8c but you looked at c...8cdc with
> gdb.  What's at c...8c8c?
> 
>> please let me know if you need more information.
> 
> The .config would be useful, but don't spam everyone on cc with it,
> just send it to me privately.
> 
> Paul.

Hi Paul,

Similar call trace was seen in 2.6.24-rc3-git2 kernel while bootup, I have attached the
boot log to bugzilla (http://bugzilla.kernel.org/attachment.cgi?id=15666&action=view).
When looking for the last good one, we found that the kernel oops seems to be reproducible 
from the 2.6.24-rc8-git3 kernel onwards.

Thanks to nishanth for pointing it out, Please let me know if you need more information.

-- 
Thanks & Regards,
Kamalesh Babulal,
Linux Technology Center,
IBM, ISTL.

Re: [BUG] 2.6.25-rc2-git4 - Regression Kernel oops while running kernbench and tbench on powerpc

[Paul Mackerras]

Kamalesh Babulal writes:

> The kernel oops after applying the patch. Some time it takes more than
> one run to reproduce it, it was reproducible in the second run this
> time.
> 
>  Unrecoverable exception 4100 at c000000000008c8c
> Oops: Unrecoverable exception, sig: 6 [#1]
> SMP NR_CPUS=128 NUMA pSeries
> Modules linked in:
> NIP: c000000000008c8c LR: 000000000ff0135c CTR: 000000000ff012f0
> REGS: c000000772343bb0 TRAP: 4100   Not tainted  (2.6.25-rc8-autotest)
> MSR: 8000000000001030 <ME,IR,DR>  CR: 44044228  XER: 00000000
> TASK = c00000077cfa0900[13437] 'cc1' THREAD: c000000772340000 CPU: 2
> GPR00: 0000000000004000 c000000772343e30 00000000000000bb 000000000000d032 
> GPR04: 00000000000000bb 0000000000000400 000000000000000a 0000000000000002 
> GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 
> GPR12: 0000000000000000 c000000000734000 0000000000000064 00000000ffe6df08 
> GPR16: 00000000105b0000 00000000105b0000 0000000010440000 00000000105b0000 
> GPR20: 00000000ffe6e008 00000000105b0000 00000000105b0000 000000000000000a 
> GPR24: 000000000ffec408 0000000000000001 00000000ffe6ddca 0000000000000400 
> GPR28: 000000000ffec408 00000000f7ff8000 000000000ffebff4 0000000000000400 
> NIP [c000000000008c8c] restore+0x8c/0xc0
> LR [000000000ff0135c] 0xff0135c
> Call Trace:
> [c000000772343e30] [c000000000008cd4] do_work+0x14/0x2c (unreliable)
> Instruction dump:
> 7c840078 7c810164 70604000 41820028 60000000 7c4c42e6 e88d01f0 f84d01f0 
> 7c841050 e84d01e8 7c422214 f84d01e8 <e9a100d8> 7c7b03a6 e84101a0 7c4ff120 

That looks like the bug that was supposed to be fixed by commit
44387e9ff25267c78a99229aca55ed750e9174c7, which is in 2.6.25-rc7 and
later.  

What was the SHA1 ID of the head commit for the kernel source that
gave you this oops?  Did you have any other patches besides the one I
sent you applied?

Paul.

Re: [BUG] 2.6.25-rc2-git4 - Regression Kernel oops while running kernbench and tbench on powerpc

[Kamalesh Babulal]

Paul Mackerras wrote:
> Kamalesh Babulal writes:
> 
>> The kernel oops after applying the patch. Some time it takes more than
>> one run to reproduce it, it was reproducible in the second run this
>> time.
>>
>>  Unrecoverable exception 4100 at c000000000008c8c
>> Oops: Unrecoverable exception, sig: 6 [#1]
>> SMP NR_CPUS=128 NUMA pSeries
>> Modules linked in:
>> NIP: c000000000008c8c LR: 000000000ff0135c CTR: 000000000ff012f0
>> REGS: c000000772343bb0 TRAP: 4100   Not tainted  (2.6.25-rc8-autotest)
>> MSR: 8000000000001030 <ME,IR,DR>  CR: 44044228  XER: 00000000
>> TASK = c00000077cfa0900[13437] 'cc1' THREAD: c000000772340000 CPU: 2
>> GPR00: 0000000000004000 c000000772343e30 00000000000000bb 000000000000d032 
>> GPR04: 00000000000000bb 0000000000000400 000000000000000a 0000000000000002 
>> GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 
>> GPR12: 0000000000000000 c000000000734000 0000000000000064 00000000ffe6df08 
>> GPR16: 00000000105b0000 00000000105b0000 0000000010440000 00000000105b0000 
>> GPR20: 00000000ffe6e008 00000000105b0000 00000000105b0000 000000000000000a 
>> GPR24: 000000000ffec408 0000000000000001 00000000ffe6ddca 0000000000000400 
>> GPR28: 000000000ffec408 00000000f7ff8000 000000000ffebff4 0000000000000400 
>> NIP [c000000000008c8c] restore+0x8c/0xc0
>> LR [000000000ff0135c] 0xff0135c
>> Call Trace:
>> [c000000772343e30] [c000000000008cd4] do_work+0x14/0x2c (unreliable)
>> Instruction dump:
>> 7c840078 7c810164 70604000 41820028 60000000 7c4c42e6 e88d01f0 f84d01f0 
>> 7c841050 e84d01e8 7c422214 f84d01e8 <e9a100d8> 7c7b03a6 e84101a0 7c4ff120 
> 
> That looks like the bug that was supposed to be fixed by commit
> 44387e9ff25267c78a99229aca55ed750e9174c7, which is in 2.6.25-rc7 and
> later.  
> 
> What was the SHA1 ID of the head commit for the kernel source that
> gave you this oops?  Did you have any other patches besides the one I
> sent you applied?
> 
> Paul.

The SHA1 ID of the kernel is 0e81a8ae37687845f7cdfa2adce14ea6a5f1dd34 (2.6.25-rc8) 
and the source seems to have the patch 44387e9ff25267c78a99229aca55ed750e9174c7.

The kernel was patched only the patch you gave me (http://lkml.org/lkml/2008/4/8/42). 

-- 
Thanks & Regards,
Kamalesh Babulal,
Linux Technology Center,
IBM, ISTL.

Re: [BUG] 2.6.25-rc2-git4 - Regression Kernel oops while running kernbench and tbench on powerpc

[Paul Mackerras]

Kamalesh Babulal writes:

> The SHA1 ID of the kernel is 0e81a8ae37687845f7cdfa2adce14ea6a5f1dd34 (2.6.25-rc8) 
> and the source seems to have the patch 44387e9ff25267c78a99229aca55ed750e9174c7.
> 
> The kernel was patched only the patch you gave me (http://lkml.org/lkml/2008/4/8/42). 

Please try again with both that patch and the one below.  Once again
it won't fix the bug but will give us more information.  When the oops
occurs, the kernel will print a lot of debug information that should
help locate the problem.

Paul.

diff --git a/arch/powerpc/kernel/asm-offsets.c b/arch/powerpc/kernel/asm-offsets.c
index e932b43..f16db50 100644
--- a/arch/powerpc/kernel/asm-offsets.c
+++ b/arch/powerpc/kernel/asm-offsets.c
@@ -144,6 +144,9 @@ int main(void)
 	DEFINE(PACA_SLBSHADOWPTR, offsetof(struct paca_struct, slb_shadow_ptr));
 	DEFINE(PACA_DATA_OFFSET, offsetof(struct paca_struct, data_offset));
 	DEFINE(PACA_TRAP_SAVE, offsetof(struct paca_struct, trap_save));
+	DEFINE(PACASLBLOG, offsetof(struct paca_struct, slblog));
+	DEFINE(PACASLBLOGIX, offsetof(struct paca_struct, slblog_ix));
+	DEFINE(PACALASTSLB, offsetof(struct paca_struct, last_slb));
 
 	DEFINE(SLBSHADOW_STACKVSID,
 	       offsetof(struct slb_shadow, save_area[SLB_NUM_BOLTED - 1].vsid));
diff --git a/arch/powerpc/kernel/entry_64.S b/arch/powerpc/kernel/entry_64.S
index 148a354..663df17 100644
--- a/arch/powerpc/kernel/entry_64.S
+++ b/arch/powerpc/kernel/entry_64.S
@@ -419,6 +419,18 @@ END_FTR_SECTION_IFSET(CPU_FTR_1T_SEGMENT)
 	slbmte	r7,r0
 	isync
 
+	ld	r4,PACASLBLOGIX(r13)
+	addi	r4,r4,1
+	clrldi	r4,r4,64-6
+	std	r4,PACASLBLOGIX(r13)
+	add	r4,r4,r13
+	addi	r4,r4,PACASLBLOG
+	li	r5,4
+	std	r5,0(r4)
+	mftb	r5
+	std	r5,8(r4)
+	std	r6,16(r4)
+	std	r0,24(r4)
 2:
 	clrrdi	r7,r8,THREAD_SHIFT	/* base of new stack */
 	/* Note: this uses SWITCH_FRAME_SIZE rather than INT_FRAME_SIZE
@@ -533,6 +545,17 @@ END_FW_FTR_SECTION_IFSET(FW_FEATURE_ISERIES)
 
 	stdcx.	r0,0,r1		/* to clear the reservation */
 
+	li	r4,0
+	slbmfee	r2,r4
+	std	r2,PACALASTSLB(r13)
+	slbmfev	r2,r4
+	std	r2,PACALASTSLB+8(r13)
+	li	r4,1
+	slbmfee	r2,r4
+	std	r2,PACALASTSLB+16(r13)
+	slbmfev	r2,r4
+	std	r2,PACALASTSLB+24(r13)
+
 	/*
 	 * Clear RI before restoring r13.  If we are returning to
 	 * userspace and we take an exception after restoring r13,
diff --git a/arch/powerpc/kernel/traps.c b/arch/powerpc/kernel/traps.c
index 4b5b7ff..c918f33 100644
--- a/arch/powerpc/kernel/traps.c
+++ b/arch/powerpc/kernel/traps.c
@@ -1141,6 +1141,40 @@ void SPEFloatingPointException(struct pt_regs *regs)
 }
 #endif
 
+static void dump_unrecov_slb(void)
+{
+#ifdef CONFIG_PPC64
+	long entry, rstart;
+	unsigned long esid, vsid;
+
+	printk(KERN_EMERG "SLB contents now:\n");
+	for (entry = 0; entry < 64; ++entry) {
+		asm volatile("slbmfee  %0,%1" : "=r" (esid) : "r" (entry));
+		if (esid == 0)
+			/* valid bit is clear along with everything else */
+			continue;
+		asm volatile("slbmfev  %0,%1" : "=r" (vsid) : "r" (entry));
+		printk(KERN_EMERG "%d: %.16lx %.16lx\n", entry, esid, vsid);
+	}
+
+	printk(KERN_EMERG "SLB 0-1 at last exception exit:\n");
+	printk(KERN_EMERG "0: %.16lx %.16lx\n", get_paca()->last_slb[0][0],
+	       get_paca()->last_slb[0][1]);
+	printk(KERN_EMERG "1: %.16lx %.16lx\n", get_paca()->last_slb[1][0],
+	       get_paca()->last_slb[1][1]);
+	printk(KERN_EMERG "SLB update log:\n");
+	rstart = entry = get_paca()->slblog_ix;
+	do {
+		printk(KERN_EMERG "%d: %lx %lx %.16lx %.16lx\n", entry,
+		       get_paca()->slblog[entry][0],
+		       get_paca()->slblog[entry][1],
+		       get_paca()->slblog[entry][2],
+		       get_paca()->slblog[entry][3]);
+		entry = (entry + 1) % 63;
+	} while (entry != rstart);
+#endif
+}
+
 /*
  * We enter here if we get an unrecoverable exception, that is, one
  * that happened at a point where the RI (recoverable interrupt) bit
@@ -1151,6 +1185,8 @@ void unrecoverable_exception(struct pt_regs *regs)
 {
 	printk(KERN_EMERG "Unrecoverable exception %lx at %lx\n",
 	       regs->trap, regs->nip);
+	if (regs->trap == 0x4100)
+		dump_unrecov_slb();
 	die("Unrecoverable exception", regs, SIGABRT);
 }
 
diff --git a/arch/powerpc/mm/slb.c b/arch/powerpc/mm/slb.c
index 906daed..235edf7 100644
--- a/arch/powerpc/mm/slb.c
+++ b/arch/powerpc/mm/slb.c
@@ -105,6 +105,7 @@ void slb_flush_and_rebolt(void)
 	 * appropriately too. */
 	unsigned long linear_llp, vmalloc_llp, lflags, vflags;
 	unsigned long ksp_esid_data, ksp_vsid_data;
+	long logix;
 
 	WARN_ON(!irqs_disabled());
 
@@ -144,6 +145,13 @@ void slb_flush_and_rebolt(void)
 		        "r"(ksp_vsid_data),
 		        "r"(ksp_esid_data)
 		     : "memory");
+	logix = get_paca()->slblog_ix;
+	logix = (logix + 1) & 63;
+	get_paca()->slblog_ix = logix;
+	get_paca()->slblog[logix][0] = 3;
+	get_paca()->slblog[logix][1] = mftb();
+	get_paca()->slblog[logix][2] = ksp_esid_data;
+	get_paca()->slblog[logix][3] = ksp_vsid_data;
 }
 
 void slb_vmalloc_update(void)
@@ -192,6 +200,7 @@ void switch_slb(struct task_struct *tsk, struct mm_struct *mm)
 	unsigned long pc = KSTK_EIP(tsk);
 	unsigned long stack = KSTK_ESP(tsk);
 	unsigned long unmapped_base;
+	long logix;
 
 	if (!cpu_has_feature(CPU_FTR_NO_SLBIE_B) &&
 	    offset <= SLB_CACHE_ENTRIES) {
@@ -204,6 +213,14 @@ void switch_slb(struct task_struct *tsk, struct mm_struct *mm)
 				<< SLBIE_SSIZE_SHIFT;
 			slbie_data |= SLBIE_C; /* C set for user addresses */
 			asm volatile("slbie %0" : : "r" (slbie_data));
+
+			logix = get_paca()->slblog_ix;
+			logix = (logix + 1) & 63;
+			get_paca()->slblog_ix = logix;
+			get_paca()->slblog[logix][0] = 2;
+			get_paca()->slblog[logix][1] = mftb();
+			get_paca()->slblog[logix][2] = slbie_data;
+			get_paca()->slblog[logix][3] = 0;
 		}
 		asm volatile("isync" : : : "memory");
 	} else {
diff --git a/arch/powerpc/mm/slb_low.S b/arch/powerpc/mm/slb_low.S
index 657f6b3..8c7ce20 100644
--- a/arch/powerpc/mm/slb_low.S
+++ b/arch/powerpc/mm/slb_low.S
@@ -249,6 +249,20 @@ _GLOBAL(slb_compare_rr_to_size)
 	 */
 	slbmte	r11,r10
 
+	ld	r3,PACASLBLOGIX(r13)
+	addi	r3,r3,1
+	clrldi	r3,r3,64-6
+	std	r3,PACASLBLOGIX(r13)
+	sldi	r3,r3,5
+	add	r3,r3,r13
+	addi	r3,r3,PACASLBLOG
+	li	r9,1
+	std	r9,0(r3)
+	mftb	r9
+	std	r9,8(r3)
+	std	r11,16(r3)
+	std	r10,24(r3)
+
 	/* we're done for kernel addresses */
 	crclr	4*cr0+eq		/* set result to "success" */
 	bgelr	cr7
diff --git a/arch/powerpc/platforms/pseries/ras.c b/arch/powerpc/platforms/pseries/ras.c
index a1ab25c..959ef26 100644
--- a/arch/powerpc/platforms/pseries/ras.c
+++ b/arch/powerpc/platforms/pseries/ras.c
@@ -325,6 +325,8 @@ static int recover_mce(struct pt_regs *regs, struct rtas_error_log * err)
 
 	if (err->disposition == RTAS_DISP_FULLY_RECOVERED) {
 		/* Platform corrected itself */
+		printk(KERN_ERR "FWNMI: platform corrected error %.16lx\n",
+		       *(unsigned long *)err);
 		nonfatal = 1;
 	} else if ((regs->msr & MSR_RI) &&
 		   user_mode(regs) &&
diff --git a/include/asm-powerpc/paca.h b/include/asm-powerpc/paca.h
index 748b35a..6280b82 100644
--- a/include/asm-powerpc/paca.h
+++ b/include/asm-powerpc/paca.h
@@ -115,6 +115,11 @@ struct paca_struct {
 	u64 system_time;		/* accumulated system TB ticks */
 	u64 startpurr;			/* PURR/TB value snapshot */
 	u64 startspurr;			/* SPURR value snapshot */
+
+	/* SLB update log */
+	long slblog_ix;
+	u64 slblog[64][4];
+	u64 last_slb[2][2];
 };
 
 extern struct paca_struct paca[];

Re: [BUG] 2.6.25-rc2-git4 - Regression Kernel oops while running kernbench and tbench on powerpc

[Kamalesh Babulal]

Paul Mackerras wrote:
> Kamalesh Babulal writes:
> 
>> The SHA1 ID of the kernel is 0e81a8ae37687845f7cdfa2adce14ea6a5f1dd34 (2.6.25-rc8) 
>> and the source seems to have the patch 44387e9ff25267c78a99229aca55ed750e9174c7.
>>
>> The kernel was patched only the patch you gave me (http://lkml.org/lkml/2008/4/8/42). 
> 
> Please try again with both that patch and the one below.  Once again
> it won't fix the bug but will give us more information.  When the oops
> occurs, the kernel will print a lot of debug information that should
> help locate the problem.
> 
> Paul.
> 
> diff --git a/arch/powerpc/kernel/asm-offsets.c b/arch/powerpc/kernel/asm-offsets.c
> index e932b43..f16db50 100644
> --- a/arch/powerpc/kernel/asm-offsets.c
> +++ b/arch/powerpc/kernel/asm-offsets.c
> @@ -144,6 +144,9 @@ int main(void)
>  	DEFINE(PACA_SLBSHADOWPTR, offsetof(struct paca_struct, slb_shadow_ptr));
>  	DEFINE(PACA_DATA_OFFSET, offsetof(struct paca_struct, data_offset));
>  	DEFINE(PACA_TRAP_SAVE, offsetof(struct paca_struct, trap_save));
> +	DEFINE(PACASLBLOG, offsetof(struct paca_struct, slblog));
> +	DEFINE(PACASLBLOGIX, offsetof(struct paca_struct, slblog_ix));
> +	DEFINE(PACALASTSLB, offsetof(struct paca_struct, last_slb));
> 
>  	DEFINE(SLBSHADOW_STACKVSID,
>  	       offsetof(struct slb_shadow, save_area[SLB_NUM_BOLTED - 1].vsid));
> diff --git a/arch/powerpc/kernel/entry_64.S b/arch/powerpc/kernel/entry_64.S
> index 148a354..663df17 100644
> --- a/arch/powerpc/kernel/entry_64.S
> +++ b/arch/powerpc/kernel/entry_64.S
> @@ -419,6 +419,18 @@ END_FTR_SECTION_IFSET(CPU_FTR_1T_SEGMENT)
>  	slbmte	r7,r0
>  	isync
> 
> +	ld	r4,PACASLBLOGIX(r13)
> +	addi	r4,r4,1
> +	clrldi	r4,r4,64-6
> +	std	r4,PACASLBLOGIX(r13)
> +	add	r4,r4,r13
> +	addi	r4,r4,PACASLBLOG
> +	li	r5,4
> +	std	r5,0(r4)
> +	mftb	r5
> +	std	r5,8(r4)
> +	std	r6,16(r4)
> +	std	r0,24(r4)
>  2:
>  	clrrdi	r7,r8,THREAD_SHIFT	/* base of new stack */
>  	/* Note: this uses SWITCH_FRAME_SIZE rather than INT_FRAME_SIZE
> @@ -533,6 +545,17 @@ END_FW_FTR_SECTION_IFSET(FW_FEATURE_ISERIES)
> 
>  	stdcx.	r0,0,r1		/* to clear the reservation */
> 
> +	li	r4,0
> +	slbmfee	r2,r4
> +	std	r2,PACALASTSLB(r13)
> +	slbmfev	r2,r4
> +	std	r2,PACALASTSLB+8(r13)
> +	li	r4,1
> +	slbmfee	r2,r4
> +	std	r2,PACALASTSLB+16(r13)
> +	slbmfev	r2,r4
> +	std	r2,PACALASTSLB+24(r13)
> +
>  	/*
>  	 * Clear RI before restoring r13.  If we are returning to
>  	 * userspace and we take an exception after restoring r13,
> diff --git a/arch/powerpc/kernel/traps.c b/arch/powerpc/kernel/traps.c
> index 4b5b7ff..c918f33 100644
> --- a/arch/powerpc/kernel/traps.c
> +++ b/arch/powerpc/kernel/traps.c
> @@ -1141,6 +1141,40 @@ void SPEFloatingPointException(struct pt_regs *regs)
>  }
>  #endif
> 
> +static void dump_unrecov_slb(void)
> +{
> +#ifdef CONFIG_PPC64
> +	long entry, rstart;
> +	unsigned long esid, vsid;
> +
> +	printk(KERN_EMERG "SLB contents now:\n");
> +	for (entry = 0; entry < 64; ++entry) {
> +		asm volatile("slbmfee  %0,%1" : "=r" (esid) : "r" (entry));
> +		if (esid == 0)
> +			/* valid bit is clear along with everything else */
> +			continue;
> +		asm volatile("slbmfev  %0,%1" : "=r" (vsid) : "r" (entry));
> +		printk(KERN_EMERG "%d: %.16lx %.16lx\n", entry, esid, vsid);
> +	}
> +
> +	printk(KERN_EMERG "SLB 0-1 at last exception exit:\n");
> +	printk(KERN_EMERG "0: %.16lx %.16lx\n", get_paca()->last_slb[0][0],
> +	       get_paca()->last_slb[0][1]);
> +	printk(KERN_EMERG "1: %.16lx %.16lx\n", get_paca()->last_slb[1][0],
> +	       get_paca()->last_slb[1][1]);
> +	printk(KERN_EMERG "SLB update log:\n");
> +	rstart = entry = get_paca()->slblog_ix;
> +	do {
> +		printk(KERN_EMERG "%d: %lx %lx %.16lx %.16lx\n", entry,
> +		       get_paca()->slblog[entry][0],
> +		       get_paca()->slblog[entry][1],
> +		       get_paca()->slblog[entry][2],
> +		       get_paca()->slblog[entry][3]);
> +		entry = (entry + 1) % 63;
> +	} while (entry != rstart);
> +#endif
> +}
> +
>  /*
>   * We enter here if we get an unrecoverable exception, that is, one
>   * that happened at a point where the RI (recoverable interrupt) bit
> @@ -1151,6 +1185,8 @@ void unrecoverable_exception(struct pt_regs *regs)
>  {
>  	printk(KERN_EMERG "Unrecoverable exception %lx at %lx\n",
>  	       regs->trap, regs->nip);
> +	if (regs->trap == 0x4100)
> +		dump_unrecov_slb();
>  	die("Unrecoverable exception", regs, SIGABRT);
>  }
> 
> diff --git a/arch/powerpc/mm/slb.c b/arch/powerpc/mm/slb.c
> index 906daed..235edf7 100644
> --- a/arch/powerpc/mm/slb.c
> +++ b/arch/powerpc/mm/slb.c
> @@ -105,6 +105,7 @@ void slb_flush_and_rebolt(void)
>  	 * appropriately too. */
>  	unsigned long linear_llp, vmalloc_llp, lflags, vflags;
>  	unsigned long ksp_esid_data, ksp_vsid_data;
> +	long logix;
> 
>  	WARN_ON(!irqs_disabled());
> 
> @@ -144,6 +145,13 @@ void slb_flush_and_rebolt(void)
>  		        "r"(ksp_vsid_data),
>  		        "r"(ksp_esid_data)
>  		     : "memory");
> +	logix = get_paca()->slblog_ix;
> +	logix = (logix + 1) & 63;
> +	get_paca()->slblog_ix = logix;
> +	get_paca()->slblog[logix][0] = 3;
> +	get_paca()->slblog[logix][1] = mftb();
> +	get_paca()->slblog[logix][2] = ksp_esid_data;
> +	get_paca()->slblog[logix][3] = ksp_vsid_data;
>  }
> 
>  void slb_vmalloc_update(void)
> @@ -192,6 +200,7 @@ void switch_slb(struct task_struct *tsk, struct mm_struct *mm)
>  	unsigned long pc = KSTK_EIP(tsk);
>  	unsigned long stack = KSTK_ESP(tsk);
>  	unsigned long unmapped_base;
> +	long logix;
> 
>  	if (!cpu_has_feature(CPU_FTR_NO_SLBIE_B) &&
>  	    offset <= SLB_CACHE_ENTRIES) {
> @@ -204,6 +213,14 @@ void switch_slb(struct task_struct *tsk, struct mm_struct *mm)
>  				<< SLBIE_SSIZE_SHIFT;
>  			slbie_data |= SLBIE_C; /* C set for user addresses */
>  			asm volatile("slbie %0" : : "r" (slbie_data));
> +
> +			logix = get_paca()->slblog_ix;
> +			logix = (logix + 1) & 63;
> +			get_paca()->slblog_ix = logix;
> +			get_paca()->slblog[logix][0] = 2;
> +			get_paca()->slblog[logix][1] = mftb();
> +			get_paca()->slblog[logix][2] = slbie_data;
> +			get_paca()->slblog[logix][3] = 0;
>  		}
>  		asm volatile("isync" : : : "memory");
>  	} else {
> diff --git a/arch/powerpc/mm/slb_low.S b/arch/powerpc/mm/slb_low.S
> index 657f6b3..8c7ce20 100644
> --- a/arch/powerpc/mm/slb_low.S
> +++ b/arch/powerpc/mm/slb_low.S
> @@ -249,6 +249,20 @@ _GLOBAL(slb_compare_rr_to_size)
>  	 */
>  	slbmte	r11,r10
> 
> +	ld	r3,PACASLBLOGIX(r13)
> +	addi	r3,r3,1
> +	clrldi	r3,r3,64-6
> +	std	r3,PACASLBLOGIX(r13)
> +	sldi	r3,r3,5
> +	add	r3,r3,r13
> +	addi	r3,r3,PACASLBLOG
> +	li	r9,1
> +	std	r9,0(r3)
> +	mftb	r9
> +	std	r9,8(r3)
> +	std	r11,16(r3)
> +	std	r10,24(r3)
> +
>  	/* we're done for kernel addresses */
>  	crclr	4*cr0+eq		/* set result to "success" */
>  	bgelr	cr7
> diff --git a/arch/powerpc/platforms/pseries/ras.c b/arch/powerpc/platforms/pseries/ras.c
> index a1ab25c..959ef26 100644
> --- a/arch/powerpc/platforms/pseries/ras.c
> +++ b/arch/powerpc/platforms/pseries/ras.c
> @@ -325,6 +325,8 @@ static int recover_mce(struct pt_regs *regs, struct rtas_error_log * err)
> 
>  	if (err->disposition == RTAS_DISP_FULLY_RECOVERED) {
>  		/* Platform corrected itself */
> +		printk(KERN_ERR "FWNMI: platform corrected error %.16lx\n",
> +		       *(unsigned long *)err);
>  		nonfatal = 1;
>  	} else if ((regs->msr & MSR_RI) &&
>  		   user_mode(regs) &&
> diff --git a/include/asm-powerpc/paca.h b/include/asm-powerpc/paca.h
> index 748b35a..6280b82 100644
> --- a/include/asm-powerpc/paca.h
> +++ b/include/asm-powerpc/paca.h
> @@ -115,6 +115,11 @@ struct paca_struct {
>  	u64 system_time;		/* accumulated system TB ticks */
>  	u64 startpurr;			/* PURR/TB value snapshot */
>  	u64 startspurr;			/* SPURR value snapshot */
> +
> +	/* SLB update log */
> +	long slblog_ix;
> +	u64 slblog[64][4];
> +	u64 last_slb[2][2];
>  };
> 
>  extern struct paca_struct paca[];
Hi Paul,

After applying the patch above and the patch posted on http://lkml.org/lkml/2008/4/8/42
the bug had the following information,

 Unrecoverable exception 4100 at c000000000008d4c
SLB contents now:
0: c000000008000000 0000408f92c94500
1: d000000008000000 0000f09b89af5400
2: c000000020000000 0000420e6f8ca500
3: 0000000010000000 0000947fa10bac80
4: 00000000f0000000 00009ef7aa634c80
5: 0000000040000000 000096bdec30bc80
8: 00000000f0000000 00002292895c1c80
9: 0000000040000000 00001a58cb298c80
10: 0000000010000000 0000181a80047c80
12: 00000000f0000000 0000273e59afdc80
13: 0000000040000000 00001f049b7d4c80
14: 0000000010000000 00001cc650583c80
16: 00000000f0000000 00007bbb0a7b3c80
17: 0000000040000000 000073814c48ac80
18: 0000000010000000 0000714301239c80
20: 00000000f0000000 00009ef7aa634c80
21: 0000000040000000 000096bdec30bc80
22: 0000000010000000 0000947fa10bac80
23: c000000718000000 0000950f4be7f500
24: c000000728000000 000095ceba49a500
25: cf00000008000000 0000d59aca40f500
26: 0000000018000000 00004e06613b8c80
27: 00000000f8000000 0000587e6a932c80
28: 0000000048000000 00005044ac609c80
29: c000000778000000 0000998be2321500
30: 00000000f0000000 000008ad8a1b8c80
31: 0000000040000000 00000073cbe8fc80
32: 0000000010000000 0000fe3580c3dc80
33: c000000028000000 0000420e6f8ca500
34: c000000758000000 0000980d056eb500
36: 00000000f0000000 00007bbb0a7b3c80
37: 0000000040000000 000073814c48ac80
38: 0000000010000000 0000714301239c80
39: c000000038000000 000042cdddee5500
40: c000000768000000 000098cc73d06500
41: c000000738000000 0000968e28ab5500
43: 00000000f0000000 000095a009bbcc80
44: 0000000040000000 00008d664b893c80
45: 0000000010000000 00008b2800642c80
47: 00000000f0000000 00009ef7aa634c80
48: 0000000040000000 000096bdec30bc80
49: 0000000010000000 0000947fa10bac80
51: 00000000f0000000 00007bbb0a7b3c80
52: 0000000040000000 000073814c48ac80
53: cf00000018000000 0000d65a38a2a500
54: 0000000010000000 0000714301239c80
55: c000000748000000 0000974d970d0500
57: 00000000f0000000 00009ef7aa634c80
58: 0000000040000000 000096bdec30bc80
59: 0000000010000000 0000947fa10bac80
61: 00000000f0000000 0000f5fe48cc7c80
62: 0000000040000000 0000edc48a99ec80
63: 0000000010000000 0000eb863f74dc80
SLB 0-1 at last exception exit:
0: c000000008000000 0000408f92c94500
1: d000000008000000 0000f09b89af5400
SLB update log:
4: 1 1fa087dccefc17 0000998be2321500 c00000077800001d
5: 2 1fa087dbeb2091 0000000018000000 0000000000000000
6: 1 1fa087dbeb20ac 000093c032a9fc80 0000000008000038
7: 1 1fa087dbeb20bd 00009ef7aa634c80 00000000f8000039
8: 1 1fa087dbeb20d1 000096bdec30bc80 000000004800003a
9: 1 1fa087dbeb37d5 0000947fa10bac80 000000001800003b
10: 2 1fa087dc26370a 0000000008000000 0000000000000000
11: 2 1fa087dc26370f 00000000f8000000 0000000000000000
12: 2 1fa087dc26372f 0000000048000000 0000000000000000
13: 2 1fa087dc263734 0000000018000000 0000000000000000
14: 1 1fa087dc26375f 0000eac6d1132c80 000000000800003c
15: 1 1fa087dc263772 0000f5fe48cc7c80 00000000f800003d
16: 1 1fa087dc263787 0000edc48a99ec80 000000004800003e
17: 1 1fa087dc263bc6 0000eb863f74dc80 000000001800003f
18: 2 1fa087dc264698 0000000008000000 0000000000000000
19: 2 1fa087dc26469e 00000000f8000000 0000000000000000
20: 2 1fa087dc2646a3 0000000048000000 0000000000000000
21: 2 1fa087dc2646a8 0000000018000000 0000000000000000
22: 1 1fa087dc2646be 0000947fa10bac80 0000000018000003
23: 1 1fa087dc2646cd 00009ef7aa634c80 00000000f8000004
24: 1 1fa087dc2646e2 000096bdec30bc80 0000000048000005
25: 1 1fa087dc264829 000093c032a9fc80 0000000008000006
26: 2 1fa087dc7695e9 0000000018000000 0000000000000000
27: 2 1fa087dc7695ee 00000000f8000000 0000000000000000
28: 2 1fa087dc7695f6 0000000048000000 0000000000000000
29: 2 1fa087dc7695fc 0000000008000000 0000000000000000
30: 1 1fa087dc769623 0000175b11a2cc80 0000000008000007
31: 1 1fa087dc769636 00002292895c1c80 00000000f8000008
32: 1 1fa087dc76964b 00001a58cb298c80 0000000048000009
33: 1 1fa087dc76a03d 0000181a80047c80 000000001800000a
34: 2 1fa087dc7840e0 0000000008000000 0000000000000000
35: 2 1fa087dc7840e5 00000000f8000000 0000000000000000
36: 2 1fa087dc784103 0000000048000000 0000000000000000
37: 2 1fa087dc784108 0000000018000000 0000000000000000
38: 1 1fa087dc784134 00001c06e1f68c80 000000000800000b
39: 1 1fa087dc784145 0000273e59afdc80 00000000f800000c
40: 1 1fa087dc78415a 00001f049b7d4c80 000000004800000d
41: 1 1fa087dc78542a 00001cc650583c80 000000001800000e
42: 2 1fa087dc84f844 0000000008000000 0000000000000000
43: 2 1fa087dc84f849 00000000f8000000 0000000000000000
44: 2 1fa087dc84f869 0000000048000000 0000000000000000
45: 2 1fa087dc84f86e 0000000018000000 0000000000000000
46: 1 1fa087dc84f891 0000708392c1ec80 000000000800000f
47: 1 1fa087dc84f8a5 00007bbb0a7b3c80 00000000f8000010
48: 1 1fa087dc84f8c3 000073814c48ac80 0000000048000011
49: 1 1fa087dc84fb2a 0000714301239c80 0000000018000012
50: 2 1fa087dc851369 0000000008000000 0000000000000000
51: 2 1fa087dc85136f 00000000f8000000 0000000000000000
52: 2 1fa087dc851374 0000000048000000 0000000000000000
53: 2 1fa087dc851379 0000000018000000 0000000000000000
54: 1 1fa087dc8513a2 000093c032a9fc80 0000000008000013
55: 1 1fa087dc8513b5 00009ef7aa634c80 00000000f8000014
56: 1 1fa087dc8513c5 000096bdec30bc80 0000000048000015
57: 1 1fa087dc85158f 0000947fa10bac80 0000000018000016
58: 1 1fa087dc858603 0000950f4be7f500 c000000718000017
59: 1 1fa087dc85aa02 000095ceba49a500 c000000728000018
60: 1 1fa087dcb5b5ea 0000d59aca40f500 cf00000008000019
61: 2 1fa087dccefa5a 0000000008000000 0000000000000000
62: 2 1fa087dccefa5f 00000000f8000000 0000000000000000
0: 2 1fa087dccefa69 0000000018000000 0000000000000000
1: 1 1fa087dccefa8f 00004e06613b8c80 000000001800001a
2: 1 1fa087dccefaa4 0000587e6a932c80 00000000f800001b
3: 1 1fa087dccefac6 00005044ac609c80 000000004800001c
Oops: Unrecoverable exception, sig: 6 [#1]
SMP NR_CPUS=128 NUMA pSeries
Modules linked in:
NIP: c000000000008d4c LR: 00000000102e9790 CTR: 00000000102686c0
REGS: c00000077304fbb0 TRAP: 4100   Not tainted  (2.6.25-rc8-autotest)
MSR: 8000000000001030 <ME,IR,DR>  CR: 28002488  XER: 20000000
TASK = c000000774bb3200[9954] 'cc1' THREAD: c00000077304c000 CPU: 1
GPR00: 0000000000004000 c00000077304fe30 00000000102e929c 000000000000d032 
GPR04: 00000000000000bc 0000000000000000 0000000000000000 0000000000000000 
GPR08: 0000000000000037 0000000010440000 00000000f765d1c0 00000000f765c240 
GPR12: 0000000048002488 00000000105ba630 0000000010030000 0000000010030000 
GPR16: 00000000105b0000 00000000105b0000 0000000010440000 00000000ff9d92d8 
GPR20: 000000001043b8f4 00000000102686c0 00000000ff9d91d8 0000000000000000 
GPR24: 0000000000000000 0000000010071140 0000000000000000 0000000000000000 
GPR28: 00000000105b39bc 00000000f765c530 00000000f7653770 00000000f764fbe0 
NIP [c000000000008d4c] restore+0xcc/0xe8
LR [00000000102e9790] 0x102e9790
Call Trace:
[c00000077304fe30] [c000000000008d7c] do_work+0x14/0x2c (unreliable)
Instruction dump:
e88d01f0 f84d01f0 7c841050 e84d01e8 7c422214 f84d01e8 e9a100d8 7c7b03a6 
e84101a0 7c4ff120 e8410170 7c5a03a6 <e8010070> e8410080 e8610088 e8810090 
---[ end trace 1d1912fbf2b044ad ]---

-- 
Thanks & Regards,
Kamalesh Babulal,
Linux Technology Center,
IBM, ISTL.

Re: [BUG] 2.6.25-rc2-git4 - Regression Kernel oops while running kernbench and tbench on powerpc

[Paul Mackerras]

Kamalesh Babulal writes:

> After applying the patch above and the patch posted on
> http://lkml.org/lkml/2008/4/8/42
> the bug had the following information,

Thanks.  The patch below, against Linus' current git tree, fixes one
bug that might be the cause of the problem, and also attempts to
detect the erroneous situation earlier and fix it up, and also print
some debug information.  Please try to reproduce the problem with this
patch applied, and if there are any console log messages starting with
SLB: or FWNMI:, please send me the console log.

Paul.

diff --git a/arch/powerpc/kernel/entry_64.S b/arch/powerpc/kernel/entry_64.S
index c0db5b7..f7f0962 100644
--- a/arch/powerpc/kernel/entry_64.S
+++ b/arch/powerpc/kernel/entry_64.S
@@ -439,6 +439,19 @@ END_FTR_SECTION_IFSET(CPU_FTR_1T_SEGMENT)
 	mr	r1,r8		/* start using new stack pointer */
 	std	r7,PACAKSAVE(r13)
 
+	/* check that SLB entry 2 contains the right thing */
+	clrrdi	r6,r1,28
+	clrldi.	r0,r6,2
+	beq	3f
+	li	r0,2
+	slbmfee	r7,r0
+	oris	r6,r6,SLB_ESID_V@h
+	cmpd	r6,r7
+	beq	3f
+	bl	bad_slb_switch
+	ld	r3,PACACURRENT(r13)
+	addi	r3,r3,THREAD
+3:
 	ld	r6,_CCR(r1)
 	mtcrf	0xFF,r6
 
@@ -540,6 +553,19 @@ END_FW_FTR_SECTION_IFSET(FW_FEATURE_ISERIES)
 	ld	r4,_XER(r1)
 	mtspr	SPRN_XER,r4
 
+	/* check that SLB entry 2 contains the right thing */
+	clrrdi	r6,r1,28	/* stack ESID */
+	clrldi.	r0,r6,2
+	beq	57f
+	li	r0,2
+	slbmfee	r7,r0
+	oris	r6,r6,SLB_ESID_V@h
+	cmpd	r6,r7
+	beq	57f
+	addi	r3,r1,STACK_FRAME_OVERHEAD
+	bl	bad_slb_exc
+	ld	r3,_MSR(r1)
+57:
 	REST_8GPRS(5, r1)
 
 	andi.	r0,r3,MSR_RI
diff --git a/arch/powerpc/kernel/smp.c b/arch/powerpc/kernel/smp.c
index be35ffa..c938134 100644
--- a/arch/powerpc/kernel/smp.c
+++ b/arch/powerpc/kernel/smp.c
@@ -45,6 +45,7 @@
 #include <asm/system.h>
 #include <asm/mpic.h>
 #include <asm/vdso_datapage.h>
+#include <asm/mmu.h>
 #ifdef CONFIG_PPC64
 #include <asm/paca.h>
 #endif
@@ -580,6 +581,10 @@ int __devinit start_secondary(void *unused)
 	atomic_inc(&init_mm.mm_count);
 	current->active_mm = &init_mm;
 
+	/* Bolt in the entry for the kernel stack now */
+	if (cpu_has_feature(CPU_FTR_SLB))
+		slb_flush_and_rebolt();
+
 	smp_store_cpu_info(cpu);
 	set_dec(tb_ticks_per_jiffy);
 	preempt_disable();
diff --git a/arch/powerpc/mm/slb.c b/arch/powerpc/mm/slb.c
index 906daed..bb7765b 100644
--- a/arch/powerpc/mm/slb.c
+++ b/arch/powerpc/mm/slb.c
@@ -309,3 +309,34 @@ void slb_initialize(void)
 	 * one. */
 	asm volatile("isync":::"memory");
 }
+
+static void dump_slb(void)
+{
+	long entry;
+	unsigned long esid, vsid;
+
+	printk(KERN_EMERG "SLB contents now:\n");
+	for (entry = 0; entry < 64; ++entry) {
+		asm volatile("slbmfee  %0,%1" : "=r" (esid) : "r" (entry));
+		if (esid == 0)
+			/* valid bit is clear along with everything else */
+			continue;
+		asm volatile("slbmfev  %0,%1" : "=r" (vsid) : "r" (entry));
+		printk(KERN_EMERG "%d: %.16lx %.16lx\n", entry, esid, vsid);
+	}
+}
+
+void bad_slb_exc(struct pt_regs *regs)
+{
+	printk(KERN_EMERG "SLB: stack not bolted on exception return\n");
+	dump_slb();
+	slb_flush_and_rebolt();
+	show_regs(regs);
+}
+
+void bad_slb_switch(void)
+{
+	printk(KERN_EMERG "SLB: stack not bolted on context switch\n");
+	dump_slb();
+	slb_flush_and_rebolt();
+}
diff --git a/arch/powerpc/platforms/pseries/ras.c b/arch/powerpc/platforms/pseries/ras.c
index a1ab25c..ed68083 100644
--- a/arch/powerpc/platforms/pseries/ras.c
+++ b/arch/powerpc/platforms/pseries/ras.c
@@ -325,6 +325,8 @@ static int recover_mce(struct pt_regs *regs, struct rtas_error_log * err)
 
 	if (err->disposition == RTAS_DISP_FULLY_RECOVERED) {
 		/* Platform corrected itself */
+		printk(KERN_ALERT "FWNMI: platform corrected error %.16lx\n",
+		       *(unsigned long *)err);
 		nonfatal = 1;
 	} else if ((regs->msr & MSR_RI) &&
 		   user_mode(regs) &&

Re: [BUG] 2.6.25-rc2-git4 - Regression Kernel oops while running kernbench and tbench on powerpc

[Kamalesh Babulal]

Paul Mackerras wrote:
> Kamalesh Babulal writes:
> 
>> After applying the patch above and the patch posted on
>> http://lkml.org/lkml/2008/4/8/42
>> the bug had the following information,
> 
> Thanks.  The patch below, against Linus' current git tree, fixes one
> bug that might be the cause of the problem, and also attempts to
> detect the erroneous situation earlier and fix it up, and also print
> some debug information.  Please try to reproduce the problem with this
> patch applied, and if there are any console log messages starting with
> SLB: or FWNMI:, please send me the console log.
> 
> Paul.
> 
> diff --git a/arch/powerpc/kernel/entry_64.S b/arch/powerpc/kernel/entry_64.S
> index c0db5b7..f7f0962 100644
> --- a/arch/powerpc/kernel/entry_64.S
> +++ b/arch/powerpc/kernel/entry_64.S
> @@ -439,6 +439,19 @@ END_FTR_SECTION_IFSET(CPU_FTR_1T_SEGMENT)
>  	mr	r1,r8		/* start using new stack pointer */
>  	std	r7,PACAKSAVE(r13)
> 
> +	/* check that SLB entry 2 contains the right thing */
> +	clrrdi	r6,r1,28
> +	clrldi.	r0,r6,2
> +	beq	3f
> +	li	r0,2
> +	slbmfee	r7,r0
> +	oris	r6,r6,SLB_ESID_V@h
> +	cmpd	r6,r7
> +	beq	3f
> +	bl	bad_slb_switch
> +	ld	r3,PACACURRENT(r13)
> +	addi	r3,r3,THREAD
> +3:
>  	ld	r6,_CCR(r1)
>  	mtcrf	0xFF,r6
> 
> @@ -540,6 +553,19 @@ END_FW_FTR_SECTION_IFSET(FW_FEATURE_ISERIES)
>  	ld	r4,_XER(r1)
>  	mtspr	SPRN_XER,r4
> 
> +	/* check that SLB entry 2 contains the right thing */
> +	clrrdi	r6,r1,28	/* stack ESID */
> +	clrldi.	r0,r6,2
> +	beq	57f
> +	li	r0,2
> +	slbmfee	r7,r0
> +	oris	r6,r6,SLB_ESID_V@h
> +	cmpd	r6,r7
> +	beq	57f
> +	addi	r3,r1,STACK_FRAME_OVERHEAD
> +	bl	bad_slb_exc
> +	ld	r3,_MSR(r1)
> +57:
>  	REST_8GPRS(5, r1)
> 
>  	andi.	r0,r3,MSR_RI
> diff --git a/arch/powerpc/kernel/smp.c b/arch/powerpc/kernel/smp.c
> index be35ffa..c938134 100644
> --- a/arch/powerpc/kernel/smp.c
> +++ b/arch/powerpc/kernel/smp.c
> @@ -45,6 +45,7 @@
>  #include <asm/system.h>
>  #include <asm/mpic.h>
>  #include <asm/vdso_datapage.h>
> +#include <asm/mmu.h>
>  #ifdef CONFIG_PPC64
>  #include <asm/paca.h>
>  #endif
> @@ -580,6 +581,10 @@ int __devinit start_secondary(void *unused)
>  	atomic_inc(&init_mm.mm_count);
>  	current->active_mm = &init_mm;
> 
> +	/* Bolt in the entry for the kernel stack now */
> +	if (cpu_has_feature(CPU_FTR_SLB))
> +		slb_flush_and_rebolt();
> +
>  	smp_store_cpu_info(cpu);
>  	set_dec(tb_ticks_per_jiffy);
>  	preempt_disable();
> diff --git a/arch/powerpc/mm/slb.c b/arch/powerpc/mm/slb.c
> index 906daed..bb7765b 100644
> --- a/arch/powerpc/mm/slb.c
> +++ b/arch/powerpc/mm/slb.c
> @@ -309,3 +309,34 @@ void slb_initialize(void)
>  	 * one. */
>  	asm volatile("isync":::"memory");
>  }
> +
> +static void dump_slb(void)
> +{
> +	long entry;
> +	unsigned long esid, vsid;
> +
> +	printk(KERN_EMERG "SLB contents now:\n");
> +	for (entry = 0; entry < 64; ++entry) {
> +		asm volatile("slbmfee  %0,%1" : "=r" (esid) : "r" (entry));
> +		if (esid == 0)
> +			/* valid bit is clear along with everything else */
> +			continue;
> +		asm volatile("slbmfev  %0,%1" : "=r" (vsid) : "r" (entry));
> +		printk(KERN_EMERG "%d: %.16lx %.16lx\n", entry, esid, vsid);
> +	}
> +}
> +
> +void bad_slb_exc(struct pt_regs *regs)
> +{
> +	printk(KERN_EMERG "SLB: stack not bolted on exception return\n");
> +	dump_slb();
> +	slb_flush_and_rebolt();
> +	show_regs(regs);
> +}
> +
> +void bad_slb_switch(void)
> +{
> +	printk(KERN_EMERG "SLB: stack not bolted on context switch\n");
> +	dump_slb();
> +	slb_flush_and_rebolt();
> +}
> diff --git a/arch/powerpc/platforms/pseries/ras.c b/arch/powerpc/platforms/pseries/ras.c
> index a1ab25c..ed68083 100644
> --- a/arch/powerpc/platforms/pseries/ras.c
> +++ b/arch/powerpc/platforms/pseries/ras.c
> @@ -325,6 +325,8 @@ static int recover_mce(struct pt_regs *regs, struct rtas_error_log * err)
> 
>  	if (err->disposition == RTAS_DISP_FULLY_RECOVERED) {
>  		/* Platform corrected itself */
> +		printk(KERN_ALERT "FWNMI: platform corrected error %.16lx\n",
> +		       *(unsigned long *)err);
>  		nonfatal = 1;
>  	} else if ((regs->msr & MSR_RI) &&
>  		   user_mode(regs) &&

Hi Paul,

Thanks, after applying the patch the oops is not reproducible on the machine. The console
log had no message starting with SLB: or FWNMI:. I have updated the bugzilla also.

Tested-by: Kamalesh Babulal <kamalesh@linux.vnet.ibm.com>

-- 
Thanks & Regards,
Kamalesh Babulal,
Linux Technology Center,
IBM, ISTL.

Re: [BUG] 2.6.25-rc2-git4 - Regression Kernel oops while running kernbench and tbench on powerpc

[Paul Mackerras]

Kamalesh Babulal writes:

> Thanks, after applying the patch the oops is not reproducible on the machine. The console
> log had no message starting with SLB: or FWNMI:. I have updated the bugzilla also.
> 
> Tested-by: Kamalesh Babulal <kamalesh@linux.vnet.ibm.com>

Could you test Linus' current git tree and see if you can reproduce
the same problem now?  The patch I sent upstream was a little
different from the one you tested, though it should have the same
effect, and I would like to be sure that it is just as effective at
fixing the bug.

Thanks,
Paul.

Re: [BUG] 2.6.25-rc2-git4 - Regression Kernel oops while running kernbench and tbench on powerpc

[Kamalesh Babulal]

Paul Mackerras wrote:
> Kamalesh Babulal writes:
> 
>> Thanks, after applying the patch the oops is not reproducible on the machine. The console
>> log had no message starting with SLB: or FWNMI:. I have updated the bugzilla also.
>>
>> Tested-by: Kamalesh Babulal <kamalesh@linux.vnet.ibm.com>
> 
> Could you test Linus' current git tree and see if you can reproduce
> the same problem now?  The patch I sent upstream was a little
> different from the one you tested, though it should have the same
> effect, and I would like to be sure that it is just as effective at
> fixing the bug.
> 
> Thanks,
> Paul.

Hi Paul,

The patch has the same effect. I tested the 2.6.26-rc1-git7 kernel and
the oops is not reproducible.

-- 
Thanks & Regards,
Kamalesh Babulal,
Linux Technology Center,
IBM, ISTL.