Re: [Bug 10071] kernel hang in inet_init (PREEMPT_RCU problem)

Re: [Bug 10071] kernel hang in inet_init (PREEMPT_RCU problem)

[Jarek Poplawski]

On Thu, Feb 28, 2008 at 12:53:03PM -0800, bugme-daemon@bugzilla.kernel.org wrote:
> http://bugzilla.kernel.org/show_bug.cgi?id=10071
...
> ------- Comment #17 from yannick.dirou@axetic.com  2008-02-28 12:53 -------
> bingo with classic rcu

Congratulations Yannick!

So, it seems mainly RCU vs. timer(?) problem. I send CC of this message
to RCU author, linux-kernel and netdev lists.

I'm not sure how much time you could/will to offer for this debugging.
Anyway, if it's possible I think it would be interesting to check with
CONFIG_PREEMPT_RCU = y again, but CONFIG_HOTPLUG_CPU = n ("Processor
type and features --> Support for suspend on SMP and ...") in the
meantime.

Many thanks,
Jarek P.


> [    0.770854] NET: Registered protocol family 2
> [    0.770868] 4294668327 net/ipv4/af_inet.c 1387
> [    0.770884] 4294668327 net/ipv4/af_inet.c 1390
> [    0.770888] 4294668327 net/ipv4/af_inet.c 1393
> [    0.770891] 4294668327 net/ipv4/af_inet.c 1402
> [    0.770893] 4294668327 net/ipv4/af_inet.c 1407
> [    0.773078] 0 4294668328l
> [    0.774036] 0 4294668329l
> [    0.775037] 0 4294668330l
> [    0.775040] 4294668330 net/ipv4/af_inet.c 1419
> [    0.775077] 4294668330 net/ipv4/af_inet.c 1426
> [    0.775077] 4294668330 net/ipv4/route.c 2998
> [    0.775077] 4294668330 net/ipv4/route.c 3009
> [    0.775077] 4294668330 net/ipv4/route.c 3016
> [    0.775077] IP route cache hash table entries: 32768 (order: 5, 131072
> bytes)
> [    0.775077] 4294668330 net/ipv4/route.c 3027
> [    0.775492] TCP established hash table entries: 131072 (order: 8, 1048576
> bytes)
> [    0.776318] TCP bind hash table entries: 65536 (order: 9, 2359296 bytes)
> [    0.785670] TCP: Hash tables configured (established 131072 bind 65536)
> [    0.785715] TCP reno registered
> 
> 
> -- 
> Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
> ------- You are receiving this mail because: -------
> You are on the CC list for the bug, or are watching someone who is.
> You are the assignee for the bug, or are watching the assignee.

Re: [Bug 10071] kernel hang in inet_init (PREEMPT_RCU problem)

[Paul E. McKenney]

On Thu, Feb 28, 2008 at 10:34:57PM +0100, Jarek Poplawski wrote:
> On Thu, Feb 28, 2008 at 12:53:03PM -0800, bugme-daemon@bugzilla.kernel.org wrote:
> > http://bugzilla.kernel.org/show_bug.cgi?id=10071
> ...
> > ------- Comment #17 from yannick.dirou@axetic.com  2008-02-28 12:53 -------
> > bingo with classic rcu
> 
> Congratulations Yannick!
> 
> So, it seems mainly RCU vs. timer(?) problem. I send CC of this message
> to RCU author, linux-kernel and netdev lists.

Interesting!  Any chance of a pointer to the original messages?
A quick Google search and a look at LKML showed me nothing.  :-/

						Thanx, Paul

> I'm not sure how much time you could/will to offer for this debugging.
> Anyway, if it's possible I think it would be interesting to check with
> CONFIG_PREEMPT_RCU = y again, but CONFIG_HOTPLUG_CPU = n ("Processor
> type and features --> Support for suspend on SMP and ...") in the
> meantime.
> 
> Many thanks,
> Jarek P.
> 
> 
> > [    0.770854] NET: Registered protocol family 2
> > [    0.770868] 4294668327 net/ipv4/af_inet.c 1387
> > [    0.770884] 4294668327 net/ipv4/af_inet.c 1390
> > [    0.770888] 4294668327 net/ipv4/af_inet.c 1393
> > [    0.770891] 4294668327 net/ipv4/af_inet.c 1402
> > [    0.770893] 4294668327 net/ipv4/af_inet.c 1407
> > [    0.773078] 0 4294668328l
> > [    0.774036] 0 4294668329l
> > [    0.775037] 0 4294668330l
> > [    0.775040] 4294668330 net/ipv4/af_inet.c 1419
> > [    0.775077] 4294668330 net/ipv4/af_inet.c 1426
> > [    0.775077] 4294668330 net/ipv4/route.c 2998
> > [    0.775077] 4294668330 net/ipv4/route.c 3009
> > [    0.775077] 4294668330 net/ipv4/route.c 3016
> > [    0.775077] IP route cache hash table entries: 32768 (order: 5, 131072
> > bytes)
> > [    0.775077] 4294668330 net/ipv4/route.c 3027
> > [    0.775492] TCP established hash table entries: 131072 (order: 8, 1048576
> > bytes)
> > [    0.776318] TCP bind hash table entries: 65536 (order: 9, 2359296 bytes)
> > [    0.785670] TCP: Hash tables configured (established 131072 bind 65536)
> > [    0.785715] TCP reno registered
> > 
> > 
> > -- 
> > Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
> > ------- You are receiving this mail because: -------
> > You are on the CC list for the bug, or are watching someone who is.
> > You are the assignee for the bug, or are watching the assignee.

Re: [Bug 10071] kernel hang in inet_init (PREEMPT_RCU problem)

[Paul E. McKenney]

On Thu, Feb 28, 2008 at 04:58:23PM -0800, Paul E. McKenney wrote:
> On Thu, Feb 28, 2008 at 10:34:57PM +0100, Jarek Poplawski wrote:
> > On Thu, Feb 28, 2008 at 12:53:03PM -0800, bugme-daemon@bugzilla.kernel.org wrote:
> > > http://bugzilla.kernel.org/show_bug.cgi?id=10071
> > ...
> > > ------- Comment #17 from yannick.dirou@axetic.com  2008-02-28 12:53 -------
> > > bingo with classic rcu
> > 
> > Congratulations Yannick!
> > 
> > So, it seems mainly RCU vs. timer(?) problem. I send CC of this message
> > to RCU author, linux-kernel and netdev lists.
> 
> Interesting!  Any chance of a pointer to the original messages?
> A quick Google search and a look at LKML showed me nothing.  :-/

Thanks to Raphael for pointing me at the above bugzilla URL!

Yannick,

Would you be willing to try your test again after applying the
patch in http://lkml.org/lkml/2008/2/2/11 and rebuilding your
kernel?

This patch does not appear to be in 2.6.25-rc3.  :-(

I am reposting this patch separately.

							Thanx, Paul

Re: [Bug 10071] kernel hang in inet_init (PREEMPT_RCU problem)

[Yannick]

(In reply to comment #20)
 > Reply-To: paulmck@linux.vnet.ibm.com
 >
 > Would you be willing to try your test again after applying the
 > patch in http://lkml.org/lkml/2008/2/2/11 and rebuilding your
 > kernel?
 >
 > This patch does not appear to be in 2.6.25-rc3.  :-(
 >
 > I am reposting this patch separately.
 >
 >                                                         Thanx, Paul

Congratulations Paul and thanks a lot to Jarek , It works,
no more hang or slow boot :
[1.061675] Calling initcall 0xc03b8e10: inet_init+0x0/0x3ba()
[1.061895] NET: Registered protocol family 2
[1.339007] IP route cache hash table entries: 32768 (order: 5, 131072 bytes)
[1.339470] TCP established hash table entries: 131072 (order: 8, 1048576 
bytes)
[1.340375] TCP bind hash table entries: 65536 (order: 9, 2359296 bytes)
[1.349809] TCP: Hash tables configured (established 131072 bind 65536)
[1.349906] TCP reno registered
[1.098514] initcall 0xc03b8e10: inet_init+0x0/0x3ba() returned 0.
[1.098604] initcall 0xc03b8e10 ran for 32 msecs: inet_init+0x0/0x3ba()

Yannick

Re: [Bug 10071] kernel hang in inet_init (PREEMPT_RCU problem)

[Paul E. McKenney]

On Fri, Feb 29, 2008 at 07:53:23AM +0100, Yannick wrote:
> (In reply to comment #20)
> > Reply-To: paulmck@linux.vnet.ibm.com
> >
> > Would you be willing to try your test again after applying the
> > patch in http://lkml.org/lkml/2008/2/2/11 and rebuilding your
> > kernel?
> >
> > This patch does not appear to be in 2.6.25-rc3.  :-(
> >
> > I am reposting this patch separately.
> >
> >                                                         Thanx, Paul
> 
> Congratulations Paul and thanks a lot to Jarek , It works,
> no more hang or slow boot :
> [1.061675] Calling initcall 0xc03b8e10: inet_init+0x0/0x3ba()
> [1.061895] NET: Registered protocol family 2
> [1.339007] IP route cache hash table entries: 32768 (order: 5, 131072 bytes)
> [1.339470] TCP established hash table entries: 131072 (order: 8, 1048576 
> bytes)
> [1.340375] TCP bind hash table entries: 65536 (order: 9, 2359296 bytes)
> [1.349809] TCP: Hash tables configured (established 131072 bind 65536)
> [1.349906] TCP reno registered
> [1.098514] initcall 0xc03b8e10: inet_init+0x0/0x3ba() returned 0.
> [1.098604] initcall 0xc03b8e10 ran for 32 msecs: inet_init+0x0/0x3ba()

Thank you very much for testing this, Yannick!

Andrew and Ingo have this patch on track to merge into a later -rc
of 2.6.25, so we will get there.  ;-)

						Thanx, Paul