LKML Archive on lore.kernel.org
help / color / mirror / Atom feed
From: Peter Zijlstra <a.p.zijlstra@chello.nl>
To: Ed Tomlinson <edt@aei.ca>
Cc: paulmck@linux.vnet.ibm.com, Steven Rostedt <rostedt@goodmis.org>,
	Sergey Senozhatsky <sergey.senozhatsky@gmail.com>,
	Ingo Molnar <mingo@elte.hu>, Thomas Gleixner <tglx@linutronix.de>,
	Andrew Morton <akpm@linux-foundation.org>,
	Dipankar Sarma <dipankar@in.ibm.com>,
	linux-kernel@vger.kernel.org
Subject: Re: INFO: possible circular locking dependency detected
Date: Fri, 15 Jul 2011 13:29:22 +0200	[thread overview]
Message-ID: <1310729362.2586.325.camel@twins> (raw)
In-Reply-To: <201107150705.46248.edt@aei.ca>

On Fri, 2011-07-15 at 07:05 -0400, Ed Tomlinson wrote:
> Jul 14 23:21:18 grover kernel: [  920.659426] -> #1 (rcu_node_level_0){..-...}:
> Jul 14 23:21:18 grover kernel: [  920.659426]        [<ffffffff8108b7e5>] lock_acquire+0x95/0x140
> Jul 14 23:21:18 grover kernel: [  920.659426]        [<ffffffff8157808b>] _raw_spin_lock+0x3b/0x50
> Jul 14 23:21:18 grover kernel: [  920.659426]        [<ffffffff810ba797>] __rcu_read_unlock+0x197/0x2d0
> Jul 14 23:21:18 grover kernel: [  920.659426]        [<ffffffff8103f2f5>] select_task_rq_fair+0x585/0xa80
> Jul 14 23:21:18 grover kernel: [  920.659426]        [<ffffffff8104633b>] try_to_wake_up+0x17b/0x360
> Jul 14 23:21:18 grover kernel: [  920.659426]        [<ffffffff81046575>] wake_up_process+0x15/0x20
> Jul 14 23:21:18 grover kernel: [  920.659426]        [<ffffffff810528f4>] irq_exit+0xb4/0x100
> Jul 14 23:21:18 grover kernel: [  920.659426]        [<ffffffff8158197e>] smp_apic_timer_interrupt+0x6e/0x99
> Jul 14 23:21:18 grover kernel: [  920.659426]        [<ffffffff81580c53>] apic_timer_interrupt+0x13/0x20
> Jul 14 23:21:18 grover kernel: [  920.659426]        [<ffffffff810ba6e9>] __rcu_read_unlock+0xe9/0x2d0 
> Jul 14 23:21:18 grover kernel: [  920.659426]        [<ffffffff814c20d4>] sock_def_readable+0x94/0xc0

Ed, are you perchance running with force_irqthreads?

Paul, what appears to be happening here is that some rcu_read_unlock()
gets interrupted, possibly before calling rcu_read_unlock_special(),
possibly not if the interrupt is itself the timer interrupt.

Supposing ->rcu_read_unlock_special is set before, any wakeup happening
from an interrupt hitting __rcu_read_unlock():

void __rcu_read_unlock(void)
{
        struct task_struct *t = current;
                
        barrier();  /* needed if we ever invoke rcu_read_unlock in rcutree.c */
        --t->rcu_read_lock_nesting;
        barrier();  /* decrement before load of ->rcu_read_unlock_special */
        if (t->rcu_read_lock_nesting == 0 &&
            unlikely(ACCESS_ONCE(t->rcu_read_unlock_special)))
                rcu_read_unlock_special(t);
#ifdef CONFIG_PROVE_LOCKING
        WARN_ON_ONCE(ACCESS_ONCE(t->rcu_read_lock_nesting) < 0);
#endif /* #ifdef CONFIG_PROVE_LOCKING */
}

After --t->rcu_read_lock_nesting, but before calling
rcu_read_unlock_special(), will trigger this lock inversion.

The alternative case, ->rcu_read_unlock_special is not set yet, it can
be set if the interrupt hitting in that same spot above, is the timer
interrupt, and the wakeup happens either from the softirq ran from the
hard IRQ tail, or as I suspect here happens, the wakeup of ksoftirqd/#.



  reply	other threads:[~2011-07-15 11:29 UTC|newest]

Thread overview: 55+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2011-07-14 14:49 Sergey Senozhatsky
2011-07-14 16:41 ` Peter Zijlstra
2011-07-14 16:57   ` Paul E. McKenney
2011-07-14 19:16     ` Sergey Senozhatsky
2011-07-14 19:15   ` Sergey Senozhatsky
2011-07-14 19:34     ` Paul E. McKenney
2011-07-14 19:38       ` Dave Jones
2011-07-14 20:33         ` Paul E. McKenney
2011-07-14 19:38       ` Sergey Senozhatsky
2011-07-14 16:58 ` Steven Rostedt
2011-07-14 17:02   ` Steven Rostedt
2011-07-14 17:05     ` Paul E. McKenney
2011-07-14 17:32       ` Steven Rostedt
2011-07-14 17:46         ` Steven Rostedt
2011-07-14 19:18           ` Paul E. McKenney
2011-07-14 19:41             ` Steven Rostedt
2011-07-14 20:33               ` Paul E. McKenney
2011-07-15 11:05             ` Ed Tomlinson
2011-07-15 11:29               ` Peter Zijlstra [this message]
2011-07-15 11:35                 ` Ed Tomlinson
2011-07-15 11:39                 ` Peter Zijlstra
2011-07-15 18:11                   ` Paul E. McKenney
2011-07-15 12:42                 ` Paul E. McKenney
2011-07-15 13:07                   ` Peter Zijlstra
2011-07-15 14:36                     ` Paul E. McKenney
2011-07-15 15:04                       ` Peter Zijlstra
2011-07-15 15:59                         ` Paul E. McKenney
2011-07-15 16:11                           ` Peter Zijlstra
2011-07-15 16:56                             ` Paul E. McKenney
2011-07-15 21:48                               ` Ed Tomlinson
2011-07-15 22:04                                 ` Paul E. McKenney
2011-07-16 19:42                                   ` Ed Tomlinson
2011-07-17  0:02                                     ` Paul E. McKenney
2011-07-17  1:56                                       ` Ed Tomlinson
2011-07-17 14:28                                         ` Paul E. McKenney
2011-07-18 15:15                                           ` Paul E. McKenney
2011-07-18  9:29                                     ` Peter Zijlstra
2011-07-18 15:29                                       ` Paul E. McKenney
2011-07-15 16:55                     ` Steven Rostedt
2011-07-15 17:03                       ` Paul E. McKenney
2011-07-15 17:16                         ` Steven Rostedt
2011-07-15 17:24                           ` Paul E. McKenney
2011-07-15 17:42                             ` Steven Rostedt
2011-07-15 18:33                               ` Paul E. McKenney
  -- strict thread matches above, loose matches on Subject: below --
2011-08-07 16:22 Justin P. Mattock
2011-08-11 20:57 ` Justin P. Mattock
2009-12-06 10:11 Richard Zidlicky
2009-10-10 23:09 John Kacur
2007-02-08 15:03 Pedro M. López
2006-10-16 14:05 alpha @ steudten Engineering
2006-10-16 14:32 ` Nick Piggin
2006-10-16 15:42   ` Randy Dunlap
2006-10-16 15:46     ` Nick Piggin
2006-10-19  6:02   ` Andrew Morton
2006-10-19  6:30     ` Nick Piggin

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1310729362.2586.325.camel@twins \
    --to=a.p.zijlstra@chello.nl \
    --cc=akpm@linux-foundation.org \
    --cc=dipankar@in.ibm.com \
    --cc=edt@aei.ca \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mingo@elte.hu \
    --cc=paulmck@linux.vnet.ibm.com \
    --cc=rostedt@goodmis.org \
    --cc=sergey.senozhatsky@gmail.com \
    --cc=tglx@linutronix.de \
    --subject='Re: INFO: possible circular locking dependency detected' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).