LKML Archive on lore.kernel.org
help / color / mirror / Atom feed
From: Jens Axboe <jens.axboe@oracle.com>
To: Suresh Siddha <suresh.b.siddha@intel.com>
Cc: paulmck@linux.vnet.ibm.com, mingo@elte.hu,
	jeremy.fitzhardinge@citrix.com, nickpiggin@yahoo.com.au,
	torvalds@linux-foundation.org, linux-kernel@vger.kernel.org,
	asit.k.mallick@intel.com
Subject: Re: [patch] generic-ipi: fix the smp_mb() placement
Date: Thu, 30 Oct 2008 08:20:30 +0100	[thread overview]
Message-ID: <20081030072029.GK31673@kernel.dk> (raw)
In-Reply-To: <20081029224229.GK30573@linux-os.sc.intel.com>

On Wed, Oct 29 2008, Suresh Siddha wrote:
> While looking at some other issue recently, we encountered this smp_mb()
> placement issue.  x86 specific code also needs some similar fixes. Patch for
> that will follow soon.
> 
> Please review the appended generic-ipi fix.

Looks good, nice debugging! A few comments below.

> 
> thanks,
> suresh
> ---
> 
> From: Suresh Siddha <suresh.b.siddha@intel.com>
> Subject: generic-ipi: fix the smp_mb() placement
> 
> smp_mb() is needed (to make the memory operations visible globally) before
> sending the ipi on the sender and the receiver (on Alpha atleast) needs
> smp_read_barrier_depends() in the handler before reading the call_single_queue
> list in a lock-free fashion.
> 
> On x86, x2apic mode register accesses for sending IPI's don't have serializing
> semantics. So the need for smp_mb() before sending the IPI becomes more
> critical in x2apic mode.
> 
> Remove the unnecessary smp_mb() in csd_flag_wait(), as the presence of that
> smp_mb() doesn't mean anything on the sender, when the ipi receiver is not
> doing any thing special (like memory fence) after clearing the CSD_FLAG_WAIT.
> 
> Signed-off-by: Suresh Siddha <suresh.b.siddha@intel.com>
> ---
> 
> diff --git a/kernel/smp.c b/kernel/smp.c
> index f362a85..75c8dde 100644
> --- a/kernel/smp.c
> +++ b/kernel/smp.c
> @@ -51,10 +51,6 @@ static void csd_flag_wait(struct call_single_data *data)
>  {
>  	/* Wait for response */
>  	do {
> -		/*
> -		 * We need to see the flags store in the IPI handler
> -		 */
> -		smp_mb();
>  		if (!(data->flags & CSD_FLAG_WAIT))
>  			break;
>  		cpu_relax();
> @@ -76,6 +72,11 @@ static void generic_exec_single(int cpu, struct call_single_data *data)
>  	list_add_tail(&data->list, &dst->list);
>  	spin_unlock_irqrestore(&dst->lock, flags);
>  
> +	/*
> +	 * Make the list addition visible before sending the ipi.
> +	 */
> +	smp_mb();
> +
>  	if (ipi)
>  		arch_send_call_function_single_ipi(cpu);

We can downgrade this to a smp_wmb().

>  
> @@ -157,7 +158,7 @@ void generic_smp_call_function_single_interrupt(void)
>  	 * Need to see other stores to list head for checking whether
>  	 * list is empty without holding q->lock
>  	 */
> -	smp_mb();
> +	smp_read_barrier_depends();
>  	while (!list_empty(&q->list)) {
>  		unsigned int data_flags;
>  
> @@ -191,7 +192,7 @@ void generic_smp_call_function_single_interrupt(void)
>  		/*
>  		 * See comment on outer loop
>  		 */
> -		smp_mb();
> +		smp_read_barrier_depends();
>  	}
>  }
>  
> @@ -370,6 +371,11 @@ int smp_call_function_mask(cpumask_t mask, void (*func)(void *), void *info,
>  	list_add_tail_rcu(&data->csd.list, &call_function_queue);
>  	spin_unlock_irqrestore(&call_function_lock, flags);
>  
> +	/*
> +	 * Make the list addition visible before sending the ipi.
> +	 */
> +	smp_mb();
> +
>  	/* Send a message to all CPUs in the map */
>  	arch_send_call_function_ipi(mask);

Ditto

-- 
Jens Axboe


  reply	other threads:[~2008-10-30  7:21 UTC|newest]

Thread overview: 16+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2008-10-29 22:42 Suresh Siddha
2008-10-30  7:20 ` Jens Axboe [this message]
2008-10-30 16:30   ` Suresh Siddha
2008-10-30 17:25     ` Jens Axboe
2008-10-30 18:53 ` Ingo Molnar
2008-10-30 20:23   ` Suresh Siddha
2008-10-31  5:10     ` Jeremy Fitzhardinge
2008-10-31  9:39       ` Ingo Molnar
2008-10-31 11:12         ` Jeremy Fitzhardinge
2008-10-31 16:53           ` Suresh Siddha
2008-10-31 20:30             ` Jeremy Fitzhardinge
2008-11-03 10:17               ` Ingo Molnar
2008-11-03 23:48                 ` Jeremy Fitzhardinge
2008-11-04  9:19                   ` Ingo Molnar
2008-11-04 22:25                     ` Suresh Siddha
2008-11-05  9:20                       ` Jens Axboe

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20081030072029.GK31673@kernel.dk \
    --to=jens.axboe@oracle.com \
    --cc=asit.k.mallick@intel.com \
    --cc=jeremy.fitzhardinge@citrix.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mingo@elte.hu \
    --cc=nickpiggin@yahoo.com.au \
    --cc=paulmck@linux.vnet.ibm.com \
    --cc=suresh.b.siddha@intel.com \
    --cc=torvalds@linux-foundation.org \
    --subject='Re: [patch] generic-ipi: fix the smp_mb() placement' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).