LKML Archive on lore.kernel.org
help / color / mirror / Atom feed
From: Jens Axboe <jens.axboe@oracle.com>
To: Nick Piggin <npiggin@suse.de>
Cc: linux-kernel@vger.kernel.org, Alan.Brunelle@hp.com,
	arjan@linux.intel.com, dgc@sgi.com
Subject: Re: IO queuing and complete affinity with threads (was Re: [PATCH 0/8] IO queuing and complete affinity)
Date: Fri, 8 Feb 2008 08:59:55 +0100	[thread overview]
Message-ID: <20080208075954.GA15220@kernel.dk> (raw)
In-Reply-To: <20080208075324.GG9730@wotan.suse.de>

On Fri, Feb 08 2008, Nick Piggin wrote:
> On Fri, Feb 08, 2008 at 08:47:47AM +0100, Jens Axboe wrote:
> > On Fri, Feb 08 2008, Nick Piggin wrote:
> > > On Thu, Feb 07, 2008 at 07:25:45PM +0100, Jens Axboe wrote:
> > > > Hi,
> > > > 
> > > > Here's a variant using kernel threads only, the nasty arch bits are then
> > > > not needed. Works for me, no performance testing (that's a hint for Alan
> > > > to try and queue up some testing for this variant as well :-)
> > > 
> > > Well this stuff looks pretty nice (although I'm not sure whether the
> > > softirq->thread changes are a good idea for performance, I guess we'll
> > > see).
> > 
> > Yeah, that is indeed an open question and why I have two seperate
> > patches for now (io-cpu-affinity branch and io-cpu-affinity-kthread
> > branch). As Ingo mentioned, this is how softirqs are handled in the -rt
> > branch already.
>  
> True, although there are some IO workloads where -rt falls behind
> mainline. May not be purely due to irq threads though, of course.

It's certainly an area that needs to be investigated.

> > > You still don't have the option that the Intel patch gave, that is,
> > > to submit on the completer. I guess that you could do it somewhat
> > > generically by having a cpuid in the request queue, and update that
> > > with the completing cpu.
> > 
> > Not sure what you mean, if setting queue_affinity doesn't accomplish it.
> > If you know the completing CPU to begin with, surely you can just set
> > the queuing affinity appropriately?
> 
> And if you don't?

Well if you don't ask for anything, you wont get anything :-)
As I mentioned, the patch is a playing ground for trying various setups.
Everything defaults to 'do as usual', set options to setup certain test
scenarios.

> > > At least they reported it to be the most efficient scheme in their
> > > testing, and Dave thought that migrating completions out to submitters
> > > might be a bottleneck in some cases.
> > 
> > More so than migrating submitters to completers? The advantage of only
> > movign submitters is that you get rid of the completion locking. Apart
> > from that, the cost should be the same, especially for the thread based
> > solution.
> 
> Not specifically for the block layer, but higher layers like xfs.

True, but that's parallel to the initial statement - that migrating
completers is most costly than migrating submitters. So I'd like Dave to
expand on why he thinks that migrating completers it more costly than
submitters, APART from the locking associated with adding the request to
a remote CPU list.

-- 
Jens Axboe


  reply	other threads:[~2008-02-08  8:00 UTC|newest]

Thread overview: 43+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2008-02-07  9:18 [PATCH 0/8] IO queuing and complete affinity Jens Axboe
2008-02-07  9:18 ` [PATCH 1/8] block: split softirq handling into blk-softirq.c Jens Axboe
2008-02-07  9:18 ` [PATCH 2/8] Add interface for queuing work on a specific CPU Jens Axboe
2008-02-07  9:45   ` Andrew Morton
2008-02-07  9:49     ` Jens Axboe
2008-02-07 17:44       ` Harvey Harrison
2008-02-11 10:51     ` Oleg Nesterov
2008-02-07  9:19 ` [PATCH 3/8] block: make kblockd_schedule_work() take the queue as parameter Jens Axboe
2008-02-07  9:19 ` [PATCH 4/8] x86: add support for remotely triggering the block softirq Jens Axboe
2008-02-07 10:07   ` Ingo Molnar
2008-02-07 10:17     ` Jens Axboe
2008-02-07 10:25       ` Ingo Molnar
2008-02-07 10:31         ` Jens Axboe
2008-02-07 10:38           ` Ingo Molnar
2008-02-07 14:18             ` Jens Axboe
2008-02-07 10:49           ` [patch] block layer: kmemcheck fixes Ingo Molnar
2008-02-07 17:42             ` Linus Torvalds
2008-02-07 17:55               ` Jens Axboe
2008-02-07 19:31               ` Ingo Molnar
2008-02-07 20:06                 ` Jens Axboe
2008-02-08  1:22               ` David Miller
2008-02-08  1:28                 ` Linus Torvalds
2008-02-08 15:09                 ` Arjan van de Ven
2008-02-08 22:44                   ` Nick Piggin
2008-02-08 22:56                     ` Arjan van de Ven
2008-02-08 23:58                       ` Nick Piggin
2008-02-08 11:38               ` Jens Axboe
2008-02-07  9:19 ` [PATCH 5/8] x86-64: add support for remotely triggering the block softirq Jens Axboe
2008-02-07  9:19 ` [PATCH 6/8] ia64: " Jens Axboe
2008-02-07  9:19 ` [PATCH 7/8] kernel: add generic softirq interface for triggering a remote softirq Jens Axboe
2008-02-07  9:19 ` [PATCH 8/8] block: add test code for testing CPU affinity Jens Axboe
2008-02-07 15:16 ` [PATCH 0/8] IO queuing and complete affinity Alan D. Brunelle
2008-02-07 18:25 ` IO queuing and complete affinity with threads (was Re: [PATCH 0/8] IO queuing and complete affinity) Jens Axboe
2008-02-07 20:40   ` Alan D. Brunelle
2008-02-08  7:38   ` Nick Piggin
2008-02-08  7:47     ` Jens Axboe
2008-02-08  7:53       ` Nick Piggin
2008-02-08  7:59         ` Jens Axboe [this message]
2008-02-08  8:12           ` Nick Piggin
2008-02-08  8:24             ` Jens Axboe
2008-02-08  8:33               ` Nick Piggin
2008-02-11  5:22           ` David Chinner
2008-02-12  8:28             ` Jeremy Higdon

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20080208075954.GA15220@kernel.dk \
    --to=jens.axboe@oracle.com \
    --cc=Alan.Brunelle@hp.com \
    --cc=arjan@linux.intel.com \
    --cc=dgc@sgi.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=npiggin@suse.de \
    --subject='Re: IO queuing and complete affinity with threads (was Re: [PATCH 0/8] IO queuing and complete affinity)' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).