LKML Archive on lore.kernel.org
help / color / mirror / Atom feed
From: Peter Zijlstra <a.p.zijlstra@chello.nl>
To: Paul Jackson <pj@sgi.com>
Cc: Max Krasnyanskiy <maxk@qualcomm.com>,
	mingo@elte.hu, tglx@linutronix.de, oleg@tv-sign.ru,
	rostedt@goodmis.org, linux-kernel@vger.kernel.org,
	rientjes@google.com
Subject: Re: [RFC/PATCH] cpuset: cpuset irq affinities
Date: Wed, 05 Mar 2008 09:37:52 +0100	[thread overview]
Message-ID: <1204706272.6241.79.camel@lappy> (raw)
In-Reply-To: <20080304191159.a02be58b.pj@sgi.com>


On Tue, 2008-03-04 at 19:11 -0600, Paul Jackson wrote:
> Max K wrote:
> > Yeah, that would definitely be awkward.
> 
> Yeah - agreed - awkward.
> 
> Forget that idea (allowing the same irq in multiple 'irqs' files.)
> 
> It seems to me that we get into trouble trying to cram that 'system'
> cpuset into the cpuset hierarchy, where that system cpuset is there to
> hold a list of irqs, but is only partially a good fit for the existing
> cpuset hierarchy.
> 
> Could this irq configuration be partly a system-wide configuration
> decision (which irqs are 'system' irqs), and partly a per-cpuset
> decision -- which cpusets (such as a real-time one) want to disable
> the usual system irqs that everyone else gets.
> 
> The cpuset portion of this should take only a single per-cpuset Boolean
> flag -- which if set True (1), asks the system to "please leave my CPUs
> off the list of CPUs receiving the usual system irqs."
> 
> Then the list of "usual system irqs" would be established in some /proc
> or /sys configuration.  Such irqs would be able to go to any CPUs
> except those CPUs which found themselves in a cpuset with the above
> per-cpuset Boolean flag set True (1).

How about we make this in-kernel boot set, that by default contains all
IRQs, all unbounded kthreads and all of user-space.

To be compatible with your existing clients you only need to move all
the IRQs to the root domain.

(Upgrading a kernel would require distributing some new userspace
anyway, right? - and we could offer a .config option to disable the boot
set for those who do upgrade kernels without upgrading user-space).

Then, once you want to make use of the new features, you have to update
your batch scheduler to only make use of load_balance and not
cpus_exclusive (as they're only interested in sched_domains, right?)

So if you want to do IRQ isolation and batch scheduling on the same
machine (as is not possible now) you need to update userspace as said
before; so that it allows for the overlapping cpuset.

For example, on a 32 cpu machine:

/cgroup/boot 0-1 (kthreads - initial userspace)
/cgroup/irqs 0-27 (most irqs)
/cgroup/batch_A 2-5
/cgroup/batch_B 6-13
/cgroup/another_big_app 14-27
/cgroup/RT-domain 28-31 (my special irq)

So by providing a .config option for strict backward compatibility, a
simple way for runtime compatibility (moving all IRQs to the root) which
should be easy to do if the kernel upgrade is accompanied by a (limited)
user-space upgrade.

And once all the features need to be used together (something that is
now not possible - so new usage) then the code that relies on
cpus_exclusive to create sched_domains needs to be changed to use
load_balance instead.

Does that sound like a feasible plan?

> How does all this interact with /proc/irq/N/smp_affinity?

Much the same way the cpuset cpus_allowed interacts with a task's
cpus_allowed. That is, cs->cpus_allowed is a mask on top of the provided
affinity.

If for some reason the cs->cpus_allowed changes in such a way that the
user-specified mask becomes empty (irq->cpus_allowed & cs->cpus_allowed
== 0), then print a message and set it to the full mask
(irq->cpus_allowed = cs->cpus_allowed).

If for some reason the cs->cpus_allowed changes in such a way that the
mask is physically impossible (set_irq_affinity(cs->cpus_allowed)
fails), then print a message and move the IRQ to the parent set.




  reply	other threads:[~2008-03-05  8:38 UTC|newest]

Thread overview: 94+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2008-02-27 22:21 [RFC/PATCH 0/4] CPUSET driven CPU isolation Peter Zijlstra
2008-02-27 22:21 ` [RFC/PATCH 1/4] sched: remove isolcpus Peter Zijlstra
2008-02-27 23:57   ` Max Krasnyanskiy
2008-02-28 10:19     ` Peter Zijlstra
2008-02-28 19:36       ` Max Krasnyansky
2008-02-27 22:21 ` [RFC/PATCH 2/4] cpuset: system sets Peter Zijlstra
2008-02-27 23:39   ` Paul Jackson
2008-02-28  1:53     ` Max Krasnyanskiy
2008-02-27 23:52   ` Max Krasnyanskiy
2008-02-28  0:11     ` Paul Jackson
2008-02-28  0:29       ` Steven Rostedt
2008-02-28  1:45         ` Max Krasnyanskiy
2008-02-28  3:41           ` Steven Rostedt
2008-02-28  4:58             ` Max Krasnyansky
2008-02-27 22:21 ` [RFC/PATCH 3/4] genirq: system set irq affinities Peter Zijlstra
2008-02-28  0:10   ` Max Krasnyanskiy
2008-02-28 10:19     ` Peter Zijlstra
2008-02-27 22:21 ` [RFC/PATCH 4/4] kthread: system set kthread affinities Peter Zijlstra
2008-02-27 23:38 ` [RFC/PATCH 0/4] CPUSET driven CPU isolation Max Krasnyanskiy
2008-02-28 10:19   ` Peter Zijlstra
2008-02-28 17:33     ` Max Krasnyanskiy
2008-02-28  7:50 ` Ingo Molnar
2008-02-28  8:08   ` Paul Jackson
2008-02-28  9:08     ` Ingo Molnar
2008-02-28  9:17       ` Paul Jackson
2008-02-28  9:32         ` David Rientjes
2008-02-28 10:12           ` David Rientjes
2008-02-28 10:26             ` Peter Zijlstra
2008-02-28 17:37             ` Paul Jackson
2008-02-28 21:24               ` David Rientjes
2008-02-28 22:46                 ` Paul Jackson
2008-02-28 23:00                   ` David Rientjes
2008-02-29  0:16                     ` Paul Jackson
2008-02-29  1:05                       ` David Rientjes
2008-02-29  3:34                         ` Paul Jackson
2008-02-29  4:00                           ` David Rientjes
2008-02-29  6:53                             ` Paul Jackson
2008-02-28 10:46         ` Ingo Molnar
2008-02-28 17:47           ` Paul Jackson
2008-02-28 20:11           ` Max Krasnyansky
2008-02-28 20:13             ` Paul Jackson
2008-02-28 20:26               ` Max Krasnyansky
2008-02-28 20:27                 ` Paul Jackson
2008-02-28 20:45                   ` Max Krasnyansky
2008-02-28 20:23       ` Max Krasnyansky
2008-02-28 17:48   ` Max Krasnyanskiy
2008-02-29  8:31   ` Andrew Morton
2008-02-29  8:36     ` Andrew Morton
2008-02-29  9:10     ` Ingo Molnar
2008-02-29 18:06       ` Max Krasnyanskiy
2008-02-28 12:12 ` Mark Hounschell
2008-02-28 19:57   ` Max Krasnyansky
2008-02-29 18:55 ` [RFC/PATCH] cpuset: cpuset irq affinities Peter Zijlstra
2008-02-29 19:02   ` Ingo Molnar
2008-02-29 20:52     ` Max Krasnyanskiy
2008-02-29 21:03       ` Peter Zijlstra
2008-02-29 21:20         ` Max Krasnyanskiy
2008-03-03 11:57           ` Peter Zijlstra
2008-03-03 17:36             ` Paul Jackson
2008-03-03 17:57               ` Peter Zijlstra
2008-03-03 18:10                 ` Paul Jackson
2008-03-03 18:18                   ` Peter Zijlstra
2008-03-04  7:35                     ` Paul Jackson
2008-03-04 11:06                       ` Peter Zijlstra
2008-03-04 19:52                         ` Max Krasnyanskiy
2008-03-05  1:11                           ` Paul Jackson
2008-03-05  8:37                             ` Peter Zijlstra [this message]
2008-03-05  8:50                               ` Ingo Molnar
2008-03-05 12:35                                 ` Paul Jackson
2008-03-05 12:43                                   ` Ingo Molnar
2008-03-05 17:44                                     ` Paul Jackson
2008-03-05 19:17                               ` Max Krasnyansky
2008-03-06 13:47                               ` Paul Jackson
2008-03-06 15:21                                 ` Peter Zijlstra
2008-03-07  3:40                                   ` Paul Jackson
2008-03-07  6:39                                     ` Paul Jackson
2008-03-07  8:47                                       ` Paul Menage
2008-03-07 14:57                                         ` Paul Jackson
2008-03-03 18:41                   ` Paul Menage
2008-03-03 18:52                     ` Paul Jackson
2008-03-04  5:26                       ` Paul Menage
2008-03-04  6:15                         ` Paul Jackson
2008-03-04  6:21                           ` Paul Menage
2008-03-04  6:26                             ` Paul Jackson
2008-03-04  6:34                               ` Paul Menage
2008-03-04  6:51                                 ` Paul Jackson
2008-02-29 20:55   ` Paul Jackson
2008-02-29 21:14     ` Peter Zijlstra
2008-02-29 21:29       ` Ingo Molnar
2008-02-29 21:32       ` Ingo Molnar
2008-02-29 21:42       ` Max Krasnyanskiy
2008-02-29 22:00         ` Paul Jackson
2008-02-29 21:53       ` Paul Jackson
2008-03-02  5:18   ` Christoph Hellwig

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1204706272.6241.79.camel@lappy \
    --to=a.p.zijlstra@chello.nl \
    --cc=linux-kernel@vger.kernel.org \
    --cc=maxk@qualcomm.com \
    --cc=mingo@elte.hu \
    --cc=oleg@tv-sign.ru \
    --cc=pj@sgi.com \
    --cc=rientjes@google.com \
    --cc=rostedt@goodmis.org \
    --cc=tglx@linutronix.de \
    --subject='Re: [RFC/PATCH] cpuset: cpuset irq affinities' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).