LKML Archive on
help / color / mirror / Atom feed
From: Max Krasnyanskiy <>
To: Paul Jackson <>
Subject: Re: Integrating cpusets and cpu isolation [was Re: [CPUISOL] CPU isolation extensions]
Date: Mon, 04 Feb 2008 15:19:15 -0800	[thread overview]
Message-ID: <> (raw)
In-Reply-To: <>

Paul Jackson wrote:
> Max wrote:
>> Looks like I failed to explain what I'm trying to achieve. So let me try again.
> Well done.  I read through that, expecting to disagree or at least
> to not understand at some point, and got all the way through nodding
> my head in agreement.  Good.
> Whether the earlier confusions were lack of clarity in the presentation,
> or lack of competence in my brain ... well guess I don't want to ask that
> question ;).

> Well ... just one minor point:
> Max wrote in reply to pj:
>>> The cpu_isolated_map is a file static variable known only within
>>> the kernel/sched.c file; this should not change.
>> I completely disagree. In fact I think all the cpu_xxx_map (online, present, isolated)
>> variables do not belong in the scheduler code. I'm thinking of submitting a patch that
>> factors them out into kernel/cpumask.c We already have cpumask.h.
> Huh?  Why would you want to do that?
>     For one thing, the map being discussed here, cpu_isolated_map,
>     is only used in sched.c, so why publish it wider?
>     And for another thing, we already declare externs in cpumask.h for
>     the other, more widely used, cpu_*_map variables cpu_possible_map,
>     cpu_online_map, and cpu_present_map.
Well, to address #2 and #3 isolated map will need to be exported as well.
Those other maps do not really have much to do with the scheduler code.
That's why I think either kernel/cpumask.c or kernel/cpu.c is a better place for them.

> Other than that detail, we seem to be communicating and in agreement on
> your first item, isolating CPU scheduler load balancing.  Good.
> On your other two items, irq and workqueue isolation, which I had
> suggested doing via cpuset sched_load_balance, I now agree that that
> wasn't a good idea.
> I am still a little surprised at using isolation extensions to
> disable irqs on select CPUs; but others have thought far more about
> irqs than I have, so I'll be quiet.
Please note that we're not talking about completely disabling IRQs. We're talking about
not routing them to the isolated CPUs by default. It's still possible to explicitly reroute an IRQ
to the isolated CPU.
Why is this needed  ? It is actually very easy to explain. IRQs are the major source of latency 
and overhead. IRQ handlers themselves are mostly ok but they typically schedule soft irqs, work 
queues and timers on the same CPU where an IRQ is handled. In other words if an isolated CPU is 
receiving IRQs it's not really isolated, because it's running a whole bunch of different kernel 
code (ie we're talking latencies, cache usage, etc). 
If course some folks may want to explicitly route certain  IRQs to the isolated CPUs. For example 
if an app depends on the network stack it may make sense to route an IRQ from the NIC to the same 
CPU the app is running on.


  reply	other threads:[~2008-02-04 23:20 UTC|newest]

Thread overview: 36+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2008-01-28  4:09 [CPUISOL] CPU isolation extensions maxk
2008-01-28  4:09 ` [PATCH] [CPUISOL] Add config options for CPU isolation maxk
2008-01-28  4:09   ` [PATCH] [CPUISOL] Export CPU isolation bits maxk
2008-01-28  4:09     ` [PATCH] [CPUISOL] Do not route IRQs to the CPUs isolated at boot maxk
2008-01-28  4:09       ` [PATCH] [CPUISOL] Support for workqueue isolation maxk
2008-01-28  4:09         ` [PATCH] [CPUISOL] Isolated CPUs should be ignored by the "stop machine" maxk
2008-01-28  9:08 ` [CPUISOL] CPU isolation extensions Peter Zijlstra
2008-01-28 14:59   ` Paul Jackson
2008-01-28 16:34     ` Steven Rostedt
2008-01-28 16:44       ` Peter Zijlstra
2008-01-28 18:54         ` Max Krasnyanskiy
2008-01-28 18:46       ` Max Krasnyanskiy
2008-01-28 19:00         ` Steven Rostedt
2008-01-28 20:22           ` Peter Zijlstra
2008-01-28 21:42             ` Max Krasnyanskiy
2008-02-05  0:32             ` CPU isolation and workqueues [was Re: [CPUISOL] CPU isolation extensions] Max Krasnyanskiy
2008-01-28 18:37     ` [CPUISOL] CPU isolation extensions Max Krasnyanskiy
2008-01-28 19:06       ` Paul Jackson
2008-01-28 21:47         ` Max Krasnyanskiy
2008-01-31 19:06         ` Integrating cpusets and cpu isolation [was Re: [CPUISOL] CPU isolation extensions] Max Krasnyanskiy
2008-02-02  6:16           ` Paul Jackson
2008-02-03  5:57             ` Max Krasnyansky
2008-02-03  7:53               ` Paul Jackson
2008-02-04  6:03                 ` Max Krasnyansky
2008-02-04 10:54                   ` Paul Jackson
2008-02-04 23:19                     ` Max Krasnyanskiy [this message]
2008-02-05  2:46                       ` Paul Jackson
2008-02-05  4:08                         ` Max Krasnyansky
2008-01-28 18:32   ` [CPUISOL] CPU isolation extensions Max Krasnyanskiy
2008-01-28 19:10     ` Paul Jackson
2008-01-28 23:41     ` Daniel Walker
2008-01-29  0:12       ` Max Krasnyanskiy
2008-01-29  1:33         ` Daniel Walker
2008-02-04  6:53           ` Max Krasnyansky
2008-01-31 12:16 ` Mark Hounschell
2008-01-31 19:13   ` Max Krasnyanskiy

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \ \ \ \ \ \ \ \ \
    --subject='Re: Integrating cpusets and cpu isolation [was Re: [CPUISOL] CPU isolation extensions]' \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).