LKML Archive on
help / color / mirror / Atom feed
From: Mark Hounschell <>
Cc:, Mark Hounschell <>
Subject: Re: [CPUISOL] CPU isolation extensions
Date: Thu, 31 Jan 2008 07:16:13 -0500	[thread overview]
Message-ID: <> (raw)
In-Reply-To: <> wrote:
> Following patch series extends CPU isolation support. Yes, most people want to virtuallize 
> CPUs these days and I want to isolate them :).
> The primary idea here is to be able to use some CPU cores as dedicated engines for running
> user-space code with minimal kernel overhead/intervention, think of it as an SPE in the 
> Cell processor.
> We've had scheduler support for CPU isolation ever since O(1) scheduler went it. 
> I'd like to extend it further to avoid kernel activity on those CPUs as much as possible.
> In fact that the primary distinction that I'm making between say "CPU sets" and 
> "CPU isolation". "CPU sets" let you manage user-space load while "CPU isolation" provides
> a way to isolate a CPU as much as possible (including kernel activities).
> I'm personally using this for hard realtime purposes. With CPU isolation it's very easy to 
> achieve single digit usec worst case and around 200 nsec average response times on off-the-shelf
> multi- processor/core systems under exteme system load. I'm working with legal folks on releasing 
> hard RT user-space framework for that.
> I can also see other application like simulators and stuff that can benefit from this.
> I've been maintaining this stuff since around 2.6.18 and it's been running in production
> environment for a couple of years now. It's been tested on all kinds of machines, from NUMA
> boxes like HP xw9300/9400 to tiny uTCA boards like Mercury AXA110.
> The messiest part used to be SLAB garbage collector changes. With the new SLUB all that mess 
> goes away (ie no changes necessary). Also CFS seems to handle CPU hotplug much better than O(1) 
> did (ie domains are recomputed dynamically) so that isolation can be done at any time (via sysfs). 
> So this seems like a good time to merge. 
> Anyway. The patchset consist of 5 patches. First three are very simple and non-controversial.
> They simply make "CPU isolation" a configurable feature, export cpu_isolated_map and provide
> some helper functions to access it (just like cpu_online() and friends).
> Last two patches add support for isolating CPUs from running workqueus and stop machine.
> More details in the individual patch descriptions.
> Ideally I'd like all of this to go in during this merge window. If people think it's acceptable 
> Linus or Andrew (or whoever is more appropriate Ingo maybe) can pull this patch set from
> 	git://

It's good to see hear from someone else that thinks a multi-processor
box _should_ be able to run a CPU intensive (%100) RT app on one of the
processors without adversely affecting or being affected by the others.
I have had issues that were _traced_ back to the fact that I am doing
just that. All I got was, you can't do that or we don't support that
kind of thing in the Linux kernel.

One example,  Andrew Mortons feedback to the LKML thread "floppy.c soft

Good luck with this. I hope this gets someones attention.

BTW, I have tried your patches against a vanilla 2.6.24 kernel but am
not successful.

# echo '1' > /sys/devices/system/cpu/cpu1/isolated
bash: echo: write error: Device or resource busy

The cpuisol=1 cmdline option yields:

harley:# cat /sys/devices/system/cpu/cpu1/isolated

harley:# cat /proc/cmdline
root=/dev/sda3 vga=normal apm=off selinux=0 noresume splash=silent
kmalloc=192M cpuisol=1


  parent reply	other threads:[~2008-01-31 13:10 UTC|newest]

Thread overview: 36+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2008-01-28  4:09 [CPUISOL] CPU isolation extensions maxk
2008-01-28  4:09 ` [PATCH] [CPUISOL] Add config options for CPU isolation maxk
2008-01-28  4:09   ` [PATCH] [CPUISOL] Export CPU isolation bits maxk
2008-01-28  4:09     ` [PATCH] [CPUISOL] Do not route IRQs to the CPUs isolated at boot maxk
2008-01-28  4:09       ` [PATCH] [CPUISOL] Support for workqueue isolation maxk
2008-01-28  4:09         ` [PATCH] [CPUISOL] Isolated CPUs should be ignored by the "stop machine" maxk
2008-01-28  9:08 ` [CPUISOL] CPU isolation extensions Peter Zijlstra
2008-01-28 14:59   ` Paul Jackson
2008-01-28 16:34     ` Steven Rostedt
2008-01-28 16:44       ` Peter Zijlstra
2008-01-28 18:54         ` Max Krasnyanskiy
2008-01-28 18:46       ` Max Krasnyanskiy
2008-01-28 19:00         ` Steven Rostedt
2008-01-28 20:22           ` Peter Zijlstra
2008-01-28 21:42             ` Max Krasnyanskiy
2008-02-05  0:32             ` CPU isolation and workqueues [was Re: [CPUISOL] CPU isolation extensions] Max Krasnyanskiy
2008-01-28 18:37     ` [CPUISOL] CPU isolation extensions Max Krasnyanskiy
2008-01-28 19:06       ` Paul Jackson
2008-01-28 21:47         ` Max Krasnyanskiy
2008-01-31 19:06         ` Integrating cpusets and cpu isolation [was Re: [CPUISOL] CPU isolation extensions] Max Krasnyanskiy
2008-02-02  6:16           ` Paul Jackson
2008-02-03  5:57             ` Max Krasnyansky
2008-02-03  7:53               ` Paul Jackson
2008-02-04  6:03                 ` Max Krasnyansky
2008-02-04 10:54                   ` Paul Jackson
2008-02-04 23:19                     ` Max Krasnyanskiy
2008-02-05  2:46                       ` Paul Jackson
2008-02-05  4:08                         ` Max Krasnyansky
2008-01-28 18:32   ` [CPUISOL] CPU isolation extensions Max Krasnyanskiy
2008-01-28 19:10     ` Paul Jackson
2008-01-28 23:41     ` Daniel Walker
2008-01-29  0:12       ` Max Krasnyanskiy
2008-01-29  1:33         ` Daniel Walker
2008-02-04  6:53           ` Max Krasnyansky
2008-01-31 12:16 ` Mark Hounschell [this message]
2008-01-31 19:13   ` Max Krasnyanskiy

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \ \ \ \ \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).