LKML Archive on lore.kernel.org
help / color / mirror / Atom feed
From: Andrew Theurer <habanero@us.ibm.com>
To: Andi Kleen <ak@muc.de>, "Bryan O'Sullivan" <bos@serpentine.com>
Cc: brettspamacct@fastclick.com,
	"Martin J. Bligh" <mbligh@aracnet.com>, Andi Kleen <ak@muc.de>,
	linux-kernel@vger.kernel.org
Subject: Re: How can I optimize a process on a NUMA architecture(x86-64 specifically)?
Date: Mon, 24 May 2004 17:00:57 -0500	[thread overview]
Message-ID: <200405241700.57249.habanero@us.ibm.com> (raw)
In-Reply-To: <20040523142806.GA33866@colin2.muc.de>

On Sunday 23 May 2004 09:28, Andi Kleen wrote:
> On Sat, May 22, 2004 at 05:28:09PM -0700, Bryan O'Sullivan wrote:
> > On Fri, 2004-05-21 at 16:42, Brett E. wrote:
> > > Right now, 5 processes are running taking up a good deal of the CPU
> > > doing memory-intensive work(cacheing) and I notice that none of the
> > > processes seem to have CPU affinity.
> >
> > I don't know what kind of system you're running on, but if it's a
> > multi-CPU Opteron, it is normally a sufficient fudge to just use
> > sched_setaffinity to bind individual processes to specific CPUs.  The
> > mainline kernel memory allocator does the right thing in that case, and
> > allocates memory locally when it can.
> >
> > You can use the taskset command to get at this from the command line, so
> > you may not even need to modify your code.
>
> Linus also merged the NUMA API support into mainline now with 2.6.7rc1, so
> you can use numactl for more finegrained tuning.

FYI Brett, some Opteron systems have a BIOS option to interleave memory.  If 
you are going to make use of NUMA, I think you want to not interleave.

Also, if you have a 25% imbalance within a domain/node, the scheduler can have 
a tendency to bounce around a task for fairness.  That might be why you are 
seeing little/no affinity to a cpu (even top might be causing some of this).   
Not sure what the threshold is between domains/nodes, but I am curious if it 
still happens with CONFIG_NUMA on.  If these are long lived cpu bound 
processes, I would try to have the number of processes be a multiple of the 
number of cpus.

-Andrew Theurer

  reply	other threads:[~2004-05-24 22:01 UTC|newest]

Thread overview: 22+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
     [not found] <1Y6yr-eM-11@gated-at.bofh.it>
     [not found] ` <1YbRm-4iF-11@gated-at.bofh.it>
     [not found]   ` <1Yma3-4cF-3@gated-at.bofh.it>
     [not found]     ` <1YmjP-4jX-37@gated-at.bofh.it>
2004-05-21 19:17       ` How can I optimize a process on a NUMA architecture(x86-64 specifically)? Andi Kleen
     [not found]       ` <1YmMN-4Kh-17@gated-at.bofh.it>
     [not found]         ` <1Yn67-50q-7@gated-at.bofh.it>
2004-05-21 19:19           ` Andi Kleen
2004-05-21 20:32             ` Martin J. Bligh
2004-05-21 23:42               ` Brett E.
2004-05-22  6:13                 ` Martin J. Bligh
2004-05-22  7:41                   ` Andi Kleen
2004-05-23  0:28                 ` Bryan O'Sullivan
2004-05-23 14:28                   ` Andi Kleen
2004-05-24 22:00                     ` Andrew Theurer [this message]
2004-05-25  0:27                       ` Scott Robert Ladd
2004-05-25  1:09                       ` Brett E.
     [not found]     ` <1YRnC-3vk-5@gated-at.bofh.it>
2004-05-23 11:57       ` Andi Kleen
2004-05-21  0:51 Brett E.
2004-05-21  1:29 ` Jesse Barnes
2004-05-21  6:37 ` Martin J. Bligh
2004-05-21 17:27   ` Brett E.
2004-05-21 17:46     ` Martin J. Bligh
2004-05-21 18:14       ` Brett E.
2004-05-21 18:30         ` Martin J. Bligh
2004-05-21 18:58         ` Jesse Barnes
2004-05-21 19:08           ` Martin J. Bligh
2004-05-23  2:49     ` David Schwartz

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=200405241700.57249.habanero@us.ibm.com \
    --to=habanero@us.ibm.com \
    --cc=ak@muc.de \
    --cc=bos@serpentine.com \
    --cc=brettspamacct@fastclick.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mbligh@aracnet.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).