LKML Archive on
help / color / mirror / Atom feed
From: "Brett E." <>
To: "Martin J. Bligh" <>
Cc: linux-kernel mailing list <>,
Subject: Re: How can I optimize a process on a NUMA architecture(x86-64 specifically)?
Date: Fri, 21 May 2004 11:14:17 -0700	[thread overview]
Message-ID: <> (raw)
In-Reply-To: <74030000.1085161614@flay>

Martin J. Bligh wrote:

>>>>Say you have a bunch of single-threaded processes on a NUMA machine. 
>>>>Does the kernel make sure to prefer allocations using a certain CPU's 
>>>>memory, preferring to run a given process on the CPU which contains 
>>>>its memory?  Or should I use the NUMA API(libnuma) to spell this out 
>>>>to the kernel? Does the kernel do the right thing in this case?
>>>The kernel will generally do the right thing (process local alloc) by
>>>default. In 99% of cases, you don't want to muck with it - unless you're
>>>running one single app dominating the whole system, and nothing else is
>>>going on, you probably don't want to specify anything explicitly.
>>Let's say I have a 2 way opteron and want to run 4 long-lived processes.   I fork and exec to create 1 of the processes, it chooses to run on processor 0 since processor 1 is overloaded at that time, so its homenode is processor 0. 
> There is no such thing as a homenode. What you describe is more or less why
> we ditched that concept.
>>I fork and exec another, it chooses processor 0 since processors 1 is overloaded at that time. .. Let's say an uneven distribution is chosen for all 4 processes, with all processes mapped to processor 0. So they allocate on node 0 yet the scheduler will map these to both processors since CPU should be balanced. In this case, you will have a situation where the second processor will have to fetch memory from the other processor's memory.
> Each process will allocate local to the CPU it is running on when it does the
> allocate, so it's difficult to get as off-kilter as you describe (though it
> is still possible).

So could process 0 run on processor 0, allocating local to processor 0, 
then run on processor 1, allocating local to processor 1, this way 
allocating to both processors?  So over time process 0's allocations 
would be split up between both processors, defeating NUMA.  The homenode 
concept + explicit CPU pinning seems useful in that they allow you to 
take advantage of NUMA better. Without these two things the kernel will 
just allocate on the currently running CPU whatever that may be when in 
fact a preference must be given to a CPU at some point, hopefully early 
on in the life of the process, in order to take advantage of NUMA.

I'm trying to play devil's advocate by the way so bear with me, you've 
been very helpful and I'm learning a great deal from you.  :)



  reply	other threads:[~2004-05-21 18:14 UTC|newest]

Thread overview: 22+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2004-05-21  0:51 How can I optimize a process on a NUMA architecture(x86-64 specifically)? Brett E.
2004-05-21  1:29 ` Jesse Barnes
2004-05-21  6:37 ` Martin J. Bligh
2004-05-21 17:27   ` Brett E.
2004-05-21 17:46     ` Martin J. Bligh
2004-05-21 18:14       ` Brett E. [this message]
2004-05-21 18:30         ` Martin J. Bligh
2004-05-21 18:58         ` Jesse Barnes
2004-05-21 19:08           ` Martin J. Bligh
2004-05-23  2:49     ` David Schwartz
     [not found] <>
     [not found] ` <>
     [not found]   ` <>
     [not found]     ` <>
2004-05-21 19:17       ` Andi Kleen
     [not found]       ` <>
     [not found]         ` <>
2004-05-21 19:19           ` Andi Kleen
2004-05-21 20:32             ` Martin J. Bligh
2004-05-21 23:42               ` Brett E.
2004-05-22  6:13                 ` Martin J. Bligh
2004-05-22  7:41                   ` Andi Kleen
2004-05-23  0:28                 ` Bryan O'Sullivan
2004-05-23 14:28                   ` Andi Kleen
2004-05-24 22:00                     ` Andrew Theurer
2004-05-25  0:27                       ` Scott Robert Ladd
2004-05-25  1:09                       ` Brett E.
     [not found]     ` <>
2004-05-23 11:57       ` Andi Kleen

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \ \ \ \ \ \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).