Linux-Fsdevel Archive on
help / color / mirror / Atom feed
From: Xi Wang <>
To: Peter Zijlstra <>
Cc: Ingo Molnar <>,
	Juri Lelli <>,
	Vincent Guittot <>,
	Dietmar Eggemann <>,
	Steven Rostedt <>,
	Mel Gorman <>, LKML <>,,,
Subject: Re: [PATCH] sched: Make select_idle_sibling search domain configurable
Date: Tue, 28 Jul 2020 10:54:22 -0700	[thread overview]
Message-ID: <> (raw)
In-Reply-To: <>

On Tue, Jul 28, 2020 at 3:39 AM <> wrote:
> On Tue, Jul 28, 2020 at 12:01:31AM -0700, Xi Wang wrote:
> > The scope of select_idle_sibling idle cpu search is LLC. This
> > becomes a problem for the AMD CCX architecture, as the sd_llc is only
> > 4 cores. On a many core machine, the range of search is too small to
> > reach a satisfactory level of statistical multiplexing / efficient
> > utilization of short idle time slices.
> >
> > With this patch idle sibling search is detached from LLC and it
> > becomes run time configurable. To reduce search and migration
> > overheads, a presearch domain is added. The presearch domain will be
> > searched first before the "main search" domain, e.g.:
> >
> > sysctl_sched_wake_idle_domain == 2 ("MC" domain)
> > sysctl_sched_wake_idle_presearch_domain == 1 ("DIE" domain)
> >
> > Presearch will go through 4 cores of a CCX. If no idle cpu is found
> > during presearch, full search will go through the remaining cores of
> > a cpu socket.
> *groan*, this is horrific :-(
> It is also in direct conflict with people wanting to make it smaller.
> On top of that, a domain number is a terrible terrible interface. They
> aren't even available without SCHED_DEBUG on.
> What is the inter-L3 latency? Going by this that had better be awesome.
> And if this Infinity Fabric stuff if highly effective in effectively
> merging L3s -- analogous to what Intel does with it's cache slices, then
> should we not change the AMD topology setup instead of this 'thing'?
> Also, this commit:
>   051f3ca02e46 ("sched/topology: Introduce NUMA identity node sched domain")
> seems to suggest L3 is actually bigger. Suravee, can you please comment?

I think 051f3ca02e46 was still saying 4 cores sharing an L3 but there
is another numa layer which is 8 cores or 2 * 4 core L3 groups. This
should be the chiplet layer.

I don't have precise data but some anecdotes are: The latency
difference between inter 4 core group access and inter 8 core group
access is not huge. Also my experience from Intel machines was that
accessing L3 data across numa domains (also across cpu socket) was not
too bad until the link bandwidth was saturated. I am hoping the
bandwidth situation is better for AMD as L3 groups are smaller.
Another factor is sometimes the trade off is spending 10s of us of
sched overhead vs time slicing at ~12.5ms latency.

What makes the decision trickly is the configuration can depend on
applications and the scale of the system. For a system with 8 cores,
running it the old way with 2 * 4 core LLCs might be the best
decision. For a system with a lot more cores, the number of threads on
the machine would also scale up, which means more potential to create
a dynamic imbalance. I have another (even more horrific) patch for
auto configuring the sysctls, which has (nnext is the size of the next
higher sched domain):

* Widen the range of idle core search if llc domain is too small, both in
* absolute sense and when compared to the next higher level domain.
if (nllc < min(24, nnext / 4)) {
        sysctl_sched_wake_idle_domain = next_level;
        sysctl_sched_wake_idle_presearch_domain = llc_level;
        /* Also make new idle search domain params more like default llc */
        sysctl_sched_wake_idle_domain_tune_flags = 1;


  reply	other threads:[~2020-07-28 17:53 UTC|newest]

Thread overview: 8+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-07-28  7:01 Xi Wang
2020-07-28 10:00 ` kernel test robot
2020-07-28 10:02 ` kernel test robot
2020-07-28 10:39 ` peterz
2020-07-28 17:54   ` Xi Wang [this message]
2020-08-03 22:38     ` Xi Wang
2020-08-04  1:19 ` kernel test robot
2020-08-04  1:19 ` [RFC PATCH] sched: __rebuild_sched_domains() can be static kernel test robot

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \ \ \ \ \ \ \ \ \ \ \ \ \ \
    --subject='Re: [PATCH] sched: Make select_idle_sibling search domain configurable' \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).