Netdev Archive on
help / color / mirror / Atom feed
From: Bjorn Helgaas <>
To: "Billie Alsup (balsup)" <>
Cc: Paul Menzel <>,
	Bjorn Helgaas <>,
	Guohan Lu <>,
	"Madhava Reddy Siddareddygari (msiddare)" <>,
	"" <>,
	"" <>,
	"David S. Miller" <>,
	Jakub Kicinski <>,
	"" <>,
	Sergey Miroshnichenko <>
Subject: Re: [RFC][PATCH] PCI: Reserve address space for powered-off devices behind PCIe bridges
Date: Thu, 15 Jul 2021 13:56:49 -0500	[thread overview]
Message-ID: <20210715185649.GA1984276@bjorn-Precision-5520> (raw)
In-Reply-To: <>

On Thu, Jul 15, 2021 at 06:12:25PM +0000, Billie Alsup (balsup) wrote:
> It took me a while to figure out that the "New Outlook" option
> doesn't actually allow sending plain text, so I have to switch to
> "Old Outlook" mode.

Since you've gone to that much trouble already, also note and 

BTW, the attribution in the email you quoted below got corrupted in
such a way that it appears that I wrote the whole thing, instead of 
what actually happened, which is that I wrote a half dozen lines of
response to your email.  Linux uses old skool email conventions ;)

> It is not clear as to what parameters Linux would use to consider a
> window broken.

By "broken," I just mean things that are prohibited by the PCI spec,
like "region doesn't fit in a window of an upstream device" or
"non-prefetchable region placed in a prefetchable window."

> But if the kernel preserves some bridge window
> assignment, then it seems feasible for our BIOS to run this same
> algorithm (reading PLX persistent scratch registers to determine
> window sizes).  I will raise this possibility with our own kernel
> team to discuss with the bios team.  We can also look more closely
> at the resource_alignment options to see if that might suffice.
> Thanks for the information!

> From: Bjorn Helgaas <>
> Date: Thursday, July 15, 2021 at 10:14 AM
> To: "Billie Alsup (balsup)" <>
> Cc: Paul Menzel <>, Bjorn Helgaas <>, Guohan Lu <>, "Madhava Reddy Siddareddygari (msiddare)" <>, "" <>, "" <>, "David S. Miller" <>, Jakub Kicinski <>, "" <>, Sergey Miroshnichenko <>
> Subject: Re: [RFC][PATCH] PCI: Reserve address space for powered-off devices behind PCIe bridges
> On Thu, Jul 15, 2021 at 04:52:26PM +0000, Billie Alsup (balsup) wrote:
> We are aware of how Cisco device specific this code is, and hadn't
> intended to upstream it.  This code was originally written for an
> older kernel version (4.8.28-WR9.0.0.26_cgl).  I am not the original
> author; I just ported it into various SONiC linux kernels.  We use
> ACPI with SONiC (although not on our non-SONiC products), so I
> thought I might be able to define such windows within the ACPI tree
> and have some generic code to read such configuration information
> from the ACPI tables,. However, initial attempts failed so I went
> with the existing approach.  I believe we did look at the hpmmiosize
> parameter, but iirc it applied to each bridge, rather than being a
> pool of address space to dynamically parcel out as necessary.
> Right.  I mentioned "pci=resource_alignment=" because it claims to be
> able to specify window sizes for specific bridges.  But I haven't
> exercised that myself.
> There are multiple bridges involved in the hardware (there are 8
> hot-plug fabric cards, each with multiple PCI devices).  Devices on
> the card are in multiple power zones, so all devices are not
> immediately visible to the pci scanning code.  The top level bridge
> reserves close to 5G.  The 2nd level (towards the fabric cards)
> reserve 4.5G.  The 3rd level has 9 bridges each reserving 512M.  The
> 4th level reserves 384M (with a 512M alignment restriction iirc).
> The 5th level reserves 384M (again with an alignment restriction).
> That defines the bridge hierarchy visible at boot.  Things behind
> that 5th level are hot-plugged where there are two more bridge
> levels and 5 devices (1 requiring 2x64M blocks and 4 requiring
> 1x64M).
> I'm not sure if the Cisco kernel team has revisited the hpmmiosize
> and resource_alignment parameters since this initial implementation.
> Reading the description of Sergey's patches, he seems to be
> approaching the same problem from a different direction.   It is
> unclear if such an approach is practical for our environment.   It
> would require updates to all of our SONiC drivers to support
> stopping/remapping/restarting, and it is unclear if that is
> acceptable.  It is certainly less preferable to pre-reserving the
> required space.  For our embedded product, we know exactly what
> devices will be plugged in, and allowing that to be pre-programmed
> into the PLX eeprom gives us the flexibility we need.  
> If you know up front what devices are possible and how much space they
> need, possibly your firmware could assign the bridge windows you need.
> Linux generally does not change window assignments unless they are
> broken.
> Bjorn

  reply	other threads:[~2021-07-15 18:58 UTC|newest]

Thread overview: 7+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
     [not found] <>
2021-07-15 17:13 ` Bjorn Helgaas
2021-07-15 18:12   ` Billie Alsup (balsup)
2021-07-15 18:56     ` Bjorn Helgaas [this message]
2021-07-15 19:49       ` Billie Alsup (balsup)
2021-07-15 19:56         ` Bjorn Helgaas
     [not found] <>
2021-07-15 13:50 ` Paul Menzel
2021-07-15 14:07   ` Bjorn Helgaas

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20210715185649.GA1984276@bjorn-Precision-5520 \ \ \ \ \ \ \ \ \ \ \ \ \
    --subject='Re: [RFC][PATCH] PCI: Reserve address space for powered-off devices behind PCIe bridges' \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).