Netdev Archive on lore.kernel.org
help / color / mirror / Atom feed
From: Bjorn Helgaas <helgaas@kernel.org>
To: "Billie Alsup (balsup)" <balsup@cisco.com>
Cc: Paul Menzel <pmenzel@molgen.mpg.de>,
	Bjorn Helgaas <bhelgaas@google.com>,
	Guohan Lu <lguohan@gmail.com>,
	"Madhava Reddy Siddareddygari (msiddare)" <msiddare@cisco.com>,
	"linux-pci@vger.kernel.org" <linux-pci@vger.kernel.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"David S. Miller" <davem@davemloft.net>,
	Jakub Kicinski <kuba@kernel.org>,
	"netdev@vger.kernel.org" <netdev@vger.kernel.org>,
	Sergey Miroshnichenko <s.miroshnichenko@yadro.com>
Subject: Re: [RFC][PATCH] PCI: Reserve address space for powered-off devices behind PCIe bridges
Date: Thu, 15 Jul 2021 13:56:49 -0500	[thread overview]
Message-ID: <20210715185649.GA1984276@bjorn-Precision-5520> (raw)
In-Reply-To: <545AA576-42A5-47A7-A08A-062582B1569A@cisco.com>

On Thu, Jul 15, 2021 at 06:12:25PM +0000, Billie Alsup (balsup) wrote:
> It took me a while to figure out that the "New Outlook" option
> doesn't actually allow sending plain text, so I have to switch to
> "Old Outlook" mode.

Since you've gone to that much trouble already, also note
http://vger.kernel.org/majordomo-info.html and
https://people.kernel.org/tglx/notes-about-netiquette 

BTW, the attribution in the email you quoted below got corrupted in
such a way that it appears that I wrote the whole thing, instead of 
what actually happened, which is that I wrote a half dozen lines of
response to your email.  Linux uses old skool email conventions ;)

> It is not clear as to what parameters Linux would use to consider a
> window broken.

By "broken," I just mean things that are prohibited by the PCI spec,
like "region doesn't fit in a window of an upstream device" or
"non-prefetchable region placed in a prefetchable window."

> But if the kernel preserves some bridge window
> assignment, then it seems feasible for our BIOS to run this same
> algorithm (reading PLX persistent scratch registers to determine
> window sizes).  I will raise this possibility with our own kernel
> team to discuss with the bios team.  We can also look more closely
> at the resource_alignment options to see if that might suffice.
> Thanks for the information!

> From: Bjorn Helgaas <helgaas@kernel.org>
> Date: Thursday, July 15, 2021 at 10:14 AM
> To: "Billie Alsup (balsup)" <balsup@cisco.com>
> Cc: Paul Menzel <pmenzel@molgen.mpg.de>, Bjorn Helgaas <bhelgaas@google.com>, Guohan Lu <lguohan@gmail.com>, "Madhava Reddy Siddareddygari (msiddare)" <msiddare@cisco.com>, "linux-pci@vger.kernel.org" <linux-pci@vger.kernel.org>, "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>, "David S. Miller" <davem@davemloft.net>, Jakub Kicinski <kuba@kernel.org>, "netdev@vger.kernel.org" <netdev@vger.kernel.org>, Sergey Miroshnichenko <s.miroshnichenko@yadro.com>
> Subject: Re: [RFC][PATCH] PCI: Reserve address space for powered-off devices behind PCIe bridges
> 
> On Thu, Jul 15, 2021 at 04:52:26PM +0000, Billie Alsup (balsup) wrote:
> We are aware of how Cisco device specific this code is, and hadn't
> intended to upstream it.  This code was originally written for an
> older kernel version (4.8.28-WR9.0.0.26_cgl).  I am not the original
> author; I just ported it into various SONiC linux kernels.  We use
> ACPI with SONiC (although not on our non-SONiC products), so I
> thought I might be able to define such windows within the ACPI tree
> and have some generic code to read such configuration information
> from the ACPI tables,. However, initial attempts failed so I went
> with the existing approach.  I believe we did look at the hpmmiosize
> parameter, but iirc it applied to each bridge, rather than being a
> pool of address space to dynamically parcel out as necessary.
> 
> Right.  I mentioned "pci=resource_alignment=" because it claims to be
> able to specify window sizes for specific bridges.  But I haven't
> exercised that myself.
> 
> There are multiple bridges involved in the hardware (there are 8
> hot-plug fabric cards, each with multiple PCI devices).  Devices on
> the card are in multiple power zones, so all devices are not
> immediately visible to the pci scanning code.  The top level bridge
> reserves close to 5G.  The 2nd level (towards the fabric cards)
> reserve 4.5G.  The 3rd level has 9 bridges each reserving 512M.  The
> 4th level reserves 384M (with a 512M alignment restriction iirc).
> The 5th level reserves 384M (again with an alignment restriction).
> That defines the bridge hierarchy visible at boot.  Things behind
> that 5th level are hot-plugged where there are two more bridge
> levels and 5 devices (1 requiring 2x64M blocks and 4 requiring
> 1x64M).
> 
> I'm not sure if the Cisco kernel team has revisited the hpmmiosize
> and resource_alignment parameters since this initial implementation.
> Reading the description of Sergey's patches, he seems to be
> approaching the same problem from a different direction.   It is
> unclear if such an approach is practical for our environment.   It
> would require updates to all of our SONiC drivers to support
> stopping/remapping/restarting, and it is unclear if that is
> acceptable.  It is certainly less preferable to pre-reserving the
> required space.  For our embedded product, we know exactly what
> devices will be plugged in, and allowing that to be pre-programmed
> into the PLX eeprom gives us the flexibility we need.  
> 
> If you know up front what devices are possible and how much space they
> need, possibly your firmware could assign the bridge windows you need.
> Linux generally does not change window assignments unless they are
> broken.
> 
> Bjorn
> 
> 

  reply	other threads:[~2021-07-15 18:58 UTC|newest]

Thread overview: 7+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
     [not found] <BYAPR11MB3527AEB1E4969C0833D1697ED9129@BYAPR11MB3527.namprd11.prod.outlook.com>
2021-07-15 17:13 ` Bjorn Helgaas
2021-07-15 18:12   ` Billie Alsup (balsup)
2021-07-15 18:56     ` Bjorn Helgaas [this message]
2021-07-15 19:49       ` Billie Alsup (balsup)
2021-07-15 19:56         ` Bjorn Helgaas
     [not found] <20210713073124.177027-1-pmenzel@molgen.mpg.de>
2021-07-15 13:50 ` Paul Menzel
2021-07-15 14:07   ` Bjorn Helgaas

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20210715185649.GA1984276@bjorn-Precision-5520 \
    --to=helgaas@kernel.org \
    --cc=balsup@cisco.com \
    --cc=bhelgaas@google.com \
    --cc=davem@davemloft.net \
    --cc=kuba@kernel.org \
    --cc=lguohan@gmail.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-pci@vger.kernel.org \
    --cc=msiddare@cisco.com \
    --cc=netdev@vger.kernel.org \
    --cc=pmenzel@molgen.mpg.de \
    --cc=s.miroshnichenko@yadro.com \
    --subject='Re: [RFC][PATCH] PCI: Reserve address space for powered-off devices behind PCIe bridges' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).