Netdev Archive on lore.kernel.org
help / color / mirror / Atom feed
From: "Billie Alsup (balsup)" <balsup@cisco.com>
To: Bjorn Helgaas <helgaas@kernel.org>
Cc: Paul Menzel <pmenzel@molgen.mpg.de>,
	Bjorn Helgaas <bhelgaas@google.com>,
	Guohan Lu <lguohan@gmail.com>,
	"Madhava Reddy Siddareddygari (msiddare)" <msiddare@cisco.com>,
	"linux-pci@vger.kernel.org" <linux-pci@vger.kernel.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"David S. Miller" <davem@davemloft.net>,
	Jakub Kicinski <kuba@kernel.org>,
	"netdev@vger.kernel.org" <netdev@vger.kernel.org>,
	Sergey Miroshnichenko <s.miroshnichenko@yadro.com>
Subject: Re: [RFC][PATCH] PCI: Reserve address space for powered-off devices behind PCIe bridges
Date: Thu, 15 Jul 2021 18:12:25 +0000	[thread overview]
Message-ID: <545AA576-42A5-47A7-A08A-062582B1569A@cisco.com> (raw)
In-Reply-To: <20210715171352.GA1974727@bjorn-Precision-5520>

It took me a while to figure out that the "New Outlook" option doesn't actually allow sending plain text, so I have to switch to "Old Outlook" mode.

It is not clear as to what parameters Linux would use to consider a window broken.  But if the kernel preserves some bridge window assignment, then it seems feasible for our BIOS to run this same algorithm (reading PLX persistent scratch registers to determine window sizes).  I will raise this possibility with our own kernel team to discuss with the bios team.  We can also look more closely at the resource_alignment options to see if that might suffice. Thanks for the information!


From: Bjorn Helgaas <helgaas@kernel.org>
Date: Thursday, July 15, 2021 at 10:14 AM
To: "Billie Alsup (balsup)" <balsup@cisco.com>
Cc: Paul Menzel <pmenzel@molgen.mpg.de>, Bjorn Helgaas <bhelgaas@google.com>, Guohan Lu <lguohan@gmail.com>, "Madhava Reddy Siddareddygari (msiddare)" <msiddare@cisco.com>, "linux-pci@vger.kernel.org" <linux-pci@vger.kernel.org>, "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>, "David S. Miller" <davem@davemloft.net>, Jakub Kicinski <kuba@kernel.org>, "netdev@vger.kernel.org" <netdev@vger.kernel.org>, Sergey Miroshnichenko <s.miroshnichenko@yadro.com>
Subject: Re: [RFC][PATCH] PCI: Reserve address space for powered-off devices behind PCIe bridges

On Thu, Jul 15, 2021 at 04:52:26PM +0000, Billie Alsup (balsup) wrote:
We are aware of how Cisco device specific this code is, and hadn't
intended to upstream it.  This code was originally written for an
older kernel version (4.8.28-WR9.0.0.26_cgl).  I am not the original
author; I just ported it into various SONiC linux kernels.  We use
ACPI with SONiC (although not on our non-SONiC products), so I
thought I might be able to define such windows within the ACPI tree
and have some generic code to read such configuration information
from the ACPI tables,. However, initial attempts failed so I went
with the existing approach.  I believe we did look at the hpmmiosize
parameter, but iirc it applied to each bridge, rather than being a
pool of address space to dynamically parcel out as necessary.

Right.  I mentioned "pci=resource_alignment=" because it claims to be
able to specify window sizes for specific bridges.  But I haven't
exercised that myself.

There are multiple bridges involved in the hardware (there are 8
hot-plug fabric cards, each with multiple PCI devices).  Devices on
the card are in multiple power zones, so all devices are not
immediately visible to the pci scanning code.  The top level bridge
reserves close to 5G.  The 2nd level (towards the fabric cards)
reserve 4.5G.  The 3rd level has 9 bridges each reserving 512M.  The
4th level reserves 384M (with a 512M alignment restriction iirc).
The 5th level reserves 384M (again with an alignment restriction).
That defines the bridge hierarchy visible at boot.  Things behind
that 5th level are hot-plugged where there are two more bridge
levels and 5 devices (1 requiring 2x64M blocks and 4 requiring
1x64M).

I'm not sure if the Cisco kernel team has revisited the hpmmiosize
and resource_alignment parameters since this initial implementation.
Reading the description of Sergey's patches, he seems to be
approaching the same problem from a different direction.   It is
unclear if such an approach is practical for our environment.   It
would require updates to all of our SONiC drivers to support
stopping/remapping/restarting, and it is unclear if that is
acceptable.  It is certainly less preferable to pre-reserving the
required space.  For our embedded product, we know exactly what
devices will be plugged in, and allowing that to be pre-programmed
into the PLX eeprom gives us the flexibility we need.  

If you know up front what devices are possible and how much space they
need, possibly your firmware could assign the bridge windows you need.
Linux generally does not change window assignments unless they are
broken.

Bjorn



  reply	other threads:[~2021-07-15 18:20 UTC|newest]

Thread overview: 7+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
     [not found] <BYAPR11MB3527AEB1E4969C0833D1697ED9129@BYAPR11MB3527.namprd11.prod.outlook.com>
2021-07-15 17:13 ` Bjorn Helgaas
2021-07-15 18:12   ` Billie Alsup (balsup) [this message]
2021-07-15 18:56     ` Bjorn Helgaas
2021-07-15 19:49       ` Billie Alsup (balsup)
2021-07-15 19:56         ` Bjorn Helgaas
     [not found] <20210713073124.177027-1-pmenzel@molgen.mpg.de>
2021-07-15 13:50 ` Paul Menzel
2021-07-15 14:07   ` Bjorn Helgaas

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=545AA576-42A5-47A7-A08A-062582B1569A@cisco.com \
    --to=balsup@cisco.com \
    --cc=bhelgaas@google.com \
    --cc=davem@davemloft.net \
    --cc=helgaas@kernel.org \
    --cc=kuba@kernel.org \
    --cc=lguohan@gmail.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-pci@vger.kernel.org \
    --cc=msiddare@cisco.com \
    --cc=netdev@vger.kernel.org \
    --cc=pmenzel@molgen.mpg.de \
    --cc=s.miroshnichenko@yadro.com \
    --subject='Re: [RFC][PATCH] PCI: Reserve address space for powered-off devices behind PCIe bridges' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).