LKML Archive on
help / color / mirror / Atom feed
From: Logan Gunthorpe <>
To: Bjorn Helgaas <>,
Subject: Re: [PATCH 2/2] NTB: PCI Quirk to Enable Switchtec NT Functionality with IOMMU On
Date: Thu, 17 May 2018 10:06:09 -0600	[thread overview]
Message-ID: <> (raw)
In-Reply-To: <>

On 17/05/18 07:45 AM, Bjorn Helgaas wrote:
> On Thu, May 17, 2018 at 05:00:13AM -0700, wrote:
>> From: Doug Meyer <>
>> Here we add the PCI quirk for the Microsemi Switchtec parts to allow
>> non-transparent bridging to work when the IOMMU is turned on.
> I'm not an NTB expert, but it *sounds* like you're specifically fixing
> DMA initiated by an NT endpoint.  I assume other aspects of
> non-transparent bridging, e.g., routing of config accesses,
> interrupts, doorbells, etc., might work even without this quirk.

Yes, that's correct.

>> When a requestor on one NT endpoint accesses memory on another NT
>> endpoint, it does this via a devfn proxy ID. Proxy IDs are uniquely
>> assigned on a per-requestor basis within the NT hardware, and are not
>> visible via PCI enumeration. As a result, a memory access from a peer
>> NT endpoint will have an unrecognized requestor ID devfn which the
>> IOMMU will reject.
> It would be very helpful if you could include a reference to the
> relevant section of a publicly available spec.

I'm not aware of any public specs on this. The basic idea is that a peer
accesses a BAR memory window on it's own side and the NTB translates it
to a memory request on the local side. This request has it's requester
ID translated through a LUT seeing the requester ID on the initiating
side isn't valid on the receiving side. The LUT has multiple entries so
that multiple requester IDs can be translated and the responses routed
back to the original requester.

> Who assigns these proxy IDs?  Quirks run before drivers claim the
> device, so it looks like this assumes the proxy ID assignments are
> either hard-wired into the device or programmed by firmware.  If the
> latter, how would they be programmed for hot-added NTBs?

The local side of the requester ID LUT are fixed by the hardware so we
can determine all the possible IDs coming from the NTB device just by
reading some registers. The peer's side of the LUT are programmed with
all possible source requester IDs by the driver but these don't have any
effect on the ID's on the receiving side.

> I'm wondering if all this could or should be done in the switchtec
> driver instead of in a quirk.  But I really don't know how that driver
> works.

We'd like it to be done in the driver too but it seems
pci_add_dma_alias() must be called before the driver is initialized and
therefore in a quirk. Presently, it seems nearly all calls to that
function are in quirks.c for this reason.



  reply	other threads:[~2018-05-17 16:06 UTC|newest]

Thread overview: 13+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-05-17 12:00 [PATCH 0/2] PCI Quirk Patchset for Microsemi Switchtec NTB dmeyer
2018-05-17 12:00 ` [PATCH 1/2] NTB: Migrate PCI Constants to Cannonical PCI Header dmeyer
2018-05-17 13:25   ` Bjorn Helgaas
2018-05-17 12:00 ` [PATCH 2/2] NTB: PCI Quirk to Enable Switchtec NT Functionality with IOMMU On dmeyer
2018-05-17 13:45   ` Bjorn Helgaas
2018-05-17 16:06     ` Logan Gunthorpe [this message]
     [not found]       ` <>
2018-05-22 21:51         ` Bjorn Helgaas
2018-05-22 22:13           ` Alex Williamson
2018-05-22 22:23           ` Logan Gunthorpe
2018-05-23 13:33             ` Bjorn Helgaas
2018-05-23 20:21               ` Logan Gunthorpe
2018-05-17 15:48   ` Logan Gunthorpe
2018-05-22 21:08     ` Doug Meyer

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \ \ \ \ \ \ \ \ \ \ \ \ \
    --subject='Re: [PATCH 2/2] NTB: PCI Quirk to Enable Switchtec NT Functionality with IOMMU On' \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).