LKML Archive on lore.kernel.org
help / color / mirror / Atom feed
From: Marc Zyngier <marc.zyngier@arm.com>
To: Jiang Liu <jiang.liu@linux.intel.com>,
	Yijing Wang <wangyijing@huawei.com>,
	Bjorn Helgaas <bhelgaas@google.com>,
	Thomas Gleixner <tglx@linutronix.de>
Cc: "linux-arm-kernel@lists.infradead.org" 
	<linux-arm-kernel@lists.infradead.org>,
	"linux-pci@vger.kernel.org" <linux-pci@vger.kernel.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	Will Deacon <Will.Deacon@arm.com>,
	"suravee.suthikulpanit@amd.com" <suravee.suthikulpanit@amd.com>
Subject: Re: [PATCH 2/6] PCI/MSI: add hooks to populate the msi_domain field
Date: Tue, 09 Dec 2014 14:59:34 +0000	[thread overview]
Message-ID: <54870E56.7000909@arm.com> (raw)
In-Reply-To: <548708CC.5000405@linux.intel.com>

On 09/12/14 14:35, Jiang Liu wrote:
> On 2014/12/9 22:27, Marc Zyngier wrote:
>> On 09/12/14 14:11, Jiang Liu wrote:
>>> On 2014/12/9 22:03, Marc Zyngier wrote:
>>>> Hi Gerry,
>>>>
>>>> On 09/12/14 12:47, Jiang Liu wrote:
>>>>> On 2014/12/9 20:12, Marc Zyngier wrote:
>>>>>> Yijing,
>>>>>>
>>>>>> On 09/12/14 11:57, Yijing Wang wrote:
>>>>>>>>>> +void __weak pcibios_set_phb_msi_domain(struct pci_bus *bus)
>>>>>>>>>> +{
>>>>>>>>>> +}
>>>>>>>>>> +
>>>>>>>>>> +static void pci_set_bus_msi_domain(struct pci_bus *bus)
>>>>>>>>>> +{
>>>>>>>>>> +	struct pci_dev *bridge = bus->self;
>>>>>>>>>> +
>>>>>>>>>> +	if (!bridge)
>>>>>>>>>> +		pcibios_set_phb_msi_domain(bus);
>>>>>>>>>> +	else
>>>>>>>>>> +		dev_set_msi_domain(&bus->dev, dev_get_msi_domain(&bridge->dev));
>>>>>>>>>> +}
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi Marc, we can not assume pci devices under same phb share the same msi irq domain,
>>>>>>>>> now in x86, pci devices under the same phb may associate different msi irq domain.
>>>>>>>
>>>>>>> Hi Marc,
>>>>>>>
>>>>>>>>
>>>>>>>> Well, this is not supposed to be a perfect solution yet, but instead a
>>>>>>>> basis for discussion. What I'd like to find out is:
>>>>>>>>
>>>>>>>> - What is the minimum granularity for associating a device with its MSI
>>>>>>>> domain in existing platforms?
>>>>>>>
>>>>>>> PCI device, after Gerry's msi irq domain patchset which now in linux-next,
>>>>>>> in x86, we will find msi irq domain by pci_dev.
>>>>>>
>>>>>> Are you *really* associating the MSI domain on a per pci-device basis?
>>>>>> That is, you have devices on the same PCI bus talking to different MSI hw?
>>>>> Hi Marc,
>>>>> 	This is a little wild:(
>>>>> 	On x86 platform with Intel VT-d(not the case for AMD-v),
>>>>> interrupt remapping is tight to DMA remapping (IOMMU) unit.
>>>>> For most common cases, IOMMU unit manages PCI bus and its sub-hierarchy.
>>>>> But it may also manage a specific PCI device. This is typically used to
>>>>> provide QoS for audio device by using dedicated IOMMU unit to avoid
>>>>> resource contention on DMA remapping tables. BIOS uses ACPI table to
>>>>> report PCI bus/device to IOMMU unit mapping relationship. (To be honest,
>>>>> I have no really experience with such a hardware platform yet, just for
>>>>> theoretical analysis)
>>>>> 	On the other hand, we now support hierarchy irqdomain. So to
>>>>> support per-PCI IOMMU unit case, we need maintain irqdomain at PCI
>>>>> device level.
>>>>> 	This piece of code from your [4/6] is flexible enough, which
>>>>> retrieves msi_domain from PCI device, then fallback to PCI bus,
>>>>> then fallback to platform specific method.
>>>>> 	domain = dev_get_msi_domain(&dev->dev);
>>>>> 	if (!domain && dev->bus->msi)
>>>>>  		domain = dev->bus->msi->domain;
>>>>>  	if (!domain)
>>>>>  		domain = arch_get_pci_msi_domain(dev);
>>>>
>>>> OK. But what I'd really like to see is a way to setup the
>>>> device<->domain binding as early as possible, without having to use more
>>>> conditional code in pci_msi_get_domain.
>>>>
>>>> IOW, can we do something similar to what pci_set_bus_msi_domain and
>>>> pci_set_msi_domain do in this patch?
>>> Hi Marc,
>>> 	I have checked x86 code, we could set pci_dev->msi_domain
>>> when creating PCI devices, just need to find some hook points
>>> into PCI core next step. If arch code doesn't set pci_dev->msi_domain,
>>> PCI MSI core may provide a default way to set pci_dev->msi_domain.
>>> This may make the implementation simpler, I guess:)
>>
>> Right. So following your earlier suggestion, I could make
>> pci_set_msi_domain a weak symbol and let arch code override this.
>>
>> My preference would have been to have arch code to create a set of
>> arch-independent data structures describing the topology, and use that
>> for everything, but maybe that's a bit ambitious for a start.
>>
>> I'll rework the series to make the symbols weak.
> Hi Marc,
> 	I think we may not need the weak symbol at all. With following
> draft patch, the PCI MSI core may simply do:
> 	if (pci_dev->dev.msi_domain == NULL)
> 		dev_set_msi_domain(&dev->dev,
> 			 dev_get_msi_domain(&dev->bus->dev));
> -----------------------------------------------------------------------
> Note: the patch won't pass compilation, just to show the key idea:)
> 
> diff --git a/arch/x86/kernel/apic/msi.c b/arch/x86/kernel/apic/msi.c
> index da163da5fdee..8147d25d4349 100644
> --- a/arch/x86/kernel/apic/msi.c
> +++ b/arch/x86/kernel/apic/msi.c
> @@ -67,6 +67,23 @@ static struct irq_chip pci_msi_controller = {
>         .flags                  = IRQCHIP_SKIP_SET_WAKE,
>  };
> 
> +struct irq_domain *x86_get_pci_msi_domain(struct pci_dev *dev)
> +{
> +       struct irq_domain *domain;
> +       struct irq_alloc_info info;
> +
> +       init_irq_alloc_info(&info, NULL);
> +       info.type = X86_IRQ_ALLOC_TYPE_MSI;
> +       info.msi_dev = dev;
> +       domain = irq_remapping_get_irq_domain(&info);
> +       if (domain == NULL)
> +               domain = msi_default_domain;
> +       if (domain == NULL)
> +               domain = ERR_PTR(-ENOSYS);
> +
> +       return domain;
> +}
> +
>  int native_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)
>  {
>         struct irq_domain *domain;
> diff --git a/arch/x86/pci/common.c b/arch/x86/pci/common.c
> index 7b20bccf3648..a26f30a8bb8f 100644
> --- a/arch/x86/pci/common.c
> +++ b/arch/x86/pci/common.c
> @@ -652,6 +652,9 @@ int pcibios_add_device(struct pci_dev *dev)
>                 pa_data = data->next;
>                 iounmap(data);
>         }
> +
> +       dev->dev.msi_domain = x86_get_pci_msi_domain(dev);
> +
>         return 0;
>  }

Right. So you set the msi_domain using the pcibios_add_device callback.
That will require some minimal surgery (the call to pci_set_msi_domain
happens before the pcibios call, so it needs to be relocated after), but
that seems like a sensible solution to me.

Thanks!

	M.
-- 
Jazz is not dead. It just smells funny...

  reply	other threads:[~2014-12-09 14:59 UTC|newest]

Thread overview: 23+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2014-12-08 20:12 [PATCH 0/6] Introducing per-device MSI domain Marc Zyngier
2014-12-08 20:12 ` [PATCH 1/6] device core: Introduce per-device MSI domain pointer Marc Zyngier
2014-12-08 20:12 ` [PATCH 2/6] PCI/MSI: add hooks to populate the msi_domain field Marc Zyngier
2014-12-09  2:03   ` Yijing Wang
2014-12-09 10:02     ` Marc Zyngier
2014-12-09 11:57       ` Yijing Wang
2014-12-09 12:12         ` Marc Zyngier
2014-12-09 12:24           ` Yijing Wang
2014-12-09 12:47           ` Jiang Liu
2014-12-09 14:03             ` Marc Zyngier
2014-12-09 14:11               ` Jiang Liu
2014-12-09 14:27                 ` Marc Zyngier
2014-12-09 14:35                   ` Jiang Liu
2014-12-09 14:59                     ` Marc Zyngier [this message]
2014-12-09 15:42                       ` Jiang Liu
2014-12-09 12:57   ` Jiang Liu
2015-01-27  0:40   ` Bjorn Helgaas
2015-01-27  0:43     ` Bjorn Helgaas
2014-12-08 20:12 ` [PATCH 3/6] PCI/MSI: of: add support for OF-provided msi_domain Marc Zyngier
2015-01-27  0:41   ` Bjorn Helgaas
2014-12-08 20:12 ` [PATCH 4/6] PCI/MSI: Let pci_msi_get_domain use struct device's msi_domain Marc Zyngier
2014-12-08 20:12 ` [PATCH 5/6] irqchip: GICv2m: Get rid of struct msi_controller Marc Zyngier
2014-12-08 20:12 ` [PATCH 6/6] irqchip: gicv3-its: " Marc Zyngier

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=54870E56.7000909@arm.com \
    --to=marc.zyngier@arm.com \
    --cc=Will.Deacon@arm.com \
    --cc=bhelgaas@google.com \
    --cc=jiang.liu@linux.intel.com \
    --cc=linux-arm-kernel@lists.infradead.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-pci@vger.kernel.org \
    --cc=suravee.suthikulpanit@amd.com \
    --cc=tglx@linutronix.de \
    --cc=wangyijing@huawei.com \
    --subject='Re: [PATCH 2/6] PCI/MSI: add hooks to populate the msi_domain field' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).