LKML Archive on lore.kernel.org
help / color / mirror / Atom feed
From: Bjorn Helgaas <helgaas@kernel.org>
To: Thomas Gleixner <tglx@linutronix.de>
Cc: LKML <linux-kernel@vger.kernel.org>,
	Alex Williamson <alex.williamson@redhat.com>,
	"Raj, Ashok" <ashok.raj@intel.com>,
	"David S. Miller" <davem@davemloft.net>,
	Bjorn Helgaas <bhelgaas@google.com>,
	linux-pci@vger.kernel.org, Kevin Tian <kevin.tian@intel.com>,
	Marc Zyngier <maz@kernel.org>, Ingo Molnar <mingo@kernel.org>,
	x86@kernel.org
Subject: Re: [patch 1/8] PCI/MSI: Enable and mask MSIX early
Date: Thu, 22 Jul 2021 16:43:29 -0500	[thread overview]
Message-ID: <20210722214329.GA349464@bjorn-Precision-5520> (raw)
In-Reply-To: <20210721192650.106154171@linutronix.de>

s/MSIX/MSI-X/ in subject

On Wed, Jul 21, 2021 at 09:11:27PM +0200, Thomas Gleixner wrote:
> The ordering of MSI-X enable in hardware is disfunctional:

s/disfunctional/dysfunctional/, isn't English wonderful ;)

>  1) MSI-X is disabled in the control register
>  2) Various setup functions
>  3) pci_msi_setup_msi_irqs() is invoked which ends up accessing
>     the MSI-X table entries
>  4) MSI-X is enabled and masked in the control register with the
>     comment that enabling is required for some hardware to access
>     the MSI-X table
> 
> #4 obviously contradicts #3. The history of this is an issue with the NIU

Annoyingly, if you "git rebase" and reword this commit log, it drops
this line and the one a few lines below because they start with "#".
Should be obvious, but took me a few iterations to see what was
happening.

> hardware. When #4 was introduced the table access actually happened in
> msix_program_entries() which was invoked after enabling and masking MSI-X.
> 
> This was changed in commit d71d6432e105 ("PCI/MSI: Kill redundant call of
> irq_set_msi_desc() for MSI-X interrupts") which removed the table write
> from msix_program_entries().
> 
> Interestingly enough nobody noticed and either NIU still works or it did
> not get any testing with a kernel 3.19 or later.
> 
> Nevertheless this is inconsistent and there is no reason why MSI-X can't be
> enabled and masked in the control register early on, i.e. move #4 above to
> #1. This preserves the NIU workaround and has no side effects on other
> hardware.
>
> Fixes: d71d6432e105 ("PCI/MSI: Kill redundant call of irq_set_msi_desc() for MSI-X interrupts")
> Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
> Cc: David S. Miller <davem@davemloft.net>
> Cc: Bjorn Helgaas <bhelgaas@google.com>
> Cc: linux-pci@vger.kernel.org
> ---
>  drivers/pci/msi.c |   28 +++++++++++++++-------------
>  1 file changed, 15 insertions(+), 13 deletions(-)
> 
> --- a/drivers/pci/msi.c
> +++ b/drivers/pci/msi.c
> @@ -772,18 +772,25 @@ static int msix_capability_init(struct p
>  	u16 control;
>  	void __iomem *base;
>  
> -	/* Ensure MSI-X is disabled while it is set up */
> -	pci_msix_clear_and_set_ctrl(dev, PCI_MSIX_FLAGS_ENABLE, 0);
> +	/*
> +	 * Some devices require MSI-X to be enabled before the MSI-X
> +	 * registers can be accessed.  Mask all the vectors to prevent
> +	 * interrupts coming in before they're fully set up.
> +	 */
> +	pci_msix_clear_and_set_ctrl(dev, 0, PCI_MSIX_FLAGS_MASKALL |
> +				    PCI_MSIX_FLAGS_ENABLE);
>  
>  	pci_read_config_word(dev, dev->msix_cap + PCI_MSIX_FLAGS, &control);
>  	/* Request & Map MSI-X table region */
>  	base = msix_map_region(dev, msix_table_size(control));
> -	if (!base)
> -		return -ENOMEM;
> +	if (!base) {
> +		ret = -ENOMEM;
> +		goto out_disable;
> +	}
>  
>  	ret = msix_setup_entries(dev, base, entries, nvec, affd);
>  	if (ret)
> -		return ret;
> +		goto out_disable;
>  
>  	ret = pci_msi_setup_msi_irqs(dev, nvec, PCI_CAP_ID_MSIX);
>  	if (ret)
> @@ -794,14 +801,6 @@ static int msix_capability_init(struct p
>  	if (ret)
>  		goto out_free;
>  
> -	/*
> -	 * Some devices require MSI-X to be enabled before we can touch the
> -	 * MSI-X registers.  We need to mask all the vectors to prevent
> -	 * interrupts coming in before they're fully set up.
> -	 */
> -	pci_msix_clear_and_set_ctrl(dev, 0,
> -				PCI_MSIX_FLAGS_MASKALL | PCI_MSIX_FLAGS_ENABLE);
> -
>  	msix_program_entries(dev, entries);
>  
>  	ret = populate_msi_sysfs(dev);
> @@ -836,6 +835,9 @@ static int msix_capability_init(struct p
>  out_free:
>  	free_msi_irqs(dev);
>  
> +out_disable:
> +	pci_msix_clear_and_set_ctrl(dev, PCI_MSIX_FLAGS_ENABLE, 0);
> +
>  	return ret;
>  }
>  
> 

  parent reply	other threads:[~2021-07-22 21:43 UTC|newest]

Thread overview: 30+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-07-21 19:11 [patch 0/8] PCI/MSI, x86: Cure a couple of inconsistencies Thomas Gleixner
2021-07-21 19:11 ` [patch 1/8] PCI/MSI: Enable and mask MSIX early Thomas Gleixner
2021-07-21 21:38   ` Raj, Ashok
2021-07-21 22:51     ` Thomas Gleixner
2021-07-22 21:43   ` Bjorn Helgaas [this message]
2021-07-27 20:33     ` Thomas Gleixner
2021-07-21 19:11 ` [patch 2/8] PCI/MSI: Mask all unused MSI-X entries Thomas Gleixner
2021-07-21 22:23   ` Raj, Ashok
2021-07-21 22:57     ` Thomas Gleixner
2021-07-22 13:46       ` Marc Zyngier
2021-07-28 10:04         ` Thomas Gleixner
2021-07-22 21:45   ` Bjorn Helgaas
2021-07-21 19:11 ` [patch 3/8] PCI/MSI: Enforce that MSI-X table entry is masked for update Thomas Gleixner
2021-07-21 22:32   ` Raj, Ashok
2021-07-21 22:59     ` Thomas Gleixner
2021-07-22 21:46   ` Bjorn Helgaas
2021-07-21 19:11 ` [patch 4/8] PCI/MSI: Enforce MSI[X] entry updates to be visible Thomas Gleixner
2021-07-22 21:48   ` Bjorn Helgaas
     [not found]     ` <CAHp75VdNi4rMuRz8UrW9Haf_Ge8KmNJ0w9ykheqkVhmpXHTUyg@mail.gmail.com>
2021-07-23  8:14       ` Marc Zyngier
2021-07-21 19:11 ` [patch 5/8] PCI/MSI: Simplify msi_verify_entries() Thomas Gleixner
2021-07-21 19:11 ` [patch 6/8] genirq: Provide IRQCHIP_AFFINITY_PRE_STARTUP Thomas Gleixner
2021-07-22 15:12   ` Marc Zyngier
2021-07-28 10:40     ` Thomas Gleixner
2021-07-21 19:11 ` [patch 7/8] x86/ioapic: Force affinity setup before startup Thomas Gleixner
2021-07-21 19:11 ` [patch 8/8] x86/msi: " Thomas Gleixner
2021-07-21 21:10 ` [patch 0/8] PCI/MSI, x86: Cure a couple of inconsistencies Raj, Ashok
2021-07-21 22:39   ` Thomas Gleixner
2021-07-22 15:17 ` Marc Zyngier
2021-07-22 21:43 ` Bjorn Helgaas
2021-07-27 20:38   ` Thomas Gleixner

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20210722214329.GA349464@bjorn-Precision-5520 \
    --to=helgaas@kernel.org \
    --cc=alex.williamson@redhat.com \
    --cc=ashok.raj@intel.com \
    --cc=bhelgaas@google.com \
    --cc=davem@davemloft.net \
    --cc=kevin.tian@intel.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-pci@vger.kernel.org \
    --cc=maz@kernel.org \
    --cc=mingo@kernel.org \
    --cc=tglx@linutronix.de \
    --cc=x86@kernel.org \
    --subject='Re: [patch 1/8] PCI/MSI: Enable and mask MSIX early' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).