pci-ftpci100: race condition in masking/unmasking interrupts

pci-ftpci100: race condition in masking/unmasking interrupts

[Pali Rohár]

Hello!

I do not see any entry in MAINTAINERS file for pci-ftpci100.c driver, so
I'm not sure to whom should I address this issue...

During pci-aardvark review, Marc pointed one issue which is currently
available also in pci-ftpci100.c driver.

When masking or unmasking interrupts there is read-modify-write sequence
for FARADAY_PCI_CTRL2 register without any locking and is not atomic:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/controller/pci-ftpci100.c?h=v5.13#n270

So there is race condition when masking/unmasking more interrupts at the
same time.

Re: pci-ftpci100: race condition in masking/unmasking interrupts

[Linus Walleij]

On Wed, Aug 18, 2021 at 1:47 PM Pali Rohár <pali@kernel.org> wrote:

> I do not see any entry in MAINTAINERS file for pci-ftpci100.c driver, so
> I'm not sure to whom should I address this issue...

It's me.

> During pci-aardvark review, Marc pointed one issue which is currently
> available also in pci-ftpci100.c driver.
>
> When masking or unmasking interrupts there is read-modify-write sequence
> for FARADAY_PCI_CTRL2 register without any locking and is not atomic:
>
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/controller/pci-ftpci100.c?h=v5.13#n270
>
> So there is race condition when masking/unmasking more interrupts at the
> same time.

I thought those operations were called in atomic context.
How did you fix it?

Yours,
Linus Walleij

Re: pci-ftpci100: race condition in masking/unmasking interrupts

[Pali Rohár]

On Tuesday 07 September 2021 13:22:37 Linus Walleij wrote:
> On Wed, Aug 18, 2021 at 1:47 PM Pali Rohár <pali@kernel.org> wrote:
> 
> > I do not see any entry in MAINTAINERS file for pci-ftpci100.c driver, so
> > I'm not sure to whom should I address this issue...
> 
> It's me.

Ok! So could you send update for MAINTAINERS file for this driver?

> > During pci-aardvark review, Marc pointed one issue which is currently
> > available also in pci-ftpci100.c driver.
> >
> > When masking or unmasking interrupts there is read-modify-write sequence
> > for FARADAY_PCI_CTRL2 register without any locking and is not atomic:
> >
> > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/controller/pci-ftpci100.c?h=v5.13#n270
> >
> > So there is race condition when masking/unmasking more interrupts at the
> > same time.
> 
> I thought those operations were called in atomic context.

I guess that they cannot be as for performance reasons you could want to
mask or unmask more interrupts in parallel on more CPUs.

> How did you fix it?

Guarding all read-modify-write operations on register by raw spin lock. See:
https://lore.kernel.org/linux-pci/20210820155020.3000-1-pali@kernel.org/

Re: pci-ftpci100: race condition in masking/unmasking interrupts

[Marc Zyngier]

Hi Linus,

On Tue, 07 Sep 2021 12:22:37 +0100,
Linus Walleij <linus.walleij@linaro.org> wrote:
> 
> On Wed, Aug 18, 2021 at 1:47 PM Pali Rohár <pali@kernel.org> wrote:
> 
> > I do not see any entry in MAINTAINERS file for pci-ftpci100.c driver, so
> > I'm not sure to whom should I address this issue...
> 
> It's me.
> 
> > During pci-aardvark review, Marc pointed one issue which is currently
> > available also in pci-ftpci100.c driver.
> >
> > When masking or unmasking interrupts there is read-modify-write sequence
> > for FARADAY_PCI_CTRL2 register without any locking and is not atomic:
> >
> > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/controller/pci-ftpci100.c?h=v5.13#n270
> >
> > So there is race condition when masking/unmasking more interrupts at the
> > same time.
> 
> I thought those operations were called in atomic context.
> How did you fix it?

They are.

But that doesn't mean that you cannot have two CPUs dealing with two
different interrupts at the same time (using disable_irq(), for
example). When that happens, your interrupt masking becomes a bit
soup. irq_ack() also gets in the way, as it does a RMW of the same
register. If the underlying HW is strictly UP, you're safe. But even
in this case, you could have some locking that gets elided at compile
time.

I also don't understand why you always clear the interrupt status
every time you mask/unmask an interrupt.

I came up with the following patchlet, which is completely untested
(not even compile-tested).

Thanks,

	M.

diff --git a/drivers/pci/controller/pci-ftpci100.c b/drivers/pci/controller/pci-ftpci100.c
index 88980a44461d..dd1697e61206 100644
--- a/drivers/pci/controller/pci-ftpci100.c
+++ b/drivers/pci/controller/pci-ftpci100.c
@@ -120,6 +120,7 @@ struct faraday_pci_variant {
 };
 
 struct faraday_pci {
+	raw_spinlock_t lock
 	struct device *dev;
 	void __iomem *base;
 	struct irq_domain *irqdomain;
@@ -270,34 +271,41 @@ static struct pci_ops faraday_pci_ops = {
 static void faraday_pci_ack_irq(struct irq_data *d)
 {
 	struct faraday_pci *p = irq_data_get_irq_chip_data(d);
+	unsigned long flags;
 	unsigned int reg;
 
+	raw_spin_lock_irqsave(&p->lock, flags);
 	faraday_raw_pci_read_config(p, 0, 0, FARADAY_PCI_CTRL2, 4, &reg);
 	reg &= ~(0xF << PCI_CTRL2_INTSTS_SHIFT);
 	reg |= BIT(irqd_to_hwirq(d) + PCI_CTRL2_INTSTS_SHIFT);
 	faraday_raw_pci_write_config(p, 0, 0, FARADAY_PCI_CTRL2, 4, reg);
+	raw_spin_unlock_irqrestore(&p->lock, flags);
 }
 
 static void faraday_pci_mask_irq(struct irq_data *d)
 {
 	struct faraday_pci *p = irq_data_get_irq_chip_data(d);
+	unsigned long flags;
 	unsigned int reg;
 
+	raw_spin_lock_irqsave(&p->lock, flags);
 	faraday_raw_pci_read_config(p, 0, 0, FARADAY_PCI_CTRL2, 4, &reg);
-	reg &= ~((0xF << PCI_CTRL2_INTSTS_SHIFT)
-		 | BIT(irqd_to_hwirq(d) + PCI_CTRL2_INTMASK_SHIFT));
+	reg &= ~BIT(irqd_to_hwirq(d) + PCI_CTRL2_INTMASK_SHIFT);
 	faraday_raw_pci_write_config(p, 0, 0, FARADAY_PCI_CTRL2, 4, reg);
+	raw_spin_unlock_irqrestore(&p->lock, flags);
 }
 
 static void faraday_pci_unmask_irq(struct irq_data *d)
 {
 	struct faraday_pci *p = irq_data_get_irq_chip_data(d);
+	unsigned long flags;
 	unsigned int reg;
 
+	raw_spin_lock_irqsave(&p->lock, flags);
 	faraday_raw_pci_read_config(p, 0, 0, FARADAY_PCI_CTRL2, 4, &reg);
-	reg &= ~(0xF << PCI_CTRL2_INTSTS_SHIFT);
 	reg |= BIT(irqd_to_hwirq(d) + PCI_CTRL2_INTMASK_SHIFT);
 	faraday_raw_pci_write_config(p, 0, 0, FARADAY_PCI_CTRL2, 4, reg);
+	raw_spin_unlock_irqrestore(&p->lock, flags);
 }
 
 static void faraday_pci_irq_handler(struct irq_desc *desc)
@@ -441,6 +449,8 @@ static int faraday_pci_probe(struct platform_device *pdev)
 	host->sysdata = p;
 	p->dev = dev;
 
+	raw_spin_lock_init(&p->lock);
+
 	/* Retrieve and enable optional clocks */
 	clk = devm_clk_get(dev, "PCLK");
 	if (IS_ERR(clk))

-- 
Without deviation from the norm, progress is not possible.