LKML Archive on lore.kernel.org
help / color / mirror / Atom feed
From: Muli Ben-Yehuda <muli@il.ibm.com>
To: Greg KH <greg@kroah.com>
Cc: jdmason@kudzu.us, bzolnier@gmail.com, linux-ide@vger.kernel.org,
	linux-pci@atrey.karlin.mff.cuni.cz, linux-kernel@vger.kernel.org,
	discuss@x86-64.org
Subject: Re: pci_get_device_reverse(), why does Calgary need this?
Date: Fri, 15 Feb 2008 09:48:27 +0200	[thread overview]
Message-ID: <20080215074827.GE4334@rhun.haifa.ibm.com> (raw)
In-Reply-To: <20080215071703.GA9022@kroah.com>

On Thu, Feb 14, 2008 at 11:17:03PM -0800, Greg KH wrote:

> Hm, that's wierd.  I thought I got something, until I realized that
> you are doing a lot of logic before you ever even determine that
> your hardware is present in the system.  Why are you calling
> calgary_locate_bbars() and doing all of that work?  Or am I mising
> something in the code flow here?

Ok, it's all starting to come back to me. See below.

> Also, it looks like you use the pci_get_device() to find the pci
> device, then you do somethings, and then drop the device, never to
> use it again.
> 
> So, a traditional "probe" might not work as well, but it could be
> used if you want to.

We have a two stage initialization process. First, we need to be
initialized very early in order to be able to allocate relatively
large contiguous chunks of RAM. That requires detecting whether we're
on a Calgary/CalIOC2 system very early (detect_calgary(), called from
pci_iommu_alloc()). Then we have "regular" PCI initialization at a
later stage (calgary_iommu_init(), called from pci_iommu_init()),
which also calls calgary_locate_bbars(). Unlike a regular PCI device
which has its BARs in the configuration space, we need to probe BIOS
provided tables to find where our BARs are before we do anything else
with the "device representation" of the IOMMU. Note that the same
two-stage initialization scheme is used by all other x86 IOMMUs, for
similar reasons.

Now, Calgary is an isolation-capable IOMMU which has per-root-bus (as
opposed to global or per-BDF) IO address space. The reason we want to
access the pci_dev of each Calgary bus is to hang off of it the IOMMU
tables for all devices under that bus so that we end up using the
right translation table when a device asks for a DMA mapping. Hence
the loop in calgary_init, which loops over all pci_dev's of Calgary
busses, raises the ref count for each pci_dev to protect against
hot-unplug, and hooks up the IOMMU table for everything under that bus
to the pci_dev for the bus. Unlike a regular driver, our main entry
points are via the DMA-API, which takes a device's pci_dev, not
Calgary's. We then walk up the bus hierarchy and find the Calgary
pci_dev that is an ancestor of this device, and that gives us the
right IOMMU table to us. Note that this has to occur *before* PCI
device initialization, since we want to turn translation on before any
device will try to DMA.

In conclusion, our usage doesn't seem lika a good fit for the probe
approach, although it could probably converted provided we got the
ordering right with regards to regular PCI device
initialization. Doesn't seem to be worth the effort.

Cheers,
Muli


  reply	other threads:[~2008-02-15  7:48 UTC|newest]

Thread overview: 45+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2008-02-13  0:15 "ide=reverse" do we still " Greg KH
2008-02-13  0:16 ` pci_get_device_reverse(), why does Calgary " Greg KH
2008-02-13  2:17   ` Alan Cox
2008-02-13  4:45     ` Greg KH
2008-02-13 12:34       ` Bartlomiej Zolnierkiewicz
2008-02-13 17:28         ` Greg KH
2008-02-13 18:16           ` Greg KH
2008-02-13 22:20             ` Bartlomiej Zolnierkiewicz
2008-02-13 23:41               ` Greg KH
2008-02-14  0:02                 ` Bartlomiej Zolnierkiewicz
2008-02-14  4:58                   ` Greg KH
2008-02-14 13:09               ` Sergei Shtylyov
2008-02-14  7:44             ` [discuss] " Andreas Jaeger
2008-02-14 12:11               ` Bartlomiej Zolnierkiewicz
2008-02-15  7:17                 ` Greg KH
2008-02-20  0:39                 ` Greg KH
2008-02-13  9:32   ` Muli Ben-Yehuda
2008-02-13 17:32     ` Greg KH
2008-02-13 17:47       ` Muli Ben-Yehuda
2008-02-13 18:14         ` Greg KH
2008-02-15  7:17           ` Greg KH
2008-02-15  7:48             ` Muli Ben-Yehuda [this message]
2008-02-15 15:20               ` Greg KH
2008-02-15 15:31                 ` Sam Ravnborg
2008-02-15 15:46                   ` yong xue
2008-02-15 18:28                   ` Greg KH
2008-02-17  7:53                     ` Muli Ben-Yehuda
2008-02-13  1:41 ` "ide=reverse" do we still " Rene Herman
2008-02-13  4:44   ` Greg KH
2008-02-13 12:06     ` Rene Herman
2008-02-13 12:16       ` Michael Ellerman
2008-02-13 12:46         ` Rene Herman
2008-02-13 22:39           ` Michael Ellerman
2008-02-13 12:57       ` Rene Herman
2008-02-14 17:16     ` Bill Davidsen
2008-02-15 13:58       ` Matthew Wilcox
2008-02-13  2:43 ` Ken Moffat
2008-02-13  4:43   ` Greg KH
2008-02-13 15:32     ` Ken Moffat
2008-02-19 15:08       ` Ken Moffat
2008-02-13  7:54 ` [discuss] " Dirk GOUDERS
2008-02-13  8:26   ` Greg KH
2008-02-13  8:54     ` Dirk GOUDERS
2008-02-13 20:00     ` Dirk GOUDERS
2008-02-13 20:48       ` Greg KH

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20080215074827.GE4334@rhun.haifa.ibm.com \
    --to=muli@il.ibm.com \
    --cc=bzolnier@gmail.com \
    --cc=discuss@x86-64.org \
    --cc=greg@kroah.com \
    --cc=jdmason@kudzu.us \
    --cc=linux-ide@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-pci@atrey.karlin.mff.cuni.cz \
    --subject='Re: pci_get_device_reverse(), why does Calgary need this?' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).