PCI userland access non-mmap APIs, kernel access to legacy space

PCI userland access non-mmap APIs, kernel access to legacy space

[Benjamin Herrenschmidt]

Hi folks !

Before I start actually writing code, I want to do a quick brain dump
here to see if there are any objections.

So a problem we have is that on non-x86, we expect to be able to access
PCI IO space via mmap'ing of sysfs based "resource" files.

However, there are various PCI host bridges out there that don't make
that a practical solution. So far, I have two cases on ppc of bridges
that either use indirect access for the IO space or require specific HW
workarounds on accesses that involve spinlocks (to properly deal with
the non-posted aspect of IO transactions). In addition, Lennert tells me
that similar issues exist on ARM.

Thus, I would like to introduce an additional read/write interface to
sysfs "resource" files (and then get Ian's libpciaccess use that when
mmap is unavailable).

In fact, it's funny in a way becauase the "legacy" files interface is
already read/write based (exclusively even, mmap isn't supported) for
legacy IOs.

I beleive that doing so is better than trying to extend /dev/port to
understand multiple domains, >64K IO space, etc... etc...., because you
don't have to understand all the remapping tricks that the operating
system might be doing behind the scene, just open the resource file (or
the legacy file of that bus for legacy accesses) and read/write to it.

Any objection ?

Also, while at it, I would like to introduce a pair of in-kernel
interfaces:

int pci_translate_legacy_resource(struct pci_bus *bus, struct resource *res);

Which takes a resource containing an absolute legacy range on that bus
(for example, an IO resource with the VGA port numbers, or a memory
resource with the VGA text mode aperture). That resource is then
"converted" for the given PCI bus. For x86, that function would boil
down to "return 0;".

The reason is that on multiple domains machines, it's actually not
trivial to figure out where the legacy space of a given domain is
mapped, if available at all. The above function is a way to solve that
problem, allowing things like vgacon to work in multi-domains setup
without half of the current arch specific hacks in there.

That would be also allowed to fail (some platforms can't give access to
legacy memory or legacy IOs at all).

We could probably even implement the pci sysfs "legacy" interface on top
of these as well, thus making it generic to all platforms.

Any objection there too ?

Cheers,
Ben.

Re: PCI userland access non-mmap APIs, kernel access to legacy space

[Jesse Barnes]

On Sunday, June 17, 2007 7:16:44 Benjamin Herrenschmidt wrote:
> Hi folks !
>
> Before I start actually writing code, I want to do a quick brain dump
> here to see if there are any objections.
>
> So a problem we have is that on non-x86, we expect to be able to
> access PCI IO space via mmap'ing of sysfs based "resource" files.
>
> However, there are various PCI host bridges out there that don't make
> that a practical solution. So far, I have two cases on ppc of bridges
> that either use indirect access for the IO space or require specific
> HW workarounds on accesses that involve spinlocks (to properly deal
> with the non-posted aspect of IO transactions). In addition, Lennert
> tells me that similar issues exist on ARM.
>
> Thus, I would like to introduce an additional read/write interface to
> sysfs "resource" files (and then get Ian's libpciaccess use that when
> mmap is unavailable).
>
> In fact, it's funny in a way becauase the "legacy" files interface is
> already read/write based (exclusively even, mmap isn't supported) for
> legacy IOs.
>
> I beleive that doing so is better than trying to extend /dev/port to
> understand multiple domains, >64K IO space, etc... etc...., because
> you don't have to understand all the remapping tricks that the
> operating system might be doing behind the scene, just open the
> resource file (or the legacy file of that bus for legacy accesses)
> and read/write to it.
>
> Any objection ?

Sounds like a good idea.  Originally, the "legacy_io" file was 
implemented this way because I knew of machines that couldn't properly 
handle it as a mmappable interface.  But when I pinged people about the 
resource files, I didn't hear anything, so they were just done with 
mmap.

But adding read/write is fine with me.  In fact, having read/write hooks 
for the memory backed resource files is also nice, since it makes 
dealing with them on the command line a bit easier.

> Also, while at it, I would like to introduce a pair of in-kernel
> interfaces:

What's the other half of the pair?

> int pci_translate_legacy_resource(struct pci_bus *bus, struct
> resource *res);
>
> Which takes a resource containing an absolute legacy range on that
> bus (for example, an IO resource with the VGA port numbers, or a
> memory resource with the VGA text mode aperture). That resource is
> then "converted" for the given PCI bus. For x86, that function would
> boil down to "return 0;".
>
> The reason is that on multiple domains machines, it's actually not
> trivial to figure out where the legacy space of a given domain is
> mapped, if available at all. The above function is a way to solve
> that problem, allowing things like vgacon to work in multi-domains
> setup without half of the current arch specific hacks in there.
>
> That would be also allowed to fail (some platforms can't give access
> to legacy memory or legacy IOs at all).
>
> We could probably even implement the pci sysfs "legacy" interface on
> top of these as well, thus making it generic to all platforms.
>
> Any objection there too ?

Yeah, sounds like a good additional abstraction for that interface.  
Should make porting it to other arches easier (I talked with kyle about 
this last week, so maybe this'll give him motivation to do it for 
parisc).

Jesse

Re: PCI userland access non-mmap APIs, kernel access to legacy space

[Benjamin Herrenschmidt]

> But adding read/write is fine with me.  In fact, having read/write hooks 
> for the memory backed resource files is also nice, since it makes 
> dealing with them on the command line a bit easier.

Plan is to use iomap so I get both IO and MMIO for (almost) free :-)

> > Also, while at it, I would like to introduce a pair of in-kernel
> > interfaces:
> 
> What's the other half of the pair?

Heh, indeed, there is none, I initially had a separate function for mem
and for io and at the last minute though "ugh, let's just use a struct
resource !" :-)

> Yeah, sounds like a good additional abstraction for that interface.  
> Should make porting it to other arches easier (I talked with kyle about 
> this last week, so maybe this'll give him motivation to do it for 
> parisc).

Ok, good. I'll do some code asap.

Ben.