[RFT] e100 driver on ARM

[RFT] e100 driver on ARM

[Jeff Garzik]

One of the last steps necessary to deprecate the eepro100 driver is to 
ensure that e100 works everywhere that eepro100 does.

The eepro100 removal has been blocked for almost a year by a vague 
suggestion from Russell that e100 doesn't work on ARM.  But he doesn't 
have that machine anymore.  So, we're stuck in limbo.

This is a call to anyone who can test an Intel 10/100 chip on the ARM 
platform, in an effort to see where we are.  I'm looking for answers to 
the following two questions:

1) Does e100 driver work on ARM?

2) If not, does the "e100-sbit" branch of 
git://git.kernel.org/pub/scm/linux/kernel/git/jgarzik/netdev-2.6.git 
work on ARM?

FWIW, the e100-sbit branch has been in Andrew Morton's -mm tree since 
Nov 2005.

Below is the commit message for the e100-sbit change, in case anyone is 
interested.  I'm also hoping that Intel will help solve this problem, 
but poking Intel hasn't produced very much :(

	Jeff


> commit 32c1459bb3814274b3c5e0c5ed4efc6c0aa89eb4
> Author: Scott Feldman <sfeldma@pobox.com>
> Date:   Wed Nov 9 02:18:52 2005 -0500
> 
>     [netdrvr e100] experiment with doing RX in a similar manner to eepro100
> 
>     I was going to say that eepro100's speedo_rx_link() does the same DMA
>     abuse as e100, but then I noticed one little detail: eepro100 sets  both
>     EL (end of list) and S (suspend) bits in the RFD as it chains it  to the
>     RFD list.  e100 was only setting the EL bit.  Hmmm, that's  interesting.
>     That means that if HW reads a RFD with the S-bit set,  it'll process
>     that RFD and then suspend the receive unit.  The  receive unit will
>     resume when SW clears the S-bit.  There is no need  for SW to restart
>     the receive unit.  Which means a lot of the receive  unit state tracking
>     code in the driver goes away.
> 
>     So here's a patch against 2.6.14.  (Sorry for inlining it; the mailer
>     I'm using now will mess with the word wrap).  I can't test this on
>     XScale (unless someone has an e100 module for Gumstix :) .  It should
>     be doing exactly what eepro100 does with RFDs.  I don't believe this
>     change will introduce a performance hit because the S-bit and EL-bit  go
>     hand-in-hand meaning if we're going to suspend because of the S- bit,
>     we're on the last resource anyway, so we'll have to wait for SW  to
>     replenish.




-- 
VGER BF report: U 0.499999

Re: [RFT] e100 driver on ARM

[Lennert Buytenhek]

On Mon, Sep 04, 2006 at 06:39:29AM -0400, Jeff Garzik wrote:

> 1) Does e100 driver work on ARM?

FWIW, e100 seems to work okay for me on an intel ixp2400 (xscale based)
board, an ixp2850 (xscale based) board and an ixp2350 (xscale3 based)
board.  ixp2350 works both with hardware coherency turned on (cpu
snoops bus) and turned off (manual dma cache clean/invalidate as usual.)

As for the other ARM platforms that I'm interested in / have hardware
for / maintain, the at91/ep93xx/pxa270 don't have PCI, and the other
two (iop32x/iop33x) I can't test because I don't have such systems with
e100 NICs, but I expect those would work, since they're both xscale
based like the ixp2400, and the ixp2400 works.


cheers,
Lennert

-- 
VGER BF report: H 6.97804e-11

Re: [RFT] e100 driver on ARM

[Catalin Marinas]

Jeff Garzik <jeff@garzik.org> wrote:
> The eepro100 removal has been blocked for almost a year by a vague
> suggestion from Russell that e100 doesn't work on ARM.  But he doesn't
> have that machine anymore.  So, we're stuck in limbo.

Russell might have tested it on an Integrator/AP (not sure
though). IIRC, I tested the e100.c driver on this platform with 2.6.12
and it was OK at that time (the platform is now discontinued by ARM
Ltd).

> 1) Does e100 driver work on ARM?

I tested the e100.c driver a few months ago on the Versatile/PB and
RealView/EB platforms with PCI backplanes attached. The driver seemed
to work well (after fixing some PCI hardware problems on our
platforms). I couldn't get the eepro100.c driver to work though, but
didn't have any reason to investigate further.

-- 
Catalin

Re: [RFT] e100 driver on ARM

[Kok, Auke]

Lennert Buytenhek wrote:
> On Mon, Sep 04, 2006 at 06:39:29AM -0400, Jeff Garzik wrote:
> 
>> 1) Does e100 driver work on ARM?
> 
> FWIW, e100 seems to work okay for me on an intel ixp2400 (xscale based)
> board, an ixp2850 (xscale based) board and an ixp2350 (xscale3 based)
> board.  ixp2350 works both with hardware coherency turned on (cpu
> snoops bus) and turned off (manual dma cache clean/invalidate as usual.)
> 
> As for the other ARM platforms that I'm interested in / have hardware
> for / maintain, the at91/ep93xx/pxa270 don't have PCI, and the other
> two (iop32x/iop33x) I can't test because I don't have such systems with
> e100 NICs, but I expect those would work, since they're both xscale
> based like the ixp2400, and the ixp2400 works.

I just got an iop342 board dropped on my lap. Once it's running, I'll make sure 
to make this the first thing to test.

Cheers,

Auke

Re: [RFT] e100 driver on ARM

[David Acker]

Kok, Auke wrote:
> Lennert Buytenhek wrote:
>> On Mon, Sep 04, 2006 at 06:39:29AM -0400, Jeff Garzik wrote:
>>
>>> 1) Does e100 driver work on ARM?
>>
>> FWIW, e100 seems to work okay for me on an intel ixp2400 (xscale based)
>> board, an ixp2850 (xscale based) board and an ixp2350 (xscale3 based)
>> board.  ixp2350 works both with hardware coherency turned on (cpu
>> snoops bus) and turned off (manual dma cache clean/invalidate as usual.)
>>
>> As for the other ARM platforms that I'm interested in / have hardware
>> for / maintain, the at91/ep93xx/pxa270 don't have PCI, and the other
>> two (iop32x/iop33x) I can't test because I don't have such systems with
>> e100 NICs, but I expect those would work, since they're both xscale
>> based like the ixp2400, and the ixp2400 works.
> 
> I just got an iop342 board dropped on my lap. Once it's running, I'll 
> make sure to make this the first thing to test.
> 

I have a pxa255 based system with PCI added to it.  The e100 would have 
memory corruption in its receive buffers detected by slab debugging 
unless I put in the patch to use the S-bit.

Here is a link to the patch posting:
http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.20-rc3/2.6.20-rc3-mm1/broken-out/git-netdev-all.patch
Search for e100.c.

http://www-gatago.com/linux/kernel/15457063.html - This discussion seems 
to hit the issue.

There appears to be a race on the cache line where the EL bit and the 
next packet info live. In my case the hardware appeared to write to a 
free packet.  The S-bit seems to make the hardware stop and spin on the 
bit, while the EL bit seems to let the hardware try to use that packet.

This race would occur less often when the receive buffer chain is always 
refilled before the hardware can use them up.  On our 400 Mhz Xscale, we 
can use up all 256 buffers if the PCI bus has another busy device on it. 
  In our case it is an 802.11g miniPCI card and our software was routing 
all ethernet packets to the wireless interface and vice versa while TCP 
streams were running accross these connections.
-Ack

Re: [RFT] e100 driver on ARM

[Lennart Sorensen]

On Thu, Mar 29, 2007 at 01:17:38AM -0400, David Acker wrote:
> I have a pxa255 based system with PCI added to it.  The e100 would have 
> memory corruption in its receive buffers detected by slab debugging 
> unless I put in the patch to use the S-bit.
> 
> Here is a link to the patch posting:
> http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.20-rc3/2.6.20-rc3-mm1/broken-out/git-netdev-all.patch
> Search for e100.c.
> 
> http://www-gatago.com/linux/kernel/15457063.html - This discussion seems 
> to hit the issue.
> 
> There appears to be a race on the cache line where the EL bit and the 
> next packet info live. In my case the hardware appeared to write to a 
> free packet.  The S-bit seems to make the hardware stop and spin on the 
> bit, while the EL bit seems to let the hardware try to use that packet.
> 
> This race would occur less often when the receive buffer chain is always 
> refilled before the hardware can use them up.  On our 400 Mhz Xscale, we 
> can use up all 256 buffers if the PCI bus has another busy device on it. 
>  In our case it is an 802.11g miniPCI card and our software was routing 
> all ethernet packets to the wireless interface and vice versa while TCP 
> streams were running accross these connections.

Which PCI host controller are you using with the PXA255?  We tried using
a PXA255 based system with a PCI controller a couple of years ago and
have to change to a different cpu in the end due to the PCI controller
simply not being valid PCI.  The PXA255 wasn't designed for PCI, and I
get the impression that non of the PCI companion chips for it do a good
enough job to actually add it correctly.

--
Len Sorensen

Re: [RFT] e100 driver on ARM

[David Acker]

Lennart Sorensen wrote:
> Which PCI host controller are you using with the PXA255?  We tried using
> a PXA255 based system with a PCI controller a couple of years ago and
> have to change to a different cpu in the end due to the PCI controller
> simply not being valid PCI.  The PXA255 wasn't designed for PCI, and I
> get the impression that non of the PCI companion chips for it do a good
> enough job to actually add it correctly.
> 

Sorry for the delay in responding...my wife and I just had twins!

We are using the IT8152G RISC-to-PCI companion chip.
-Ack

Re: [RFT] e100 driver on ARM

[Lennart Sorensen]

On Mon, Apr 16, 2007 at 11:07:36AM -0400, David Acker wrote:
> Lennart Sorensen wrote:
> >Which PCI host controller are you using with the PXA255?  We tried using
> >a PXA255 based system with a PCI controller a couple of years ago and
> >have to change to a different cpu in the end due to the PCI controller
> >simply not being valid PCI.  The PXA255 wasn't designed for PCI, and I
> >get the impression that non of the PCI companion chips for it do a good
> >enough job to actually add it correctly.
> >
> 
> Sorry for the delay in responding...my wife and I just had twins!
> 
> We are using the IT8152G RISC-to-PCI companion chip.

Well the IT8152G+PXA255 combination used on the SBC we tried a couple of
years ago did not work.  The PCI bus had errors and the SBC maker gave
up trying to fix it.  We switched to a Geode SC1200 based board instead
which works fine PCI wise.

My suspicision (although it is only that) is that the PXA255 trying to
access memory may cause interruptions in PCI bus master transfers, which
is of course not permitted by the PCI spec (at least the way I read it).
We tried it with RTL8139, AMD 972 (both ethernet) as well as a number of
T1/E1 and DDS wan cards from sangoma.  The wan cards had the most issues
with it (they drivers and hardware would get out of sync due to PCI bus
problems), while the ethernet just had occational packet corruption.  I
will certainly never consider using a PXA + ITE pci controller
combination ever again.  Too bad since the performance of the PXA is
amazing.  The PXA chips are not designed to speak to PCI, and the ITE
companion chip doesn't quite do the job of pretending it was.  I would
expect problems if you do pci bus master transfers and/or any kind of
PCI bus traffic load.

--
Len Sorensen

Re: [RFT] e100 driver on ARM

[David Acker]

Lennart Sorensen wrote:
> Well the IT8152G+PXA255 combination used on the SBC we tried a couple of
> years ago did not work.  The PCI bus had errors and the SBC maker gave
> up trying to fix it.  We switched to a Geode SC1200 based board instead
> which works fine PCI wise.
> 

I don't think this is it.  Other PCI devices work fine on this board 
include several 802.11 radios.  The S-bit patch totally fixes the problem.

Here is a quote from Russell that describes what I believe is the main 
problem:
http://www-gatago.com/linux/kernel/15457063.html
"
Has e100 actually been fixed to use the PCI DMA API correctly yet?
Looking at it, it doesn't look like it, so until it does, eepro100
is the far better bet for platforms needing working DMA API.

What I'm talking about is e100's apparant belief that it can modify
rfd's in the receive ring on a non-cache coherent architecture and
expect the data around it to remain unaffected (see e100_rx_alloc_skb):

struct rfd {
u16 status;
u16 command;
u32 link;
u32 rbd;
u16 actual_size;
u16 size;
};

it touches command and link. This means that the whole rfd plus
maybe the following or preceding 16 bytes get loaded into a cache
line (assuming cache lines of 32 bytes), and that data written
out again at sync. However, it does this on what seems to be an
active receive chain.

So, both the CPU _and_ the device own the same data. Which is a
violation of the DMA API.
"

I think that the S-bit patch fixes it because the hardware spins on the 
s-bit instead of using the packet.  With just the el-bit, the hardware 
tries to use the same cache line that the software is updating.

Can someone from Intel let us know if I understand the hardware's 
handling of the S and EL bits?  If my interpretation is correct, can the 
s-bit patch be applied?  It seems like the correct way to lock out the 
hardware while a packet is being updated.  I have not seen a reason 
given not to apply the patch.

-Ack

Re: [RFT] e100 driver on ARM

[Lennert Buytenhek]

On Thu, Apr 26, 2007 at 09:41:22AM -0400, David Acker wrote:

> Here is a quote from Russell that describes what I believe is the main 
> problem:
> http://www-gatago.com/linux/kernel/15457063.html
> "
> Has e100 actually been fixed to use the PCI DMA API correctly yet?
> Looking at it, it doesn't look like it, so until it does, eepro100
> is the far better bet for platforms needing working DMA API.
> 
> What I'm talking about is e100's apparant belief that it can modify
> rfd's in the receive ring on a non-cache coherent architecture and
> expect the data around it to remain unaffected (see e100_rx_alloc_skb):
> 
> struct rfd {
> u16 status;
> u16 command;
> u32 link;
> u32 rbd;
> u16 actual_size;
> u16 size;
> };
> 
> it touches command and link. This means that the whole rfd plus
> maybe the following or preceding 16 bytes get loaded into a cache
> line (assuming cache lines of 32 bytes), and that data written
> out again at sync. However, it does this on what seems to be an
> active receive chain.
> 
> So, both the CPU _and_ the device own the same data. Which is a
> violation of the DMA API.
> "
> 
> I think that the S-bit patch fixes it because the hardware spins on the 
> s-bit instead of using the packet.  With just the el-bit, the hardware 
> tries to use the same cache line that the software is updating.
> 
> Can someone from Intel let us know if I understand the hardware's 
> handling of the S and EL bits?  If my interpretation is correct, can the 
> s-bit patch be applied?  It seems like the correct way to lock out the 
> hardware while a packet is being updated.  I have not seen a reason 
> given not to apply the patch.

This is all a while ago now, but wasn't the e100 S-bit patch originally
written by Intel people in response to the very same quote by Russell
King that you've quoted above?  The S-bit patch should probably just
be applied, IMHO.

Re: [RFT] e100 driver on ARM

[Jeff Garzik]

Lennert Buytenhek wrote:
> This is all a while ago now, but wasn't the e100 S-bit patch originally
> written by Intel people in response to the very same quote by Russell
> King that you've quoted above?

Correct.

I just wanted to make sure it didn't kill any boxes.

	Jeff

Re: [RFT] e100 driver on ARM

[Kok, Auke]

Jeff Garzik wrote:
> Lennert Buytenhek wrote:
>> This is all a while ago now, but wasn't the e100 S-bit patch originally
>> written by Intel people in response to the very same quote by Russell
>> King that you've quoted above?
> 
> Correct.
> 
> I just wanted to make sure it didn't kill any boxes.

Neither did we. On top of that we didn't have the equipment to test on ARM on 
until recently, and the system that I got recently will not even load the kernel 
with an e100 NIC in the PCI slot (way way way before e100.ko loads, e1000 works 
just fine). that doesn't help either.

Meanwhile we've not sat still and jesse wrote a patch to have e100 use IO 
optionally for plagued platforms which seems to fix some of these issues, and 
Jeff Kirsher has been actively tracking a e100 IPMI issue on a very specific 
platform.

Jeff, I think I should just push the IO patch and the sbit code to Andrew and 
have it sit there. That is a vastly larger test resource than we currently can 
generate for this. If needed we just let is sit there for a whole release cycle 
before moving it to #upstream.

seems like a good idea, right?

Auke

Re: [RFT] e100 driver on ARM

[Jeff Garzik]

Kok, Auke wrote:
> Jeff Garzik wrote:
>> Lennert Buytenhek wrote:
>>> This is all a while ago now, but wasn't the e100 S-bit patch originally
>>> written by Intel people in response to the very same quote by Russell
>>> King that you've quoted above?
>>
>> Correct.
>>
>> I just wanted to make sure it didn't kill any boxes.
> 
> Neither did we. On top of that we didn't have the equipment to test on 
> ARM on until recently, and the system that I got recently will not even 
> load the kernel with an e100 NIC in the PCI slot (way way way before 
> e100.ko loads, e1000 works just fine). that doesn't help either.
> 
> Meanwhile we've not sat still and jesse wrote a patch to have e100 use 
> IO optionally for plagued platforms which seems to fix some of these 
> issues, and Jeff Kirsher has been actively tracking a e100 IPMI issue on 
> a very specific platform.
> 
> Jeff, I think I should just push the IO patch and the sbit code to 
> Andrew and have it sit there. That is a vastly larger test resource than 
> we currently can generate for this. If needed we just let is sit there 
> for a whole release cycle before moving it to #upstream.

The sbit code has been in -mm via netdev-2.6.git#ALL for many months 
now.  I am certainly open to adding any number of "for -mm only" style 
patches that you wish to pass along, for /any/ driver: e100, e1000, ixgb.

	Jeff

Re: [RFT] e100 driver on ARM

[H. Peter Anvin]

Lennart Sorensen wrote:
> 
> My suspicision (although it is only that) is that the PXA255 trying to
> access memory may cause interruptions in PCI bus master transfers, which
> is of course not permitted by the PCI spec (at least the way I read it).

Why wouldn't that be permitted?  It, in fact, happens all the time (the
host bridge withdraws the GNT# line and raises STOP#, which does a
Termination With Data of the bus transfer.)  This is a normal event and
if you can't handle it you won't work with many host bridges at all.

	-hpa

Re: [RFT] e100 driver on ARM

[Kok, Auke]

Jeff Garzik wrote:
> Kok, Auke wrote:
>> Jeff, I think I should just push the IO patch and the sbit code to 
>> Andrew and have it sit there. That is a vastly larger test resource than 
>> we currently can generate for this. If needed we just let is sit there 
>> for a whole release cycle before moving it to #upstream.
> 
> The sbit code has been in -mm via netdev-2.6.git#ALL for many months 
> now.  I am certainly open to adding any number of "for -mm only" style 
> patches that you wish to pass along, for /any/ driver: e100, e1000, ixgb.

I think we should then just bite the bullet and move it over to #upstream for 
2.6.22, that gives us another full release cycle to monitor it.

I'll post 2 patches in a second. the first is for upstream and allows the driver 
to run with an invalid mac address in case the user overrides the eeprom 
checksum (in which case the mac address could likely be wrong). The second one 
is the I/O mode patch and for Andrew (MM), allthough the I/O mode is disabled by 
default, so it should be safe to put in upstream as well, but I leave that up to 
you to decide.

Auke

Re: [RFT] e100 driver on ARM

[Lennart Sorensen]

On Thu, Apr 26, 2007 at 09:19:34AM -0700, H. Peter Anvin wrote:
> Why wouldn't that be permitted?  It, in fact, happens all the time (the
> host bridge withdraws the GNT# line and raises STOP#, which does a
> Termination With Data of the bus transfer.)  This is a normal event and
> if you can't handle it you won't work with many host bridges at all.

Well there must have been something else wrong then.  Certainly I saw
data corruption on a rtl8139.  No problems with the same hardware using
a geode SC1200, so I have no idea.  I liked the speed of the PXA255 a
lot better than the slow poke SC1200.

--
Len Sorensen