net: tx timeouts with skge, 8139too, dmfe drivers/NICs

net: tx timeouts with skge, 8139too, dmfe drivers/NICs

[Marin Mitov]

Hi all,

I experience very rare freezes at heavy outbound traffic 
(sending ~4GB DVD image to another host(s) on the same LAN) 
using skge driver (NIC on the mobo) as well as (recently tested)
using rtl8139 or dmfe NICs on the PCI bus. There is a single 
switch between them (tested with another one just to exclude
a faulty switch).

skge <--> Marvell 88E8001 chip
8139too <--> Realtek 8136B chip
dmfe <--> Davicom DM9102 chip

Symptoms are similar: tx timeouts and no more net activity.
KDE desktop works, computational programs - work, the machine 
is usable, but cannot ping, nor can be ping-ed anymore.
rmmod && modprobe the respective modules repairs the problem.
Simple surfing/e-mailing from it do not trigger the problem.

The machine is used as LTSP server for old PCs (as X terminals)
(mostly outbound traffic) and is not usable as such due to this
problem.

The kernel is 2.6.24.2-SMP/x86_32 (PREEMPT or not - NO difference).

As far as this happens with 3 different NICs/drivers could it be
a problem in the (common for all of them) networking subsystem?

As far as many persons are working on this machine only limited 
testing could be done.

Thank you in advance for your suggestions, help (and patches).

Regards.

Marin Mitov

Re: net: tx timeouts with skge, 8139too, dmfe drivers/NICs

[Jeff Garzik]

Marin Mitov wrote:
> Hi all,
> 
> I experience very rare freezes at heavy outbound traffic 
> (sending ~4GB DVD image to another host(s) on the same LAN) 
> using skge driver (NIC on the mobo) as well as (recently tested)
> using rtl8139 or dmfe NICs on the PCI bus. There is a single 
> switch between them (tested with another one just to exclude
> a faulty switch).
> 
> skge <--> Marvell 88E8001 chip
> 8139too <--> Realtek 8136B chip
> dmfe <--> Davicom DM9102 chip
> 
> Symptoms are similar: tx timeouts and no more net activity.
> KDE desktop works, computational programs - work, the machine 
> is usable, but cannot ping, nor can be ping-ed anymore.
> rmmod && modprobe the respective modules repairs the problem.
> Simple surfing/e-mailing from it do not trigger the problem.
> 
> The machine is used as LTSP server for old PCs (as X terminals)
> (mostly outbound traffic) and is not usable as such due to this
> problem.
> 
> The kernel is 2.6.24.2-SMP/x86_32 (PREEMPT or not - NO difference).
> 
> As far as this happens with 3 different NICs/drivers could it be
> a problem in the (common for all of them) networking subsystem?

A TX timeout (like hardware timeouts, in general) is a very generic 
behavior, with many causes.

In general, when you see timeouts with varied hardware and drivers, 
you're almost always dealing with a problem with interrupt delivery, or 
a generic system problem, rather than bugs in the network stack or all 
three drivers.

	Jeff

Re: net: tx timeouts with skge, 8139too, dmfe drivers/NICs

[Marin Mitov]

On Monday 25 February 2008 10:53:01 pm you wrote:
> Marin Mitov wrote:
> > Hi all,
> >
> > I experience very rare freezes at heavy outbound traffic
> > (sending ~4GB DVD image to another host(s) on the same LAN)
> > using skge driver (NIC on the mobo) as well as (recently tested)
> > using rtl8139 or dmfe NICs on the PCI bus. There is a single
> > switch between them (tested with another one just to exclude
> > a faulty switch).
> >
> > skge <--> Marvell 88E8001 chip
> > 8139too <--> Realtek 8136B chip
> > dmfe <--> Davicom DM9102 chip
> >
> > Symptoms are similar: tx timeouts and no more net activity.
> > KDE desktop works, computational programs - work, the machine
> > is usable, but cannot ping, nor can be ping-ed anymore.
> > rmmod && modprobe the respective modules repairs the problem.
> > Simple surfing/e-mailing from it do not trigger the problem.
> >
> > The machine is used as LTSP server for old PCs (as X terminals)
> > (mostly outbound traffic) and is not usable as such due to this
> > problem.
> >
> > The kernel is 2.6.24.2-SMP/x86_32 (PREEMPT or not - NO difference).
> >
> > As far as this happens with 3 different NICs/drivers could it be
> > a problem in the (common for all of them) networking subsystem?
>
> A TX timeout (like hardware timeouts, in general) is a very generic
> behavior, with many causes.
>
> In general, when you see timeouts with varied hardware and drivers,
> you're almost always dealing with a problem with interrupt delivery, or

All the drivers are using #INTA on PCI bus (no MSI/MSI-X).

"problem with interrupt delivery" - you suspect interrupts incorrectly
 disabled (lost) in the drivers or faulty hardware(motherboard)?

> a generic system problem, rather than bugs in the network stack or all

"a generic system problem" - bad config or faulty hardware(motherboard)?

Where I should look for the problem?

Just for info: the system is very stable - uptime (if no power outages) could
be a month or more (rebooting for kernel changes or updates).

Marin Mitov

> three drivers.
>
> 	Jeff
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: net: tx timeouts with skge, 8139too, dmfe drivers/NICs

[Stephen Hemminger]

On Mon, 25 Feb 2008 23:36:06 +0200
Marin Mitov <mitov@issp.bas.bg> wrote:

> On Monday 25 February 2008 10:53:01 pm you wrote:
> > Marin Mitov wrote:
> > > Hi all,
> > >
> > > I experience very rare freezes at heavy outbound traffic
> > > (sending ~4GB DVD image to another host(s) on the same LAN)
> > > using skge driver (NIC on the mobo) as well as (recently tested)
> > > using rtl8139 or dmfe NICs on the PCI bus. There is a single
> > > switch between them (tested with another one just to exclude
> > > a faulty switch).
> > >
> > > skge <--> Marvell 88E8001 chip
> > > 8139too <--> Realtek 8136B chip
> > > dmfe <--> Davicom DM9102 chip
> > >
> > > Symptoms are similar: tx timeouts and no more net activity.
> > > KDE desktop works, computational programs - work, the machine
> > > is usable, but cannot ping, nor can be ping-ed anymore.
> > > rmmod && modprobe the respective modules repairs the problem.
> > > Simple surfing/e-mailing from it do not trigger the problem.
> > >
> > > The machine is used as LTSP server for old PCs (as X terminals)
> > > (mostly outbound traffic) and is not usable as such due to this
> > > problem.
> > >
> > > The kernel is 2.6.24.2-SMP/x86_32 (PREEMPT or not - NO difference).
> > >
> > > As far as this happens with 3 different NICs/drivers could it be
> > > a problem in the (common for all of them) networking subsystem?
> >
> > A TX timeout (like hardware timeouts, in general) is a very generic
> > behavior, with many causes.
> >
> > In general, when you see timeouts with varied hardware and drivers,
> > you're almost always dealing with a problem with interrupt delivery, or
> 
> All the drivers are using #INTA on PCI bus (no MSI/MSI-X).
> 
> "problem with interrupt delivery" - you suspect interrupts incorrectly
>  disabled (lost) in the drivers or faulty hardware(motherboard)?
> 
> > a generic system problem, rather than bugs in the network stack or all
> 
> "a generic system problem" - bad config or faulty hardware(motherboard)?
> 
> Where I should look for the problem?
> 
> Just for info: the system is very stable - uptime (if no power outages) could
> be a month or more (rebooting for kernel changes or updates).
> 
> Marin Mitov

Make sure the interrupt is showing up as level triggered in /proc/interrupts.
The BIOS may be configuring it as edge-triggered and that won't work with
Ethernet drivers that use NAPI.

Re: net: tx timeouts with skge, 8139too, dmfe drivers/NICs

[Marin Mitov]

Hi Stephen,

> Make sure the interrupt is showing up as level triggered in
> /proc/interrupts. The BIOS may be configuring it as edge-triggered and that
> won't work with Ethernet drivers that use NAPI.

for: skge <--> Marvell 88E8001 chip
cat /proc/interrupts gives (AMD64 X2 SMP):
           CPU0       CPU1
 21:   11691000   11933174   IO-APIC-fasteoi   eth0

It is neither IO-APIC-edge, nor IO-APIC-level.

Could it be the problem?

Marin Mitov

Re: net: tx timeouts with skge, 8139too, dmfe drivers/NICs

[Stephen Hemminger]

On Tue, 26 Feb 2008 00:09:46 +0200
Marin Mitov <mitov@issp.bas.bg> wrote:

> Hi Stephen,
> 
> > Make sure the interrupt is showing up as level triggered in
> > /proc/interrupts. The BIOS may be configuring it as edge-triggered and that
> > won't work with Ethernet drivers that use NAPI.
> 
> for: skge <--> Marvell 88E8001 chip
> cat /proc/interrupts gives (AMD64 X2 SMP):
>            CPU0       CPU1
>  21:   11691000   11933174   IO-APIC-fasteoi   eth0
> 
> It is neither IO-APIC-edge, nor IO-APIC-level.
> 
> Could it be the problem?
> 
> Marin Mitov

No. that isn't the problem.

Re: net: tx timeouts with skge, 8139too, dmfe drivers/NICs

[Marin Mitov]

On Monday 25 February 2008 10:53:01 pm you wrote:
> > As far as this happens with 3 different NICs/drivers could it be
> > a problem in the (common for all of them) networking subsystem?
> 
> A TX timeout (like hardware timeouts, in general) is a very generic 
> behavior, with many causes.
> 
> In general, when you see timeouts with varied hardware and drivers, 
> you're almost always dealing with a problem with interrupt delivery, or 
> a generic system problem, rather than bugs in the network stack or all 
> three drivers.

Well, this gave me a direction of research. 

Using printk in various parts of  skge driver, as well as modifying it to
collect different statistics (used via ethtool -S eth0), the following observations
had been made when it freezes:

1. interrupts are generated (status register shows there are pending
interrupts and they are NOT masked), but irq_handler is NOT invoked.

2. Looking on the cat /proc/interrups shows that when skge is working
both CPUs receive any IRQs. When skge freezes NO CPU receives skge's
interrupts, CPU[0] receives any others IRQs, but skge's, CPU[1] do not
receive any IRQ above the line (see bellow), but receives LOC: and RES:
below the line.
#cat /proc/interrups
           CPU0       CPU1
  0:         85          1   IO-APIC-edge      timer
  1:      34078          9   IO-APIC-edge      i8042
  6:          1          4   IO-APIC-edge      floppy
  7:        216          1   IO-APIC-edge      parport0
  8:          0          1   IO-APIC-edge      rtc
  9:          0          0   IO-APIC-fasteoi   acpi
 12:     893003    1390080   IO-APIC-edge      i8042
 14:      59682     286628   IO-APIC-edge      ide0
 15:    5458527         12   IO-APIC-edge      ide1
 16:   60547054          1   IO-APIC-fasteoi   mga@pci:0000:01:00.0
 17:    1634623     914447   IO-APIC-fasteoi   sata_via
 18:       7768          7   IO-APIC-fasteoi   sata_promise
 19:          0          0   IO-APIC-fasteoi   ehci_hcd:usb1, uhci_hcd:usb2, uhci_hcd:usb3, uhci_hcd:usb4, uhci_hcd:usb5
 20:     535380          1   IO-APIC-fasteoi   VIA8237
 21:   30780380   31448992   IO-APIC-fasteoi   eth0
---------line added by me----------------------------------
NMI:          0          0   Non-maskable interrupts
LOC:  154311126  154736178   Local timer interrupts
RES:    1325239    2423719   Rescheduling interrupts
CAL:      40893        456   function call interrupts
TLB:      52651      29184   TLB shootdowns
TRM:          0          0   Thermal event interrupts
SPU:          0          0   Spurious interrupts
ERR:          0
MIS:          0

That looks like IRQs are somehow disabled (at IO-APIC/LAPIC?)
at some priority and bellow.

Here is the place to say that after freezing, ifconfig down/up (+routing info)
does NOT solve the problem, while rmmod/modprobe the driver, makes it work 
again.

So, I moved the functions request_irq()/free_irq() from driver's probe()/release() 
methods to open()/stop() methods. Thus modified, when skge freezes, 
ifconfig down/up makes it work again (no need to rmmod/modprobe).

That makes me think that somehow skge's IRQ is disabled OUT of the driver
and free_irq()/request_irq() clears the problem. Am I wrong?

Could it be possible? How could this happen?

Any comments/suggestions/patches wellcome.

Regards

Marin Mitov