Ethernet over Kernel Sockets

Ethernet over Kernel Sockets

[Lukas Razik]

Hello all!

As you know, some network cards doesn't have a 'eth' interface under linux.
Because of that I'm developing a net_device based driver which doesn't 
transmit and receive directly through a real network card but through 
UDP kernel sockets.
That means:
If my net_device->hard_start_xmit function gets an packet to transmit 
(in interrupt context), there will be a work struct queued into a 
workqueue and the packet will be processed by the workqueue later on (in 
process context).
On the receiver side I have a kernel thread which blocks on 
sock_recvmsg() and if there comes a UDP message, then it will be 
processed and a sk_buff will be passed to the kernel.
The current state is that all works stable but I've bad transmission 
rates and bad ping times.

For example I've to systems and each has one Gigabit-Ethernet card:
System 1: ifconfig eth0 192.168.0.1
System 2: ifconfig eth0 192.168.0.2

If I load my driver then I get an additional eth interface (for example):
System 1: ifconfig eth1 192.168.1.1
System 2: ifconfig eth1 192.168.1.2

So the eth1 interfaces are based on sockets which use the eth0 (Gigabit) 
interfaces for the communication.
Now, if I measure the transmission between 192.168.0.1 and 192.168.0.2 
then I get transmission rates that are normal for Gigabit-Ethernet 
(~25µsec (PingPong) and ~900Mbit/s).
If I measure the transmission through the eth1 interfaces between 
192.168.1.1 and 192.168.1.2 then I only get ~1msec (PingPong) and only 
about ~400Mbit/s.

The interesting thing is, that I always get PingPongs of 1ms (= one 
jiffie). Although if I measure with 100MBit- or other non-Ethernet cards.
Maybe it's because I run the socket_recvmsg() function in a kernel 
thread which blocks on the function and must wait to be executed by the 
scheduler.

Now I don't know how I could solve this problem:
How can I force the kernel to process a received message immediately 
when it comes in through an UDP kernel socket?

I hope that someone of you can help me....
The source code you can find here:
http://net.razik.de/ethos.tar.gz

Regards and Many Thanks for any help!
Lukas

Re: Ethernet over Kernel Sockets

[Pavel Machek]

On Fri 2008-02-29 21:54:21, Lukas Razik wrote:
> Hello all!
> 
> As you know, some network cards doesn't have a 'eth' 
> interface under linux.
> Because of that I'm developing a net_device based driver 
> which doesn't transmit and receive directly through a 
> real network card but through UDP kernel sockets.

What is it good for?

> So the eth1 interfaces are based on sockets which use 
> the eth0 (Gigabit) interfaces for the communication.
> Now, if I measure the transmission between 192.168.0.1 
> and 192.168.0.2 then I get transmission rates that are 
> normal for Gigabit-Ethernet (~25?sec (PingPong) and 
> ~900Mbit/s).
> If I measure the transmission through the eth1 
> interfaces between 192.168.1.1 and 192.168.1.2 then I 
> only get ~1msec (PingPong) and only about ~400Mbit/s.
> 
> The interesting thing is, that I always get PingPongs of 
> 1ms (= one jiffie). Although if I measure with 100MBit- 
> or other non-Ethernet cards.

Try increasing HZ :-)... or maybe it is a gigabit interrupt mitigation
issue?

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: Ethernet over Kernel Sockets

[Lukas Razik]

Hello Pavel!

Thanks for your answer!

>>As you know, some network cards doesn't have a 'eth' 
>>interface under linux.
>>Because of that I'm developing a net_device based driver 
>>which doesn't transmit and receive directly through a 
>>real network card but through UDP kernel sockets.
> 
> What is it good for?

It's good for network cards which have a socket implementation but no 
'eth' interface which is needed by some applications...

>>The interesting thing is, that I always get PingPongs of 
>>1ms (= one jiffie). Although if I measure with 100MBit- 
>>or other non-Ethernet cards.
> 
> Try increasing HZ :-)... or maybe it is a gigabit interrupt mitigation
> issue?

Yes, that would have been a nice solution for an OS-freak (like me and 
sure like you) but there are also other people :-/ who want to run more 
than the really nice linux kernel on their machines... ;-)))

The performance brake was the kernel thread for receiving packets. My 
solution are socket callbacks.
The idea I've got from someone of the kernelnewbies mailing list and 
some other people. Now I've PingPong times of ~30µsec (with Ethernet 
over Sockets over Ethernet, which natively has ~23µsec).

For interested people:
http://mail.nl.linux.org/kernelnewbies/2008-03/msg00005.html

Regards,
Lukas