LKML Archive on lore.kernel.org
help / color / mirror / Atom feed
* Re: netatalk slow after system upgrade (possibly kernel problem?)
@ 2008-01-27 13:34 Rachel Greenham
  0 siblings, 0 replies; 5+ messages in thread
From: Rachel Greenham @ 2008-01-27 13:34 UTC (permalink / raw)
  To: linux-kernel

(this thread came up in searching about the problem myself, so joined  
to post this, hence the 'broken' threading, I expect:)

Andrew Morton asked:

> It would be interesting if this could be repeated on bare hardware,  
> so we
> can eliminate the possibility that it is some weird interaction with  
> vmware.

I can confirm this; I'm seeing exactly these symptoms where the linux  
box running netatalk is running natively, not in VMWare. VMWare server  
is *installed* but only running in the sense that the vmnet interfaces  
are active.

Which begs an obvious test... There's no change when I kill the vmware  
networking (and all other) services. It still reckons it's going to  
take 24 minutes to transfer a file a little under a gigabyte.

Although, in fact, this thread does present a likely temporary  
workaround; to install a linux with an older kernel into vmware and  
use that to serve my content. :-D

That Linux system is Ubuntu Gutsy all up to date on amd64.

rachel@mab:~$ uname -a
Linux mab.local 2.6.22-14-generic #1 SMP Tue Dec 18 05:28:27 UTC 2007  
x86_64 GNU/Linux

netatalk 2.0.3-6ubuntu1 (built with ssl)

I'm getting between 500KB/s and 1MB/s on copies from that machine to a  
couple of macs on the local (gigabit) LAN. (They're both Intel macs  
running Leopard all up to date.)

Conversely I'm getting between 20MB/s and 40MB/s on copies from the  
macs to the netatalk server. (The variance is between the macs; one of  
them seems to be copying about twice as fast as the other probably due  
to issues of their own, but both clearly show that reading from  
netatalk is heavily crippled.

NB: I also posted this as a bug on ubuntu launchpad here:

https://bugs.launchpad.net/ubuntu/+source/netatalk/+bug/186327

But I noticed your precise question and thought I'd answer it. Don't  
know how useful I can be in here but I can certainly respond to  
requests for further info.

-- 
Rachel


^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: netatalk slow after system upgrade (possibly kernel problem?)
  2008-01-27  6:00 ` Andrew Morton
  2008-02-05 15:26   ` Michael Monnerie
@ 2008-03-12  2:26   ` Michael Monnerie
  1 sibling, 0 replies; 5+ messages in thread
From: Michael Monnerie @ 2008-03-12  2:26 UTC (permalink / raw)
  To: Andrew Morton; +Cc: netatalk-admins, linux-kernel, netdev

[-- Attachment #1: Type: text/plain, Size: 3400 bytes --]

(full quote below, because this message was in January)

I did some new tests, see http://zmi.at/x/wire.delayack2.pcap
Dump from SUSE10.3, Apple client had delayed ACK set to 2 with
sysctl -w net.inet.tcp.delayed_ack=2, and also MacOS10.5 has this 
problem. It was suggested I could have this problem:
http://www.helios.de/support/ti.phtml?110

On the same hardware, I can start the old SUSE10.0 at the same time in 
parallel with the new SUSE10.3, both are virtual machines in VMware, 
and the old machine is always fast and the new one always slow. From 
the dump I would say it's the fault of MacOS - sometimes a very long 
delay before an ACK. But then again, it's fast when copying to the old 
SUSE linux box. Very strange.

On Sonntag, 27. Januar 2008 Andrew Morton wrote:
> (cc netdev)
>
> > On Fri, 25 Jan 2008 12:55:42 +0100 Michael Monnerie
> > <michael.monnerie@it-management.at> wrote: Dear lists,
> >
> > I've been spending a LOT of time trying to find out where's the
> > problem, but can't find it and therefore seek urgent help now. We
> > have the following system:
> >
> > Server with VMware server
> > -> VM running a webserver and netatalk
> > -> 2 other VMs not related
> >
> > The VM with netatalk was SUSE 10.0 with kernel 2.6.13-15.15-smp
> > (from SUSE), and things were pretty fun and quick. Then we upgraded
> > to SUSE 10.2 and now 10.3, where everything EXCEPT netatalk runs
> > perfect. Since this upgrade, Apple clients (MacOS X) now do READ
> > very very slowly (about 512KB/s over the gigabit LAN), while
> > writing to the server still is normal (>20MB/s). I've even retried
> > with the newest kernel 2.6.23.13, tried different
> > /proc/sys/net/ipv4/tcp_congestion_control (cubic, reno, bic, etc.)
> > and nothing helps. I've then tried to install Samba and found that
> > we have similar problems reading with it from MacOS clients. Now
> > I'm pretty sure it should be something with the linux kernel, but I
> > don't understand what.
> >
> > Here are the wireshark dumps in pcap format:
> > http://zmi.at/x/atalk-write-fast.pcap
> > -> you can see writing to the server (192.168.120.9) is normal and
> > fast
> >
> > http://zmi.at/x/atalk-read-slow.pcap
> > -> reading is horribly slow. Lots of "unknown", because of netatalk
> > or what?
> >
> > http://zmi.at/x/unknown-atalk.pcap
> > -> another dump while reading, you see "unknown" reads. I'm not
> > sure if it's just wireshark not understanding the packets or
> > netatalk.
> >
> > And trying with samba:
> > http://zmi.at/x/smb-read-slow.pcap
> > http://zmi.at/x/smb-write-quick.pcap
> > you can see that it's also slow.
> >
> > Now why did it work with the old 2.6.13 kernel? I still have that
> > old VM, and when I start it, it is always perfectly fast. Only
> > newer versions are slow. Can somebody give me a hint please?
>
> It would be interesting if this could be repeated on bare hardware,
> so we can eliminate the possibility that it is some weird interaction
> with vmware.



-- 
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0676/846 914 666                      .network.your.ideas.
// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
// Keyserver: www.keyserver.net                   Key-ID: 1C1209B4

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 194 bytes --]

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: netatalk slow after system upgrade (possibly kernel problem?)
  2008-01-27  6:00 ` Andrew Morton
@ 2008-02-05 15:26   ` Michael Monnerie
  2008-03-12  2:26   ` Michael Monnerie
  1 sibling, 0 replies; 5+ messages in thread
From: Michael Monnerie @ 2008-02-05 15:26 UTC (permalink / raw)
  To: Andrew Morton; +Cc: netatalk-admins, linux-kernel, netdev

[-- Attachment #1: Type: text/plain, Size: 3792 bytes --]

(I let full quote here for reference, my text is on the bottom)

On Sonntag, 27. Januar 2008 Andrew Morton wrote:
> (cc netdev)
>
> > On Fri, 25 Jan 2008 12:55:42 +0100 Michael Monnerie
> > <michael.monnerie@it-management.at> wrote: Dear lists,
> >
> > I've been spending a LOT of time trying to find out where's the
> > problem, but can't find it and therefore seek urgent help now. We
> > have the following system:
> >
> > Server with VMware server
> > -> VM running a webserver and netatalk
> > -> 2 other VMs not related
> >
> > The VM with netatalk was SUSE 10.0 with kernel 2.6.13-15.15-smp
> > (from SUSE), and things were pretty fun and quick. Then we upgraded
> > to SUSE 10.2 and now 10.3, where everything EXCEPT netatalk runs
> > perfect. Since this upgrade, Apple clients (MacOS X) now do READ
> > very very slowly (about 512KB/s over the gigabit LAN), while
> > writing to the server still is normal (>20MB/s). I've even retried
> > with the newest kernel 2.6.23.13, tried different
> > /proc/sys/net/ipv4/tcp_congestion_control (cubic, reno, bic, etc.)
> > and nothing helps. I've then tried to install Samba and found that
> > we have similar problems reading with it from MacOS clients. Now
> > I'm pretty sure it should be something with the linux kernel, but I
> > don't understand what.
> >
> > Here are the wireshark dumps in pcap format:
> > http://zmi.at/x/atalk-write-fast.pcap
> > -> you can see writing to the server (192.168.120.9) is normal and
> > fast
> >
> > http://zmi.at/x/atalk-read-slow.pcap
> > -> reading is horribly slow. Lots of "unknown", because of netatalk
> > or what?
> >
> > http://zmi.at/x/unknown-atalk.pcap
> > -> another dump while reading, you see "unknown" reads. I'm not
> > sure if it's just wireshark not understanding the packets or
> > netatalk.
> >
> > And trying with samba:
> > http://zmi.at/x/smb-read-slow.pcap
> > http://zmi.at/x/smb-write-quick.pcap
> > you can see that it's also slow.
> >
> > Now why did it work with the old 2.6.13 kernel? I still have that
> > old VM, and when I start it, it is always perfectly fast. Only
> > newer versions are slow. Can somebody give me a hint please?
>
> It would be interesting if this could be repeated on bare hardware,
> so we can eliminate the possibility that it is some weird interaction
> with vmware.

I tested now on the same machine, outside of VMware: It is blazing fast, 
just as it should be - kernel release 2.6.18.8-0.5-ZMI, where -ZMI 
means I compiled the kernel myself. Still, it runs quick within the VM 
with kernel 2.6.13-15.15-smp from SUSE 10.0, but slow with more recent 
kernels (I couldn't test every combination of course). 

I tried but cannot compile 2.6.13 within SUSE 10.3, it gives lots of 
errors because libs changed.

It would be very nice to find the source of the problem. I uploaded the 
kernel config options files onto my server for reference, HTH:

VMware with SUSE 10.0 and fast netatalk/samba:
http://zmi.at/x/config2.6.13-15.15.txt

VMware host, outside of VMware, also fast:
http://zmi.at/x/config2.6.18.8-host.txt

VMware with SUSE 10.2 and kernel 2.6.22-9-ZMI, slow:
http://zmi.at/x/config2.6.22.9-slow.txt

VMware with SUSE 10.3 and kernel 2.6.23.13-ZMI, slow:
http://zmi.at/x/config2.6.23.13-slow.txt

The problem seems to be somewhere between the kernel and VMware, but I 
do not know what to do.

mfg zmi
-- 
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0676/846 914 666                      .network.your.ideas.
// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
// Keyserver: www.keyserver.net                   Key-ID: 1C1209B4

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 194 bytes --]

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: netatalk slow after system upgrade (possibly kernel problem?)
  2008-01-25 11:55 Michael Monnerie
@ 2008-01-27  6:00 ` Andrew Morton
  2008-02-05 15:26   ` Michael Monnerie
  2008-03-12  2:26   ` Michael Monnerie
  0 siblings, 2 replies; 5+ messages in thread
From: Andrew Morton @ 2008-01-27  6:00 UTC (permalink / raw)
  To: Michael Monnerie; +Cc: netatalk-admins, linux-kernel, netdev


(cc netdev)

> On Fri, 25 Jan 2008 12:55:42 +0100 Michael Monnerie <michael.monnerie@it-management.at> wrote:
> Dear lists,
> 
> I've been spending a LOT of time trying to find out where's the problem, 
> but can't find it and therefore seek urgent help now. We have the 
> following system:
> 
> Server with VMware server
> -> VM running a webserver and netatalk
> -> 2 other VMs not related
> 
> The VM with netatalk was SUSE 10.0 with kernel 2.6.13-15.15-smp (from 
> SUSE), and things were pretty fun and quick. Then we upgraded to SUSE 
> 10.2 and now 10.3, where everything EXCEPT netatalk runs perfect. Since 
> this upgrade, Apple clients (MacOS X) now do READ very very slowly 
> (about 512KB/s over the gigabit LAN), while writing to the server still 
> is normal (>20MB/s). I've even retried with the newest kernel 
> 2.6.23.13, tried different /proc/sys/net/ipv4/tcp_congestion_control 
> (cubic, reno, bic, etc.) and nothing helps. I've then tried to install 
> Samba and found that we have similar problems reading with it from 
> MacOS clients. Now I'm pretty sure it should be something with the 
> linux kernel, but I don't understand what.
> 
> Here are the wireshark dumps in pcap format:
> http://zmi.at/x/atalk-write-fast.pcap
> -> you can see writing to the server (192.168.120.9) is normal and fast
> 
> http://zmi.at/x/atalk-read-slow.pcap
> -> reading is horribly slow. Lots of "unknown", because of netatalk or 
> what?
> 
> http://zmi.at/x/unknown-atalk.pcap
> -> another dump while reading, you see "unknown" reads. I'm not sure if 
> it's just wireshark not understanding the packets or netatalk.
> 
> And trying with samba:
> http://zmi.at/x/smb-read-slow.pcap
> http://zmi.at/x/smb-write-quick.pcap
> you can see that it's also slow.
> 
> Now why did it work with the old 2.6.13 kernel? I still have that old 
> VM, and when I start it, it is always perfectly fast. Only newer 
> versions are slow. Can somebody give me a hint please?

It would be interesting if this could be repeated on bare hardware, so we
can eliminate the possibility that it is some weird interaction with vmware.


^ permalink raw reply	[flat|nested] 5+ messages in thread

* netatalk slow after system upgrade (possibly kernel problem?)
@ 2008-01-25 11:55 Michael Monnerie
  2008-01-27  6:00 ` Andrew Morton
  0 siblings, 1 reply; 5+ messages in thread
From: Michael Monnerie @ 2008-01-25 11:55 UTC (permalink / raw)
  To: netatalk-admins; +Cc: linux-kernel

[-- Attachment #1: Type: text/plain, Size: 2170 bytes --]

Dear lists,

I've been spending a LOT of time trying to find out where's the problem, 
but can't find it and therefore seek urgent help now. We have the 
following system:

Server with VMware server
-> VM running a webserver and netatalk
-> 2 other VMs not related

The VM with netatalk was SUSE 10.0 with kernel 2.6.13-15.15-smp (from 
SUSE), and things were pretty fun and quick. Then we upgraded to SUSE 
10.2 and now 10.3, where everything EXCEPT netatalk runs perfect. Since 
this upgrade, Apple clients (MacOS X) now do READ very very slowly 
(about 512KB/s over the gigabit LAN), while writing to the server still 
is normal (>20MB/s). I've even retried with the newest kernel 
2.6.23.13, tried different /proc/sys/net/ipv4/tcp_congestion_control 
(cubic, reno, bic, etc.) and nothing helps. I've then tried to install 
Samba and found that we have similar problems reading with it from 
MacOS clients. Now I'm pretty sure it should be something with the 
linux kernel, but I don't understand what.

Here are the wireshark dumps in pcap format:
http://zmi.at/x/atalk-write-fast.pcap
-> you can see writing to the server (192.168.120.9) is normal and fast

http://zmi.at/x/atalk-read-slow.pcap
-> reading is horribly slow. Lots of "unknown", because of netatalk or 
what?

http://zmi.at/x/unknown-atalk.pcap
-> another dump while reading, you see "unknown" reads. I'm not sure if 
it's just wireshark not understanding the packets or netatalk.

And trying with samba:
http://zmi.at/x/smb-read-slow.pcap
http://zmi.at/x/smb-write-quick.pcap
you can see that it's also slow.

Now why did it work with the old 2.6.13 kernel? I still have that old 
VM, and when I start it, it is always perfectly fast. Only newer 
versions are slow. Can somebody give me a hint please?

mfg zmi
-- 
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0676/846 914 666                      .network.your.ideas.
// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
// Keyserver: www.keyserver.net                   Key-ID: 1C1209B4

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 194 bytes --]

^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2008-03-12  2:27 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2008-01-27 13:34 netatalk slow after system upgrade (possibly kernel problem?) Rachel Greenham
  -- strict thread matches above, loose matches on Subject: below --
2008-01-25 11:55 Michael Monnerie
2008-01-27  6:00 ` Andrew Morton
2008-02-05 15:26   ` Michael Monnerie
2008-03-12  2:26   ` Michael Monnerie

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).