LKML Archive on lore.kernel.org
help / color / mirror / Atom feed
* netatalk slow after system upgrade (possibly kernel problem?)
@ 2008-01-25 11:55 Michael Monnerie
  2008-01-26  3:33 ` [Netatalk-admins] " Didier
  2008-01-27  6:00 ` Andrew Morton
  0 siblings, 2 replies; 6+ messages in thread
From: Michael Monnerie @ 2008-01-25 11:55 UTC (permalink / raw)
  To: netatalk-admins; +Cc: linux-kernel

[-- Attachment #1: Type: text/plain, Size: 2170 bytes --]

Dear lists,

I've been spending a LOT of time trying to find out where's the problem, 
but can't find it and therefore seek urgent help now. We have the 
following system:

Server with VMware server
-> VM running a webserver and netatalk
-> 2 other VMs not related

The VM with netatalk was SUSE 10.0 with kernel 2.6.13-15.15-smp (from 
SUSE), and things were pretty fun and quick. Then we upgraded to SUSE 
10.2 and now 10.3, where everything EXCEPT netatalk runs perfect. Since 
this upgrade, Apple clients (MacOS X) now do READ very very slowly 
(about 512KB/s over the gigabit LAN), while writing to the server still 
is normal (>20MB/s). I've even retried with the newest kernel 
2.6.23.13, tried different /proc/sys/net/ipv4/tcp_congestion_control 
(cubic, reno, bic, etc.) and nothing helps. I've then tried to install 
Samba and found that we have similar problems reading with it from 
MacOS clients. Now I'm pretty sure it should be something with the 
linux kernel, but I don't understand what.

Here are the wireshark dumps in pcap format:
http://zmi.at/x/atalk-write-fast.pcap
-> you can see writing to the server (192.168.120.9) is normal and fast

http://zmi.at/x/atalk-read-slow.pcap
-> reading is horribly slow. Lots of "unknown", because of netatalk or 
what?

http://zmi.at/x/unknown-atalk.pcap
-> another dump while reading, you see "unknown" reads. I'm not sure if 
it's just wireshark not understanding the packets or netatalk.

And trying with samba:
http://zmi.at/x/smb-read-slow.pcap
http://zmi.at/x/smb-write-quick.pcap
you can see that it's also slow.

Now why did it work with the old 2.6.13 kernel? I still have that old 
VM, and when I start it, it is always perfectly fast. Only newer 
versions are slow. Can somebody give me a hint please?

mfg zmi
-- 
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0676/846 914 666                      .network.your.ideas.
// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
// Keyserver: www.keyserver.net                   Key-ID: 1C1209B4

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 194 bytes --]

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [Netatalk-admins] netatalk slow after system upgrade (possibly kernel problem?)
  2008-01-25 11:55 netatalk slow after system upgrade (possibly kernel problem?) Michael Monnerie
@ 2008-01-26  3:33 ` Didier
  2008-01-27  6:00 ` Andrew Morton
  1 sibling, 0 replies; 6+ messages in thread
From: Didier @ 2008-01-26  3:33 UTC (permalink / raw)
  To: Michael Monnerie, netatalk-admins; +Cc: linux-kernel

Hi,
On Fri, 25 Jan 2008 12:55:42 +0100, Michael Monnerie wrote
> Dear lists,
> 
> I've been spending a LOT of time trying to find out where's the 
> problem, but can't find it and therefore seek urgent help now. We 
> have the following system:
Did you try to force the server MTU to 1500 (it looks like you have jumbo
frame enabled) ?

Some interesting TCP/IP packets though :) 
 
OT as it was working before but if it's a simple LAN you may have a flaky/wire
in the loop too.

Didier


^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: netatalk slow after system upgrade (possibly kernel problem?)
  2008-01-25 11:55 netatalk slow after system upgrade (possibly kernel problem?) Michael Monnerie
  2008-01-26  3:33 ` [Netatalk-admins] " Didier
@ 2008-01-27  6:00 ` Andrew Morton
  2008-02-05 15:26   ` Michael Monnerie
  2008-03-12  2:26   ` Michael Monnerie
  1 sibling, 2 replies; 6+ messages in thread
From: Andrew Morton @ 2008-01-27  6:00 UTC (permalink / raw)
  To: Michael Monnerie; +Cc: netatalk-admins, linux-kernel, netdev


(cc netdev)

> On Fri, 25 Jan 2008 12:55:42 +0100 Michael Monnerie <michael.monnerie@it-management.at> wrote:
> Dear lists,
> 
> I've been spending a LOT of time trying to find out where's the problem, 
> but can't find it and therefore seek urgent help now. We have the 
> following system:
> 
> Server with VMware server
> -> VM running a webserver and netatalk
> -> 2 other VMs not related
> 
> The VM with netatalk was SUSE 10.0 with kernel 2.6.13-15.15-smp (from 
> SUSE), and things were pretty fun and quick. Then we upgraded to SUSE 
> 10.2 and now 10.3, where everything EXCEPT netatalk runs perfect. Since 
> this upgrade, Apple clients (MacOS X) now do READ very very slowly 
> (about 512KB/s over the gigabit LAN), while writing to the server still 
> is normal (>20MB/s). I've even retried with the newest kernel 
> 2.6.23.13, tried different /proc/sys/net/ipv4/tcp_congestion_control 
> (cubic, reno, bic, etc.) and nothing helps. I've then tried to install 
> Samba and found that we have similar problems reading with it from 
> MacOS clients. Now I'm pretty sure it should be something with the 
> linux kernel, but I don't understand what.
> 
> Here are the wireshark dumps in pcap format:
> http://zmi.at/x/atalk-write-fast.pcap
> -> you can see writing to the server (192.168.120.9) is normal and fast
> 
> http://zmi.at/x/atalk-read-slow.pcap
> -> reading is horribly slow. Lots of "unknown", because of netatalk or 
> what?
> 
> http://zmi.at/x/unknown-atalk.pcap
> -> another dump while reading, you see "unknown" reads. I'm not sure if 
> it's just wireshark not understanding the packets or netatalk.
> 
> And trying with samba:
> http://zmi.at/x/smb-read-slow.pcap
> http://zmi.at/x/smb-write-quick.pcap
> you can see that it's also slow.
> 
> Now why did it work with the old 2.6.13 kernel? I still have that old 
> VM, and when I start it, it is always perfectly fast. Only newer 
> versions are slow. Can somebody give me a hint please?

It would be interesting if this could be repeated on bare hardware, so we
can eliminate the possibility that it is some weird interaction with vmware.


^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: netatalk slow after system upgrade (possibly kernel problem?)
  2008-01-27  6:00 ` Andrew Morton
@ 2008-02-05 15:26   ` Michael Monnerie
       [not found]     ` <73F08C2B-C3F9-4846-962E-A3D2692804E0@supported.de>
  2008-03-12  2:26   ` Michael Monnerie
  1 sibling, 1 reply; 6+ messages in thread
From: Michael Monnerie @ 2008-02-05 15:26 UTC (permalink / raw)
  To: Andrew Morton; +Cc: netatalk-admins, linux-kernel, netdev

[-- Attachment #1: Type: text/plain, Size: 3792 bytes --]

(I let full quote here for reference, my text is on the bottom)

On Sonntag, 27. Januar 2008 Andrew Morton wrote:
> (cc netdev)
>
> > On Fri, 25 Jan 2008 12:55:42 +0100 Michael Monnerie
> > <michael.monnerie@it-management.at> wrote: Dear lists,
> >
> > I've been spending a LOT of time trying to find out where's the
> > problem, but can't find it and therefore seek urgent help now. We
> > have the following system:
> >
> > Server with VMware server
> > -> VM running a webserver and netatalk
> > -> 2 other VMs not related
> >
> > The VM with netatalk was SUSE 10.0 with kernel 2.6.13-15.15-smp
> > (from SUSE), and things were pretty fun and quick. Then we upgraded
> > to SUSE 10.2 and now 10.3, where everything EXCEPT netatalk runs
> > perfect. Since this upgrade, Apple clients (MacOS X) now do READ
> > very very slowly (about 512KB/s over the gigabit LAN), while
> > writing to the server still is normal (>20MB/s). I've even retried
> > with the newest kernel 2.6.23.13, tried different
> > /proc/sys/net/ipv4/tcp_congestion_control (cubic, reno, bic, etc.)
> > and nothing helps. I've then tried to install Samba and found that
> > we have similar problems reading with it from MacOS clients. Now
> > I'm pretty sure it should be something with the linux kernel, but I
> > don't understand what.
> >
> > Here are the wireshark dumps in pcap format:
> > http://zmi.at/x/atalk-write-fast.pcap
> > -> you can see writing to the server (192.168.120.9) is normal and
> > fast
> >
> > http://zmi.at/x/atalk-read-slow.pcap
> > -> reading is horribly slow. Lots of "unknown", because of netatalk
> > or what?
> >
> > http://zmi.at/x/unknown-atalk.pcap
> > -> another dump while reading, you see "unknown" reads. I'm not
> > sure if it's just wireshark not understanding the packets or
> > netatalk.
> >
> > And trying with samba:
> > http://zmi.at/x/smb-read-slow.pcap
> > http://zmi.at/x/smb-write-quick.pcap
> > you can see that it's also slow.
> >
> > Now why did it work with the old 2.6.13 kernel? I still have that
> > old VM, and when I start it, it is always perfectly fast. Only
> > newer versions are slow. Can somebody give me a hint please?
>
> It would be interesting if this could be repeated on bare hardware,
> so we can eliminate the possibility that it is some weird interaction
> with vmware.

I tested now on the same machine, outside of VMware: It is blazing fast, 
just as it should be - kernel release 2.6.18.8-0.5-ZMI, where -ZMI 
means I compiled the kernel myself. Still, it runs quick within the VM 
with kernel 2.6.13-15.15-smp from SUSE 10.0, but slow with more recent 
kernels (I couldn't test every combination of course). 

I tried but cannot compile 2.6.13 within SUSE 10.3, it gives lots of 
errors because libs changed.

It would be very nice to find the source of the problem. I uploaded the 
kernel config options files onto my server for reference, HTH:

VMware with SUSE 10.0 and fast netatalk/samba:
http://zmi.at/x/config2.6.13-15.15.txt

VMware host, outside of VMware, also fast:
http://zmi.at/x/config2.6.18.8-host.txt

VMware with SUSE 10.2 and kernel 2.6.22-9-ZMI, slow:
http://zmi.at/x/config2.6.22.9-slow.txt

VMware with SUSE 10.3 and kernel 2.6.23.13-ZMI, slow:
http://zmi.at/x/config2.6.23.13-slow.txt

The problem seems to be somewhere between the kernel and VMware, but I 
do not know what to do.

mfg zmi
-- 
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0676/846 914 666                      .network.your.ideas.
// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
// Keyserver: www.keyserver.net                   Key-ID: 1C1209B4

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 194 bytes --]

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [Netatalk-admins] netatalk slow after system upgrade (possibly kernel problem?)
       [not found]     ` <73F08C2B-C3F9-4846-962E-A3D2692804E0@supported.de>
@ 2008-02-07  0:49       ` Michael Monnerie
  0 siblings, 0 replies; 6+ messages in thread
From: Michael Monnerie @ 2008-02-07  0:49 UTC (permalink / raw)
  To: Ralph Böhme; +Cc: netatalk-admins, linux-kernel, netdev, Andrew Morton

[-- Attachment #1: Type: text/plain, Size: 1453 bytes --]

I received this e-mail, which could be the resolution to this problem. I 
will test it with the customer tomorrow.

Is there a workaround possible from the Linux server, instead of 
modifying all clients? As it runs with kernel 2.6.13, but not with 
2.6.23, maybe some behaviour change via sysctl could help here.

On Mittwoch, 6. Februar 2008 Ralph Böhme wrote:
> Am 05.02.2008 um 16:26 schrieb Michael Monnerie:
> > Still, it runs quick within the VM
> > with kernel 2.6.13-15.15-smp from SUSE 10.0, but slow with more
> > recent kernels (I couldn't test every combination of course).
>
> Use the sniffer, Luke!
>
> If you find an ongoing sequence of ~5 packets sent by the server,
> ~200ms pause, client ACK, next turn, then you might me experiencing
> this:
>
> http://www.helios.de/support/ti.phtml?110
>
> Proposed solution:
> sysctl -w net.inet.tcp.delayed_ack=2 to be set on the client side.
> Note: the issue should be resolved with OS X 10.5.
>
> It seems as if /some/ server side TCP stacks don't like OS X 10.4
> delayed ACKs implementation.
>
> HTH,
> -Ralph



-- 
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0676/846 914 666                      .network.your.ideas.
// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
// Keyserver: www.keyserver.net                   Key-ID: 1C1209B4

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 194 bytes --]

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: netatalk slow after system upgrade (possibly kernel problem?)
  2008-01-27  6:00 ` Andrew Morton
  2008-02-05 15:26   ` Michael Monnerie
@ 2008-03-12  2:26   ` Michael Monnerie
  1 sibling, 0 replies; 6+ messages in thread
From: Michael Monnerie @ 2008-03-12  2:26 UTC (permalink / raw)
  To: Andrew Morton; +Cc: netatalk-admins, linux-kernel, netdev

[-- Attachment #1: Type: text/plain, Size: 3400 bytes --]

(full quote below, because this message was in January)

I did some new tests, see http://zmi.at/x/wire.delayack2.pcap
Dump from SUSE10.3, Apple client had delayed ACK set to 2 with
sysctl -w net.inet.tcp.delayed_ack=2, and also MacOS10.5 has this 
problem. It was suggested I could have this problem:
http://www.helios.de/support/ti.phtml?110

On the same hardware, I can start the old SUSE10.0 at the same time in 
parallel with the new SUSE10.3, both are virtual machines in VMware, 
and the old machine is always fast and the new one always slow. From 
the dump I would say it's the fault of MacOS - sometimes a very long 
delay before an ACK. But then again, it's fast when copying to the old 
SUSE linux box. Very strange.

On Sonntag, 27. Januar 2008 Andrew Morton wrote:
> (cc netdev)
>
> > On Fri, 25 Jan 2008 12:55:42 +0100 Michael Monnerie
> > <michael.monnerie@it-management.at> wrote: Dear lists,
> >
> > I've been spending a LOT of time trying to find out where's the
> > problem, but can't find it and therefore seek urgent help now. We
> > have the following system:
> >
> > Server with VMware server
> > -> VM running a webserver and netatalk
> > -> 2 other VMs not related
> >
> > The VM with netatalk was SUSE 10.0 with kernel 2.6.13-15.15-smp
> > (from SUSE), and things were pretty fun and quick. Then we upgraded
> > to SUSE 10.2 and now 10.3, where everything EXCEPT netatalk runs
> > perfect. Since this upgrade, Apple clients (MacOS X) now do READ
> > very very slowly (about 512KB/s over the gigabit LAN), while
> > writing to the server still is normal (>20MB/s). I've even retried
> > with the newest kernel 2.6.23.13, tried different
> > /proc/sys/net/ipv4/tcp_congestion_control (cubic, reno, bic, etc.)
> > and nothing helps. I've then tried to install Samba and found that
> > we have similar problems reading with it from MacOS clients. Now
> > I'm pretty sure it should be something with the linux kernel, but I
> > don't understand what.
> >
> > Here are the wireshark dumps in pcap format:
> > http://zmi.at/x/atalk-write-fast.pcap
> > -> you can see writing to the server (192.168.120.9) is normal and
> > fast
> >
> > http://zmi.at/x/atalk-read-slow.pcap
> > -> reading is horribly slow. Lots of "unknown", because of netatalk
> > or what?
> >
> > http://zmi.at/x/unknown-atalk.pcap
> > -> another dump while reading, you see "unknown" reads. I'm not
> > sure if it's just wireshark not understanding the packets or
> > netatalk.
> >
> > And trying with samba:
> > http://zmi.at/x/smb-read-slow.pcap
> > http://zmi.at/x/smb-write-quick.pcap
> > you can see that it's also slow.
> >
> > Now why did it work with the old 2.6.13 kernel? I still have that
> > old VM, and when I start it, it is always perfectly fast. Only
> > newer versions are slow. Can somebody give me a hint please?
>
> It would be interesting if this could be repeated on bare hardware,
> so we can eliminate the possibility that it is some weird interaction
> with vmware.



-- 
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0676/846 914 666                      .network.your.ideas.
// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
// Keyserver: www.keyserver.net                   Key-ID: 1C1209B4

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 194 bytes --]

^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2008-03-12  2:27 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2008-01-25 11:55 netatalk slow after system upgrade (possibly kernel problem?) Michael Monnerie
2008-01-26  3:33 ` [Netatalk-admins] " Didier
2008-01-27  6:00 ` Andrew Morton
2008-02-05 15:26   ` Michael Monnerie
     [not found]     ` <73F08C2B-C3F9-4846-962E-A3D2692804E0@supported.de>
2008-02-07  0:49       ` [Netatalk-admins] " Michael Monnerie
2008-03-12  2:26   ` Michael Monnerie

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).