LKML Archive on lore.kernel.org help / color / mirror / Atom feed
* netatalk slow after system upgrade (possibly kernel problem?) @ 2008-01-25 11:55 Michael Monnerie 2008-01-26 3:33 ` [Netatalk-admins] " Didier 2008-01-27 6:00 ` Andrew Morton 0 siblings, 2 replies; 6+ messages in thread From: Michael Monnerie @ 2008-01-25 11:55 UTC (permalink / raw) To: netatalk-admins; +Cc: linux-kernel [-- Attachment #1: Type: text/plain, Size: 2170 bytes --] Dear lists, I've been spending a LOT of time trying to find out where's the problem, but can't find it and therefore seek urgent help now. We have the following system: Server with VMware server -> VM running a webserver and netatalk -> 2 other VMs not related The VM with netatalk was SUSE 10.0 with kernel 2.6.13-15.15-smp (from SUSE), and things were pretty fun and quick. Then we upgraded to SUSE 10.2 and now 10.3, where everything EXCEPT netatalk runs perfect. Since this upgrade, Apple clients (MacOS X) now do READ very very slowly (about 512KB/s over the gigabit LAN), while writing to the server still is normal (>20MB/s). I've even retried with the newest kernel 2.6.23.13, tried different /proc/sys/net/ipv4/tcp_congestion_control (cubic, reno, bic, etc.) and nothing helps. I've then tried to install Samba and found that we have similar problems reading with it from MacOS clients. Now I'm pretty sure it should be something with the linux kernel, but I don't understand what. Here are the wireshark dumps in pcap format: http://zmi.at/x/atalk-write-fast.pcap -> you can see writing to the server (192.168.120.9) is normal and fast http://zmi.at/x/atalk-read-slow.pcap -> reading is horribly slow. Lots of "unknown", because of netatalk or what? http://zmi.at/x/unknown-atalk.pcap -> another dump while reading, you see "unknown" reads. I'm not sure if it's just wireshark not understanding the packets or netatalk. And trying with samba: http://zmi.at/x/smb-read-slow.pcap http://zmi.at/x/smb-write-quick.pcap you can see that it's also slow. Now why did it work with the old 2.6.13 kernel? I still have that old VM, and when I start it, it is always perfectly fast. Only newer versions are slow. Can somebody give me a hint please? mfg zmi -- // Michael Monnerie, Ing.BSc ----- http://it-management.at // Tel: 0676/846 914 666 .network.your.ideas. // PGP Key: "curl -s http://zmi.at/zmi.asc | gpg --import" // Fingerprint: AC19 F9D5 36ED CD8A EF38 500E CE14 91F7 1C12 09B4 // Keyserver: www.keyserver.net Key-ID: 1C1209B4 [-- Attachment #2: This is a digitally signed message part. --] [-- Type: application/pgp-signature, Size: 194 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Netatalk-admins] netatalk slow after system upgrade (possibly kernel problem?) 2008-01-25 11:55 netatalk slow after system upgrade (possibly kernel problem?) Michael Monnerie @ 2008-01-26 3:33 ` Didier 2008-01-27 6:00 ` Andrew Morton 1 sibling, 0 replies; 6+ messages in thread From: Didier @ 2008-01-26 3:33 UTC (permalink / raw) To: Michael Monnerie, netatalk-admins; +Cc: linux-kernel Hi, On Fri, 25 Jan 2008 12:55:42 +0100, Michael Monnerie wrote > Dear lists, > > I've been spending a LOT of time trying to find out where's the > problem, but can't find it and therefore seek urgent help now. We > have the following system: Did you try to force the server MTU to 1500 (it looks like you have jumbo frame enabled) ? Some interesting TCP/IP packets though :) OT as it was working before but if it's a simple LAN you may have a flaky/wire in the loop too. Didier ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: netatalk slow after system upgrade (possibly kernel problem?) 2008-01-25 11:55 netatalk slow after system upgrade (possibly kernel problem?) Michael Monnerie 2008-01-26 3:33 ` [Netatalk-admins] " Didier @ 2008-01-27 6:00 ` Andrew Morton 2008-02-05 15:26 ` Michael Monnerie 2008-03-12 2:26 ` Michael Monnerie 1 sibling, 2 replies; 6+ messages in thread From: Andrew Morton @ 2008-01-27 6:00 UTC (permalink / raw) To: Michael Monnerie; +Cc: netatalk-admins, linux-kernel, netdev (cc netdev) > On Fri, 25 Jan 2008 12:55:42 +0100 Michael Monnerie <michael.monnerie@it-management.at> wrote: > Dear lists, > > I've been spending a LOT of time trying to find out where's the problem, > but can't find it and therefore seek urgent help now. We have the > following system: > > Server with VMware server > -> VM running a webserver and netatalk > -> 2 other VMs not related > > The VM with netatalk was SUSE 10.0 with kernel 2.6.13-15.15-smp (from > SUSE), and things were pretty fun and quick. Then we upgraded to SUSE > 10.2 and now 10.3, where everything EXCEPT netatalk runs perfect. Since > this upgrade, Apple clients (MacOS X) now do READ very very slowly > (about 512KB/s over the gigabit LAN), while writing to the server still > is normal (>20MB/s). I've even retried with the newest kernel > 2.6.23.13, tried different /proc/sys/net/ipv4/tcp_congestion_control > (cubic, reno, bic, etc.) and nothing helps. I've then tried to install > Samba and found that we have similar problems reading with it from > MacOS clients. Now I'm pretty sure it should be something with the > linux kernel, but I don't understand what. > > Here are the wireshark dumps in pcap format: > http://zmi.at/x/atalk-write-fast.pcap > -> you can see writing to the server (192.168.120.9) is normal and fast > > http://zmi.at/x/atalk-read-slow.pcap > -> reading is horribly slow. Lots of "unknown", because of netatalk or > what? > > http://zmi.at/x/unknown-atalk.pcap > -> another dump while reading, you see "unknown" reads. I'm not sure if > it's just wireshark not understanding the packets or netatalk. > > And trying with samba: > http://zmi.at/x/smb-read-slow.pcap > http://zmi.at/x/smb-write-quick.pcap > you can see that it's also slow. > > Now why did it work with the old 2.6.13 kernel? I still have that old > VM, and when I start it, it is always perfectly fast. Only newer > versions are slow. Can somebody give me a hint please? It would be interesting if this could be repeated on bare hardware, so we can eliminate the possibility that it is some weird interaction with vmware. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: netatalk slow after system upgrade (possibly kernel problem?) 2008-01-27 6:00 ` Andrew Morton @ 2008-02-05 15:26 ` Michael Monnerie [not found] ` <73F08C2B-C3F9-4846-962E-A3D2692804E0@supported.de> 2008-03-12 2:26 ` Michael Monnerie 1 sibling, 1 reply; 6+ messages in thread From: Michael Monnerie @ 2008-02-05 15:26 UTC (permalink / raw) To: Andrew Morton; +Cc: netatalk-admins, linux-kernel, netdev [-- Attachment #1: Type: text/plain, Size: 3792 bytes --] (I let full quote here for reference, my text is on the bottom) On Sonntag, 27. Januar 2008 Andrew Morton wrote: > (cc netdev) > > > On Fri, 25 Jan 2008 12:55:42 +0100 Michael Monnerie > > <michael.monnerie@it-management.at> wrote: Dear lists, > > > > I've been spending a LOT of time trying to find out where's the > > problem, but can't find it and therefore seek urgent help now. We > > have the following system: > > > > Server with VMware server > > -> VM running a webserver and netatalk > > -> 2 other VMs not related > > > > The VM with netatalk was SUSE 10.0 with kernel 2.6.13-15.15-smp > > (from SUSE), and things were pretty fun and quick. Then we upgraded > > to SUSE 10.2 and now 10.3, where everything EXCEPT netatalk runs > > perfect. Since this upgrade, Apple clients (MacOS X) now do READ > > very very slowly (about 512KB/s over the gigabit LAN), while > > writing to the server still is normal (>20MB/s). I've even retried > > with the newest kernel 2.6.23.13, tried different > > /proc/sys/net/ipv4/tcp_congestion_control (cubic, reno, bic, etc.) > > and nothing helps. I've then tried to install Samba and found that > > we have similar problems reading with it from MacOS clients. Now > > I'm pretty sure it should be something with the linux kernel, but I > > don't understand what. > > > > Here are the wireshark dumps in pcap format: > > http://zmi.at/x/atalk-write-fast.pcap > > -> you can see writing to the server (192.168.120.9) is normal and > > fast > > > > http://zmi.at/x/atalk-read-slow.pcap > > -> reading is horribly slow. Lots of "unknown", because of netatalk > > or what? > > > > http://zmi.at/x/unknown-atalk.pcap > > -> another dump while reading, you see "unknown" reads. I'm not > > sure if it's just wireshark not understanding the packets or > > netatalk. > > > > And trying with samba: > > http://zmi.at/x/smb-read-slow.pcap > > http://zmi.at/x/smb-write-quick.pcap > > you can see that it's also slow. > > > > Now why did it work with the old 2.6.13 kernel? I still have that > > old VM, and when I start it, it is always perfectly fast. Only > > newer versions are slow. Can somebody give me a hint please? > > It would be interesting if this could be repeated on bare hardware, > so we can eliminate the possibility that it is some weird interaction > with vmware. I tested now on the same machine, outside of VMware: It is blazing fast, just as it should be - kernel release 2.6.18.8-0.5-ZMI, where -ZMI means I compiled the kernel myself. Still, it runs quick within the VM with kernel 2.6.13-15.15-smp from SUSE 10.0, but slow with more recent kernels (I couldn't test every combination of course). I tried but cannot compile 2.6.13 within SUSE 10.3, it gives lots of errors because libs changed. It would be very nice to find the source of the problem. I uploaded the kernel config options files onto my server for reference, HTH: VMware with SUSE 10.0 and fast netatalk/samba: http://zmi.at/x/config2.6.13-15.15.txt VMware host, outside of VMware, also fast: http://zmi.at/x/config2.6.18.8-host.txt VMware with SUSE 10.2 and kernel 2.6.22-9-ZMI, slow: http://zmi.at/x/config2.6.22.9-slow.txt VMware with SUSE 10.3 and kernel 2.6.23.13-ZMI, slow: http://zmi.at/x/config2.6.23.13-slow.txt The problem seems to be somewhere between the kernel and VMware, but I do not know what to do. mfg zmi -- // Michael Monnerie, Ing.BSc ----- http://it-management.at // Tel: 0676/846 914 666 .network.your.ideas. // PGP Key: "curl -s http://zmi.at/zmi.asc | gpg --import" // Fingerprint: AC19 F9D5 36ED CD8A EF38 500E CE14 91F7 1C12 09B4 // Keyserver: www.keyserver.net Key-ID: 1C1209B4 [-- Attachment #2: This is a digitally signed message part. --] [-- Type: application/pgp-signature, Size: 194 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
[parent not found: <73F08C2B-C3F9-4846-962E-A3D2692804E0@supported.de>]
* Re: [Netatalk-admins] netatalk slow after system upgrade (possibly kernel problem?) [not found] ` <73F08C2B-C3F9-4846-962E-A3D2692804E0@supported.de> @ 2008-02-07 0:49 ` Michael Monnerie 0 siblings, 0 replies; 6+ messages in thread From: Michael Monnerie @ 2008-02-07 0:49 UTC (permalink / raw) To: Ralph Böhme; +Cc: netatalk-admins, linux-kernel, netdev, Andrew Morton [-- Attachment #1: Type: text/plain, Size: 1453 bytes --] I received this e-mail, which could be the resolution to this problem. I will test it with the customer tomorrow. Is there a workaround possible from the Linux server, instead of modifying all clients? As it runs with kernel 2.6.13, but not with 2.6.23, maybe some behaviour change via sysctl could help here. On Mittwoch, 6. Februar 2008 Ralph Böhme wrote: > Am 05.02.2008 um 16:26 schrieb Michael Monnerie: > > Still, it runs quick within the VM > > with kernel 2.6.13-15.15-smp from SUSE 10.0, but slow with more > > recent kernels (I couldn't test every combination of course). > > Use the sniffer, Luke! > > If you find an ongoing sequence of ~5 packets sent by the server, > ~200ms pause, client ACK, next turn, then you might me experiencing > this: > > http://www.helios.de/support/ti.phtml?110 > > Proposed solution: > sysctl -w net.inet.tcp.delayed_ack=2 to be set on the client side. > Note: the issue should be resolved with OS X 10.5. > > It seems as if /some/ server side TCP stacks don't like OS X 10.4 > delayed ACKs implementation. > > HTH, > -Ralph -- // Michael Monnerie, Ing.BSc ----- http://it-management.at // Tel: 0676/846 914 666 .network.your.ideas. // PGP Key: "curl -s http://zmi.at/zmi.asc | gpg --import" // Fingerprint: AC19 F9D5 36ED CD8A EF38 500E CE14 91F7 1C12 09B4 // Keyserver: www.keyserver.net Key-ID: 1C1209B4 [-- Attachment #2: This is a digitally signed message part. --] [-- Type: application/pgp-signature, Size: 194 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: netatalk slow after system upgrade (possibly kernel problem?) 2008-01-27 6:00 ` Andrew Morton 2008-02-05 15:26 ` Michael Monnerie @ 2008-03-12 2:26 ` Michael Monnerie 1 sibling, 0 replies; 6+ messages in thread From: Michael Monnerie @ 2008-03-12 2:26 UTC (permalink / raw) To: Andrew Morton; +Cc: netatalk-admins, linux-kernel, netdev [-- Attachment #1: Type: text/plain, Size: 3400 bytes --] (full quote below, because this message was in January) I did some new tests, see http://zmi.at/x/wire.delayack2.pcap Dump from SUSE10.3, Apple client had delayed ACK set to 2 with sysctl -w net.inet.tcp.delayed_ack=2, and also MacOS10.5 has this problem. It was suggested I could have this problem: http://www.helios.de/support/ti.phtml?110 On the same hardware, I can start the old SUSE10.0 at the same time in parallel with the new SUSE10.3, both are virtual machines in VMware, and the old machine is always fast and the new one always slow. From the dump I would say it's the fault of MacOS - sometimes a very long delay before an ACK. But then again, it's fast when copying to the old SUSE linux box. Very strange. On Sonntag, 27. Januar 2008 Andrew Morton wrote: > (cc netdev) > > > On Fri, 25 Jan 2008 12:55:42 +0100 Michael Monnerie > > <michael.monnerie@it-management.at> wrote: Dear lists, > > > > I've been spending a LOT of time trying to find out where's the > > problem, but can't find it and therefore seek urgent help now. We > > have the following system: > > > > Server with VMware server > > -> VM running a webserver and netatalk > > -> 2 other VMs not related > > > > The VM with netatalk was SUSE 10.0 with kernel 2.6.13-15.15-smp > > (from SUSE), and things were pretty fun and quick. Then we upgraded > > to SUSE 10.2 and now 10.3, where everything EXCEPT netatalk runs > > perfect. Since this upgrade, Apple clients (MacOS X) now do READ > > very very slowly (about 512KB/s over the gigabit LAN), while > > writing to the server still is normal (>20MB/s). I've even retried > > with the newest kernel 2.6.23.13, tried different > > /proc/sys/net/ipv4/tcp_congestion_control (cubic, reno, bic, etc.) > > and nothing helps. I've then tried to install Samba and found that > > we have similar problems reading with it from MacOS clients. Now > > I'm pretty sure it should be something with the linux kernel, but I > > don't understand what. > > > > Here are the wireshark dumps in pcap format: > > http://zmi.at/x/atalk-write-fast.pcap > > -> you can see writing to the server (192.168.120.9) is normal and > > fast > > > > http://zmi.at/x/atalk-read-slow.pcap > > -> reading is horribly slow. Lots of "unknown", because of netatalk > > or what? > > > > http://zmi.at/x/unknown-atalk.pcap > > -> another dump while reading, you see "unknown" reads. I'm not > > sure if it's just wireshark not understanding the packets or > > netatalk. > > > > And trying with samba: > > http://zmi.at/x/smb-read-slow.pcap > > http://zmi.at/x/smb-write-quick.pcap > > you can see that it's also slow. > > > > Now why did it work with the old 2.6.13 kernel? I still have that > > old VM, and when I start it, it is always perfectly fast. Only > > newer versions are slow. Can somebody give me a hint please? > > It would be interesting if this could be repeated on bare hardware, > so we can eliminate the possibility that it is some weird interaction > with vmware. -- // Michael Monnerie, Ing.BSc ----- http://it-management.at // Tel: 0676/846 914 666 .network.your.ideas. // PGP Key: "curl -s http://zmi.at/zmi.asc | gpg --import" // Fingerprint: AC19 F9D5 36ED CD8A EF38 500E CE14 91F7 1C12 09B4 // Keyserver: www.keyserver.net Key-ID: 1C1209B4 [-- Attachment #2: This is a digitally signed message part. --] [-- Type: application/pgp-signature, Size: 194 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2008-03-12 2:27 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2008-01-25 11:55 netatalk slow after system upgrade (possibly kernel problem?) Michael Monnerie 2008-01-26 3:33 ` [Netatalk-admins] " Didier 2008-01-27 6:00 ` Andrew Morton 2008-02-05 15:26 ` Michael Monnerie [not found] ` <73F08C2B-C3F9-4846-962E-A3D2692804E0@supported.de> 2008-02-07 0:49 ` [Netatalk-admins] " Michael Monnerie 2008-03-12 2:26 ` Michael Monnerie
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).