ncpfs and TCP vs UDP

ncpfs and TCP vs UDP

[Pierre Ossman]

Hi Petr,

I was hoping you could give me some input on another concern I have.
Which of TCP and UDP is the preferred transport for NCP? The client for
Windows seems to use TCP, which would suggest that that is the most
tested dialect. I also did a quick test with bonnie++:

UDP:

Version  1.03 -----Sequential Output------ --Sequential Input- --Random-
             -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
        500M  1569  19  1714  10   826   9  1414  18  1569   9  65.2   3
             ------Sequential Create------ --------Random Create--------
             -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
       files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
          16   460  14   897  16   556  15   491  15  1131  13   320   6
poseidon.drzeus.cx,500M,1569,19,1714,10,826,9,1414,18,1569,9,65.2,3,16,460,14,897,16,556,15,491,15,1131,13,320,6

TCP:

Version  1.03 -----Sequential Output------ --Sequential Input- --Random-
             -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
        500M  1653  19  3711  11  2312   9  1419  18  4489   6  91.5   1
             ------Sequential Create------ --------Random Create--------
             -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
       files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
          16   483  14   912  15   587  14   483  14  1176  13   287   6
poseidon.drzeus.cx,500M,1653,19,3711,11,2312,9,1419,18,4489,6,91.5,1,16,483,14,912,15,587,14,483,14,1176,13,287,6

TCP has roughly twice the throughput of UDP. Note that this was done
against a vmware machine that wasn't exactly maxing out the network.

Input welcome.

Rgds
-- 
     -- Pierre Ossman

  Linux kernel, MMC maintainer        http://www.kernel.org
  PulseAudio, core developer          http://pulseaudio.org
  rdesktop, core developer          http://www.rdesktop.org

Re: ncpfs and TCP vs UDP

[vandrove]

Quoting Pierre Ossman <drzeus-list@drzeus.cx>:

> Hi Petr,

Hello,

> I was hoping you could give me some input on another concern I have.
> Which of TCP and UDP is the preferred transport for NCP? The client for
> Windows seems to use TCP, which would suggest that that is the most
> tested dialect. I also did a quick test with bonnie++:

TCP is definitely preferred.  There are couple of reasons why you should prefer TCP:

(1) There is server configuration option to disable NCP/UDP.  You cannot disable
NCP/TCP that easily.

(2) TCP (NCP over TCP) retransmits only missing data, and it can ask for
retransmit much sooner as it knows what link latency is.  NCP/UDP can only ask
for complete packet retransmission, and it has no good idea what's link latency
because there is no ACK from server when it receives request - you can only
resend after usual link latency + time for process request, so you'll wait
longer for retransmit, and on retransmit you need to send again complete request
(which can be 64KB of data if you use 64KB buffer size...)

(3) To avoid problems with retransmits ncpfs uses default buffer size 60KB for
TCP (SOCK_STREAM), while 1KB for UDP/IPX (it must be multiple of sector size, so
using 1.4KB is not an option).  So if you read 1 page, you get 1 request/reply
when using TCP, but 4 requests/replies in UDP/IPX.  And as all this is fully
synchronous, and for today's link latency is dominating factor, it will take 4
times longer...

(4) Theoretically with TCP you should never need retransmits as TCP takes care
of that.  Unfortunately due to server implementation you still cannot have more
than one request in flight (at least with NW5 - I never tried it with NW6 as
spec says I should not do that, and NW5 implementation agreed with spec).

> Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
>         500M  1569  19  1714  10   826   9  1414  18  1569   9  65.2   3
>              ------Sequential Create------ --------Random Create--------
>              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
>        files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
>           16   460  14   897  16   556  15   491  15  1131  13   320   6

What was 'Block' size ?  For 1KB block size you should get more or less same
speed for TCP and UDP.  If you have link with big latency and big throughput,
then for 2KB block TCP should be 2 times faster, for 60KB block 60 times faster,
and above 60KB block difference should stay on 1:60.

You can try bumping NCP/UDP block size to 60KB as well, but my exprience with
switches and NetWare is that in such cases you can get into situation with
deterministic packet loss - like that from 40 UDP packets sent back to back to
server every 12th gets lost - and if you resend, again 12th packet gets lost,
again and again until you have some luck, or until client decides that server is
dead.
                                                          Petr

Re: ncpfs and TCP vs UDP

[Jan Engelhardt]

On Jan 26 2007 22:20, vandrove@vc.cvut.cz wrote:
>Quoting Pierre Ossman <drzeus-list@drzeus.cx>:
>
>TCP is definitely preferred.  There are couple of reasons why you should
>prefer TCP:
>
>(1) There is server configuration option to disable NCP/UDP.  You cannot
>disable NCP/TCP that easily.
>
>(2) TCP (NCP over TCP) retransmits only missing data, and it can ask for
>retransmit much sooner as it knows what link latency is.  NCP/UDP can only ask
>for complete packet retransmission, and it has no good idea what's link
>latency because there is no ACK from server when it receives request - you can
>only resend after usual link latency + time for process request, so you'll
>wait longer for retransmit, and on retransmit you need to send again complete
>request (which can be 64KB of data if you use 64KB buffer size...)
>
>(3) To avoid problems with retransmits ncpfs uses default buffer size 60KB for
>TCP (SOCK_STREAM), while 1KB for UDP/IPX (it must be multiple of sector size,
>so using 1.4KB is not an option).  So if you read 1 page, you get 1
>request/reply when using TCP, but 4 requests/replies in UDP/IPX.  And as all
>this is fully synchronous, and for today's link latency is dominating factor,
>it will take 4 times longer...
[and (4)]

Well, probably the same reason as NFS over UDP is discouraged. See nfs(5)
section WARNINGS (in short: IP fragment ID can wrap quite fast on Gigabit)



	-`J'
-- 

Re: ncpfs and TCP vs UDP

[Pierre Ossman]

Jan Engelhardt wrote:
> Well, probably the same reason as NFS over UDP is discouraged. See nfs(5)
> section WARNINGS (in short: IP fragment ID can wrap quite fast on Gigabit)
>
>   

I have no such warning in my nfs(5), but I am aware of this yes.
Somewhat amusing that both nfs and ncpfs tend to default to using udp
with this in mind. ;)

Rgds

-- 
     -- Pierre Ossman

  Linux kernel, MMC maintainer        http://www.kernel.org
  PulseAudio, core developer          http://pulseaudio.org
  rdesktop, core developer          http://www.rdesktop.org

Re: ncpfs and TCP vs UDP

[Jan Engelhardt]

>> Well, probably the same reason as NFS over UDP is discouraged. See nfs(5)
>> section WARNINGS (in short: IP fragment ID can wrap quite fast on Gigabit)
>
>I have no such warning in my nfs(5), but I am aware of this yes.
>Somewhat amusing that both nfs and ncpfs tend to default to using udp
>with this in mind. ;)

nfs has already been "fixed" to prefer tcp over udp (i.e. the default is tcp)


	-`J'
--