Re: Integration of SCST in the mainstream Linux kernel

["Nicholas A. Bellinger" <nab@linux-iscsi.org>]

On Mon, 2008-02-04 at 15:12 -0800, Nicholas A. Bellinger wrote:
> On Mon, 2008-02-04 at 17:00 -0600, James Bottomley wrote:
> > On Mon, 2008-02-04 at 22:43 +0000, Alan Cox wrote:
> > > > better. So for example, I personally suspect that ATA-over-ethernet is way 
> > > > better than some crazy SCSI-over-TCP crap, but I'm biased for simple and 
> > > > low-level, and against those crazy SCSI people to begin with.
> > > 
> > > Current ATAoE isn't. It can't support NCQ. A variant that did NCQ and IP
> > > would probably trash iSCSI for latency if nothing else.
> > 
> > Actually, there's also FCoE now ... which is essentially SCSI
> > encapsulated in Fibre Channel Protocols (FCP) running over ethernet with
> > Jumbo frames.  It does the standard SCSI TCQ, so should answer all the
> > latency pieces.  Intel even has an implementation:
> > 
> > http://www.open-fcoe.org/
> > 
> > I tend to prefer the low levels as well.  The whole disadvantage for IP
> > as regards iSCSI was the layers of protocols on top of it for
> > addressing, authenticating, encrypting and finding any iSCSI device
> > anywhere in the connected universe.
> 
> Btw, while simple in-band discovery of iSCSI exists, the standards based
> IP storage deployments (iSCSI and iFCP) use iSNS (RFC-4171) for
> discovery and network fabric management, for things like sending state
> change notifications when a particular network portal is going away so
> that the initiator can bring up a different communication patch to a
> different network portal, etc.
> 
> > 
> > I tend to see loss of routing from operating at the MAC level to be a
> > nicely justifiable tradeoff (most storage networks tend to be hubbed or
> > switched anyway).  Plus an ethernet MAC with jumbo frames is a large
> > framed nearly lossless medium, which is practically what FCP is
> > expecting.  If you really have to connect large remote sites ... well
> > that's what tunnelling bridges are for.
> > 
> 
> Some of the points by Julo on the IPS TWG iSCSI vs. FCoE thread:
> 
>       * the network is limited in physical span and logical span (number
>         of switches)
>       * flow-control/congestion control is achieved with a mechanism
>         adequate for a limited span network (credits). The packet loss
>         rate is almost nil and that allows FCP to avoid using a
>         transport (end-to-end) layer
>       * FCP she switches are simple (addresses are local and the memory
>         requirements cam be limited through the credit mechanism)
>       * The credit mechanisms is highly unstable for large networks
>         (check switch vendors planning docs for the network diameter
>         limits) – the scaling argument
>       * Ethernet has no credit mechanism and any mechanism with a
>         similar effect increases the end point cost. Building a
>         transport layer in the protocol stack has always been the
>         preferred choice of the networking community – the community
>         argument
>       * The "performance penalty" of a complete protocol stack has
>         always been overstated (and overrated). Advances in protocol
>         stack implementation and finer tuning of the congestion control
>         mechanisms make conventional TCP/IP performing well even at 10
>         Gb/s and over. Moreover the multicore processors that become
>         dominant on the computing scene have enough compute cycles
>         available to make any "offloading" possible as a mere code
>         restructuring exercise (see the stack reports from Intel, IBM
>         etc.)
>       * Building on a complete stack makes available a wealth of
>         operational and management mechanisms built over the years by
>         the networking community (routing, provisioning, security,
>         service location etc.) – the community argument
>       * Higher level storage access over an IP network is widely
>         available and having both block and file served over the same
>         connection with the same support and management structure is
>         compelling– the community argument
>       * Highly efficient networks are easy to build over IP with optimal
>         (shortest path) routing while Layer 2 networks use bridging and
>         are limited by the logical tree structure that bridges must
>         follow. The effort to combine routers and bridges (rbridges) is
>         promising to change that but it will take some time to finalize
>         (and we don't know exactly how it will operate). Untill then the
>         scale of Layer 2 network is going to seriously limited – the
>         scaling argument
> 

Another data point from the "The "performance penalty of a complete
protocol stack has always been overstated (and overrated)" bullet above:

"As a side argument – a performance comparison made in 1998 showed SCSI
over TCP (a predecessor of the later iSCSI) to perform better than FCP
at 1Gbs for block sizes typical for OLTP (4-8KB). That was what
convinced us to take the path that lead to iSCSI – and we used plain
vanilla x86 servers with plain-vanilla NICs and Linux (with similar
measurements conducted on Windows)."

--nab