[Beowulf] SC|05: will 10Ge beat IB?

[ctierney at hypermall.net]

Quoting Mark Hahn <hahn at physics.mcmaster.ca>:

> last year, there was lots of buzz around IB as an up-and-comer,
> with some undercurrents of doubt related to latency, memory 
> consumption on large nets, proliferation of stacks, etc.
> 
> this year, the picture was a lot clearer, with at least some 
> progress on all these.  lots of vendors pushing IB, including 
> some starting to push IB for storage.  IB-storage vendors seem
> to be regarding it as a faster FC, though - to me, that's a huge
> step backwards, since FC was never really inter-operable, at least
> not in the sense that ethernet is inter-op.
> 
> further, there was also LOTS of 10G ethernet this year, and I have 
> to wonder: will 10G squeeze IB out of the market?  

That depends.  Do you think that iSCSI (or some other variant) is finally
going to take off as it was supposed to?  IB has some QoS features that
may be important when using IB for multiple purposes, like HSN and FC.
I don't know if that is true, just a theory.

Also, 10G is still quite expensive (except Myrinet 10g).  
Fibre-media converters are still expensive and as far as I have heard, no
one is pushing 10G over copper.  IB is quite cheap compared to 10G, but
it is a new technology and adoption outside of HPC is slow.  The volume
isn't there yet to bring pricing where it should be.

> 
> for people who want to do fast filesystems, 10G seems more attractive
> because of ethernet's comfort factor, interop, and technically higher 
> bandwidth.  the IB-storage vendors I talked to did not seem to be 
> thinking of eth-type interop, but rather of capturing whole shops 
> in the traditional FC/SAN fashion.

Do you want direct attached storage, or a SAN environment?  That will
drive some of the decisions.  When looking at distributed filesystems 
(Lustre, Ibrix, Terrascale) you don't need a SAN.  These are more 
appropriate for most HPC applications than shared filesystems (CXFS, StorNEXT,
Polyserve, GFS) because of scalibility of throughput, meta-data 
performance, and number of clients supported.  I'm assuming that Lustre 
starts to support distributed meta-data soon.

> 
> but for MPI, Quadrics or Infinipath are the obvious choices, since they 
> both deliver amazing latency, which makes large, tightly-coupled programs
> possible.  Myrinet 10G is attractive in many ways, as well, though its
> latency is higher, though probably still faster than IB.  10G eth is,
> of course, even higher latency than IB.

Is Myrinet 10G latency higher than Quadrics?  Myricom quoted 1.8 microseconds
at SC05, but I forgot what Quadrics is these days.  The difference between
Infinipath and other PCI-Express solutions is the bus.  If/when Myricom or
Quadrics provide HT support, their latency will be near Infinipath. If/when
Pathscale releases a PCI-E card, its latency will be worse, and similar to the
others. At this point the differences are going to be less than 1/4 microsecond
(my guess) and other features will become more important.

> 
> for people not at thes extremes, gigabit is still quite attractive,
> since it's incredibly easy, and often fast enough, especially if 
> 10G is available for federating switches or connecting fileservers.
> 
> what do you think: will 10G eth eat IB's lunch?

IB should be much cheaper than it is.  Increased volume will get it there,
but all of the IB vendors think they are special, and not just another 
ethernet. When they start thinking they are ethernet then it may have a 
fighting chance against 10G eth.

Craig



----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.