[Beowulf] SC|05: will 10Ge beat IB?

[Mark Hahn]

last year, there was lots of buzz around IB as an up-and-comer,
with some undercurrents of doubt related to latency, memory 
consumption on large nets, proliferation of stacks, etc.

this year, the picture was a lot clearer, with at least some 
progress on all these.  lots of vendors pushing IB, including 
some starting to push IB for storage.  IB-storage vendors seem
to be regarding it as a faster FC, though - to me, that's a huge
step backwards, since FC was never really inter-operable, at least
not in the sense that ethernet is inter-op.

further, there was also LOTS of 10G ethernet this year, and I have 
to wonder: will 10G squeeze IB out of the market?  

for people who want to do fast filesystems, 10G seems more attractive
because of ethernet's comfort factor, interop, and technically higher 
bandwidth.  the IB-storage vendors I talked to did not seem to be 
thinking of eth-type interop, but rather of capturing whole shops 
in the traditional FC/SAN fashion.

but for MPI, Quadrics or Infinipath are the obvious choices, since they 
both deliver amazing latency, which makes large, tightly-coupled programs
possible.  Myrinet 10G is attractive in many ways, as well, though its
latency is higher, though probably still faster than IB.  10G eth is,
of course, even higher latency than IB.

for people not at thes extremes, gigabit is still quite attractive,
since it's incredibly easy, and often fast enough, especially if 
10G is available for federating switches or connecting fileservers.

what do you think: will 10G eth eat IB's lunch?