[Beowulf] choosing a high-speed interconnect

[Matt L. Leininger]

On Tue, 2004-10-12 at 17:06 -0400, Mark Hahn wrote:
> 
> IB vendors swear up and down that they're cheaper than Myri,
> lower-latency, higher bandwidth and taste great with iced cream.
> I must admit to some skepticism in spite of lacking any IB experience ;)
> it does seem clear that upcoming PCI-e systems will let IB vendors
> drop a few more chips off their nic, and theoretically come down to 
> the $2-300/nic range.  as far as I know, switches are staying more or 
> less at the same price.  and it's worth remembering that IB still 
> doesn't have *that* much field-proof (questions regarding whether IB
> will continue to be a sole-source ecosystem, issues of integrating 
> with Linux, rumors of sticking points  regarding pinned memory, qpair 
> scaling in large clusters, handling congestion, etc.)
> 
 
  There are multiple 128 node (and greater) IB systems that are stable
and are being used for production apps.  The #7 top500 machine from
RIKEN is using IB and has been in production for over six months.  My
cluster at Sandia (about 128 nodes) is being used for IB R&D and
production science runs.  The science runs have produce many papers over
the last 9 months.  We've purchased other IB clusters ranging from 64 to
>300 nodes that are for production use.  All run great under Linux, and
you have multiple IB vendors to choose from (Voltaire, Topspin,
InfiniCon, and Mellanox).   Almost all of the IB software development is
done under Linux first and then ported to other OSes.  

   QP scaling isn't as critical an issue if the MPI implementation sets
up the connections as needed (kinda of a lazy connection setup).  Why
set up an all-to-all QP connectivity if the MPI implements an all-to-all
or collectives as tree based pt2pt algorithms.  Network congestion on
larger clusters can be reduced by using source based adaptive
(multipath) routing instead of the standard IB static routing.  

  Also remember that IB has a lot more field experience than the latest
Myricom hardware and MX software stack.  

	- Matt