[Beowulf] choosing a high-speed interconnect

[Michael T. Prinkey]

This won't help with your Opteron systems as they probably have broadcom
(tg3) NICs, but GAMMA has just released an update that supports Intel
(e1000) gigabit cards:

http://www.disi.unige.it/project/gamma/index.html

They have an MPI implementation as well:

http://www.disi.unige.it/project/gamma/mpigamma/index.html

They claim vastly improved latency and incrementally improved bandwidth on
gigabit hardware relative to TCP/IP.  We are planning to test it with the
new Xeon cluster we will be building next month.  It will be interesting
to see how it fairs with LINPACK and the MFIX CFD code.

Anyone given GAMMA a try?

Mike

On Tue, 12 Oct 2004, Joe Landman wrote:

> First questions first:
> 
>   Why do you think you need a faster network, and what aspect of fast do 
> you think you need?  Low latency?  High bandwidth?
> 
> Then...
> 
>   What codes are you running?  Across how many CPUS?  Have you done a 
> performance analysis on your system to observe "slow" runs in progress, 
> and are you convinced that the network is the issue?
> 
> We have done lots of tuning bits for customers where the issues wound up 
> being something else than what they had thought.  It is worth at least 
> looking into for your code/problems, and identifying the bottleneck (if 
> you haven't already done so).
> 
> That said, all the below require an external "switch" fabric.  All range 
> from $500-$2000 per HBA, and about $1000 or more per switch port.  
> Varies a bit.  Performance is comparible in most cases, with IB  seeming 
> to have a higher ceiling than the others.
> 
> Joe
> 
> Chris Sideroff wrote:
> 
> >I'm sure posing this may raise more questions than answer but which
> >high-speed interconnect would offer the best 'bang for the buck':
> >
> >1) myrinet
> >2) quardics qsnet
> >3) mellanox infiniband
> >
> >Currently, our 30 node dual Opteron (MSI K8D Master-FT boards) cluster
> >uses Gig/E and are looking to upgrade to a faster network. 
> >
> >As well, what are the components would one need for each setup?  The
> >reason I ask is for example the Myrinet switches accept different line
> >cards and am not sure which one to use.  Sorry if this a bit of a newbie
> >question but I have no experience with any of this kind of hardware. I
> >am reading the docs for each but thought your feedback would be good.
> >
> >Thanks
> >
> >Chris Sideroff
> >
> >_______________________________________________
> >Beowulf mailing list, Beowulf at beowulf.org
> >To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
> >  
> >
> 
>