[Beowulf] building Infiniband 4x cluster questions

[Robert Horton]

Hi,

Most of what I know about Infiniband came from the notes at
http://www.hpcadvisorycouncil.com/events/switzerland_workshop/agenda.php
(or John Hearns in his previous life!).

On Mon, 2011-11-07 at 00:01 +0100, Vincent Diepeveen wrote:
> Do i need to connect both to the same switch?
> So in short with infiniband you lose 2 ports  of the switch to 1  
> card, is that correct?

You probably want to just connect one port to a switch and leave the
other one unconnected to start with.

> CARDS:
> Do all switches work with all cards?
> Can you mix the many different cards that are out there of 4x  
> infiniband?
> If not, can you mix from mellanox the different cards theirs? So mix  
> from 1 manufacturer cards?

They will (or at least should) all work to a point but depending on what
combination you are using you may not get some features. If you want an
easy life keep it all from the same manufacturer

> Does infiniband 4x work for x64 machines?

The 4x bit is the number of links aggregated together. 4x is normal for
connections from a switch to a node, higher numbers are sometimes used
for inter-switch links. You also need to note the data rate (eg SDR,
DDR, QDR etc).

> DRIVERS:
> Drivers for cards now. Are those all open source, or does it require  
> payment? Is the source released of
> all those cards drivers, and do they integrate into linux?

You should get everything you need from the Linux kernel and / or OFED.

> MPI:
> The MPI library you can use with each card is that different  
> manufacturer from manufacturer? Free to download
> and can work with different distro's? Does it compile? Is it just a  
> library or a modified compiler?

There are quite a lot to choose from but OpenMPI is probably a good
starting point.

Hope that's some help...

Rob