[Beowulf] building Infiniband 4x cluster questions - using 1 port out of 2

[Vincent Diepeveen]

On Nov 7, 2011, at 12:44 PM, Robert Horton wrote:

> Hi,
>
> Most of what I know about Infiniband came from the notes at
> http://www.hpcadvisorycouncil.com/events/switzerland_workshop/ 
> agenda.php
> (or John Hearns in his previous life!).
>
> On Mon, 2011-11-07 at 00:01 +0100, Vincent Diepeveen wrote:
>> Do i need to connect both to the same switch?
>> So in short with infiniband you lose 2 ports  of the switch to 1
>> card, is that correct?
>
> You probably want to just connect one port to a switch and leave the
> other one unconnected to start with.

What's the second one doing, is this just in case the switch fails, a  
kind of 'backup'  port?

In my naivity i had thought that both ports together formed the  
bidirectional link to the switch.
So i thought that 1 port was for 10 gigabit upstream and the other  
port was for 10 gigabit downstream,
did i misunderstood that?

>
>> CARDS:
>> Do all switches work with all cards?
>> Can you mix the many different cards that are out there of 4x
>> infiniband?
>> If not, can you mix from mellanox the different cards theirs? So mix
>> from 1 manufacturer cards?
>
> They will (or at least should) all work to a point but depending on  
> what
> combination you are using you may not get some features. If you  
> want an
> easy life keep it all from the same manufacturer

I will load the switch (es) to the maximum number of messages a  
second it can handle,


>
>> Does infiniband 4x work for x64 machines?
>
> The 4x bit is the number of links aggregated together. 4x is normal  
> for
> connections from a switch to a node, higher numbers are sometimes used
> for inter-switch links. You also need to note the data rate (eg SDR,
> DDR, QDR etc).
>
>> DRIVERS:
>> Drivers for cards now. Are those all open source, or does it require
>> payment? Is the source released of
>> all those cards drivers, and do they integrate into linux?
>
> You should get everything you need from the Linux kernel and / or  
> OFED.
>
>> MPI:
>> The MPI library you can use with each card is that different
>> manufacturer from manufacturer? Free to download
>> and can work with different distro's? Does it compile? Is it just a
>> library or a modified compiler?
>
> There are quite a lot to choose from but OpenMPI is probably a good
> starting point.
>
> Hope that's some help...
>
> Rob
>