[Beowulf] building Infiniband 4x cluster questions

[Vincent Diepeveen]

hi,

There is a lot of infiniband 4x stuff on ebay now. Most interesting  
for me to buy some and toy with it. However i have not much clue
about infiniband as it always used to be 'out of my pricerange'. So  
looking for som expert advice here. A few questions:

Now i assume what i need is: bunch of cards, copper cables, and a  
switch, and a big dosis of luck.

There is a lot of cards on ebay with 2 ports.
10 gbit  + 10 gbit.

Do i need to connect both to the same switch?
So in short with infiniband you lose 2 ports  of the switch to 1  
card, is that correct?

CARDS:
Do all switches work with all cards?
Can you mix the many different cards that are out there of 4x  
infiniband?
If not, can you mix from mellanox the different cards theirs? So mix  
from 1 manufacturer cards?

SOCKET 1155:
Do the cards work for socket 1155 if it's pci-e versions? (of course  
watercooled nodes each)

Is there a limit on how much RAM the machine can have? (referring to  
the famous 4GB limit of the QM400 cards of quadrics)
Does infiniband 4x work for x64 machines?

DRIVERS:
Drivers for cards now. Are those all open source, or does it require  
payment? Is the source released of
all those cards drivers, and do they integrate into linux?

MPI:
The MPI library you can use with each card is that different  
manufacturer from manufacturer? Free to download
and can work with different distro's? Does it compile? Is it just a  
library or a modified compiler?

Note i assume it's possible to combine it all with pdsh.

SWITCH:
I see a bunch of topspin 120 switches there. Supposed to be 200 ns.
there is a 47 manual page, yet it doesn't mention anything about a  
password needed to login,
only the default password it mentions.

What if it already has been set, as one ebay guy mentions he didn't  
manage to login there.
Is it possible to reset that login or isn't it possible to modify  
login password?

Is it possible to combine 2 switches and have so to speak a 48 port  
switch then?

Oh btw i plan to ship messages sized 256 bytes massively over the  
switch. Would it work if i add a 2nd switch
just to act as a 2nd rail? And a 3d and a 4th rail also work? So a  
rail or 4 would it work?

Really important question that rail question.

As that would allow more messages per second. Most messages will be a  
byte or 128-256 and for sure nothing
will be bigger. Some messages are shorter. If 128 is that much faster  
i'd go for 128.

What more do i need to know?

Lots of simple questions in short! Many thanks in advance for  
answerring any question or raising new ones :)

Regards,
Vincent Diepeveen
diep at xs4all.nl