[Beowulf] building Infiniband 4x cluster questions
diep at xs4all.nl
Sun Nov 6 15:01:02 PST 2011
There is a lot of infiniband 4x stuff on ebay now. Most interesting
for me to buy some and toy with it. However i have not much clue
about infiniband as it always used to be 'out of my pricerange'. So
looking for som expert advice here. A few questions:
Now i assume what i need is: bunch of cards, copper cables, and a
switch, and a big dosis of luck.
There is a lot of cards on ebay with 2 ports.
10 gbit + 10 gbit.
Do i need to connect both to the same switch?
So in short with infiniband you lose 2 ports of the switch to 1
card, is that correct?
Do all switches work with all cards?
Can you mix the many different cards that are out there of 4x
If not, can you mix from mellanox the different cards theirs? So mix
from 1 manufacturer cards?
Do the cards work for socket 1155 if it's pci-e versions? (of course
watercooled nodes each)
Is there a limit on how much RAM the machine can have? (referring to
the famous 4GB limit of the QM400 cards of quadrics)
Does infiniband 4x work for x64 machines?
Drivers for cards now. Are those all open source, or does it require
payment? Is the source released of
all those cards drivers, and do they integrate into linux?
The MPI library you can use with each card is that different
manufacturer from manufacturer? Free to download
and can work with different distro's? Does it compile? Is it just a
library or a modified compiler?
Note i assume it's possible to combine it all with pdsh.
I see a bunch of topspin 120 switches there. Supposed to be 200 ns.
there is a 47 manual page, yet it doesn't mention anything about a
password needed to login,
only the default password it mentions.
What if it already has been set, as one ebay guy mentions he didn't
manage to login there.
Is it possible to reset that login or isn't it possible to modify
Is it possible to combine 2 switches and have so to speak a 48 port
Oh btw i plan to ship messages sized 256 bytes massively over the
switch. Would it work if i add a 2nd switch
just to act as a 2nd rail? And a 3d and a 4th rail also work? So a
rail or 4 would it work?
Really important question that rail question.
As that would allow more messages per second. Most messages will be a
byte or 128-256 and for sure nothing
will be bigger. Some messages are shorter. If 128 is that much faster
i'd go for 128.
What more do i need to know?
Lots of simple questions in short! Many thanks in advance for
answerring any question or raising new ones :)
diep at xs4all.nl
More information about the Beowulf