[Beowulf] building Infiniband 4x cluster questions
diep at xs4all.nl
Mon Nov 7 08:58:41 PST 2011
I had noticed the diff between SDR up to QDR,
the SDR cards are affordable, the QDR isn't.
The SDR's are all $50-$75 on ebay now. The QDR's i didn't find cheap
prices in that pricerange yet.
If i would want to build a network that's low latency and had a
budget of $800 or so a node of course i would
build a dolphin SCI network, as that's probably the fastest latency
card sold for a $675 or so a piece.
I do not really see a rival latency wise to Dolphin there. I bet most
manufacturers selling clusters don't use
it as they can make $100 more profit or so selling other networking
stuff, and universities usually swallow that.
So price total dominates the network. As it seems now infiniband 4x
is not going to offer enough performance.
The one-way pingpong latencies over a switch that i see of it, are
not very convincing. I see remote writes to RAM
are like nearly 10 microseconds for 4x infiniband and that card is
the only one affordable.
The old QM400's i have here are one-way pingpong 2.1 us or so, and
QM500-B's are plentyful on the net (of course big disadvantage: needs
which are a 1.3 us or so there and have SHMEM. Not seeing a cheap
switch for the QM500's though nor cables.
You see price really dominates everything here. Small cheap nodes you
cannot build if the port price, thanks to expensive network card,
more than doubles.
Power is not the real concern for now - if a factory already burns a
couple of hundreds of megawatts, a small cluster somewhere on the
a few kilowatts is not really a problem :)
On Nov 7, 2011, at 5:36 PM, Prentice Bisbal wrote:
> On 11/06/2011 06:01 PM, Vincent Diepeveen wrote:
>> There is a lot of infiniband 4x stuff on ebay now.
> Do you mean 4x, or QDR? They refer to different parts of the IB
> architecture. 4x refers to the number of lanes for the data to travel
> down and QDR refers to the data signalling rate.
> It's probably irrelevant for this conversation, but if you are just
> learning about IB, It's good to understand that difference.
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin
> To change your subscription (digest mode or unsubscribe) visit
More information about the Beowulf