[Beowulf] building Infiniband 4x cluster questions

[Vincent Diepeveen]

hi Prentice,

I had noticed the diff between SDR up to QDR,
the SDR cards are affordable, the QDR isn't.

The SDR's are all $50-$75 on ebay now. The QDR's i didn't find cheap  
prices in that pricerange yet.

If i would want to build a network that's low latency and had a  
budget of $800 or so a node of course i would
build a dolphin SCI network, as that's probably the fastest latency  
card sold for a $675 or so a piece.

I do not really see a rival latency wise to Dolphin there. I bet most  
manufacturers selling clusters don't use
it as they can make $100 more profit or so selling other networking  
stuff, and universities usually swallow that.

So price total dominates the network. As it seems now infiniband 4x  
is not going to offer enough performance.
The one-way pingpong latencies over a switch that i see of it, are  
not very convincing. I see remote writes to RAM
are like nearly 10 microseconds for 4x infiniband and that card is  
the only one affordable.

The old QM400's i have here are one-way pingpong 2.1 us or so, and  
QM500-B's are plentyful on the net (of course big disadvantage: needs  
pci-x),
which are a 1.3 us or so there and have SHMEM. Not seeing a cheap  
switch for the QM500's though nor cables.

You see price really dominates everything here. Small cheap nodes you  
cannot build if the port price, thanks to expensive network card,
more than doubles.

Power is not the real concern for now - if a factory already burns a  
couple of hundreds of megawatts, a small cluster somewhere on the  
attick eating
a few kilowatts is not really a problem :)



On Nov 7, 2011, at 5:36 PM, Prentice Bisbal wrote:

>
> On 11/06/2011 06:01 PM, Vincent Diepeveen wrote:
>> hi,
>>
>> There is a lot of infiniband 4x stuff on ebay now.
>
> Vincent,
>
> Do you mean 4x, or QDR? They refer to different parts of the IB
> architecture. 4x refers to the number of lanes for the data to travel
> down and QDR refers to the data signalling rate.
>
> It's probably irrelevant for this conversation, but if you are just
> learning about IB, It's good to understand that difference.
>
> Prentice
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin  
> Computing
> To change your subscription (digest mode or unsubscribe) visit  
> http://www.beowulf.org/mailman/listinfo/beowulf