[Beowulf] building Infiniband 4x cluster questions

[Prentice Bisbal]

Vincent,

Don't forget that between SDR and QDR, there is DDR.  If SDR is too
slow, and QDR is too expensive, DDR might be just right.

--
Goldilocks


On 11/07/2011 11:58 AM, Vincent Diepeveen wrote:
> hi Prentice,
>
> I had noticed the diff between SDR up to QDR,
> the SDR cards are affordable, the QDR isn't.
>
> The SDR's are all $50-$75 on ebay now. The QDR's i didn't find cheap
> prices in that pricerange yet.
>
> If i would want to build a network that's low latency and had a budget
> of $800 or so a node of course i would
> build a dolphin SCI network, as that's probably the fastest latency
> card sold for a $675 or so a piece.
>
> I do not really see a rival latency wise to Dolphin there. I bet most
> manufacturers selling clusters don't use
> it as they can make $100 more profit or so selling other networking
> stuff, and universities usually swallow that.
>
> So price total dominates the network. As it seems now infiniband 4x is
> not going to offer enough performance.
> The one-way pingpong latencies over a switch that i see of it, are not
> very convincing. I see remote writes to RAM
> are like nearly 10 microseconds for 4x infiniband and that card is the
> only one affordable.
>
> The old QM400's i have here are one-way pingpong 2.1 us or so, and
> QM500-B's are plentyful on the net (of course big disadvantage: needs
> pci-x),
> which are a 1.3 us or so there and have SHMEM. Not seeing a cheap
> switch for the QM500's though nor cables.
>
> You see price really dominates everything here. Small cheap nodes you
> cannot build if the port price, thanks to expensive network card,
> more than doubles.
>
> Power is not the real concern for now - if a factory already burns a
> couple of hundreds of megawatts, a small cluster somewhere on the
> attick eating
> a few kilowatts is not really a problem :)
>
>
>
> On Nov 7, 2011, at 5:36 PM, Prentice Bisbal wrote:
>
>>
>> On 11/06/2011 06:01 PM, Vincent Diepeveen wrote:
>>> hi,
>>>
>>> There is a lot of infiniband 4x stuff on ebay now.
>>
>> Vincent,
>>
>> Do you mean 4x, or QDR? They refer to different parts of the IB
>> architecture. 4x refers to the number of lanes for the data to travel
>> down and QDR refers to the data signalling rate.
>>
>> It's probably irrelevant for this conversation, but if you are just
>> learning about IB, It's good to understand that difference.
>>
>> Prentice
>> _______________________________________________
>> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
>> To change your subscription (digest mode or unsubscribe) visit
>> http://www.beowulf.org/mailman/listinfo/beowulf
>