[Beowulf] building Infiniband 4x cluster questions

[Vincent Diepeveen]

It seems the latency of DDR infiniband to do a blocked read from  
remote memory (RDMA) is between that of SDR and quadrics, with  
quadrics being a lot faster.

http://www.google.nl/url?sa=t&rct=j&q=rdma%20latency%20ddr% 
20infiniband&source=web&cd=9&ved=0CF8QFjAI&url=http%3A%2F% 
2Fwww.cse.scitech.ac.uk%2Fdisco%2Fmew18%2FPresentations%2FDay2% 
2F5th_Session% 
2FMarkLehrer.pdf&ei=tjW4ToWjOY2dOoD69esB&usg=AFQjCNEzRhG5ljCxmm1r0SMXVob 
nAbZUAQ&cad=rja

If i click there i get to a MarkLehrer.pdf

www.cse.scitech.ac.uk/disco/mew18/Presentations/.../MarkLehrer.pdf

It claims a RDMA read has latency of 1.91 us.

However i'll have to see that in my own benchmark first before i  
believe it when we hammer with many different processes
at that card at the same time. You get problems like switch latencies  
and other nasty stuff then.

This is a presentation slide and i need something that works in reality.

HP 4X DDR InfiniBand Mezzanine HCA 410533-B21 SFF-8470

they're $75 but just 2 of them available on ebay. The next 'ddr' one is

QLE7104 QLOGIC INFINIBAND 8X DDR SINGLE PORT HBA

So that's a qlogic one, $108 just 3 of them available, but we already  
get at a dangerous price level.
Remember i want well over a million reads getting done a second and i  
didn't count the pollution by writes even yet.

HP 4X DDR InfiniBand Mezzanine HCA - 2 Ports 448262-B21

They're $121 and again just 2 available.

This seems a problem with infiniband on ebay. Even if you search 16  
cards, you can each time buy 2 or so max.
As if sometimes a scientist takes 2 back home and puts 'em on ebay.  
No big 'old' masses get posted there.

The first one to offer 10, that's http://www.ebay.com/itm/HP- 
INFINIBAND-4X-DDR-PCI-E-DUAL-PORT-HCA-448397B21-/110649801200? 
pt=COMP_EN_Hubs&hash=item19c33df9f0
That 's at $192.11 a piece.

It seems DDR infiniband still isn't in my pricerange Prentice.

The QM500-B's from quadrics go for between $30 and $50 however.

On Nov 7, 2011, at 7:16 PM, Prentice Bisbal wrote:

> Vincent,
>
> Don't forget that between SDR and QDR, there is DDR.  If SDR is too
> slow, and QDR is too expensive, DDR might be just right.
>
> --
> Goldilocks
>
>
> On 11/07/2011 11:58 AM, Vincent Diepeveen wrote:
>> hi Prentice,
>>
>> I had noticed the diff between SDR up to QDR,
>> the SDR cards are affordable, the QDR isn't.
>>
>> The SDR's are all $50-$75 on ebay now. The QDR's i didn't find cheap
>> prices in that pricerange yet.
>>
>> If i would want to build a network that's low latency and had a  
>> budget
>> of $800 or so a node of course i would
>> build a dolphin SCI network, as that's probably the fastest latency
>> card sold for a $675 or so a piece.
>>
>> I do not really see a rival latency wise to Dolphin there. I bet most
>> manufacturers selling clusters don't use
>> it as they can make $100 more profit or so selling other networking
>> stuff, and universities usually swallow that.
>>
>> So price total dominates the network. As it seems now infiniband  
>> 4x is
>> not going to offer enough performance.
>> The one-way pingpong latencies over a switch that i see of it, are  
>> not
>> very convincing. I see remote writes to RAM
>> are like nearly 10 microseconds for 4x infiniband and that card is  
>> the
>> only one affordable.
>>
>> The old QM400's i have here are one-way pingpong 2.1 us or so, and
>> QM500-B's are plentyful on the net (of course big disadvantage: needs
>> pci-x),
>> which are a 1.3 us or so there and have SHMEM. Not seeing a cheap
>> switch for the QM500's though nor cables.
>>
>> You see price really dominates everything here. Small cheap nodes you
>> cannot build if the port price, thanks to expensive network card,
>> more than doubles.
>>
>> Power is not the real concern for now - if a factory already burns a
>> couple of hundreds of megawatts, a small cluster somewhere on the
>> attick eating
>> a few kilowatts is not really a problem :)
>>
>>
>>
>> On Nov 7, 2011, at 5:36 PM, Prentice Bisbal wrote:
>>
>>>
>>> On 11/06/2011 06:01 PM, Vincent Diepeveen wrote:
>>>> hi,
>>>>
>>>> There is a lot of infiniband 4x stuff on ebay now.
>>>
>>> Vincent,
>>>
>>> Do you mean 4x, or QDR? They refer to different parts of the IB
>>> architecture. 4x refers to the number of lanes for the data to  
>>> travel
>>> down and QDR refers to the data signalling rate.
>>>
>>> It's probably irrelevant for this conversation, but if you are just
>>> learning about IB, It's good to understand that difference.
>>>
>>> Prentice
>>> _______________________________________________
>>> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin  
>>> Computing
>>> To change your subscription (digest mode or unsubscribe) visit
>>> http://www.beowulf.org/mailman/listinfo/beowulf
>>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin  
> Computing
> To change your subscription (digest mode or unsubscribe) visit  
> http://www.beowulf.org/mailman/listinfo/beowulf