[Beowulf] building Infiniband 4x cluster questions

[Gilad Shainer]

RDMA read is a round trip operation and it is measured from host memory to host memory. I doubt if Quadrics had half of it for round trip operations measured from host memory to host memory. The PCI-X memory to card was around 0.7 by itself (one way)....

Gilad


-----Original Message-----
From: beowulf-bounces at beowulf.org [mailto:beowulf-bounces at beowulf.org] On Behalf Of Vincent Diepeveen
Sent: Monday, November 07, 2011 12:33 PM
To: Greg Keller
Cc: beowulf at beowulf.org
Subject: Re: [Beowulf] building Infiniband 4x cluster questions

hi Greg,

Very useful info! I already was wondering about the different timings i see for infiniband, but indeed it's the ConnectX that scores better in latency.

$289 on ebay but that's directly QDR then.

"ConnectX-2 Dual-Port VPI QDR Infiniband Mezzanine I/O Card for Dell PowerEdge M1000e-Series Blade Servers"

This 1.91 microseconds for a RDMA read is for a connectx. Not bad for Infiniband.
Only 50% slower in latency than quadrics which is pci-x of course.

Yet now needed is a cheap price for 'em :)

It seems indeed all the 'cheap' offers are the infinihost III DDR versions.

Regards,
Vincent

On Nov 7, 2011, at 9:21 PM, Greg Keller wrote:

>
>> Date: Mon, 07 Nov 2011 13:16:00 -0500
>> From: Prentice Bisbal<prentice at ias.edu>
>> Subject: Re: [Beowulf] building Infiniband 4x cluster questions
>> Cc: Beowulf Mailing List<beowulf at beowulf.org> 
>> Message-ID:<4EB82060.3050300 at ias.edu>
>> Content-Type: text/plain; charset=ISO-8859-1
>>
>> Vincent,
>>
>> Don't forget that between SDR and QDR, there is DDR.  If SDR is too 
>> slow, and QDR is too expensive, DDR might be just right.
> And for DDR a key thing is, when latency matters, "ConnectX" DDR is 
> much better than the earlier "Infinihost III" DDR cards.  We have 
> 100's of each and the ConnectX make a large impact for some codes.  
> Although nearly antique now, we actually have plans for the ConnectX 
> cards in yet another round of updated systems.  This is the 3rd 
> Generation system I have been able to re-use the cards in (Harperton, 
> Nehalem, and now Single Socket Sandy Bridge), which makes me very 
> happy.  A great investment that will likely live until PCI-Gen3 slots 
> are the norm.
> --
> Da Bears?!
>
>> --
>> Goldilocks
>>
>>
>> On 11/07/2011 11:58 AM, Vincent Diepeveen wrote:
>>>>  hi Prentice,
>>>>
>>>>  I had noticed the diff between SDR up to QDR,  the SDR cards are 
>>>> affordable, the QDR isn't.
>>>>
>>>>  The SDR's are all $50-$75 on ebay now. The QDR's i didn't find 
>>>> cheap  prices in that pricerange yet.
>>>>
>>>>  If i would want to build a network that's low latency and had a 
>>>> budget  of $800 or so a node of course i would  build a dolphin SCI 
>>>> network, as that's probably the fastest latency  card sold for a 
>>>> $675 or so a piece.
>>>>
>>>>  I do not really see a rival latency wise to Dolphin there. I bet 
>>>> most  manufacturers selling clusters don't use  it as they can make 
>>>> $100 more profit or so selling other networking  stuff, and 
>>>> universities usually swallow that.
>>>>
>>>>  So price total dominates the network. As it seems now infiniband 
>>>> 4x is  not going to offer enough performance.
>>>>  The one-way pingpong latencies over a switch that i see of it, are 
>>>> not  very convincing. I see remote writes to RAM  are like nearly 
>>>> 10 microseconds for 4x infiniband and that card is the  only one 
>>>> affordable.
>>>>
>>>>  The old QM400's i have here are one-way pingpong 2.1 us or so, and  
>>>> QM500-B's are plentyful on the net (of course big disadvantage:
>>>> needs
>>>>  pci-x),
>>>>  which are a 1.3 us or so there and have SHMEM. Not seeing a cheap  
>>>> switch for the QM500's though nor cables.
>>>>
>>>>  You see price really dominates everything here. Small cheap nodes 
>>>> you  cannot build if the port price, thanks to expensive network 
>>>> card,  more than doubles.
>>>>
>>>>  Power is not the real concern for now - if a factory already burns 
>>>> a  couple of hundreds of megawatts, a small cluster somewhere on 
>>>> the  attick eating  a few kilowatts is not really a problem:)
>>>>
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin 
> Computing To change your subscription (digest mode or unsubscribe) 
> visit http://www.beowulf.org/mailman/listinfo/beowulf

_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf