[Beowulf] InfiniBand channel bundling?

[Gilad Shainer]

The PCIe interface does provide some part of the latency reduction, but another part is coming from the InfiniBand side. Also the message rate increases between these generation in a factor of around 4X, and this is due to the new HCA design. 

Regards,
Gilad Shainer 


-----Original Message-----
From: Beowulf [mailto:beowulf-bounces at beowulf.org] On Behalf Of Brice Goglin
Sent: Thursday, October 30, 2014 8:09 AM
To: beowulf at beowulf.org
Subject: Re: [Beowulf] InfiniBand channel bundling?

Do you want to compare FDR vs QDR when used in the exact same PCIe slots? FDR often goes in Gen3 while QDR has been in Gen2 for a while.

According to the link below, Gen3 is responsible for pretty much all the improvement between QDR/Gen2 and FDR/Gen3.
http://en.community.dell.com/techcenter/high-performance-computing/b/general_hpc/archive/2012/03/06/hpc-i-o-performance-using-pci-e-gen3-slots-on-the-12th-generation-12g-poweredge-servers

Brice



Le 30/10/2014 15:41, Prentice Bisbal a écrit :
> Gilad,
>
> If end-to-end is lower for FDR, then what latency is being measured 
> for FDR that is higher than for QDR? According to Wikipedia, and the 
> Mellanox website, FDR does have a better latency than QDR (0.7 
> microseconds vs. 1.3 microseconds), but I and others on this list have 
> heard that FDR has worse latency than QDR. Have we been misinformed, 
> or does it depend on how you measure or define latency?
>
> http://en.wikipedia.org/wiki/InfiniBand
> http://www.mellanox.com/page/performance_infiniband
>
> Prentice
>
>
> On 10/29/2014 06:46 PM, Gilad Shainer wrote:
>> End-to-end FDR latency is lower than end-to-end QDR latency - per 
>> published measurments that can be found in multiple places.
>>
>> Gilad
>>
>> -----Original Message-----
>> From: Beowulf [mailto:beowulf-bounces at beowulf.org] On Behalf Of Jörg 
>> Saßmannshausen
>> Sent: Wednesday, October 29, 2014 3:45 PM
>> To: Beowulf Mailinglist
>> Subject: Re: [Beowulf] InfiniBand channel bundling?
>>
>> Hi all,
>>
>> thanks again for the wealth of information.
>>
>> Now, given that I am not interested in transporting files over the IB 
>> network but I am doing parallel calculations, I would have thought 
>> that the latency here is more important than the speed?
>> Thus, if FDR has a higher latency than QDR, does that mean my 
>> performance is decreasing when I am running a calculation between nodes?
>>
>> For those of you who are into Chemistry code: I am using VASP, cp2k, 
>> quantum espresso and cpmd mainly. All of that is plain wave code.
>>
>> All the best from a wet London
>>
>> Jörg
>>
>>
>> On Mittwoch 29 Oktober 2014 Prentice Bisbal wrote:
>>> On 10/28/2014 04:43 PM, Mark Hahn wrote:
>>>> On Tue, 28 Oct 2014, John Hearns wrote:
>>>>> Here is a very good post from Glenn Lockwood regarding FDR versus 
>>>>> dual-rail QDR:
>>>>>
>>>>> http://glennklockwood.blogspot.co.uk/2013/05/fdr-infiniband-vs-dua
>>>>> l
>>>>> -rail
>>>>> -qdr.html
>>>> indeed, very nice.  though also quite surprising - is it known that 
>>>> FDR is so terrible for latency?  seems astonishing to me.
>>> Yes, it was known to me. I had already known that FDR was worse than 
>>> QDR for latency, but I don't remember my source. I don't know if I'd 
>>> characterize it as "so terrible", though.
>>
>> --
>> *************************************************************
>> Dr. Jörg Saßmannshausen, MRSC
>> University College London
>> Department of Chemistry
>> Gordon Street
>> London
>> WC1H 0AJ
>>
>> email: j.sassmannshausen at ucl.ac.uk
>> web: http://sassy.formativ.net
>>
>> Please avoid sending me Word or PowerPoint attachments.
>> See http://www.gnu.org/philosophy/no-word-attachments.html
>> _______________________________________________
>> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin 
>> Computing To change your subscription (digest mode or unsubscribe) 
>> visit http://www.beowulf.org/mailman/listinfo/beowulf
>

_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf