[Beowulf] InfiniBand channel bundling?

[Prentice Bisbal]

Brice,

You're talking only about latency, right? From the graphs FDR/Gen3 and 
QDR/Gen3 are almost identical.

For bandwidth, QDR/Gen3 provided only a 16 - 20% improvement over 
QDR/Gen2., depending on whether they were measuring one-way or 
bidirectional bandwidth, whereas FDR seemed to provide about 40-50 % of 
the bandwidth performance difference between FDR/Gen3 and QDR/Gen2. 
Interestingly, the text of the article didn't compare FDR/Gen3 to 
QDR/Gen3, but it's clear from the graphs that FDR/Gen 3 is still way 
above QDR/Gen3 despite the 16-20% improvement Gen3 provides.

Prentice

On 10/30/2014 11:08 AM, Brice Goglin wrote:
> Do you want to compare FDR vs QDR when used in the exact same PCIe
> slots? FDR often goes in Gen3 while QDR has been in Gen2 for a while.
>
> According to the link below, Gen3 is responsible for pretty much all the
> improvement between QDR/Gen2 and FDR/Gen3.
> http://en.community.dell.com/techcenter/high-performance-computing/b/general_hpc/archive/2012/03/06/hpc-i-o-performance-using-pci-e-gen3-slots-on-the-12th-generation-12g-poweredge-servers
>
> Brice
>
>
>
> Le 30/10/2014 15:41, Prentice Bisbal a écrit :
>> Gilad,
>>
>> If end-to-end is lower for FDR, then what latency is being measured
>> for FDR that is higher than for QDR? According to Wikipedia, and the
>> Mellanox website, FDR does have a better latency than QDR (0.7
>> microseconds vs. 1.3 microseconds), but I and others on this list have
>> heard that FDR has worse latency than QDR. Have we been misinformed,
>> or does it depend on how you measure or define latency?
>>
>> http://en.wikipedia.org/wiki/InfiniBand
>> http://www.mellanox.com/page/performance_infiniband
>>
>> Prentice
>>
>>
>> On 10/29/2014 06:46 PM, Gilad Shainer wrote:
>>> End-to-end FDR latency is lower than end-to-end QDR latency - per
>>> published measurments that can be found in multiple places.
>>>
>>> Gilad
>>>
>>> -----Original Message-----
>>> From: Beowulf [mailto:beowulf-bounces at beowulf.org] On Behalf Of Jörg
>>> Saßmannshausen
>>> Sent: Wednesday, October 29, 2014 3:45 PM
>>> To: Beowulf Mailinglist
>>> Subject: Re: [Beowulf] InfiniBand channel bundling?
>>>
>>> Hi all,
>>>
>>> thanks again for the wealth of information.
>>>
>>> Now, given that I am not interested in transporting files over the IB
>>> network but I am doing parallel calculations, I would have thought
>>> that the latency here is more important than the speed?
>>> Thus, if FDR has a higher latency than QDR, does that mean my
>>> performance is decreasing when I am running a calculation between nodes?
>>>
>>> For those of you who are into Chemistry code: I am using VASP, cp2k,
>>> quantum espresso and cpmd mainly. All of that is plain wave code.
>>>
>>> All the best from a wet London
>>>
>>> Jörg
>>>
>>>
>>> On Mittwoch 29 Oktober 2014 Prentice Bisbal wrote:
>>>> On 10/28/2014 04:43 PM, Mark Hahn wrote:
>>>>> On Tue, 28 Oct 2014, John Hearns wrote:
>>>>>> Here is a very good post from Glenn Lockwood regarding FDR versus
>>>>>> dual-rail QDR:
>>>>>>
>>>>>> http://glennklockwood.blogspot.co.uk/2013/05/fdr-infiniband-vs-dual
>>>>>> -rail
>>>>>> -qdr.html
>>>>> indeed, very nice.  though also quite surprising - is it known that
>>>>> FDR is so terrible for latency?  seems astonishing to me.
>>>> Yes, it was known to me. I had already known that FDR was worse than
>>>> QDR for latency, but I don't remember my source. I don't know if I'd
>>>> characterize it as "so terrible", though.
>>> -- 
>>> *************************************************************
>>> Dr. Jörg Saßmannshausen, MRSC
>>> University College London
>>> Department of Chemistry
>>> Gordon Street
>>> London
>>> WC1H 0AJ
>>>
>>> email: j.sassmannshausen at ucl.ac.uk
>>> web: http://sassy.formativ.net
>>>
>>> Please avoid sending me Word or PowerPoint attachments.
>>> See http://www.gnu.org/philosophy/no-word-attachments.html
>>> _______________________________________________
>>> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
>>> To change your subscription (digest mode or unsubscribe) visit
>>> http://www.beowulf.org/mailman/listinfo/beowulf
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf