[Beowulf] InfiniBand channel bundling?

[Gilad Shainer]

Have no idea what the guy did in the referenced blog - not sure what was the setting, same system etc. But having higher latency with FDR versus QDR is not something that we see in any of the testing we did.

The slight increase in the switch latency has nothing to do with the data speed - but with the features set and have support for both IB and Eth on the same switch. 

The FDR switch latency of 170ns, was decreased to 130ns on the EDR switch (we already posted that info). 

 Gilad


-----Original Message-----
From: Mark Hahn [mailto:hahn at mcmaster.ca] 
Sent: Thursday, October 30, 2014 10:18 PM
To: Gilad Shainer
Cc: beowulf at beowulf.org
Subject: Re: [Beowulf] InfiniBand channel bundling?

> If you compare QDR devices to FDR devices, than FDR is showing lower 
>latency.

in the paper referenced, that is not the case.  the numbers provided are QDR 1.27 us, versus FDR 1.67 us.  although it's only 400ns, it's still >30% slower, when one might expect a speed improvement.

> What you might heard is that the FDR switches are slightly higher 
>latency than the QDR switches as they include new capabilities of link 
>level retransmission and forward error correction, but overall end to 
>end latency with FDR is lower.

that's interesting - do you mean that in order to achieve higher bandwidth, the error rate becomes a problem, necessitating RT/FEC?  I guess it's obvious from the shrinkage of allowed passive/copper cable lengths that SNR/BER is a big issue, but does this imply that going optical will reduce the latency cost for FDR?

> The EDR switch latency is lower than the FDR switch and the QDR 
> switch, so further latency decrease will be seen with EDR.

I'm puzzled by this, since at least marketing latencies of existing switches are pretty low (170ns/hop for FDR).  how much can 170ns be improved?

thanks, mark hahn.