[Beowulf] InfiniBand channel bundling?

[Peter Kjellström]

On Wed, 29 Oct 2014 12:02:49 -0400
Prentice Bisbal <prentice.bisbal at rutgers.edu> wrote:

> 
> On 10/28/2014 06:24 AM, John Hearns wrote:
> > Here is a very good post from Glenn Lockwood regarding FDR versus
> > dual-rail QDR:
> >
> >
> > http://glennklockwood.blogspot.co.uk/2013/05/fdr-infiniband-vs-dual-rail-qdr.html
> 
>  From the article:
> 
> > The vendors would have you believe that FDR is just better in every 
> > way (after all, FDR's 14Gbit peak is bigger than QDR's 10Gbit
> > peak), but I was skeptical. 
> 
> I thought that FDR was 54 Gb/s. I had to check Wikipedia. He's
> talking about 1x (single-lane) performance. Most IB is 4x.

This thread is getting thick with bad information and confusion. Let me
try to clear some up:

FDR is fourteen-rate (per lane) -> 14 * 4 lanes typical -> 56 Gbps on
the wire. Encoding is 64/66 bit and after pci-express transfer size
etc. you get around 6000 MB/s.

QDR is 10G (per lane) -> 10 * 4 lanes typical -> 40 Gbps on the wire.
Encoding is 8/10 -> 32 Gbps and after pci-express 3200-4000 MB/s.

On a certain centos-6.5 platform I see these actual numbers using
ib_write_bw/ib_write_lat (verbs):

      bw      lat(near cpu)   lat(far cpu)
--------------------------------------------------
QDR   3300*   1.12            1.54
FDR   6000    0.85            1.59

QDR hw is mlx connectx3 to mlx switchX (same switch), ivy bridge node.
FDR hw is mlx connectx3 to mlx switchX (same switch), sandy bridge node.

Of course what people refer to as QDR could also be connectx2 or
qlogic/intel. The smaller hit for "far cpu" for qdr could in my case be
an improvment from snb to ivb.

* I expected more like 4000 here but 3300 is what I got.

/Peter