[Beowulf] InfiniBand channel bundling?

[Peter Kjellström]

On Fri, 31 Oct 2014 00:59:20 -0400 (EDT)
Mark Hahn <hahn at mcmaster.ca> wrote:

> >      bw      lat(near cpu)   lat(far cpu)
> > --------------------------------------------------
> > QDR   3300*   1.12            1.54
> > FDR   6000    0.85            1.59
> 
> thanks for the numbers, though they are somewhat perplexing.

The only surprising thing IMO is the 3300 MB/s number for QDR. And now
looking closer I notice that that IVB system only used pci-gen2 for the
ib hca not gen3. So that explains that.

> hmm, so do you have an sense for what differences are due to
> host differences?  also, what does near/far mean if this is all
> on the same switch?  (simple 36pt switch, or is it a big switch
> with an internal fat tree?)

See my near/far stuff further down.

I was a bit sloppy with my description. By "same switch" I meant that
the tests were done between two nodes on the same 36 port switch (to
make the numbers as meaningful as possible).

Note that it wasn't the same switch for the two tests (two completely
different systems).

Here's a 3rd datapoint. Nehalem (external share pci-e) on connectx2 QDR
via a 36-port InfiniscaleIV:

     bw      lat(any cpu)
-------------------------
  3200       1.27

This time the 3200 was totally expected (pci-e limited). Here one can
speculate a bit regarding current CPUs (like snb/ivb) vs prev. gen with
neh/wsm. It sure was nicer to be an MPI app with 1.27us flat compared
to 0.8-1.6 variance within the node...

> > Of course what people refer to as QDR could also be connectx2 or
> > qlogic/intel. The smaller hit for "far cpu" for qdr could in my
> > case be an improvment from snb to ivb.
> 
> that kinda makes it sound like near/far is just the in-node topology
> (far means +1 QPI hop?)

Yeah a description for my sloppy "near/far cpu" would have been
nice ;-).

The latency in the near cpu column is ib_write_lat running pinned on a
core on the socket with the pci-e of the ib hca. The far number is when
I pinned the processes to a core on the socket that had to go QPI to
reach ib.

/Peter

> thanks, mark hahn.
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin
> Computing To change your subscription (digest mode or unsubscribe)
> visit http://www.beowulf.org/mailman/listinfo/beowulf