[Beowulf] 10G networking?

[Joe Landman]

On 01/23/2015 01:36 AM, Mark Hahn wrote:
> Hi all,
> I'd appreciate any comments about the state of 10G as a reasonable
> cluster network.  Have you done any recent work on 10G performance?
>
> https://lwn.net/Articles/629155/
>
> shows some excellent evidence-based work on kernel paths, but it seems 
> focused on bulk/saturation, rather than single-packet latency.
> (not that 10G MPI wouldn't be saturated in a lot of cases vs FDR!)
>
> I also read some comments recently about improvements that particularly
> benefit certain Intel adapters - anyone have comments on Intel vs 
> Broadcom (vs Mellanox, I suppose).
>
> It seems like 10gT is on the cusp of real volume-type prices: I saw a 
> quote today for a major-vendor 24pt switch for something like $140/port
> after educational discounts.  I know 10gT is somewhat higher latency
> than other PHYs - or is it worth sticking to SFP+?  Any comments on 
> latency properties of various switches?  Any nice results of recent 
> hardware and OpenMX?

I've "this is the year of 10GbE" for something like the last 6 years, 
and its not been generally true for HPC.  We're at 40GbE now, and 100GbE 
isn't far off (think months).  10GbE should be getting very cheap now, 
but its not at copper RJ45 pricing.  It is landed on motherboards, at 
least a few.

We've had very good results with 10GbE from an aggregate bandwidth 
scenario.   Latency is hovering in the low double digits microseconds 
(two way, 64 byte packets).  One way latencies in the mid single digits.

If you have extremely latency sensitive stuff, Infiniband is better.  If 
you are mixed, 40 GbE is similar latency (low single digit microseconds, 
and 3.2-ish GB/s) is much easier to configure (unless you do RoCE).

These days, you should choose between 40GbE and FDR IB (EDR is out real 
soon now) for HPC.

-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics, Inc.
e: landman at scalableinformatics.com
w: http://scalableinformatics.com
t: @scalableinfo
p: +1 734 786 8423 x121
c: +1 734 612 4615