[Beowulf] 10G networking?
deadline at eadline.org
Fri Jan 23 08:02:33 PST 2015
I have been pleasantly surprised by the performance of Chelsio.
My tests are very small scale (no switch) and the hardware I'm
testing is for my Limulus desk side cluster.
I ran netpipe TCP and got latencies of 9us back-to-back (SFP+)
Hopping through a two port adapter I got 18 us.
I then tried openMX. On omx_perf I got the following latencies,
0 hops 5.503 us (Head to n0)
1 hops 10.438 us (Head to n1)
2 hops 15.829 us (Head to n2)
The network is a simple linear connection. looks like this
head --- n0 --- n1 --- n2
I ran HPL and the results went from 395.2 GFLOPS with 1 GbE to
567.4 GFLOPS with 10 GbE. I saw similar improvements in NAS latency
sensitie codes. The white paper with the data is being prepared
These test were for Chelsio T420-SO-CR (low cost stripped down
2 port NICS) I just received a 4-port 10 GbE card that I'm going to
test with both the Chelsio T520-SO-CR (2 port) and the Mellanox MCX311A
(1 port) The idea is 4-ports on the head node connected to each of
three compute nodes, with the remaining port open for expansion.
For my purposes, small low cost desk-side HPC, 10-GbE is
an excellent option.
> Hi all,
> I'd appreciate any comments about the state of 10G as a reasonable
> cluster network. Have you done any recent work on 10G performance?
> shows some excellent evidence-based work on kernel paths, but it
> seems focused on bulk/saturation, rather than single-packet latency.
> (not that 10G MPI wouldn't be saturated in a lot of cases vs FDR!)
> I also read some comments recently about improvements that particularly
> benefit certain Intel adapters - anyone have comments on Intel vs
> Broadcom (vs Mellanox, I suppose).
> It seems like 10gT is on the cusp of real volume-type prices: I saw a
> quote today for a major-vendor 24pt switch for something like $140/port
> after educational discounts. I know 10gT is somewhat higher latency
> than other PHYs - or is it worth sticking to SFP+? Any comments on
> latency properties of various switches? Any nice results of recent
> hardware and OpenMX?
> much appreciated!
> -mark hahn
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> Mailscanner: Clean
More information about the Beowulf