[Beowulf] 10G networking?

[Gavin W. Burris]

Hi, Mark.

We just launched a new cluster, all with 10gigE.  What kind of numbers
are you looking for?  Most of our workload is data analysis or
iterations of a simulation with commercial software, though we do
validate and support Open MPI.  We just opened the doors and are still
working the kinks out, but maybe I can find time to run a benchmark
sometime soon.  My only benchmarks so far have been storage with IOzone
and single node with Intel LINPACK.  What benchmark do you suggest?

Cheers.

On 01:36AM Fri 01/23/15 -0500, Mark Hahn wrote:
> Hi all,
> I'd appreciate any comments about the state of 10G as a reasonable
> cluster network.  Have you done any recent work on 10G performance?
> 
> https://lwn.net/Articles/629155/
> 
> shows some excellent evidence-based work on kernel paths, but it seems
> focused on bulk/saturation, rather than single-packet latency.
> (not that 10G MPI wouldn't be saturated in a lot of cases vs FDR!)
> 
> I also read some comments recently about improvements that particularly
> benefit certain Intel adapters - anyone have comments on Intel vs Broadcom
> (vs Mellanox, I suppose).
> 
> It seems like 10gT is on the cusp of real volume-type prices: I saw a quote
> today for a major-vendor 24pt switch for something like $140/port
> after educational discounts.  I know 10gT is somewhat higher latency
> than other PHYs - or is it worth sticking to SFP+?  Any comments on latency
> properties of various switches?  Any nice results of recent hardware and
> OpenMX?
> 
> much appreciated!
> -mark hahn
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf

-- 
Gavin W. Burris
Senior Project Leader for Research Computing
The Wharton School
University of Pennsylvania