[Beowulf] 10G networking?

[Mark Hahn]

Hi all,
I'd appreciate any comments about the state of 10G as a reasonable
cluster network.  Have you done any recent work on 10G performance?

https://lwn.net/Articles/629155/

shows some excellent evidence-based work on kernel paths, but it 
seems focused on bulk/saturation, rather than single-packet latency.
(not that 10G MPI wouldn't be saturated in a lot of cases vs FDR!)

I also read some comments recently about improvements that particularly
benefit certain Intel adapters - anyone have comments on Intel vs 
Broadcom (vs Mellanox, I suppose).

It seems like 10gT is on the cusp of real volume-type prices: I saw a 
quote today for a major-vendor 24pt switch for something like $140/port
after educational discounts.  I know 10gT is somewhat higher latency
than other PHYs - or is it worth sticking to SFP+?  Any comments on 
latency properties of various switches?  Any nice results of recent 
hardware and OpenMX?

much appreciated!
-mark hahn