[Beowulf] 10 GbE

[Patrick Geoffray]

Hi Igor,

Igor Kozin wrote:
> - Switch latency  (btw, the data sheet says x86 inside);

AFAIK, it is using the 24-port Fulcrum chip, with has a latency of 
~300ns. The 48-port models use multiple crossbars in a Clos, partially 
(S) or fully (SX) connected. I have never benchmarked the 48-port 
version, but the datasheet says 1200ns latency for the 7148SX.

Those are good switches. Flow control works well and latency is good for 
10GbE. However, keep in mind that it would be painful to grow past one 
switch, Ethernet spanning tree and all that.

The x86 is to run the switch management stack (EOS).

> - Netxen NX3-20GxR card vs Intel 10 GbE AD DA card.

FWIW, running MX-over-Ethernet (MXoE) between two Myricom 10G-PCIE-8B 
SFP+ NICs in back-to-back (no switch), I get 2.7 us MPI latency. This is 
500ns more than with QSFP or CX4, but all dense 10GbE switches are using 
SFP+ these days.

Patrick