Experience with Force10 GigE switches in larger clusters ...

[Richard Walsh]

Maurice Hilarius wrote:

>We are setting up a cluster with their E600 in the next couple of weeks.
>I would be happy to share results privately.  Please let me know any particular 
>test/benchmark results desired.  Implementation will be with Tyan S2723 
>( Intel 7501) and dual XEON 2.4/533, 2GB RAM per node, Onboard Intel GigE 
>NIC and SysKonnect SK9821 V2 GigE NICs Software/management: RH7.3, 2.4.20 
>kernel, bproc, Rocks Cluster, etc..

I am interested in price-performance like everybody else ;-).  Latency and
bandwidth numbers measured via PMB (Pallas MPI Benchmark) would be of interest. 
Want to compare these numbers to my Cray X1, T3E, and the Myrinet numbers.

The SysKonnect card should give the best-possible NIC performance revealing 
the best-possible switch performance for the E600.  Knowing the on-board
NIC numbers for comparison would also be of interest. I was told that the switch 
supported jumbo frames. Would be nice to know that you can get them to work
and to see the MTU 9000 bandwidth numbers. I was not given pricing except for 
list for the 24-port GigE card. If you end up computing a rough price/port 
for the E600 that would be useful (for say 128 nodes). 

Performance reports for channel bonding with GigE indicate that you get
less from a bonded pair than from one NIC because packet fragments going
round-robin across the channels cause retransmits. Do you have any experience 
with GIGE channel bonding?  Will you tryit with your on-board and Syskonnet 
cards?  I was told the switch "supports" channel bonding ... not sure
what they meant.  

Anything you find interesting would likely interest me as well.

Regards,


Richard Walsh
#---------------------------------------------------
# Richard Walsh
# Project Manager, Cluster Computing, Computational
#                  Chemistry and Finance
# netASPx, Inc.
# 1200 Washington Ave. So.
# Minneapolis, MN 55415
# VOX:    612-337-3467
# FAX:    612-337-3400
# EMAIL:  rbw at networkcs.com, richard.walsh at netaspx.com
#         rbw at ahpcrc.org
#
#---------------------------------------------------
# "Why waste time learning when ignorace is
#  instantaneous?"                 -Thomas Hobbes
#---------------------------------------------------