[Fwd: Re: 32-port gigabit switch]

[Dave Vehrs]

On Thu, 2003-03-06 at 14:46, Jeffrey B. Layton wrote:
> Dave Vehrs wrote:
> 
> >Rogerio,
> >
> >If you do the math, you will find that Myrinet quickly becomes less
> >expensive than Gigabit as it scales to large node counts.  While Gigabit
> >PCI cards are inexpensive, Gigabit switches are not.  In our experience,
> >at ~64 nodes and above, Myrinet will be cheaper than Gigabit.
> >
> 
> In my experience the sweet-spot is about 128-256 nodes.
> Hopefully IB will change everything ;)
> 
> 
> >
> >Additionally the latency on Gigabit is simply to high for large
> >clusters.  Remember on clusters latency is more important than
> >throughput. 
> >
> 
> Bzzt. Wrong. Dead Wrong. "It depends" is the correct answer. (Don't
> pass GO, don't collect $200).

OK show me how to do 512 nodes or higher with Gigabit and reasonable
latencies (i.e "for large clusters").  

The grid stuff, like Mark Hahn described, only works on small clusters
and increases the hardware requirements by a huge factor (2-3 cards per
system and multiple switches).  Plus the additional admin time to
configure and maintain all the networking options.  

Its a nightmare.  Go with cheap, used Myrinet and save yourself a lot of
heartache.

> Jeff
> 
> 

-- 
David E Vehrs, System Engineer		Aspen Systems
davidv at aspsys.com			3900 Youngfield Street
Tel: +01 303 431 4606			Wheat Ridge CO 80033, USA
Fax: +01 303 431 7196			http://www.aspsys.com