Myrinet vs FastEthernet

[Patrick GEOFFRAY]

Martin Kunz wrote:

> What are your impressions concerning Myrinet compared to Fast Ethernet?
> Has anybody ever used both for some time? Is it worth the added cost,

Hi,

I have a paper about it that i need to submit somewhere. I have
compared the performance of some typical applications with Fast
Ethernet and Myrinet on a 8 Dual PIII 550 nodes cluster (so
nothing about scalability). I was using MPICH-p4 (with
shared-memory) on FE and MPICH-GM (with shared-memory) on Myrinet.
The applications were Scalapack LU (Linpack), NAS CG (Sparse), MM5
(Weather forecast) and a discrete event simulation (ATM
simulation).

For Scalapack LU, the gain with Myrinet is 17% using one processor
per node(8 nodes), 23% using 2 processors per node (4 nodes). It's
clear that the gain with a dense matrix computation like Linpack
does not justify the difference of price, it's CPU bounded,
minimum of communication.
With CG, the gain is 58% one CPU per node, 66% with 2 procs per
node. It's much more interesting. As the real world application
are very often sparse matrix computations, it can be a good
investment for 60ish% of gain.
For MM5, it's disappointing : 15% one proc per node, 10% with 2
procs per node. The runtime (RSL) seems to be expensive (on top of
MPI). The code itself seems to try to overlapp, so the SMP mode is
more expensive that the single CPU mode. Greg Lindhal uses Myrinet
for this code if i remember well, for scalability issues i guess
(Greg ?).
Finally, the discrete event simulation (TimeWarp-like) is a
perfect example : a lot (A LOT) of small messages (tiny messages)
to simulate a very large ATM network. The gain is 2450% with 8
processors on 8 nodes and 1200% with 8 processors on 4 nodes.
Latency is the big parameter, so Myrinet is the choice.

So you can see that it can be a big jump, or a small improvement,
it depends of your application as usual :-)

> (Are there any benchmark results out there? I saw the netperf site
> and had the impression that Myrinet mainly improves the *speed* by
> about a factor four, but doesn't do much to the *latency* for that
Netperf is IP performance. The part of the IP latency independant
of the network itself is quite high (kernel and protocol cost). 
 
> Any experience with system stability and general convenience/handling
> of the hardware and drivers?
I feel Myrinet hardware very stable and the drivers and MPI have
improved a lot (and will improve much more in the near future but
i am unpartial here...)
 
> How about the new generation (Myrinet 2000 if I remember correctly)?
Myrinet 2000 is a technology push : The links are upgraded to 2+2
Gb/s and the Lanai (processor on the NIC) is now Lanai9 (132 MHz
and soon 200 MHz). The clock is much faster than the Lanai7 (66
MHz) : The GM latency is around 9 us, the GM bandwidth is 225 MB/s
on a PCI 64 bits/66 Mhz (x86 Motherboard).
A new serial link will be ready in the Fall, and optical fiber
NICs also.

> Any crucial points one shouldn't neglect? Other comments?
The scalability of the switches : Up to 64 ports with a full
bisection ($40 000), 128 ports with 2 switches 64-ports ($80 000).
The overall price is still high, i hope that the decreasing price
of GigEthernet will decrease the price of Myrinet, as Myricom
keeps the same total price per port than GigEthernet :-)

> (i.e. I'm interested in whatever scraps of knowledge concerning
> Myrinet are floating around the community... :-) )
There will be a Myrinet Users Group (MUG) in Lyon, France at the
end of September 2000. The food is very nice in Lyon, so you
should cross the Channel :-)

Hope it gives you some inputs.

Patrick Geoffray
---
Aerospatiale Matra - Sycomore
Universite Lyon I - RESAM
http://lhpca.univ-lyon1.fr