Giganet vs. Myrinet vs. gigabit

[Jarrod Smith]

> 
> Hi all,
> 
> Anyone have experience with these high speed network devices? I'm starting out
> with a 32 node cluster doing bioinformatics. The jobs will be high I/O probably
> through NFS, thus the need for some high speed connectivity between the nodes.
> I'm looking for stability and possibly cross platform use since the NFS server
> may have to go to either our large SGI or Sun server.
> 

I've benchmarked Linux NFS3 over myrinet on 64bit/66MHz PCI of the
Serverworks HE-SL chipset (running some flavor of the 2.4 kernel).  
Without any special tweaks to the TCP stack, it maxed out at around
35MB/sec for transfers between a single client and server, 70MB/sec
aggregate with two clients connecting to one server, and 100MB/sec
aggregate between 3 or more clients and a single server.

This is somewhat impressive, but not really real-world performance (for
us) because to make sure we were testing the interconnect and not the
client/server disks, we had to be sure the data was already buffered on
the server, and we wrote it to /dev/null on the clients.  Otherwise the
performance was between 1/3 to 1/10th this amount.

Basically, Myrinet is so fast that you begin exposing the limititions of
disk I/O over the SCSI/IDE bus on both the clients and server.  In a more
real-world situation, the Myrinet would be idling most of the time waiting
on disk I/O unless perhaps you had striped RAID on both ends, or lots
(10+) clients talking simultaneously to one very-high-performance disk
array on the server.  And remember that the more simultaneous clients you
have connecting, the more random the reads/writes will become so the last
point might even be too optimistic.  I don't have much experience with
truly high-performance disk array I/O.

Anyway, Myrinet is nice and fast, but all of a sudden you begin exposing
bottlenecks in the system that didn't used to be bottlenecks.  In many
cases it may be overkill if all you plan to do is use it for NFS.

-- 
Jarrod A. Smith, Ph.D.
Asst. Director, Center for Structural Biology
Research Asst. Professor, Biochemistry
Vanderbilt University

On Thu, 18 Oct 2001, farul ghazali wrote:

> TIA
> 
> 
> __________________________________________________
> Do You Yahoo!?
> Make a great connection at Yahoo! Personals.
> http://personals.yahoo.com
> 
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
>