networking options

[Matthew Timothy Pratola]

Hi Dean,

Of course each app is different, but i'm interested more in high
throughput/low latency as it cannot harm apps which don't require this,
but is absolutely necessary for apps which do require this level of
performance.  In this regard, myrinet is certainly the one i've heard the
most about, but that doesn't preclude something else which is new and/or
better.

Also its unclear as to the effect of topology on a cluster of this size
(one response i've received indicated that it may not be much of a
factor).

These interconnects are definitely not cheap though :)

Matt

> > Just looking for some information and/or recommendations on the various
> > networking options available for building beowulfs.  The beowulf in
> > question would consist of ~1000 nodes.  I'm aware of myrinet, dolphonics,
> > fiber and of course the ethernet options.  Also of concern is topology.
> > If any of you can expand my knowledge (or point me to articles which will
> > help me) in this regard, it is much appreciated.  Even practical
> > experience in terms of real-world latency, throughput and reliability of
> > these various options would be helful, as well as knowing if they require
> > the use of a proprietary MPI implementation (bad for us).
> >
>
> Its best summarized by "how much money ya got?". To a lesser extent, its
> what you intend on doing with the cluster and how important absolute
> performance is to you. Each app is pretty much different as to its
> networking requirements. The cost of high throughput/low latency
> networking is on the order of $1500 or more per machine (based on a
> recent myrinet purchase). Considering that pretty darn good cluster
> nodes can be had for $2k each... On the other end of the spectrum,
> gigabit ethernet is pretty damn cheap now and helps quite nicely with
> many apps.