[Beowulf] The move to gigabit - technical questions
Robert G. Brown
rgb at phy.duke.edu
Wed Mar 16 06:41:40 PST 2005
On Tue, 15 Mar 2005, Vincent Diepeveen wrote:
> At 05:41 PM 3/14/2005 -0500, Glen Gardner wrote:
> >Gigabit will be a little faster than 100Mbit on a small cluster, but not
> >a lot.
> What is 'not a lot'.
> I would guess it's factor 10 faster in bandwidth?
(Maybe, you don't get QUITE 100% of the raw clock advantage in all
applications on all hardware, Vincent;-). However, for most
applications on most hardware you >>should<< get a signficant advantage
-- 80-95% of 10x, or 8-9.5x. Not a just "a little".
A really, really cheap switch might have problems with bisection
bandwidth and chop this down for simultaneous flat-out bidirectional
data streams, but relatively few parallel applications engage in
flat-out bidirectional communications. Even if it does, your problem is
more likely to be with resource contention (e.g. two hosts trying to
talk to a third at the same time) than it is with actual bandwidth
oversubscription. This is what Vincent is suggesting that you look into
(or let us look into:-) below.
If your particular usage pattern does create resource contention, then
you might well need to either hand-optimize the pattern to avoid
saturating your cheap hardware, create a network with cheap components
that effectively breaks up the pathological communications pattern
(which it sounds like is what you actually did) or buy better hardware
(either better gigE switches or a "real" HPC network).
However you shouldn't really trash gigE itself -- it isn't at fault and
your results aren't typical.
> >I ended up using 5 cheap gigabit switches to make a gigabit concentrator
> >for my 12 node cluster.
> >It eliminated the tendency for the network to saturate under a heavy load.
> Very interesting, can you post a connection scheme and routing table?
> >It also let me use gigabit network cards in my I/O node and controlling
> >node with a small improvement in file I/O.
> Streaming i/o or random access?
> cheapo disk arrays get what is it, 400MB/s handsdown or so?
> that's raid5 readspeed, plenty security at a raid5 array.
> >The compute nodes remaind with 100 Mbit to conserve power. The setup
> >works rather nicely.
> what type of software do you run at it,
> embarrassingly parallel software?
> >Vincent Diepeveen wrote:
> >>Good evening,
> >>It's interesting to investigate what gigabit can do for small home clusters.
> >>Any latency oriented approach is doomed to fail obviously at gigabit. But
> >>they're cheap. For 40 euro i see several getting offered already.
> >>First important question is of course how much system time those NIC's eat
> >>when fully loading their bandwidth.
> >>Example, i have an old dual k7 here with pci 2.2 (32 bits 33Mhz).
> >>Suppose i put a gigabit card in it.
> >>In say 6 messages a second i ship 8MB data at a time. Ship and send in turn.
> >>So it ships a packet of 8MB, then receives a packet of 8MB.
> >>Other than the cost of the thread to store the packet to RAM, does such a
> >>card in any way stop or block the cpu's which are 100% loaded with
> >>searching software (my chessprogram diep in this case)?
> >>What penalty other than that thread handling the message is there in terms
> >>of system time reduction to the 2 processes searching?
> >>Oh btw, i assume that gigabit can handle 48MB/s user data a second?
> >>Beowulf mailing list, Beowulf at beowulf.org
> >>To change your subscription (digest mode or unsubscribe) visit
> >Glen E. Gardner, Jr.
> >AMSAT MEMBER 10593
> >Glen.Gardner at verizon.net
Robert G. Brown http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb at phy.duke.edu
More information about the Beowulf