[Beowulf] onboard Gb lan: any opinion, sugestion or impression?

[Robert G. Brown]

On Tue, 14 Nov 2006, Jones de Andrade wrote:

> Hi all.
>
> Well, I've found some discussions within this list about brands of switches,
> and some about network card brands also.
>
> We came to a question here, in the cluster we are planning: how usefull are
> the, now so common, onboard gigabit networks (even dual networks) that are
> shipped in the motherboards? Great, bood, bad or terrible performance?
> stability issues? Much concern about it's use for clusters? Is there any
> example, or benchmark available of different onboard network chips in any
> (cluster?) application?

The answer is (as you plan the cluster) "it depends".

For some, even for many or most, cluster applications they are just
peachy.  For a grid-style cluster doing mostly embarrassingly parallel
monte carlo simulations with coarse grained access to moderate amounts
of e.g.  shared disk (or other network-accessible resources) and no
interprocessor communications to speak of they can even be "overkill" --
many a cluster has been built that was quite successful using mere
100BT.

For other cluster applications, the IPC demands go up, and you can
easily find that the scaling of your parallel application is limited by
network latency, by network bandwidth, or both.  Or you may be planning
a really BIG cluster, where even fairly coarse grained computations can
get into contention for resources only accessible through network
bottlenecks.  Here TCP/IP over gigabit ethernet is definitely better
than 100 BT ethernet, but still imposes significant overhead in terms of
latency (partly because of the TCP stack -- it isn't all a matter of
ethernet latency per se).

So the usual advice is to FIRST analyze your expected application mix
and try to determine the granularity and communications pattern(s) of
the most demanding of those applications.  If possible, base the
analysis on actual measurements -- doing it "theoretically" is possible
but requires a lot of expertise and knowledge about the network and
communications libraries (e.g. MPI, sockets, native drivers for advanced
networks) and resource bottlenecks (speeds of various ways of providing
disk access over the network, access to a primary server in master-slave
type computations) and even about the (e.g. topological) PATTERNS of
communications that will be used and how well they might be supported in
hardware.

Oh, and don't forget how all this integrates with your proposed node
hardware -- nowadays you can have a one or two gigE interface for one
CPU core, two single core CPUs, one dual core CPU, or a dual core dual
CPU all in OTC hardware -- and then there are the more expensive
alternatives.  So you could be sharing a single communications channel
with four CPU cores each running separate threads (and with possible
memory latency/bandwidth/contention issues as well) or two
communications channels could be available to a single CPU core.  Pretty
wide range to make pronouncements on...

To get you started on the KINDS of things that play into scaling
estimates you can check out my online book at

   http://www.phy.duke.edu/~rgb/Beowulf/beowulf_book.php

and look at the chapters on Amdahl's law and the figures on scaling.
There are also lots of resources that might help at

   www.clustermonkey.net

Hope this helps.

    rgb

>
> Thanks a lot in advance,
>
> Sincerally yours,
>
> Jones
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit 
> http://www.beowulf.org/mailman/listinfo/beowulf
>

-- 
Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu