[Beowulf] Confused over the term cluster

[Robert G. Brown]

On Fri, 28 Mar 2008, Kalpana Kanthasamy wrote:

> Hi,
>
> I am kinda confused with term cluster and now am sure if this where I can
> post my question. What exactly can be defined as a cluster ?, We have a 32
> node cluster in our lab. and the number 32 here is referred to 32 separate
> machines ( each with its own processor ). What about the new machines
> like,
> dual core processors or the quad core processors, are those considered
> clusters also. For example, should we refer to a quad core processor as a
> cluster of 4 processors. Hope someone out there could explain. Thanks

A "cluster" generally refers to a cluster of separate machines using a
network as an interconnect.  A machine with multiple processors (or
multiple cores, or multiple multiple core processors) isn't a cluster
per se UNLESS it is internally architected on top of what amounts to a
network.  There have been supercomputer designs that are -- IBM's SP
series for example, or the CM series in the old days -- and "bus"
oriented designs that put the various processors on a parallel structure
that isn't precisely "a network" as well.

Nowadays the nature of single motherboards has become even more
architecturally blurred as e.g.  Hypertransport has features that look
like like a network as much as a bus, so one could argue that AMD
systems with many cores that use it are "a cluster" from a certain
architectural point of view.

However, on THIS list, the issue of what is or isn't "a cluster node"
has to be taken in context.  A "node" (I think) most commonly refers to
a single physical box (motherboard) with one or more network
connections.  By common usage, we don't consider a single multicore box
to be a cluster per se, although we certainly wouldn't care if somebody
did as the issues facing the user of such a box are ALMOST ALL the same
issues that would need to be faced for multiple node clusters.

Ultimately, we view "a cluster" as ultimately being a collection of CPU
cores that can be in two or more physical boxes containing UP systems
(one core, one processor per motherboard/box) all the way up to 16-way
SMP (four quad cores per motherboard/box) or whatever the current
max-pack boxes are, connected by some sort of COTS network, which
historically may well be not terribly OTS, unless you consider
single-source direct purchases to be "shelves".

The point is that PROGRAMMING a cluster efficiently requires viewing the
entire collection of available processor cores as the resource, with a
differential matrix of interprocessor communications (IPCs) with very
specific costs/speeds (latencies and bandwidths) associated with each
link, with nonlinear aspects of communication such as just how IPCs
onboard work vs IPCs between nodes all accounted for.  Multicores and
large SMP systems have actually rather complicated cluster design,
because other bottlenecks emerge that are strongly differential for
various cluster purposes, such as how memory bandwidth and latency fares
when many cores are trying to access a large shared memory resource with
or without locking, processor affinity, NUMA, and more.

With a pile of UP PC's on a single network, there were really just a
handful of really important rates -- latency and bw to main memory in
streaming and non-streaming (cache defeating) memory access patterns,
raw CPU clock and its relation to various integer and floating point
rates, and latency and bw between nodes.  Now one has to worry about
core to core on the same die, core to core on different dies, core to
"local" memory with affinity, core to memory that "belongs" to other
cores or dies -- quite a range of possible bottlenecking rates between
cores on ONE PHYSICAL MOTHERBOARD -- and THEN one still has to worry
about the internode network and network-based IPCs, which in turn
depends on how the cores write to e.g. shared network devices!

My advice is to START with the simple model for a cluster -- one CPU
core/node, one network -- to get to where you understand the BASIC
principles of bottlenecks, the differential effect of latency and
bandwidth for different kinds of parallel code, and so on.  When you've
mastered the ideas at this level (with the help of the list as needed),
THEN this list is an excellent resource for learning more advanced
topics as well.  The ongoing discussion of NUMA and processor affinity
has been most instructive, for example, although I lack the time to
really experiment with a lot of what was discussed.

    rgb

-- 
Robert G. Brown                            Phone(cell): 1-919-280-8443
Duke University Physics Dept, Box 90305
Durham, N.C. 27708-0305
Web: http://www.phy.duke.edu/~rgb
Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php
Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977