[Beowulf] What is a "proper" machine count for a cluster

[Ashley Pittman]

On Wed, 2007-03-14 at 11:16 -0700, Mathew Shember wrote:
> I am just wondering what is a reasonable amount of computers to allocate 
> for playing around.

For most things I find a four node/dual cpu system to be adequate, at
least initially.  Inter/inter node comms will naturally happen at
different speeds and at this size you can get an appreciation for both,
as machines get bigger the ratio generally falls which is something to
be aware of.  There is very little you can do on a bigger machine that
you can't do on a machine of this size, the real problems are things you
can do on a small machine which you can't do on a big machine.

In my experience the approximate "sizes" where things become more
interesting and require additional thought are 4, 16, 64, 512 and 2048
where "size" can mean either node count or process(CPU) count depending
on what you are doing.

Suppose you want to do something fairly simple on your cluster, lets say
check for free space in /tmp across the nodes, on a four node cluster
you probably log into each node and type "df", on a 16 node cluster you
type "mpirun df -h /tmp", on a 64 node cluster you'd need to make your
terminal bigger any type "mpirun df -h /tmp | grep -v Filesystem |
sort", at 512 you could set up a cron job to save the free space to a
database somewhere (maybe use ganglia?) and have a CGI script show you a
nice histogram of nodeid vs free space.  At 2048 nodes you have to
ensure the cron jobs are staggered over time and you are also going to
be asking your boss for a bigger monitor!

Most of the scalability problems in code tend to happen at the about the
same sizes as well, at four processes you can get away with running a
surprising amount of your code serially, at 16 this isn't really
possible so you fan everything out to the nodes, at 64 you find yourself
bandwidth limited whilst fanning work out so you start using a tree
based algorithm, at 512 nodes the tree starts breaking down so you make
it wider, perhaps f-nomial and maybe add pipe-lining, at 2048 you have
to start adding synchronisation points, CTS/RTS acknowledgement,
pipe-lining becomes compulsory and you start to notice it's moving
higher up the software stack as well.

In short, four nodes is good.  Get five so you still have four in cases
where one of them is broken.  You will of course have a head node as
well, why not make that that part of the cluster as it's a bit of a
waste not to at this scale but put it at the end rather than the
beginning so it's the last node to get jobs allocated to it.  Six nodes,
twelve cores, 1gig ram per core, should allow you to experiment with
most things.  Of course you are only two nodes away from having 16 cpus
and 16 is the next step up the list, what was your budget again...

Of course when you start doing real things rather than just interesting
things it really does depend, not least on what the real things actually
are.

Ashley,