[Beowulf] What is a "proper" machine count for a cluster

[John Hearns]

Mathew Shember wrote:
> Hello all,
> 
> 
> I am just wondering what is a reasonable amount of computers to allocate 
> for playing around.
> 
> I was thinking 4 pentiums with dual gig processors.  Ram would be a gig 
> or more.
> 
Matthew, if I may put some words in your mouth what I THINK you are 
saying is that you have some surplus desktop machines which you can 
repurpose as a Linux compute cluster (Beowulf).
The list is here to help you, and it will be an excellent 
training/learning experience.
Look at the Cluster Live CD which was recently linked ot on the list.
Designate one machine as the master, and boot the rest off it.
Then look at Rocks and Warewulf.

However, your question is about "what is a reasonable amount of 
computers". That actually depends on your job load and more importantly 
your interconnect.
If you are doing serial jobs which use one machine, your cluster will 
scale, so the amount of useful work you get out goes up with machine 
count. We assume here you have enough RAM per processor.
However, for parallel work you have to have a fast interconnect.
I assume these machines have gigabit? Even if they have,
I would suggest buying some Intel PCI gigabit cards, they are very 
cheap. Put these in and get a decent quality gigabit switch to use as 
the parallel network.

Now comes the pitch.  The "sweet spot" in cluster nodes at the moment 
seems to be a dual socket, dual core Opteron or Intel machine with 2 
gigs of RAM per core, so each box is a 4-way SMP
(I will be flayed alive by the list for such cavalier numbers).
Now, such a box is likely to have oodles (scientific term) more 
processing power than four pentiums.

So I guess what I would say is - follow the learning curve which many 
folks here have done. Build that Beowulf with the four machines, get 
your code running on it. Download the evaluations of commercial 
compilers - that's an important part of Beowulfery too.
But next year, show the cluster to your bosses and get the budget for a 
cluster assembled and delivered by your friendly integrator. The 
"cluster" may be as small as one multi-core box of course.
The learning and real work you do with the four-box cluster will stand 
you in very good stead when speccing out a bigger solution.