[Beowulf] Building new cluster - estimate

[Matt Lawrence]

On Wed, 23 Jul 2008, Ivan Oleynik wrote:

> In principle, we have some experience in building and managing clusters, but
> with 40 node systems it would make sense to get a good cluster integrator to
> do the job. Can people share their recent experiences and recommend reliable
> vendors to deal with?

I suggest that you need a minimum of 16GB/node (2GB/core) and possibly 
32GB/node (4GB/node).

You also should have a dedicated system for managing the cluster, a 
commodity PC with a lot of disk is adequate.  That is the ideal place to 
host a DHCP server, tftp server (for for pxeboot), web/ftp/nfs server for 
installation and local mirror of the various repositories.  This system 
should not be accessable by the users, just the admins.

You will want to set up IPMI on all of the nodes.  You want to avoid 
touching the harware or even going into the machine room whenever 
practical.

Label both ends of each cable with source and destination.  We have been 
using LSL-79 labels from wiremarkersplus.com.  I am very fond of using 
velcro cable ties http://www.fastenation.com/class.php?id=9 for managing 
cables.  I personally bought a spool of the 8" straps for use on my 
previous job and the folks at my current job bought a spool on my 
recommendation.  Very useful.

We have also been buying custom ethernet cables in 1' increments from 
reynco.com.  Very little price difference and ir radically reduced the 
rat's nest of cabling.  Color coding by function/network is handy as well. 
Also, black ethernet cables inside of a black rack within a data center 
that is really well lit can lead to all sorts of headaches.  Release your 
inner anti-goth and use brightly colored cables.

You will want to record the MAC address to physical location mapping 
sooner rather than later.

Install blanking panels to cover unused spaces in the rack.  This will 
improve airflow.

Both Rocks and Warewulf seem to be excellent and have very friendly and 
supportive developers.  Talking to them would probably be a good idea.

A much longer message than I planned, hopefully it is of use.

-- Matt
It's not what I know that counts.
It's what I can remember in time to use.