[Beowulf] Compute Node OS on Local Disk vs. Ram Disk

[Bogdan Costescu]

On Sun, 28 Sep 2008, Jon Forrest wrote:

> There are two philosophies on where a compute node's OS and basic 
> utilities should be located:

You forget a NFS-root setup, this doesn't require memory for the RAM 
disk on which you later mount NFS dirs.

> In both cases it's important to remember to make any changes to this 
> distribution rather than just using "pdsh" or "tentakel" to 
> dynamically modify a compute node. This is so that the next time the 
> compute node boots, it gets the uptodate distribution.

I prefer to look at the nodes as disposable, instead of "let's keep 
the node up as long as possible", so I usually don't modify a running 
system. Instead I modify the node "image" and reboot the nodes after 
the current jobs finish - this is easy to do when using a queueing 
system and is easy to hide from users when the typical jobs are longer 
than the reboot time.

> However, on a modern multicore compute node this might just be a few 
> percent of the total RAM on the node.

This also depends on how much of the distribution you keep as part of 
the node "image" and how you place the application software. It's 
often the case that the application software is distributed to the 
nodes from a cluster-wide FS, either from a directory holding software 
only or from the user's home dir; extending this to also include most 
of the libraries needed by the application software (f.e. fftw) means 
that the node "image" can be made very small without putting any part 
of the distribution on NFS (I know that some people totally dislike 
system utilities coming from a NFS mounted directory or depending on 
libs coming from one).

> Approach #2 requires much less time when a node is installed,
> and a little less time when a node is booted.

I don't agree with you here as you probably have in mind a 
kickstart-based install for approach #1 running upon each node boot. I 
use for a long time a different approach - the node "image" is copied 
via rsync at boot time; the long waiting time for installing the RPMs 
and running whatever configuration scripts happens only once when the 
node "image" is prepared, the nodes only copy it - and it's a full 
copy only the first time they boot with a new disk, afterwards it's 
the rsync magic which makes it finish within seconds while making it 
look like a new disk :-)

-- 
Bogdan Costescu

IWR, University of Heidelberg, INF 368, D-69120 Heidelberg, Germany
Phone: +49 6221 54 8869/8240, Fax: +49 6221 54 8868/8850
E-mail: bogdan.costescu at iwr.uni-heidelberg.de