[Beowulf] Compute Node OS on Local Disk vs. Ram Disk
Bogdan.Costescu at iwr.uni-heidelberg.de
Tue Sep 30 02:09:29 PDT 2008
On Sun, 28 Sep 2008, Jon Forrest wrote:
> There are two philosophies on where a compute node's OS and basic
> utilities should be located:
You forget a NFS-root setup, this doesn't require memory for the RAM
disk on which you later mount NFS dirs.
> In both cases it's important to remember to make any changes to this
> distribution rather than just using "pdsh" or "tentakel" to
> dynamically modify a compute node. This is so that the next time the
> compute node boots, it gets the uptodate distribution.
I prefer to look at the nodes as disposable, instead of "let's keep
the node up as long as possible", so I usually don't modify a running
system. Instead I modify the node "image" and reboot the nodes after
the current jobs finish - this is easy to do when using a queueing
system and is easy to hide from users when the typical jobs are longer
than the reboot time.
> However, on a modern multicore compute node this might just be a few
> percent of the total RAM on the node.
This also depends on how much of the distribution you keep as part of
the node "image" and how you place the application software. It's
often the case that the application software is distributed to the
nodes from a cluster-wide FS, either from a directory holding software
only or from the user's home dir; extending this to also include most
of the libraries needed by the application software (f.e. fftw) means
that the node "image" can be made very small without putting any part
of the distribution on NFS (I know that some people totally dislike
system utilities coming from a NFS mounted directory or depending on
libs coming from one).
> Approach #2 requires much less time when a node is installed,
> and a little less time when a node is booted.
I don't agree with you here as you probably have in mind a
kickstart-based install for approach #1 running upon each node boot. I
use for a long time a different approach - the node "image" is copied
via rsync at boot time; the long waiting time for installing the RPMs
and running whatever configuration scripts happens only once when the
node "image" is prepared, the nodes only copy it - and it's a full
copy only the first time they boot with a new disk, afterwards it's
the rsync magic which makes it finish within seconds while making it
look like a new disk :-)
IWR, University of Heidelberg, INF 368, D-69120 Heidelberg, Germany
Phone: +49 6221 54 8869/8240, Fax: +49 6221 54 8868/8850
E-mail: bogdan.costescu at iwr.uni-heidelberg.de
More information about the Beowulf