[Beowulf] Compute Node OS on Local Disk vs. Ram Disk

[Jon Forrest]

Bogdan Costescu wrote:
> On Sun, 28 Sep 2008, Jon Forrest wrote:
> 
>> There are two philosophies on where a compute node's OS and basic 
>> utilities should be located:
> 
> You forget a NFS-root setup, this doesn't require memory for the RAM 
> disk on which you later mount NFS dirs.

You're right. I should have mentioned it. It does fall into the
non-local install classification, although it has fewer of
the problems.

> I prefer to look at the nodes as disposable, instead of "let's keep the 
> node up as long as possible", so I usually don't modify a running 
> system. Instead I modify the node "image" and reboot the nodes after the 
> current jobs finish - this is easy to do when using a queueing system 
> and is easy to hide from users when the typical jobs are longer than the 
> reboot time.

There are several ways to modify a running system, some more
dangerous than others. Probably the most dangerous is modifying
shared libraries and executables. Probably the least dangerous
is adding new files of any type. Most of the modifications I've
had to make involve editing text files, which hasn't caused any problems.

The trouble with rebooting nodes is that this takes human energy.
It's easier to keep nodes up as long possible, although this is not
a significant issue provided that the reboots are done as innocuously as
you describe.

>> However, on a modern multicore compute node this might just be a few 
>> percent of the total RAM on the node.
> 
> This also depends on how much of the distribution you keep as part of 
> the node "image" and how you place the application software. 

True. Some people, like the BusyBox people, have put a lot
of energy into coming up with "tiny versions of many common
UNIX utilities" in order to save memory in embedded systems.

>> Approach #2 requires much less time when a node is installed,
>> and a little less time when a node is booted.
> 
> I don't agree with you here as you probably have in mind a 
> kickstart-based install for approach #1 running upon each node boot.

True.

> I use for a long time a different approach - the node "image" is copied 
> via rsync at boot time; the long waiting time for installing the RPMs 
> and running whatever configuration scripts happens only once when the 
> node "image" is prepared, the nodes only copy it - and it's a full copy 
> only the first time they boot with a new disk, afterwards it's the rsync 
> magic which makes it finish within seconds while making it look like a 
> new disk :-)

This is a good idea. Can you write more about this?

Cordially,


-- 
Jon Forrest
Research Computing Support
College of Chemistry
173 Tan Hall
University of California Berkeley
Berkeley, CA
94720-1460
510-643-1032
jlforrest at berkeley.edu