[Beowulf] Which do you prefer local disk installed OS or NFS rooted?

[Kimmo Kallio]

Hi,

>Do you a. Install your OS on each local disk?(using whatever method,
>kickstart,systemimager, etc) or b. do you nfsroot and then use the local
>disk space as scratch and swap.

A bit different solution here: network bootable with local system disk.  
First small ramdisk is loaded over the network to get things running:  
Check/create partitions and filesystems and then copy root fs to local
drive and do pivot_root and abandon the ramdisk. Then copy all the rest
(tar.gz files) over NFS using or use cached copy on the local disk (using
checksums to verify that things are what they are supposed to be). /usr
filesystem (large, ours is almost 3G) is handled a bit differently,
version/checksum is processed first and (incremental) updates are done
from a compressed isofs image. Then some extra tricks to preserve
persistent data (/var/logs etc.).

> What benefits and drawbacks have all of you seen by doing either?

This has the performance and server independence benefits of local disk
vs. low maintenance of network bootable system (avoids local disk
corruption and system "erosion" over time). We use the system with the
same actualy system image on desktops and non-compute servers, that
explains why the system image is so large... but it's a single system
image for over 350 of our machines with plenty of room for growth.

I can see the main drawback is that this is a complicated setup. We have
a toolkit for this and I'm hoping to release it for others to download but
it still needs still quite bit of work before that.

Hope this helps,

Mr Kimmo Kallio
System Manager
European Bioinformatics Institute
Email: kallio at ebi.ac.uk