[Beowulf] Gentoo in the HPC environment

[Olli-Pekka Lehto]

On 01 Jul 2014, at 03:27, Christopher Samuel <samuel at unimelb.edu.au> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> On 26/06/14 05:08, Kilian Cavalotti wrote:
> 
>> You can just use a very minimal OS on your compute nodes, then
>> compile and install all the user facing bits in a shared location.
>> You hand an environment modules system to the users and off they
>> go.
> 
> That's pretty much exactly what I've done for the last 8 years or so,
> and it works really well.
> 
> Our current incarnation is to netboot a minimal RHEL image on diskless
> nodes (managed via xCAT) and then all the applications are in
> /usr/local including various Perl, Python and R versions with their
> associated module hell.  At the moment we've got about 638 different
> modules across 3 different architectures.
> 
> The main reason for RHEL on compute nodes is to keep hardware support
> people happy, most vendors take the Blue Brothers approach to Linux
> ("we support both kinds of Linux, RHEL *and* SLES”).

This is especially true in an environment where you have, or plan to
have parallel filesystems, InfiniBand, accelerators and libraries tightly coupled
to them (say, HDF5+Lustre w/RDMA or Tesla+MPI w/GPUDirect). In these cases
you end up with a very small subset of “common denominator” distros+versions 
that you can deal with. RHEL/CentOS 6.[N-1] (where N is the latest version) seems 
to be a relatively safe bet. 

It is possible to build things on your own but you easily end up in very uncharted 
territory. In this kind of an environment implementing Docker and/or VMs is also 
quite a bit more challenging. 

On a sidenote, EasyBuild was mentioned here earlier and it seems that they 
have IMO right idea of simplifying installs of the type of environment that Chris
was describing and many (including us) seem to hold as best practice. Just heard 
about it at ISC and haven’t had the chance to test it in practice yet, though. 

O-P 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4512 bytes
Desc: not available
URL: <http://www.beowulf.org/pipermail/beowulf/attachments/20140701/1402132a/attachment.bin>