[Beowulf] Best packaging practices/environment and containers

[Remy Dernat]

Hi,

In the first place, my apologies for those thinking that the following 
is just a troll.

So, I digging recently the packaging technologies for maintaining many 
versions of softwares. Here, we use the old "modules" tool, and we 
continue to build everything manually (from source or by building RPMs), 
and share applications through nfs.

However, after looking at EasyBuild ( 
https://hpcugent.github.io/easybuild/ ) and conda ( 
http://conda.pydata.org/docs/ ), I feel a bit lost...
EasyBuild is well known, I think, but... I found other tools like 
Hashdist ( https://hashdist.github.io/ ), Spack ( 
https://github.com/LLNL/spack ) ( and even an old interesting tool, fpm 
which is not designed for the exactly same purpose 
https://github.com/jordansissel/fpm )...

On the other hand, now, you have containers (docker, rkt, lxc, etc...), 
cgroups, namespaces and singularity ( 
http://gmkurtzer.github.io/singularity/ ).

I am asking myself (and I think I am not alone :) ), if there is still a 
good reason to avoid the use of containers (except for security matters) 
in our clusters (*) ? It could be much more easy to manage all these things.

Should I investigate deeply on the first solutions or switch directly to 
containers ? I already created a docker container once for a user, to 
address its specific needs. In that particular case, it required too 
many dependencies, and too much of my time to adapt everything on our 
cluster.

What do you suggest ?

Best,
Remy

(*) MPI/infiniband/gpu performances impacts.

-- 
Rémy Dernat
Ingénieur d'Etudes
MBB/ISE-M