[Beowulf] Best packaging practices/environment and containers
remy.dernat at univ-montp2.fr
Thu Mar 10 07:25:58 PST 2016
In the first place, my apologies for those thinking that the following
is just a troll.
So, I digging recently the packaging technologies for maintaining many
versions of softwares. Here, we use the old "modules" tool, and we
continue to build everything manually (from source or by building RPMs),
and share applications through nfs.
However, after looking at EasyBuild (
https://hpcugent.github.io/easybuild/ ) and conda (
http://conda.pydata.org/docs/ ), I feel a bit lost...
EasyBuild is well known, I think, but... I found other tools like
Hashdist ( https://hashdist.github.io/ ), Spack (
https://github.com/LLNL/spack ) ( and even an old interesting tool, fpm
which is not designed for the exactly same purpose
On the other hand, now, you have containers (docker, rkt, lxc, etc...),
cgroups, namespaces and singularity (
I am asking myself (and I think I am not alone :) ), if there is still a
good reason to avoid the use of containers (except for security matters)
in our clusters (*) ? It could be much more easy to manage all these things.
Should I investigate deeply on the first solutions or switch directly to
containers ? I already created a docker container once for a user, to
address its specific needs. In that particular case, it required too
many dependencies, and too much of my time to adapt everything on our
What do you suggest ?
(*) MPI/infiniband/gpu performances impacts.
More information about the Beowulf