[Beowulf] Best packaging practices/environment and containers

[Olli-Pekka Lehto]

IMO containers and the build tools you listed address two different layers 
and can really complement each other: Build management tools handle the 
build rules and containers, well, contain the build. :)

With a configuration management tool like EasyBuild one can define the
general build rules and target containers/VMs/bare metal with practically
the same rule files. 

Even with the advent of containers we will still have bare metal and VM 
deployments for the forseaable future. Thus writing configuration rules 
in a format that's not tied to a specific platform technology (like, for 
example, a Dockerfile) increases the reusability of the rules. 

O-P

----- Original Message -----
> From: "Remy Dernat" <remy.dernat at univ-montp2.fr>
> To: beowulf at beowulf.org
> Sent: Thursday, 10 March, 2016 17:25:58
> Subject: [Beowulf] Best packaging practices/environment and containers

> Hi,
> 
> In the first place, my apologies for those thinking that the following
> is just a troll.
> 
> So, I digging recently the packaging technologies for maintaining many
> versions of softwares. Here, we use the old "modules" tool, and we
> continue to build everything manually (from source or by building RPMs),
> and share applications through nfs.
> 
> However, after looking at EasyBuild (
> https://hpcugent.github.io/easybuild/ ) and conda (
> http://conda.pydata.org/docs/ ), I feel a bit lost...
> EasyBuild is well known, I think, but... I found other tools like
> Hashdist ( https://hashdist.github.io/ ), Spack (
> https://github.com/LLNL/spack ) ( and even an old interesting tool, fpm
> which is not designed for the exactly same purpose
> https://github.com/jordansissel/fpm )...
> 
> On the other hand, now, you have containers (docker, rkt, lxc, etc...),
> cgroups, namespaces and singularity (
> http://gmkurtzer.github.io/singularity/ ).
> 
> I am asking myself (and I think I am not alone :) ), if there is still a
> good reason to avoid the use of containers (except for security matters)
> in our clusters (*) ? It could be much more easy to manage all these things.
> 
> Should I investigate deeply on the first solutions or switch directly to
> containers ? I already created a docker container once for a user, to
> address its specific needs. In that particular case, it required too
> many dependencies, and too much of my time to adapt everything on our
> cluster.
> 
> What do you suggest ?
> 
> Best,
> Remy
> 
> (*) MPI/infiniband/gpu performances impacts.
> 
> --
> Rémy Dernat
> Ingénieur d'Etudes
> MBB/ISE-M
> 
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf