[Beowulf] Docker in HPC

[Peter Clapham]

On 28/11/13 22:44, Christopher Samuel wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 28/11/13 23:13, Bogdan Costescu wrote:
>
>> f.e. a single version of GROMACS
> But then you run into the issue of different users who say "but I must
> have GROMACS x.y.z", hence why we've already got GROMACS 4.0.7, 4.5.7
> and 4.6.3 installed on a system that was a clean install only a few
> months ago. :-/   At least the person who uses GROMACS 3.x hasn't
> asked for it (yet).
>
> Then there are the ones who have a patch they want, or built with
> different (mutually exclusive) options (for Ray).
>
This is very much our software challenge in a nutshell (ignoring our 
security aspects mentioned).

Person A requires historical versions for their own reasons where as 
person B really needs twig cvs release for their stack to operate. In 
general though the compilation step tends to be the easy part. The 
challenge often comes at run time where various groups configure their 
project area to have something along the lines of:

/software/project-a/bin/mybin -> /software/project-a/mybin-2.6.1/bin/mybin

If 2 projects have similar structures then getting the runtime 
environment right is really rather important and becomes more exciting 
where perl / python modules are involved...

Sadly it is a constant challenge to encourage users to execute their 
jobs via a wrapper script which sets specifically the environment 
required for the jobs to run within. So running jobs via Docker with a 
pre-defined hard configured environment becomes useful from a providence 
perspective. Users would know for example that running against image-a 
will give results a repeatedly both now and in say 3 years time if they 
come to re-analyse their data for paper 2 or follow up checks / 
confirmation or cross data analysis.

I suspect the larger local challenges will be making the integration as 
seem-less as possible for our users on the one hand and reducing the 
maintenance overhead for those responsible for the many applications in 
use and the churn that will develop over time.

Fingers are crossed for now but ultimately time will tell how successful 
things are
Pete






-- 
 The Wellcome Trust Sanger Institute is operated by Genome Research 
 Limited, a charity registered in England with number 1021457 and a 
 company registered in England with number 2742969, whose registered 
 office is 215 Euston Road, London, NW1 2BE.