[Beowulf] Docker in HPC

[John Hearns]

On 27 November 2013 12:01, Peter Clapham <pc7 at sanger.ac.uk> wrote:

>
> In the bio-informatics arena the local software half life is
> approximately 6-12 months. This, along with the wide range of
> applications in use rapidly creates an environment where users can cross
> link or pick up binaries or libraries that they weren't expecting.
> Rolling containers with predefined environments would not only
> potentially alleviate these potential pitfalls BUT they could provide an
> environment in which data can be re-analysed at a future date in against
> the same pre-defined environment.
>
Peter, that's very very interesting!

I must say that I first started thinking along these lines when working in
High Energy Physics.
Predating Docker by a loooooongggg way, I though VMs.
At the time, the CERN LHC analysis depended on running on Redhat Linux, on
a version which
was going out of date, or had already gone out of date (can;t remember the
version).
Hence the creation of Fermi/CERN Linux.
I thought that instead of physically installing what was soon to be an
out-of-date distro on 1000s' of machines,
wouldn't it be better to have a VM which you were sure would haev all the
libraries and dependencies for your
environment, and then ship it off to remote centres.
Of course this is being done today.


Regarding ISV codes, today you frequently see a vendor supplying an entire
tree of software,
complete with an MPI  plus a version of Python with libraries, plus X, Y,Z.
They do this because they have to ship software which works to users, and
don't want the inevitable
'Oh - my sysadmin installed Python 2.7 and your software does not work with
that' type of query.
I don't see that it is that far a step to packaging up Docker containers.

(Sorry - don't mean to single out Python here - I've just been workign with
it for the past few days).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.beowulf.org/pipermail/beowulf/attachments/20131127/91953223/attachment.html>