[Beowulf] Docker in HPC

[Lux, Jim (337C)]

From: John Hearns <hearnsj at googlemail.com<mailto:hearnsj at googlemail.com>>
Date: Wednesday, November 27, 2013 4:35 AM
To: "beowulf at beowulf.org<mailto:beowulf at beowulf.org>" <beowulf at beowulf.org<mailto:beowulf at beowulf.org>>
Subject: Re: [Beowulf] Docker in HPC



On 27 November 2013 12:29, Tim Cutts <tjrc at sanger.ac.uk<mailto:tjrc at sanger.ac.uk>> wrote:
Yes, Pete, Guy and I have been debating this stuff for some time, together with some of our informatics coders.
 Should virtualisation ever also be necessary (for example to ship ... to another site to analyse some of their data)

Well why not just clone your informatics coders?
I'm sure you have all the necessary technology at the Sanger Centre - line up your coders, take a DNA sample,
clone them and send off the clones on low cost airline flights to where they are needed.

I suppose the nine-month lead time might be a bit problematic from a project planning point of view.

---

I took a project management class on task planning, and we worked in fungible work months. (I think the instructor was born after Brooks wrote his book) Why can you not divide the reproductive work among 9X workers and get your toilers in a month?  OK, I recognize that this isn't possible today (although see below for a better idea).

Perhaps a bigger concern is the latency from birth to "productive coder".  Is there a potential application of computational chemistry here to produce pharmacological agents that will reduce that 10 year latency (minimum) to something smaller?  Perhaps with selective breeding or genetic manipulation?  Chickens and cows reach marketable size much faster today than they used to. Software developers (or STEM graduates in general) are next.  Conceivably, one could reduce the gestation period as well. These physically smaller coders (make em smarter faster, but don't waste energy on growing large bodies) will occupy less space in the office, so we can turn today's space wasteful cube farms with their 8 foot ceilings into something more reasonable.  Perhaps not to the size of the cages for battery hens, but still smaller than today's cubicle.

Next, imagine a Beowulf Cluster of Coders. Is not the whole Beowulf idea based on using commodity components in a large group to achieve what required an expensive single machine to do before?  Think of this.. No relying on specialists or single great intellects: one can harness the power of the masses.  And you'll get more consistent intellectual performance. None of that spiky curve of journals per year stuff to worry about.  And you can put your computational units in locations where environmental conditions favor optimum trades between productivity and cost.  Food and housing is MUCH cheaper in some places than in others.

In this initial implementation, just as early Beowulfs had to rely on off the shelf consumer PC on utility shelving, the cluster of coders would have to use "off the street" computational units in conventional cubicles.   But as described above, we can use pharmacology and genetic techniques proven in the farming industry to produce more "purpose designed" computational units, just as modern clusters have rack mounted processors mounted in customized rack enclosures.

We then come back to the original problem: manufacturing latency.. Here is my proposed solution:  we apply clustering at a finer scale, just as we have done with "manycore" processors incorporating multiple computational units on one chip.  Using commodity wetware, we aggressively parallelize the production process:   Take the DNA, get that embryo growing in vitro, divide it into a bunch of pieces, distribute the workload among multiple cores, and then recombine later.  There are a few practical engineering details that remain to be worked out, but now that I have disclosed the basic idea, I'll make sure my phone is turned on for the Nobel committee's call next November.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.beowulf.org/pipermail/beowulf/attachments/20131127/8a1dfb7e/attachment.html>