[Beowulf] HPC workflows

[John Hanks]

On Fri, Dec 7, 2018 at 7:04 AM Gerald Henriksen <ghenriks at gmail.com> wrote:

> On Wed, 5 Dec 2018 09:35:07 -0800, you wrote:
>
> Now obviously you could do what for example Java does with a jar file,
> and simply throw everything into a single rpm/deb and ignore the
> packaging guidelines, but then you are back to in essence creating a
> container and just hiding it behind a massive rpm/deb.
>
>
An rpm/deb packaged container? You better trademark that as soon as
possible, I had completely overlooked circular abstractions as an option :)

We have plenty of examples now about the limitations and dangers of
distributing software via models like nodejs/npm, python/pip, etc. I make
some effort to keep a build of R current with all libraries from CRAN and
Bioconductor reasonably up to date and it's an ongoing, rolling nightmare.
I'm aware that packaging it in rpm/deb would be extremely difficult and
there are plenty more examples of this. But, putting it in a container
wouldn't make my life any easier and would, in fact, just add yet another
layer of something to keep up to date. For some things a pragmatic approach
of treating it as a standalone entity loosely linked to the OS is
warranted. But at that point I'm faced with doing with N amount of effort
where N = "amount of effort to install software" or N + C where C = "work
out containerizing and extra step of pulling container when running". N + C
> N for any controlled OS environment like a cluster or, one would hope,
production environments which handle my credit card and private data. In my
view it follows then that one would only choose to do the N + C effort if
one had extra warm bodies around that
needed tasks.

I might also consider it from this perspective. I pull plugins into my vim
config without ever entertaining the thought that they should have been
packaged as rpms/debs. Maybe some software just doesn't deserve to be
packaged? Either because it already has a means of distribution in
userspace (plugins, pip, npm, cran,...) or because it's just crap software.
I accept that containers may be the best way to deal with crap, as
previously noted.


> Option 2 worked 20 years ago when we only cared about 2 or 3
> distributions of Linux and had a lot less open source / free software.
> But, unfortunately, it does not scale and so for that reason (and a
> few others) the effort to create Docker / npm, maven, etc. is the
> lesser of the options.
>
>
I think the more accurate phrase there is "lesser of the evils". Every time
I get into a discussion about this topic it feels as if the discussion is
about determining which rusty saw is the best choice to saw off my own leg
and when I suggest "maybe we shouldn't saw off our own legs?" I'm met with
a chorus of "but griznog, EVERYONE is sawing of their own leg with X, Y and
Z these days!! You simply must attend the next Leg Sawing Conference..."
Electron and nodejs is a great example. No one seems to step back and
consider that perhaps the entire concept is broken and maybe wasn't a good
idea. But unfortunately it's quite easy to blame the packaging tools and at
this point many careers and reputations are intertwined with nodejs
continuing to be popular, so the people in the best position to consider it
was a bad idea are unlikely to consider that.

In my view containers are little more than incredibly complex static
binaries and I've already sat through years and years of discussions about
the problems of static binaries and concluded they are great for a few
narrow cases but not the best solution in general. So, 'cat
every_argument_against_static_binaries_ever | sed s/static
binary/container/g'. That containers require some minimal amount of setuid
root or a daemon to wrap root access adds a significant bit of weight to
those criticisms.

I realize I'm the odd person out here and that containers are a part of the
fabric now, for better or worse. In the same way I'm forever stuck dealing
with the clusterfsck that Java is, I'll be stuck with containers. To that
end massive kudos to singularity for at least bringing some semblance of
sanity to the mix. FWIW, I think containers are better than Java, but I
will still grumble as I slowly begin dragging the rusty blade back and
forth across my thigh.

griznog
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.beowulf.org/pipermail/beowulf/attachments/20181207/4980f3a8/attachment-0001.html>