<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Hi,</p>
<p>First of all, thanks for sharing all of these informations.</p>
<p>I also participated in a similar study in 2017 :
<a class="moz-txt-link-freetext" href="https://arxiv.org/pdf/1709.10140.pdf">https://arxiv.org/pdf/1709.10140.pdf</a> <br>
</p>
<p>Eduardo is now working for Sylabs.</p>
<p>I also did some talks in France about containers in the HPC :</p>
<p> eg.:
<a class="moz-txt-link-freetext" href="http://devlog.cnrs.fr/_media/jdev2017/dev2017_p8_rdernat_short.pdf">http://devlog.cnrs.fr/_media/jdev2017/dev2017_p8_rdernat_short.pdf</a></p>
<p><br>
</p>
<p>Those technologies changed a lot since then, but conclusions are
almost the same, even if this paper
(<a class="moz-txt-link-freetext" href="https://arxiv.org/abs/1905.08415">https://arxiv.org/abs/1905.08415</a>) focused on MPI.<br>
</p>
<p>I am using Singularity since version 1 and installed it in our
HTC systems (in production) since march 2017 from version 2.2. It
works well. I noticed some issues from time to time and submitted
issues on github, or for new features. Sometimes, I even wrote
some piece of code or doc for them or for the container ecosystem.
The sylabs team is pretty active, and fixed all of those bugs,
except one (CRIU / DMTCP [*]). I am happy to have installed it
from all this time, since I am working mainly with a bioinformatic
community of users. The bioinformatic software landspace is full
of different languages with many dependencies. Before Singularity,
I installed all softwares and packages, statically. At the end
there were more than 150 applications, but the mixture of these
applications was the most complicated part. Limitations of
Singularity include MPI jobs where you still need to be very
carefull in the MPI versions you use (container vs host).<br>
</p>
<p>I am also using nvidia-docker with the user namespace for GPUs,
for about 2 years now. This is a standalone service, not connected
to our clusters, mostly used for Deep Learning. However, I tried
Singularity on it a while ago (using gpu4singularity code from
NIH; before "--nv" option appears in Singularity) and it also
works fine.</p>
<p>IMHO, the main downside of singularity, is that Sylabs are
developping new releases too quickly, and most of the time, new
releases are targeting security bugs... So, as an admin, you need
to upgrade quite often, if your OS is still compatible with it...
Alternatives are to use charliecloud, udocker or shifter. But we
also chose Singularity b/c it was the most active project, with
many contributors. Some people could also argue that Singularity
is moving too fast to a "cloud model", allowing the use of K8s,
but I not agree as HPC community is here from the beginning, for
them, and I think (almost for sure) that they won't turn their
back on them.<br>
</p>
<p>Note that some new big players could become interesting in the
future b/c big companies are developing those products
(Katacontainers, with a mix of advantages between VM and
containers, supported by Intel, using a lightweight qemu, and
Podman with RedHat from the atomic project). However, from this
time, HPC is not in their priority list, and RedHat, as well as
nvidia, already collaborated with Singularity (in many ways).<br>
</p>
<p>IMO all HPC systems should now allow users to launch
"containerized" jobs on it (and not using Docker for obvious
security reasons). We have now many scientific workflows which are
designed to run either on HPC/HTC/Grids or clouds platforms. If,
as a big HPC platform sysadmin, you don't allow that kind of jobs,
the risk for your platform is to see many people leaving the
platform to compute elsewhere (eg. in the cloud or another
platform allowing to perform containerized tasks), even if you
could argue that they won't benefit the best hardware performances
(if users already have a pipeline or a single application using
containers (many apps are now "containerized"), they won't be very
excited in converting everything to get it work on your HPC system
[**]). So you have to be proactive, and collaborate with your
users, to provide the best and secure platform to run those jobs.<br>
</p>
<p>We are currently developing a WebUI that is generating docker and
Singularity recipes [***]; every contributions are welcome (note
that some issues descriptions are still in french). This is still
a WIP, and recipes may be not secure for now.<br>
</p>
<p><br>
</p>
<p>Best regards,</p>
<p>Rémy.</p>
<p>[*] Since version 3.2, Singularity provides a way to stop/resume
jobs with the OCI subcommands using cgroup freezer.<br>
</p>
<p>[**] Ok, many developers worked hard in the past, to get their
software work on HPC systems (using OpenMPI, cuda, OpenACC, or
whatsoever), and they will continue, for sure, but most of
academic/scientific users did not have any easy access to a cloud
(and now, many users have that kind of access... ?)... What would
be the gain for their users between using an optimized app on a
HPC system, and distributing their (sequentials ?) jobs among many
clouds... (that is a real question, and I think there is no good
answer, as it depends on the size and the type of the problem and
the software used, but I can be wrong about it...) ?<br>
</p>
<p>[***] <a class="moz-txt-link-freetext" href="https://gitlab.mbb.univ-montp2.fr/jlopez/wicopa/">https://gitlab.mbb.univ-montp2.fr/jlopez/wicopa/</a> <br>
</p>
<blockquote>
<p>Contributors need an account on our gitlab; you can email me or
create an issue here
(<a class="moz-txt-link-freetext" href="https://kimura.univ-montp2.fr/calcul/helpdesk_NewTicket.html">https://kimura.univ-montp2.fr/calcul/helpdesk_NewTicket.html</a>)
to get one.<br>
</p>
</blockquote>
<div class="moz-cite-prefix">Le 26/05/2019 à 14:17, Benjamin Redling
a écrit :<br>
</div>
<blockquote type="cite"
cite="mid:3D1B25A6-1FFE-472D-B386-555BD00F22DA@uni-jena.de">
<pre class="moz-quote-pre" wrap="">Good news.
I'll try it out, again.
Am 26. Mai 2019 13:57:05 MESZ schrieb INKozin <a class="moz-txt-link-rfc2396E" href="mailto:i.n.kozin@googlemail.com"><i.n.kozin@googlemail.com></a>:
</pre>
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">for what it's worth, Singularity worked well for me last time I tried
it.
I think it was shortly after NVIDIA had announced support for it.
On Sun, 26 May 2019 at 11:11, Benjamin Redling
<a class="moz-txt-link-rfc2396E" href="mailto:benjamin.rampe@uni-jena.de"><benjamin.rampe@uni-jena.de></a>
wrote:
</pre>
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">On 23/05/2019 16.13, Loncaric, Josip via Beowulf wrote:
</pre>
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">"Charliecloud" is a more secure approach to containers in HPC:
</pre>
</blockquote>
<pre class="moz-quote-pre" wrap="">
I tried Singularity short before and during 2.3 with GPUs -- didn't
work, documented issue, maybe solved. Stopped caring.
Shortly afterwards I read about Charliecloud and tried it -- didn't
work, too many issues. Stopped caring.
So, "more secure" on paper (less lines of code) doesn't get any work
</pre>
</blockquote>
<pre class="moz-quote-pre" wrap="">done.
</pre>
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">My advice to anyone with a working setup: try it out if time permits,
but don't bother to much and definitively don't advertise it to third
parties beforehand.
Regards,
Benjamin
--
FSU Jena | <a class="moz-txt-link-freetext" href="https://JULIELab.de/Staff/Redling/">https://JULIELab.de/Staff/Redling/</a>
☎ +49 3641 9 44323
_______________________________________________
Beowulf mailing list, <a class="moz-txt-link-abbreviated" href="mailto:Beowulf@beowulf.org">Beowulf@beowulf.org</a> sponsored by Penguin
</pre>
</blockquote>
<pre class="moz-quote-pre" wrap="">Computing
</pre>
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">To change your subscription (digest mode or unsubscribe) visit
<a class="moz-txt-link-freetext" href="https://beowulf.org/cgi-bin/mailman/listinfo/beowulf">https://beowulf.org/cgi-bin/mailman/listinfo/beowulf</a>
</pre>
</blockquote>
</blockquote>
<pre class="moz-quote-pre" wrap="">
</pre>
</blockquote>
<pre class="moz-signature" cols="72">--
Dernat Rémy
Plateforme MBB - ISEM Montpellier</pre>
</body>
</html>