[Beowulf] Containers in HPC

[Bill Broadley]

On 5/23/19 5:35 AM, Jonathan Aquilina wrote:> Thanks for the great explanation
and clarification. Another question that stems from the below what mechanisms
exist in terms of security for the containers to be as secure as a VM?

As usual with security it's complicated.  Both VPSs and containers have security
advantages and disadvantages.

Generally I'd rate a container as dramatically more secure than a normal
process.  Take a normal linux desktop.  If you are running a clock, typically it
could read your screen, your keyboard, your entire /home dir, read any file
owned by the user, see what OS you are running, what processes are running, run
a screenshot etc.  Sure apparmor and selinux try to limit this, for binaries
they know about, but what about other binaries?

What's worse is any root process can do just about anything.

Containers by default do not have access to /proc, /sys, /home, can't learn much
at all about the underlying OS.  Race conditions in /tmp or /var tmp aren't
visible.  However something running in a container can attack the same OS kernel
that the user is interacting with.  So a kernel compromise gets you everything.

BTW, docker is wildly insecure, basically gives root to the users who can run
it.  But that's not required of containers.  Singularity for instance is largely
a secure version of docker.

So virtual machines have a big benefit, the VM has it's own kernel, and it's
often a simplified virtual kernel without zillions of drivers for random
hardware.  It's "hardware" is really just the underlying host kernel.  However
if a virtual machine is compromised the attacker could replace the kernel that
could attack the host kernel more directly.... almost exactly like a container
could.

A downside of containers is MUCH less visibility from the host OS.  Normally if
you are trying to see what an OS is doing you can run top, strace, look in
/proc, ask apt/dpkg/yum/dnf what packages are around, look at files, monitor tcp
connections, etc.  With a virtual machine all of those abstractions have state
stored in the guest OS kernel, not the host OS.  So the host OS can not just run
ps, top, lsof, strace, look in /proc or /sys to tell what's going on.  Most
importantly the host OS can't easy look into a guest to see if it has a
vulnerable library in use.  Of if one of the processes in the guest is the
latest bitcoin miner and/or malware.

As a sysadmin that supports users with root inside of virtual machines, this can
be a major issue.  I'd rather support users with containers.  Containers have
shorter lifetimes, less state, explicitly handled state, and are usually rebuilt
regularly from up to date sources.  Virtual machines are easy to lose where you
literally are not sure where it is, when it was patched, and what physical
hardware it's on.

VPSs tend to never die, often forgotten, with an unknown amount of state.  Sure
there are configuration management systems like puppet, chef, and ansible.  But
generally I think using containers leads to more secure default behaviors...
even if it's more work.