[Beowulf] disabling swap on cluster nodes?

[Bogdan Costescu]

On Wed, Feb 11, 2015 at 3:01 PM, Jason Riedy <jason at lovesgoodfood.com> wrote:
> You know all those system daemons that aren't doing anything while a
> compute job is running?  Those get pushed out to swap if the thing
> doing real work needs the memory.  They are swapped back into memory
> between jobs (ideally not during).

How many hundreds of daemons are running and how many hundreds of GB
of RAM do they use on the compute nodes? Of course, I'm exaggerating
here, but seriously speaking only a handful of small daemons are
enough.

Now if you think about a cron daemon, this will wake up every minute
to check whether it has stuff to do, so every minute the swap will be
exercised. A queueing system daemon often send usage/accounting data
to the master node on seconds to tens of seconds time scale, so also
not a good match for swap. A Nagios/ganglia/zabbix/etc. daemon does
the same. So what are the daemons that can stay swapped out for longer
periods?

And are these daemons using so much memory? Are we really talking
about setting up multi-GB swap for a few MB or tens of MB of daemon
code and data? My first generation Raspberry Pi (512MB RAM) does all a
node should do and much more - before touching swap.

> It'd be nice simply not to have those daemons or to force them into
> out-of-memory hibernation, but that's much more effort.

I don't quite understand, why are the daemons started at all if they
are not needed?

>  (Also, it'd
> be nice for boot times on 1+TiB nodes to be less than 15 minutes
> before even touching the OS.  ugh.  If only there were a fast scan
> of a small portion where the OS will be loaded, then the OS could
> scan the rest...)

Agreed. For a 4TB machine it's really ridiculous...

Cheers,
Bogdan