[Beowulf] disabling swap on cluster nodes?
hahn at mcmaster.ca
Wed Feb 11 06:08:14 PST 2015
> enable swap on the same nodes. While swapping (both in and out) the
if there is ever a significant period of time when you're both
swapping out and in, you're probably thrashing. I'm not advocating
thrashing. however, it's hard to think of rational arguments against
two particular ways to use swap:
- to permit the kernel to offload cold pages. this is what happens
in a normal linux system when you see a occasional bits of SO,
and an accumulation of only a few percent (of ram) in swap.
this implies no or trivial SI traffic, and simply provides more
ram to productive uses.
- to permit suspension/preemption of jobs. this quite a valuable
scheduler trick, and works surprisingly well. the suspended job
gets swapped out (SO traffic but no SI), then back in (SI with
no SO traffic). this is incremental, which is not ideally efficient,
but is pretty nice if the preemptor is not huge.
> latest generation CPU gets to perform probably about the same FLOPS as
> an 8086 with software emulated FP - is this really HPC? It's already
again, I'm not talking about thrashing, wherein the CPU is idled waiting
for pages during a storm of both SI/SO traffic. non-thrash swap usage
does not imply a waste of CPU.
sometimes I think the superstition against swap reflects an anachronistic
belief that disk IO is inefficient. (ie, perhaps exposure to PIO controllers.)
> bad enough that the RAM is mostly NUMA these days and the various
what does NUMA have to do with it? I don't think swap is going to have
some sort of pessimizing-for-numa effect - in fact, I don't see why
it wouldn't help (if a non-numa-optimal page gets swapped out, the
thread that faults it back in is presumably more numa-optimal...)
> is now free for others) and the user satisfaction (no swap=faster
> finishing) simultaneously :)
that is a false equivalency.
> For many years now I do not configure swap on my workstations or
> laptops either (using mostly Fedora), as they have enough RAM for a
> full-blown graphical desktop environment and all assorted
> applications. And when I do start to see problems,
how strange that you attribute the problem to the availability of swap.
>> we also run with overcommit=2.
> Overcommitting is unfortunately still needed for the older Fortran
> applications which statically allocate large arrays. Yes, some of
> those pesky things are still around...
overcommit=2 and swap go together very well (obviously).
More information about the Beowulf