[Beowulf] numactl and load balancing
gus at ldeo.columbia.edu
Thu Jul 23 13:25:49 PDT 2015
On 07/23/2015 03:03 PM, mathog wrote:
> Dell with 2CPU x 12core x 2 threads, shows up in procinfo as 48 cpus.
> Trying to run 30 processes 1 each on different "CPU"s by starting them
> one at a time with
> numactl -C 1-30 /$PATH/program #args...
> when 30 have started the script spins waiting for one to exit then
> another is started. "top" is showing some of these are running at 50%
> CPU, so they are being started on a CPU which already has a job going. I
> can see where that would happen, since there doesn't seem to be anything
> in numactl about load balancing. The thing is, these processes are
> _staying_ on the same CPU, never migrating to another. That I don't
> understand. I would have thought numactl sets some mask on the process
> restricting the CPUs it can move to, but would not otherwise affect it,
> so the OS should migrate it when it sees this situation. In practice it
> seems to leave it running on whichever CPU it starts on. Or does linux
> not migrate processes when they are heavily loading a single CPU, only
> when they run out of memory???
> Also "perf top" shows 81% for the program and 13% for numactl.
> The goal here is to carefully divvy up the load so that exactly 15 jobs
> run on each Numa zone, since then the data in all the inner loops will
> fit within the 30M of L3 cache on each CPU. If it puts 17 on one and 13
> the inner loop data won't fit and performance slows down dramatically.
> Looks like I need to keep track of which job is running where and
> numactl lock it to that node. (I don't think there is a queue system on
> this machine at present.)
> David Mathog
> mathog at caltech.edu
> Manager, Sequence Analysis Facility, Biology Division, Caltech
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
Two alternatives to numactl that you could try are
taskset and hwloc/hwloc-bind.
The latter, along with the lstopo utility,
may allow finer control regarding numa, cache, etc.
I hope this helps,
More information about the Beowulf