[Beowulf] Again about NUMA (numactl and taskset)

[Chris Samuel]

----- "Greg Lindahl" <lindahl at pbm.com> wrote:

> > > Your MPI (and OpenMP) should do this for you.
> > 
> > Although not always correctly, it may assume that it can
> > allocate from core 0 onwards leading to odd performance
> > issues if you happen to get two 4 CPU jobs running on the
> > same node..
> 
> Most clusters I've seen aren't used that way (whole nodes only),

I don't know of a single cluster in Australia that's managed
that way, presumably because there's far less money for such
systems than elsewhere and we have to try and do a lot more with
a lot less (we can't afford to waste idle CPUs).

The Australian state of the art peak computing facility in
Canberra just dropped out of the Top500 for instance.

> but sure, you can always consult the manual in that case.

IIRC this was with MVAPICH and we had to consult the source
to find the undocumented environment variable to fix it. :)

We switched to OpenMPI and use Torque's 2.6 kernel cpuset
support instead, with local patches.

cheers,
Chris
-- 
Christopher Samuel - (03) 9925 4751 - Systems Manager
 The Victorian Partnership for Advanced Computing
 P.O. Box 201, Carlton South, VIC 3053, Australia
VPAC is a not-for-profit Registered Research Agency