[Beowulf] big read triggers migration and slow memory IO?

[Christopher Samuel]

On 09/07/15 08:43, Jonathan Barber wrote:

> I think you're process is being moved between NUMA nodes and you're
> losing locality to the data. Try confining the process and data to the
> same NUMA node with the numactl command. 

A friend of mine who hacks on MySQL (at a variety of companies, now at
IBM) wrote yesterday about a long running "swap insanity" issue that
sounds similar - the swap they are talking about is not paging to disk
but memory getting moved between NUMA nodes.

https://www.flamingspork.com/blog/2015/07/08/the-sad-state-of-mysql-and-numa/

It would be interesting for David to try his program with one of the
suggestions in the blog to see if that makes any difference:

numactl –interleave all

The other thing that may shed some light is to run "perf top"
whilst this is going on (as root) to see if there are any
obvious culprits.

Best of luck!
Chris
-- 
 Christopher Samuel        Senior Systems Administrator
 VLSCI - Victorian Life Sciences Computation Initiative
 Email: samuel at unimelb.edu.au Phone: +61 (0)3 903 55545
 http://www.vlsci.org.au/      http://twitter.com/vlsci