[Beowulf] big read triggers migration and slow memory IO?

[Jonathan Barber]

On 9 July 2015 at 00:42, Christopher Samuel <samuel at unimelb.edu.au> wrote:

> On 09/07/15 08:43, Jonathan Barber wrote:
>
> > I think you're process is being moved between NUMA nodes and you're
> > losing locality to the data. Try confining the process and data to the
> > same NUMA node with the numactl command.
>
> A friend of mine who hacks on MySQL (at a variety of companies, now at
> IBM) wrote yesterday about a long running "swap insanity" issue that
> sounds similar - the swap they are talking about is not paging to disk
> but memory getting moved between NUMA nodes.
>
>
> https://www.flamingspork.com/blog/2015/07/08/the-sad-state-of-mysql-and-numa/
>
> It would be interesting for David to try his program with one of the
> suggestions in the blog to see if that makes any difference:
>
> numactl –interleave all
>

I think David's machine has two NUMA nodes, which means that each NUMA node
should have 256GB of RAM, which is greater than the size of the file - so
he should just bind the process to one node.


>
> The other thing that may shed some light is to run "perf top"
> whilst this is going on (as root) to see if there are any
> obvious culprits.
>

Last time I tried to diagnose this kind of problem with perf it didn't help
because although it would show which instructions were slow (presumably
because they were waiting on data from the other NUMA node) it didn't say
why they were slow. I'm not 100% sure but I think this was because the CPU
didn't have instrumentation for this.

Of course, I may have been doing it wrong :)


> Best of luck!
> Chris
> --
>  Christopher Samuel        Senior Systems Administrator
>  VLSCI - Victorian Life Sciences Computation Initiative
>  Email: samuel at unimelb.edu.au Phone: +61 (0)3 903 55545
>  http://www.vlsci.org.au/      http://twitter.com/vlsci
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
>



-- 
Jonathan Barber <jonathan.barber at gmail.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.beowulf.org/pipermail/beowulf/attachments/20150709/f1b34baf/attachment.html>