[Beowulf] big read triggers migration and slow memory IO?

[Christopher Samuel]

On 10/07/15 07:27, mathog wrote:

> Well, that seems to be it, but not quite with the same symptoms you
> observed.  khugepaged never showed up, and "perf top" never revealed
> _spin_lock_irqsave.  Instead this is what "perf top" shows in my tests:

So this sounds look like the same symptoms as reported here too:

http://structureddata.org/2012/06/18/linux-6-transparent-huge-pages-and-hadoop-workloads/

There they note that:

# Looking at the functions listed in the Flame Graph (below)
# it looked like the issue was related to virtual memory and
# the Linux source shows many of these functions are in
# linux/mm/compaction.c.

which is indeed where compation_alloc() lives.

The kernel describes that function as:

/*
 * This is a migrate-callback that "allocates" freepages by taking pages
 * from the isolated freelists in the block we are migrating to.
 */

        /*
         * Isolate free pages if necessary, and if we are not aborting due to
         * contention.
         */


Introduced by this commit:

commit 748446bb6b5a9390b546af38ec899c868a9dbcf0
Author: Mel Gorman <mel at csn.ul.ie>
Date:   Mon May 24 14:32:27 2010 -0700

    mm: compaction: memory compaction core
    
    This patch is the core of a mechanism which compacts memory in a zone by
    relocating movable pages towards the end of the zone.
    
    A single compaction run involves a migration scanner and a free scanner.
    Both scanners operate on pageblock-sized areas in the zone.  The migration
    scanner starts at the bottom of the zone and searches for all movable
    pages within each area, isolating them onto a private list called
    migratelist.  The free scanner starts at the top of the zone and searches
    for suitable areas and consumes the free pages within making them
    available for the migration scanner.  The pages isolated for migration are
    then migrated to the newly isolated free pages.
    
    [aarcange at redhat.com: Fix unsafe optimisation]
    [mel at csn.ul.ie: do not schedule work on other CPUs for compaction]
    Signed-off-by: Mel Gorman <mel at csn.ul.ie>
    Acked-by: Rik van Riel <riel at redhat.com>
    Reviewed-by: Minchan Kim <minchan.kim at gmail.com>
    Cc: KOSAKI Motohiro <kosaki.motohiro at jp.fujitsu.com>
    Cc: Christoph Lameter <cl at linux-foundation.org>
    Cc: KAMEZAWA Hiroyuki <kamezawa.hiroyu at jp.fujitsu.com>
    Signed-off-by: Andrew Morton <akpm at linux-foundation.org>
    Signed-off-by: Linus Torvalds <torvalds at linux-foundation.org>



All the best,
Chris
-- 
 Christopher Samuel        Senior Systems Administrator
 VLSCI - Victorian Life Sciences Computation Initiative
 Email: samuel at unimelb.edu.au Phone: +61 (0)3 903 55545
 http://www.vlsci.org.au/      http://twitter.com/vlsci