[Beowulf] physical memory

[Lawrence Stewart]

Does linux recombine physical memory into contiguous regions?

My impression has been "no".  Somewhere down in the guts of the kernel there is
the "slab" allocator, which maintains a data structure of free memory in power-of-two
sizes.  As memory is used, the chunks get broken up and naturally migrate towards the
small, fragmented end of the allocator.

For desktop, this makes little difference, but it would seem to reduce the availability of
huge pages.

Huge pages and contiguous memory are valuable for HPC because they reduce the rate 
of TLB misses.  In addition,
they are valuable for communications because they let IO devices use DMA to access
physical memory with fewer translation resources or misses to software.

Because of the lack of any daemon which tries to recombine regions, it has been my
experience that one must allocate all physmem resources at job launch time or worse, at
boot time if you want any guarantee of getting what you want.

Random and fragmented physical memory also has a small bad effect on caches by
producing excess misses due to overcommit of the number of "ways" of the cache.
This can mean a run after a reboot is faster, or merely introduce performance
variability.  In large jobs, that variability contributes to load imbalance, which
has a bad effect on performance.

Am I way out of date on this?  Worth fixing?

-Larry