[Beowulf] physical memory

[Mark Hahn]

> Does linux recombine physical memory into contiguous regions?
>
> My impression has been "no".  Somewhere down in the guts of the kernel there is

viewed through the mosaic of user VM, one normally can't tell.
but hugepage support is the only place I can easily imagine noticing.
(and in various vintages of kernels, there is support for rearranging normal
pages to enable hugepages.)

> the "slab" allocator, which maintains a data structure of free memory in

well, slab is for internal kernel use.  in almost any system, most memory
will be in either pagecache or user (anonymous) allocations.  these are 
not sensitive to physical layout (except the normal, optional, NUMA constraints.)

it's fairly unusual to find a machine with a lot of slab use.  it certainly
can happen - I notice it with Lustre.  (I don't know offhand whether Lustre 
actually obeys drop_caches in this sense - this usage is clearly not pagecache.)

> Huge pages and contiguous memory are valuable for HPC because they reduce the rate
> of TLB misses.

certainly, though I see an aweful lot of computing that has such a low
hot-memory footprint that such issues are irrelevant.

> they are valuable for communications because they let IO devices use DMA to access
> physical memory with fewer translation resources or misses to software.

I'd like to see numbers on that - my guess is that the few devices that include 
pagewalk hardware simply provide enough speed for all non-contig pages.
but sure, it's easy enough to imagine that this could be relevant for codes
that spend a lot of time shuffling (as opposed to computing).  big data,
perhaps GPU DB offload situations.

> Because of the lack of any daemon which tries to recombine regions, it has been my

well, it's a kernel thread, not a daemon (khugepaged).  I can't say how 
well it works, though.

-mark