[Beowulf] NASTRAN on cluster

[Mark Hahn]

> >> >easy to change, though, at least to give 2-2.5 GB on ia32.
> >>
> >> It may *not* be easy to change, depending on the distro and glibc.
> >
> >It *is* relatively easy to change. In fact in 2.4 kernels the
> >TASK_UNMAPPED_BASE (TUB) is not fixed but at 1/3 of TASK_SIZE which is
> >available physical memory - 1G (for the kernel)=3G by default. This is
> >set in <line-2.4.xx>/include/asm-i386/processor.h. You can change that
> >relatively safely to 1/12 which will let you allocate closer to ~3G
> >using mmap.
> 
> It is easy to change.  I developed a simple hack for this back in 2001,
> and a much better per-process patch was developed later.  MSC.Nastran

this solution has been known for longer than that - at last back
to 1998-ish on LKML.

> was introduced by some distros, including RHL and derivatives, that had
> the effect of pinning glibc to a specific address.  The net was that

this is "prelinking", no?  I have the impression that you can pretty
easily control which binaries are prelinked, and even reverse the change.

> On IA32, the brk/sbrk space grows up, from above your program; heap
> grows down, from the top of the address space, and the mmap space is in
> the middle (note, the heap could also allocate from space below the mmap
> point).  The patches described above move the TUMB up or down, affecting
> the partition point.

slightly odd terminology - it's the stack that grows down from ~3GB,
and the brk/sbrk space is called "heap".  since there is one space 
and three growing things, they collide.  you have to fix at least one
of them.  one interesting option is to fix the max stack size (often
only a few MB will do), and have heap and mmap arena grow towards each other.
I believe at least one distro had this option.

> > Most "legacy" Fortran codes (including the one we sell)
> >allocate memory in one large chunk, so your compiler or glibc resp.
> >will use mmap to allocate and hence you are stuck with 2G.
> 
> Subject to the above discussion.

interestingly, you can also avoid mmap entirely if you statically link.

> >A while ago Greg Lindahl posted a little code snippet the prevents
> >glibc's malloc from using mmap all together. Grep the archives for it.
> 
> This would be a useful bit of code.

mallopt(M_MMAP_MAX,0);

http://www.gnu.org/software/libc/manual/html_node/Malloc-Tunable-Parameters.html#Malloc-Tunable-Parameters