[Beowulf] large array to run

[Chris Samuel]

On Fri, 14 Dec 2007, Mark Hahn wrote:

> (it also depends on whether your glibc contains a malloc that
> switches to mmap for large allocations rather than sbrk, etc.)

I was looking at that the other day wondering why the maximum RAM & 
data segment size limits set by Torque with -l mem=2g were not being 
enforced.

After a bit of head scratching I tracked it down to the fact that 
between somewhere around glibc 2.3 the old malloc() implementation 
using brk() was ripped out and replaced with one that uses mmap() for 
allocations of 128KB or more.

Unfortunately the kernel implementation of mmap() doesn't check the 
maximum memory size (RLIMIT_RSS) or maximum data size (RLIMIT_DATA) 
limits which were being set, but only the maximum virtual RAM size 
(RLIMIT_AS) - this is documented in the setrlimit(2) man page:

>       RLIMIT_AS
>              The maximum size of the process’s virtual memory
>              (address space) in  bytes.   This  limit  affects calls
>              to brk(2), mmap(2) and  mremap(2), which fail with
>              the error ENOMEM upon exceeding  this limit.

(it also says that RLIMIT_RSS hasn't worked since 2.4.29, which seems 
to be born out by a quick grep of 2.6.24-rc3 I have to hand)

In other words you can set a low memory limit of say 10MB with:

$ ulimit -m $((10*1024)

and then run a program that allocates 2GB RAM in large chunks 
successfully and only fails when it tries to request a trivial amount 
of RAM. :-(

I've submitted a patch for Torque to set RLIMIT_AS as well as 
RLIMIT_RSS & RLIMIT_DATA.

cheers,
Chris
-- 
Christopher Samuel - (03) 9925 4751 - Systems Manager
 The Victorian Partnership for Advanced Computing
 P.O. Box 201, Carlton South, VIC 3053, Australia
VPAC is a not-for-profit Registered Research Agency
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part.
Url : http://www.scyld.com/pipermail/beowulf/attachments/20071214/9a1aab49/attachment.bin