[Beowulf] large array to run
csamuel at vpac.org
Thu Dec 13 15:00:54 PST 2007
On Fri, 14 Dec 2007, Mark Hahn wrote:
> (it also depends on whether your glibc contains a malloc that
> switches to mmap for large allocations rather than sbrk, etc.)
I was looking at that the other day wondering why the maximum RAM &
data segment size limits set by Torque with -l mem=2g were not being
After a bit of head scratching I tracked it down to the fact that
between somewhere around glibc 2.3 the old malloc() implementation
using brk() was ripped out and replaced with one that uses mmap() for
allocations of 128KB or more.
Unfortunately the kernel implementation of mmap() doesn't check the
maximum memory size (RLIMIT_RSS) or maximum data size (RLIMIT_DATA)
limits which were being set, but only the maximum virtual RAM size
(RLIMIT_AS) - this is documented in the setrlimit(2) man page:
> The maximum size of the process’s virtual memory
> (address space) in bytes. This limit affects calls
> to brk(2), mmap(2) and mremap(2), which fail with
> the error ENOMEM upon exceeding this limit.
(it also says that RLIMIT_RSS hasn't worked since 2.4.29, which seems
to be born out by a quick grep of 2.6.24-rc3 I have to hand)
In other words you can set a low memory limit of say 10MB with:
$ ulimit -m $((10*1024)
and then run a program that allocates 2GB RAM in large chunks
successfully and only fails when it tries to request a trivial amount
of RAM. :-(
I've submitted a patch for Torque to set RLIMIT_AS as well as
RLIMIT_RSS & RLIMIT_DATA.
Christopher Samuel - (03) 9925 4751 - Systems Manager
The Victorian Partnership for Advanced Computing
P.O. Box 201, Carlton South, VIC 3053, Australia
VPAC is a not-for-profit Registered Research Agency
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Size: 189 bytes
Desc: This is a digitally signed message part.
Url : http://www.scyld.com/pipermail/beowulf/attachments/20071214/9a1aab49/attachment.bin
More information about the Beowulf