[Beowulf] Re: Memory limit enforcement

[Daniel Spångberg]

On Wed, 10 Oct 2007 21:18:54 +0200, David Mathog <mathog at caltech.edu>  
wrote:

>
> I'm thinking that it shouldn't be too difficult (but what do
> I know about kernel hacking?) to allow a process to request:
>
> 1. for any child processes that may be created after the request
> 2. for any malloc() type operation
> 3. have the child share the parent's memory statistics/limits
>    as maintained by the kernel.  (Not the complete memory map, just
>    the sum of physical and virtual memory allocated.)
>
> The end result would be malloc/calloc failing in some
> child process once the parent process's counters hit up against
> the set limit(s).  This would happen no matter which child ate all the
> memory, which is the desired behavior.
>

That's probably a good start, but wouldn't help with applications which  
use a lot of shared memory. I have currently a resource limit problem with  
an application which uses *a lot* of SYSV shared memory. Essentially, on a  
4 cpu machine with 4 GB memory it starts four processes, creates four 1 GB  
shared memory segments, one per process and then attaches all four  
segments to all processes. So the virtual memory per process is about 4GB  
and when it actually uses the memory the RSS of each process also comes  
close to 4GB. We currently use a daemon which kills processes which have a  
larger RSS than their rlimits, which obviosly is quite bad for this kind  
of application.

Daniel Spångberg
UPPMAX
Uppsala University
Sweden