[Beowulf] NASTRAN on cluster

[Lombard, David N]

From: Roland Krause on Tuesday, April 12, 2005 9:32 AM
>
>--- "Lombard, David N" <david.n.lombard at intel.com> wrote:
>[...]
>> [...]
>> >
>> >on ia32, TASK_UNMAPPED_BASE, by default, is at 1GB rather than ~1.3.
>> >easy to change, though, at least to give 2-2.5 GB on ia32.
>>
>> It may *not* be easy to change, depending on the distro and glibc.
>
>It *is* relatively easy to change. In fact in 2.4 kernels the
>TASK_UNMAPPED_BASE (TUB) is not fixed but at 1/3 of TASK_SIZE which is
>available physical memory - 1G (for the kernel)=3G by default. This is
>set in <line-2.4.xx>/include/asm-i386/processor.h. You can change that
>relatively safely to 1/12 which will let you allocate closer to ~3G
>using mmap.

It is easy to change.  I developed a simple hack for this back in 2001,
and a much better per-process patch was developed later.  MSC.Nastran
pointed to both of these at various times.

However, at some point (I think during RHL8.0), a run-time optimization
was introduced by some distros, including RHL and derivatives, that had
the effect of pinning glibc to a specific address.  The net was that
changing TUMB by my primitive patch or the far-more-desirable
per-process patch caused havoc as the brk/sbrk space collided with
glibc.  IIRC, the glibc updates with this optimization were made
available back to RHL 7.2.  I can dig out the specifics for anyone
interested, but it is (was) plainly visible in the glibc spec file.

Note: I did call this glibc pinning an optimization, as it did help the
majority of users, despite the fact that it hurt some very specific
codes...

>There seems to also be some confusion - at least for me, and please
>correct me if I am wrong here - about TUB and mmap vs. brk/sbrk.

On IA32, the brk/sbrk space grows up, from above your program; heap
grows down, from the top of the address space, and the mmap space is in
the middle (note, the heap could also allocate from space below the mmap
point).  The patches described above move the TUMB up or down, affecting
the partition point.

> ...
> Most "legacy" Fortran codes (including the one we sell)
>allocate memory in one large chunk, so your compiler or glibc resp.
>will use mmap to allocate and hence you are stuck with 2G.

Subject to the above discussion.

>A while ago Greg Lindahl posted a little code snippet the prevents
>glibc's malloc from using mmap all together. Grep the archives for it.

This would be a useful bit of code.

>Finally the whole problem is history with the arrival of 2.6.9 where
>changes to the kernel were made so that the memory spaces now grow
>towards each other from opposite sides so that one now can avoid these
>tricks allthogether, at least on 64bit machines :-)

This is an ia32 problem...

-- 
David N. Lombard
 
My comments represent my opinions, not those of Intel Corporation.