Page Coloring in Alpha Linux (was Benchmarking L2 cache on the A lpha 21264 )

[Tom Morris]

There was a discussion last week about run to run performance
variability for codes which are L2 cache resident.  As a couple of 
folks said this is almost certainly due to the lack of page coloring
support in the Linux memory allocator.

We've done some testing with both Greg's and Joe's patches and,
as was indicated on the kernel mailing list, there are side effects
in terms of both allocation time and pool fragmentation.  However,
for machines which are dedicated to codes which are L2 cache
sensitive and which won't be using the memory allocator a lot, 
this could be a perfectly acceptable tradeoff.  I suspect they could
be improved upon though.

Another thing to point out is that page coloring makes the allocator
more deterministic, but not completely deterministic.  Even on 
Tru64 it's possible to get variations between runs, particularly if
they layout of stuff in memory has changed significantly (for example
after a reboot).  Good page coloring support in Linux would make
performance of cache senstive codes both more predictable and faster.

Tom