[Beowulf] Feedback on large pages in Linux

[Kevin Pedretti]

> On Fri, Jul 21, 2006 at 11:39:23PM -0400, hahn at physics.mcmaster.ca wrote:
> 
> > it would be incredibly useful if the HPC community could come up with 
> > some actual numbers on how important bigpages are.  I'm ashamed to say
> > that I haven't made any effort to measure this on any of my clusters.
> > oprofile appears to make it pretty easy to capture real data on how 
> > often TLB's are a problem.


At least on Opteron, the benefit of large pages is highly dependent on
the memory access pattern.  The main reason is that the Opteron only has
eight 2-Mbyte TLB entries, compared to 512 4-Kbyte TLB entries (see
below).  So, an app that accesses lots of little regions of memory
scattered all over the place will probably be hurt by using large
pages.  The only sure way to tell is to run your app both ways... which
is really easy to do on a machine like Red Storm (Cray XT3) because its
compute node OS, Catamount, has full support for large and small pages
("run -small_pages my_app" vs. "run -large_pages my_app").

Anybody know if recent Intel processors have the same issue?

Kevin


obtained using cpuid instruction on an Opteron 146...

L1 2-Mbyte TLB:
    DTLB entries       = 8
    ITLB entries       = 8
    DTLB associativity = full
    ITLB associativity = full

L1 4-Kbyte TLB:
    DTLB entries       = 32
    ITLB entries       = 32
    DTLB associativity = full
    ITLB associativity = full

L2 2-Mbyte TLB:
    DTLB entries       = 0
    ITLB entries       = 0
    DTLB associativity = off
    ITLB associativity = off

L2 4-Kbyte TLB:
    DTLB entries       = 512
    ITLB entries       = 512
    DTLB associativity = 4
    ITLB associativity = 4