[Beowulf] Feedback on large pages in Linux

[Tom Mitchell]

On Fri, Jul 21, 2006 at 11:39:23PM -0400, hahn at physics.mcmaster.ca wrote:
> 
> yeesh.  for what it's worth, I sat in on an OLS BOF today which 
> discussed this issue (using big pages in linux).  from what I gather,
> kernel authorities don't want to complicate the VM by mixing normal
> and big pages....
....
> (falling back from large to small pages and replacing contiguous ranges
> of small pages with larger ones), perhaps purely at user-level.
> 
> it would be incredibly useful if the HPC community could come up with 
> some actual numbers on how important bigpages are.  I'm ashamed to say
> that I haven't made any effort to measure this on any of my clusters.
> oprofile appears to make it pretty easy to capture real data on how 
> often TLB's are a problem.
> 
> have any of you tried to get a handle on this stuff?

It might pay to look at papers by Kazushige Goto.
   http://www.tacc.utexas.edu/general/news/archive/20040809_02.php
   http://www.tacc.utexas.edu/general/staff/goto/
   http://www.tacc.utexas.edu/resources/software/#blas

His blas libraries are tuned in a number of ways
including page size and the expense of TLB hits.

I mention this because numerous HPC benchmark numbers are
built to use his libraries.  As such these HPC numbers confuse
numbers obtained from less optimum libs and any TLB
related values.

i.e. compare HPC numbers where the only thing that
changes is the blas lib.

See #9:
	http://www.cs.utexas.edu/users/flame/Publications/

  "Kazushige Goto and Robert van de Geijn. On Reducing TLB
  Misses in Matrix Multiplication. FLAME Working Note #9,
  The University of Texas at Austin, Department of Computer
  Sciences. Technical Report TR-2002-55. Nov. 2002."

  http://www.cs.utexas.edu/users/flame/pubs/FLAWN9.ps.gz

In this paper are a number of tables that give a clue of the
baseline expense of TLB hits as well as resulting improvements
when this expense taken into account in the design of the library.

My thought right now is that very large pages for data would
be nifty to have if you knew you could get them.  If you cannot
guarantee them then use libs like the Goto-blas library so it
matters less.

Important, http://www.tacc.utexas.edu/resources/software/gotoblasfaq.php 
	"GOTO is my last name and does not mean "go to." (Nor
	is it pronounced "go to"; the correct pronunciation
	of my name is more like "goat-toe.")"



-- 
	T o m  M i t c h e l l 
	Found me a new place to hang my hat :-)
	Now it got bought.