[Beowulf] recommendations for cluster upgrades

[Håkon Bugge]

On May 14, 2009, at 1:20 , Rahul Nabar wrote:

> What's the best way to test the "memory bandwidth sensitivity" of my
> code? Given the fact that I have only the AMD Opterons available
> currently. Any active / passive tests that  can throw some metrics out
> at me?

A _very_ brute-force method, probably not recommended - but simple, is  
the following. Run your applications with CPU caches disabled on your  
system(s) and calculate the ratio of the elapsed time between the no- 
cache timing and the default timing (with caches enabled).

If that ratio is low (close to one), that application is almost 100%  
memory bound. If that ratio is high (100-1000), you're running mostly  
out of the CPU caches. If you have two applications which you know in  
advance know possesses (close to) 100% cache-hit and 0% cache hit, you  
can use the ratio of these as you lower-upper bound of ratios.

How to disable CPU caches?

echo disable=<N> > /proc/mtrr

where <N> is the Memory Type Range Register which covers your memory  
area (listed with type=write-back) when you do a 'cat /proc/mtrr' .

In practice, you enable the CPU caches again by rebooting.

Needless to say, if you are not able to reduce the running time of you  
apps down to a low number of 10s of seconds, the run with CPU caches  
disabled could take a while...

Another simple method for you could be to use numactl when running  
your apps on your Opteron cluster. If you have sufficient memory on  
the nodes, you can use numactl to dictate the use of memory from a  
single memory controller (a "node" in numactl parlance). This way, you  
cut your total available memory bandwidth on a node by a factor of ~2.  
If you see a significant degradation in performance, you memory  
bandwidth bound.



Thanks, Håkon