[Beowulf] recommendations for cluster upgrades
hbugge at platform.com
Thu May 14 00:44:14 PDT 2009
On May 14, 2009, at 1:20 , Rahul Nabar wrote:
> What's the best way to test the "memory bandwidth sensitivity" of my
> code? Given the fact that I have only the AMD Opterons available
> currently. Any active / passive tests that can throw some metrics out
> at me?
A _very_ brute-force method, probably not recommended - but simple, is
the following. Run your applications with CPU caches disabled on your
system(s) and calculate the ratio of the elapsed time between the no-
cache timing and the default timing (with caches enabled).
If that ratio is low (close to one), that application is almost 100%
memory bound. If that ratio is high (100-1000), you're running mostly
out of the CPU caches. If you have two applications which you know in
advance know possesses (close to) 100% cache-hit and 0% cache hit, you
can use the ratio of these as you lower-upper bound of ratios.
How to disable CPU caches?
echo disable=<N> > /proc/mtrr
where <N> is the Memory Type Range Register which covers your memory
area (listed with type=write-back) when you do a 'cat /proc/mtrr' .
In practice, you enable the CPU caches again by rebooting.
Needless to say, if you are not able to reduce the running time of you
apps down to a low number of 10s of seconds, the run with CPU caches
disabled could take a while...
Another simple method for you could be to use numactl when running
your apps on your Opteron cluster. If you have sufficient memory on
the nodes, you can use numactl to dictate the use of memory from a
single memory controller (a "node" in numactl parlance). This way, you
cut your total available memory bandwidth on a node by a factor of ~2.
If you see a significant degradation in performance, you memory
More information about the Beowulf