CPU affinity for serial jobs (was Re: [Beowulf] VMC - Virtual Machine Console)

[Joe Landman]

Bernard Li wrote:

> 
> I get the sense that CPU affinity is beneficial even for
> embarrassingly/serial jobs -- however I am curious whether anybody has

Well, yes it is.

> actual numbers to back this?  And is the potential benefits worth the

Here are some, more about process CPU->memory affinity than process 
CPU->cache affinity...

I am forcing the memory to be on a different CPU than running the code:

landman at dualcore:/big/stream-jb-2006-6-8$ numactl --physcpubind=1 
--membind=1 ./stream_d_c_omp_x86_64

Function      Rate (MB/s)   RMS time     Min time     Max time
Copy:        3122.6206       0.2050       0.2050       0.2052
Scale:       3107.5644       0.2061       0.2059       0.2065
Add:         3118.4948       0.3080       0.3078       0.3084
Triad:       3107.2203       0.3091       0.3090       0.3093

I am forcing the memory to be on the same CPU running the code:

landman at dualcore:/big/stream-jb-2006-6-8$ numactl --physcpubind=1 
--membind=0 ./stream_d_c_omp_x86_64

Function      Rate (MB/s)   RMS time     Min time     Max time
Copy:        3705.1849       0.1729       0.1727       0.1734
Scale:       3954.8385       0.1619       0.1618       0.1623
Add:         3893.1524       0.2468       0.2466       0.2472
Triad:       3857.3264       0.2490       0.2489       0.2493

This is more of a memory affinity issue, you want your code running on 
the memory local to the particular CPU.  A decade ago, we saw all sorts 
of performance degredation when processes would migrate from CPU to CPU 
on the big SGI machines, partially defeating the utility of cache (not 
to mention forcing a huge amount of inter-cpu traffic when you had 
massive invalidation storms as a result of the scheduler moving the 
process).

Simply having the scheduler leave the process on the same CPU turned out 
to be a significant win.  Especially for long running jobs that were 
very cache intensive (e.g. such as EP codes, Monte Carlo, ...)


> time/effort to set this up rather than let the default Linux scheduler
> deal with it.

The linux scheduler is actually quite reasonable about this (these days, 
with a modern kernel).  Some older kernels had problems.

> 
> Cheers,
> 
> Bernard

-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics LLC,
email: landman at scalableinformatics.com
web  : http://www.scalableinformatics.com
        http://jackrabbit.scalableinformatics.com
phone: +1 734 786 8423
fax  : +1 866 888 3112
cell : +1 734 612 4615