[Beowulf] computing on Altix?

[Mark Hahn]

> One thing you might want to check and make sure is that the number of threads
> is limited to the number of CPUs you have.  Context switch is not cheap.

it's certainly true that you don't want nthreads >> ncpus.  but context
switching is actually not that expensive - depending on the machine, etc,
for instance, generic desktops running generic linux can switch in .4 us.
that number goes up for many reasons, such as cache contention.

but for parallel code, the real point is that you would like threads 
gang-scheduled, much like gang-scheduling MPI.  if some threads in a paralle
code are preempted, any sort of synchronization will suddenly become very 
inefficient.  I'd guess that's why you experienced such a collapse - 
not merely the primary cost of the context-switches.