[Beowulf] Approach For Diagnosing Heat Related Failure?

[Bill Broadley]

I'd suggest doing a visual inspection.  Make sure all fans are not blocked by
cables, are spinning.  If that looks normal pull the CPU heat sinks and make
sure they have good coverage with the heat sink goo, but not so much that it
leaks over the edge of the chip.  When you put the heat sink back on make sure
the heat sink mount works as intended, especially on the (mostly intel?) 4
post system where an unclicked post can result in unevent heat sink pressure.

Be careful, fans moving != spinning.  I've seen some that just vibrate enough
to look like they are spinning at a casual glance and are actually not moving
much air and are contributing a fair bit of heat to the system (I.e. very hot
to the touch).

If that looks normal then I'd start swapping parts till you find the heat
sensitive one.