[Beowulf] Remote console management

[Bruce Allen]

We're getting ready to put together our next large Linux compute cluster. 
This time around, we'd like to be able to interact with the machines 
remotely.  By this I mean that if a machine is locked up, we'd like to be 
able to see what's on the console, power cycle it, mess with BIOS 
settings, and so on, WITHOUT having to drive to work, go into the cluster 
room, etc.

One possible solution is to buy nodes that have IPMI cards.  These 
piggyback on the ethernet LAN and let you interact with the machine even 
in the absence of an OS.  With the appropriate tools running on a remote 
machine, you can interact with the nodes even if they have no OS on them 
or are hung.

Another solution is to use the DB9 serial ports of the nodes.  You have an 
'administrative' box containing lots of high-port-count serial cards (eg, 
Cyclades 32 or 64 port cards) and then run a serial cable from each node 
to this box.  By remotely logging into this admin box you can access the 
serial ports of the machines, and if the BIOS has the right 
settings/support, this lets you have keyboard/console access.

Or one can do both IPMI + remote serial port access.

Could people on this list please report their experiences with these or 
other approaches?  In particular, does someone have a simple and 
inexpensive solution (say < $100/node) which lets them remotely:
  - power cycle a machine
  - examine/set BIOS values
  - look at console output even for a dead/locked/unresponsive box
  - ???

Thanks!

Bruce Allen
U. of Wisconsin - Milwaukee Physics Department