[Beowulf] Remote console management

[Joe Landman]

Hi Bruce:

   We (my company) have set something like this up for one of our 
customers with the cluster they bought, where the admins can be (in 
theory) 1/2 world away and completely control the machines.

   It is quite possible to do this, but you have to be ready for the 
sticker shock on the KVM over IP.   Lets address the remote power first.

   There are lots of power strips out there, many folks use APC units of 
one form or the other.  We use NBB-1600 from WTI 
(http://www.wti.com/nbb16.htm) to handle power control.  They work, 
quite well at that.  The only issue is that you will need a few of them 
for full racks, and you would need to set the bios to power on on 
restoration of power.  If you have an IPMI capable machine (Sun v20z), 
you plug in to the service processor via a managment network (make it 
separate from your main network, but routeable and visible from the 
WAN), and you can power cycle the machine.  You get some nice diagnostic 
bits in there for free as well.

   Now onto to the remote consoles.  I am sure I am going to regret 
saying this, but ... I have seen many people purchase cyclades, and I 
cannot say I have ever seen them being used.  One customer asked me what 
it was for, as their last cluster vendor required it as part of their 
package, and they (the customer) never used it.  In part because it was 
never attached.  IPMI gives you the ability to do a console over IPMI 
(text only).  I am not sure how well this works in general.  I prefer 
KVM over IP.  Yes, it is expensive.  Rariton KX232's are about 
6000-8000$US/unit these days, and the CIMs that you attach to the 
machines are about $100/unit.  128 CIMs will run you 12k$.  And will 
require 4 of these Raritan units.  The Raritans also have an IE-only 
(grrr!!!) interface.  Some others are saner and use either a java 
applet, or some sort of portable console.  I have heard a rumor that 
someone uses vnc, but have not looked hard enough to find out if this is 
true.

   Combining these two technologies will get you that level of control.

   That said, some of the higher end vendors management processors 
integrate all/most of these things.  It would be overkill to get one of 
those units with the aforementioned power/kvm technologies.  Raritan 
also integrates power control if you prefer single vendors.

   Joe

Bruce Allen wrote:
> We're getting ready to put together our next large Linux compute 
> cluster. This time around, we'd like to be able to interact with the 
> machines remotely.  By this I mean that if a machine is locked up, we'd 
> like to be able to see what's on the console, power cycle it, mess with 
> BIOS settings, and so on, WITHOUT having to drive to work, go into the 
> cluster room, etc.
> 
> One possible solution is to buy nodes that have IPMI cards.  These 
> piggyback on the ethernet LAN and let you interact with the machine even 
> in the absence of an OS.  With the appropriate tools running on a remote 
> machine, you can interact with the nodes even if they have no OS on them 
> or are hung.
> 
> Another solution is to use the DB9 serial ports of the nodes.  You have 
> an 'administrative' box containing lots of high-port-count serial cards 
> (eg, Cyclades 32 or 64 port cards) and then run a serial cable from each 
> node to this box.  By remotely logging into this admin box you can 
> access the serial ports of the machines, and if the BIOS has the right 
> settings/support, this lets you have keyboard/console access.
> 
> Or one can do both IPMI + remote serial port access.
> 
> Could people on this list please report their experiences with these or 
> other approaches?  In particular, does someone have a simple and 
> inexpensive solution (say < $100/node) which lets them remotely:
>  - power cycle a machine
>  - examine/set BIOS values
>  - look at console output even for a dead/locked/unresponsive box
>  - ???
> 
> Thanks!
> 
> Bruce Allen
> U. of Wisconsin - Milwaukee Physics Department
> 
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit 
> http://www.beowulf.org/mailman/listinfo/beowulf

-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics LLC,
email: landman at scalableinformatics.com
web  : http://www.scalableinformatics.com
phone: +1 734 786 8423
fax  : +1 734 786 8452
cell : +1 734 612 4615