[Beowulf] Remote console management

[Bruce Allen]

Hi Joe,

>  We (my company) have set something like this up for one of our customers 
> with the cluster they bought, where the admins can be (in theory) 1/2 world 
> away and completely control the machines.
>
>  It is quite possible to do this, but you have to be ready for the sticker 
> shock on the KVM over IP.   Lets address the remote power first.
>
>  There are lots of power strips out there, many folks use APC units of one 
> form or the other.  We use NBB-1600 from WTI (http://www.wti.com/nbb16.htm) 
> to handle power control.  They work, quite well at that.  The only issue is 
> that you will need a few of them for full racks, and you would need to set 
> the bios to power on on restoration of power.  If you have an IPMI capable 
> machine (Sun v20z), you plug in to the service processor via a managment 
> network (make it separate from your main network, but routeable and visible 
> from the WAN), and you can power cycle the machine.  You get some nice 
> diagnostic bits in there for free as well.

We don't need the power strips.  The inexpensive IPMI cards let us to both 
hard and soft power cycle.

>  Now onto to the remote consoles.  I am sure I am going to regret saying 
> this, but ... I have seen many people purchase cyclades, and I cannot say I 
> have ever seen them being used.  One customer asked me what it was for, as 
> their last cluster vendor required it as part of their package, and they (the 
> customer) never used it.  In part because it was never attached.

We WOULD attach them!

> IPMI gives you the ability to do a console over IPMI (text only).  I am 
> not sure how well this works in general.

The main issue is 'where is the console output buffered'.  Look back at my 
original posting.  I want to see the same display that someone plugging in 
a monitor would see.  This means that at least the last 24*80 bytes have 
to be buffered.  And ideally somewhat more.

> I prefer KVM over IP.  Yes, it is expensive. 
> Rariton KX232's are about 6000-8000$US/unit these days, and the CIMs that you 
> attach to the machines are about $100/unit.  128 CIMs will run you 12k$.  And 
> will require 4 of these Raritan units.  The Raritans also have an IE-only 
> (grrr!!!) interface.  Some others are saner and use either a java applet, or 
> some sort of portable console.  I have heard a rumor that someone uses vnc, 
> but have not looked hard enough to find out if this is true.
>
>  Combining these two technologies will get you that level of control.

My impression is that the a serial port terminal server
http://shopper.cnet.com/Cyclades_TS_3000___terminal_server/4014-3243_9-30091852.html
would be more cost effective, IF it works.

>  That said, some of the higher end vendors management processors integrate 
> all/most of these things.  It would be overkill to get one of those units 
> with the aforementioned power/kvm technologies.

Actually I am surprised that IPMI 2.0 doesn't already do ALL of what I 
want.  The main issue seems to be this one: where is the serial console 
output buffered?

> Raritan also integrates power control if you prefer single vendors.

I didn't know about Raritan.  Just had a look on the web.  They look to be 
about $150 - $200 per node in quantity one.  So they might be reasonable 
in quantity fifteen or so.

How can they do power control (for a locked system) with just Keyboard, 
Video and Mouse connectors?

Cheers,
 	Bruce


> Bruce Allen wrote:
>> We're getting ready to put together our next large Linux compute cluster. 
>> This time around, we'd like to be able to interact with the machines 
>> remotely.  By this I mean that if a machine is locked up, we'd like to be 
>> able to see what's on the console, power cycle it, mess with BIOS settings, 
>> and so on, WITHOUT having to drive to work, go into the cluster room, etc.
>> 
>> One possible solution is to buy nodes that have IPMI cards.  These 
>> piggyback on the ethernet LAN and let you interact with the machine even in 
>> the absence of an OS.  With the appropriate tools running on a remote 
>> machine, you can interact with the nodes even if they have no OS on them or 
>> are hung.
>> 
>> Another solution is to use the DB9 serial ports of the nodes.  You have an 
>> 'administrative' box containing lots of high-port-count serial cards (eg, 
>> Cyclades 32 or 64 port cards) and then run a serial cable from each node to 
>> this box.  By remotely logging into this admin box you can access the 
>> serial ports of the machines, and if the BIOS has the right 
>> settings/support, this lets you have keyboard/console access.
>> 
>> Or one can do both IPMI + remote serial port access.
>> 
>> Could people on this list please report their experiences with these or 
>> other approaches?  In particular, does someone have a simple and 
>> inexpensive solution (say < $100/node) which lets them remotely:
>>  - power cycle a machine
>>  - examine/set BIOS values
>>  - look at console output even for a dead/locked/unresponsive box
>>  - ???
>> 
>> Thanks!
>> 
>> Bruce Allen
>> U. of Wisconsin - Milwaukee Physics Department
>> 
>> _______________________________________________
>> Beowulf mailing list, Beowulf at beowulf.org
>> To change your subscription (digest mode or unsubscribe) visit 
>> http://www.beowulf.org/mailman/listinfo/beowulf
>
> -- 
> Joseph Landman, Ph.D
> Founder and CEO
> Scalable Informatics LLC,
> email: landman at scalableinformatics.com
> web  : http://www.scalableinformatics.com
> phone: +1 734 786 8423
> fax  : +1 734 786 8452
> cell : +1 734 612 4615
>