[Beowulf] Remote console management

[Bruce Allen]

Hi Stuart,

> My experience has not been positive in this area.  Serial consoles tend to be 
> very expensive and not provide access to the bios unless the mother board has 
> support for serial access to the bios (don't expect this in cheap compute 
> nodes).

The nodes we are currently considering (dual opterons, supermicro 
motherboard) do support serial bios access.

> We use kvm over ethernet to the head nodes of our clusters and don't bother 
> with the compute nodes.

This is similar to what we do with our existing four-year-old 296-node 
cluster and our previous (1998!) 48-node cluster.

> We looked at serially attached power boards to remotely power cycle nodes, 
> but again they were very expensive (for 150 nodes).

We can get the remote power cycle from the IPMI cards.

> It looks like all serial consoles/remote management cards/kvm's tend to be 
> around $500-$1000/node, which tends to be ~50% the cost of the compute node 
> (in our clusters).

The IPMI cards add $60 - $80 per node which works out to $30 - $40 per 
CPU.  This is reasonable.

The Cyclades TS 3000 (48 serial ports) is available in quantity 1 for $4k. 
This works out to $83 per node.

> For a large cluster (100+ nodes) and sub $100/node, the cheapest solution is 
> to give a PhD or grad student an extra $10k and get a small trolley with 
> keyboard/monitor/mouse.

We've been doing this for a number of years, though we use undergrads 
because they cost less!  But we'd like something more consistently 
available 24x7 from anywhere.

The cost of IPMI cards + a serial terminal box appears to be around 
$150/node.  Our nodes are going to be quite expensive, as we are also 
using the cluster for distributed storage.  So this cost is acceptable, 
though I would prefer to reduce it to the $100/node level if possible.

In any case the cost is much less than the $500 - $1000 per node that you 
quote.

So I would be interested in hearing the experiences of others who have 
done this.

Cheers,
 	Bruce



> On 23/09/2005, at 5:35, Bruce Allen wrote:
>
>> We're getting ready to put together our next large Linux compute cluster. 
>> This time around, we'd like to be able to interact with the machines 
>> remotely.  By this I mean that if a machine is locked up, we'd like to be 
>> able to see what's on the console, power cycle it, mess with BIOS settings, 
>> and so on, WITHOUT having to drive to work, go into the cluster room, etc.
>> 
>> One possible solution is to buy nodes that have IPMI cards.  These 
>> piggyback on the ethernet LAN and let you interact with the machine even in 
>> the absence of an OS.  With the appropriate tools running on a remote 
>> machine, you can interact with the nodes even if they have no OS on them or 
>> are hung.
>> 
>> Another solution is to use the DB9 serial ports of the nodes.  You have an 
>> 'administrative' box containing lots of high-port-count serial cards (eg, 
>> Cyclades 32 or 64 port cards) and then run a serial cable from each node to 
>> this box.  By remotely logging into this admin box you can access the 
>> serial ports of the machines, and if the BIOS has the right 
>> settings/support, this lets you have keyboard/console access.
>> 
>> Or one can do both IPMI + remote serial port access.
>> 
>> Could people on this list please report their experiences with these or 
>> other approaches?  In particular, does someone have a simple and 
>> inexpensive solution (say < $100/node) which lets them remotely:
>>  - power cycle a machine
>>  - examine/set BIOS values
>>  - look at console output even for a dead/locked/unresponsive box
>>  - ???
>> 
>> Thanks!
>> 
>> Bruce Allen
>> U. of Wisconsin - Milwaukee Physics Department
>> 
>> _______________________________________________
>> Beowulf mailing list, Beowulf at beowulf.org
>> To change your subscription (digest mode or unsubscribe) visit 
>> http://www.beowulf.org/mailman/listinfo/beowulf
>> 
>
>
> --
> Dr Stuart Midgley
> sdm900 at gmail.com
>
>