[Beowulf] Remote console management
ballen at gravity.phys.uwm.edu
Thu Sep 22 20:30:32 PDT 2005
> My experience has not been positive in this area. Serial consoles tend to be
> very expensive and not provide access to the bios unless the mother board has
> support for serial access to the bios (don't expect this in cheap compute
The nodes we are currently considering (dual opterons, supermicro
motherboard) do support serial bios access.
> We use kvm over ethernet to the head nodes of our clusters and don't bother
> with the compute nodes.
This is similar to what we do with our existing four-year-old 296-node
cluster and our previous (1998!) 48-node cluster.
> We looked at serially attached power boards to remotely power cycle nodes,
> but again they were very expensive (for 150 nodes).
We can get the remote power cycle from the IPMI cards.
> It looks like all serial consoles/remote management cards/kvm's tend to be
> around $500-$1000/node, which tends to be ~50% the cost of the compute node
> (in our clusters).
The IPMI cards add $60 - $80 per node which works out to $30 - $40 per
CPU. This is reasonable.
The Cyclades TS 3000 (48 serial ports) is available in quantity 1 for $4k.
This works out to $83 per node.
> For a large cluster (100+ nodes) and sub $100/node, the cheapest solution is
> to give a PhD or grad student an extra $10k and get a small trolley with
We've been doing this for a number of years, though we use undergrads
because they cost less! But we'd like something more consistently
available 24x7 from anywhere.
The cost of IPMI cards + a serial terminal box appears to be around
$150/node. Our nodes are going to be quite expensive, as we are also
using the cluster for distributed storage. So this cost is acceptable,
though I would prefer to reduce it to the $100/node level if possible.
In any case the cost is much less than the $500 - $1000 per node that you
So I would be interested in hearing the experiences of others who have
> On 23/09/2005, at 5:35, Bruce Allen wrote:
>> We're getting ready to put together our next large Linux compute cluster.
>> This time around, we'd like to be able to interact with the machines
>> remotely. By this I mean that if a machine is locked up, we'd like to be
>> able to see what's on the console, power cycle it, mess with BIOS settings,
>> and so on, WITHOUT having to drive to work, go into the cluster room, etc.
>> One possible solution is to buy nodes that have IPMI cards. These
>> piggyback on the ethernet LAN and let you interact with the machine even in
>> the absence of an OS. With the appropriate tools running on a remote
>> machine, you can interact with the nodes even if they have no OS on them or
>> are hung.
>> Another solution is to use the DB9 serial ports of the nodes. You have an
>> 'administrative' box containing lots of high-port-count serial cards (eg,
>> Cyclades 32 or 64 port cards) and then run a serial cable from each node to
>> this box. By remotely logging into this admin box you can access the
>> serial ports of the machines, and if the BIOS has the right
>> settings/support, this lets you have keyboard/console access.
>> Or one can do both IPMI + remote serial port access.
>> Could people on this list please report their experiences with these or
>> other approaches? In particular, does someone have a simple and
>> inexpensive solution (say < $100/node) which lets them remotely:
>> - power cycle a machine
>> - examine/set BIOS values
>> - look at console output even for a dead/locked/unresponsive box
>> - ???
>> Bruce Allen
>> U. of Wisconsin - Milwaukee Physics Department
>> Beowulf mailing list, Beowulf at beowulf.org
>> To change your subscription (digest mode or unsubscribe) visit
> Dr Stuart Midgley
> sdm900 at gmail.com
More information about the Beowulf