[Beowulf] Remote console management

[Bruce Allen]

> IPMI cards are a good idea. I work with them all the time.
> We use IPMI for remote monitoring of systems, and for power cycling.
> IPMI cards for Supermicro nodes are not expensive, and are IPM 2
> compliant.
>
> IPMI does support Serial-over-LAN, but I don't have experience with it,
> I'm not sure Linux does this.

We're quite interested to know if/how this works.  Could it eliminate the 
serial port server boxes?

> Other manufacturers have similar. The Sun Service Processors are IPMI
> compliant, and in addition allow remote access via ssh. You can do a
> terminal redirect and get full access to the BIOS, ie. you ssh into
> the SP and type 'system console' and get a serial console on the node.

Unfortunately the Sun/Newisys hardware is not an option for us.  We need 
four hot swap SATA drives per node.

>> Another solution is to use the DB9 serial ports of the nodes.  You have an
>> 'administrative' box containing lots of high-port-count serial cards (eg,
>> Cyclades 32 or 64 port cards) and then run a serial cable from each node
>> to this box.  By remotely logging into this admin box you can access the
>> serial ports of the machines, and if the BIOS has the right
>> settings/support, this lets you have keyboard/console access.
>>
>> Or one can do both IPMI + remote serial port access.

> Cyclades terminal servers are very good.
> That's a dedicated rackmount box (running Linux) with multiple serial
> lines. You telnet to a specific port and get the serial screen.
> I would spec one of these for a new cluster.
> We ship them on many of our clusters, as an option, and customers are
> always happy with them.

It's the TS 3000.  32 ports for $3995 in quantity one.

> The alternative is to use a single flying lead from the head node

Doesn't this require someone on-site?  I want remote management, not 
local.

> I would spec your new cluster with IPMI cards, plus a Cyclades.
> Have your systems supplier set all the BIOSes to do a serial redirect,
> and enable serial consoles at the Linux boot stage.
> This is standard with all our clusters.
>
> I'll give you some help if you email me off-list.

Thanks!  So far it seems quite clear.  Two other potential options 
that have been suggested are:

[1] if the BIOS can redirect console IO to a USB port, then use 
USB-ethernet converters plus a cheap oversubscribed ethernet network and a 
dedicated box to gather the output, or

[2] use the Linux netconsole driver to redirect console output to an 
ethernet port, then gather this output (eg, kernel printk()s) with a 
dedicated collection box.  The total cost of this option is one 
inexpensive 24 or 48 ports switch in each rack, pluse one more switch to 
concentrate those together.  This would cost under $20/node.

Cheers,
 	Bruce