[Beowulf] IBM LCM16 capacity

[Stuart Barkley]

On Sat, 7 Sep 2013 at 13:55 -0000, John McCulloch wrote:

...sorry for the late response, but I hope this is useful...

> Can anyone relate experience with implementing the IBM LCM16 KVM
> switch in a tiered configuration?  We are spec'ing approximately 500
> node x86 cluster.

We have several of what are LCM8s (probably, I'm not where I can
actually verify the model, but the rear panel is the same/similar).

> Since you can daisy chain up to 16 servers per port, I would expect
> the supported maximum servers to be greater than 256 with two or
> three tiered LCM16 units.  Or does each port connect only one target
> system in a tiered configuration.

We do not have multiple units tiered so I can't comment on that (but I
will down below anyway).

I think you may have misread things.  A single LCM16 should handle 256
systems.  Our ~LCM8 will handle 8 x 16 nodes.  Each of the 8 ports
connects to one rack of up 16 nodes with daisy chained dongles (see
figure 5 on your link below).  A couple of our racks have 17 nodes in
them connected to the LCM8.  The last node to power up just doesn't
get seen.  Powering off one of the other nodes make the missing one
become visible.

We manage all of our nodes with IPMI/SOL (serial over lan) which is
enough for almost everything.  SOL is a little clumsy but works well
enough.  We skip the "Virtual Media" support on the BMC.

One cluster has one LCM8 in each row of up to 8 racks.  I haven't
bothered connecting more recent compute nodes.  These newer nodes have
never seen a keyboard, monitor or mouse (or anything with USB).

Another cluster only has one LCM8 for the management nodes and I
connected a spare dongle to a 100 ft cat5 cable to drag to any system
which really needs KVM support (the main purpose seeming to be vendor
diagnostics).

With a 1U IBM keyboard/display unit these are useful for management
nodes instead of a crash cart.  Ours are similar to the one at the top
of figure 4 (without the Media/USB additions).

> The specs state only up to 256 target systems are supported in a
> tiered configuration.
>
> http://www.redbooks.ibm.com/abstracts/tips0788.html

Okay, I think I see the tiered operation now.  If you are using the
"Virtual Media" dongles you can't daisy chain the dongles so would
need one LCM16 in each rack of 16 nodes.  A 17th LCM16 would be
upstream of the others for a total of 16 x 16 nodes.

I don't really get this "Virtual Media" silliness or even putting VGA
connectors and supporting electronics on compute nodes.  With over 400
compute nodes I really don't want anything running that thinks it
wants a DVD or USB drive (even a virtual one).

Stuart Barkley
-- 
I've never been lost; I was once bewildered for three days, but never lost!
                                        --  Daniel Boone