[Beowulf] UPS & power supply instability

[David Mathog]

David Kewley <kewley at gps.caltech.edu> wrote:


> 
> The problem is this: We can fire up our cluster to about 40% of maximum 
> load and everything is fine.  But if we go over some threshold right 
> around 40% of max, the output currents from the PDUs go unstable.  

Ugh.

It might be a useful diagnostic to place a purely resistive load
on one of the UPS units and see if the Liebert is also
unstable with that.  Unless you happen to have access to a
lot of desk lamps you can construct a test rig without
spending _too_ much money by wiring a large number of
incandescent bulb sockets onto some sort of heat resistant
frame, each set of 5 100W bulbs (wired in parallel) with its
own cord.  100 W bulbs, sockets, and cords are all farly cheap. 
As a bonus you could also use this to actually test how long
the Lieberts will really provide power
without risking blowing out any computers should a defective
unit send a big spike out when it finally shuts down.

Caveat, I've never tried this on a UPS nearly that size.  Your room
has the AC to handle the power but one suspects that it will get
a bit toasty near the resulting wall of bulbs.  Bright too.

Anyway, if the Liebert is unstable with the resistive load you
(meaning a qualified electrician) could then place this load
directly on the output of the Liebert.  If it's still unstable
then the Liebert is clearly to blame.  If not, then the wiring is
suspect.

But that advice is for the "we don't have much money to diagnose
this" situation, which is usually what my position is but
hopefully doesn't apply in your case.   Since your division
already spent big $$$ on the room they can now be reasonably
expected to part with more $ to find some experienced
electrician, who has built big machine rooms before, to figure
out what's wrong.  I suggest you start calling the national
labs and other huge cluster installations to obtain the names of
some electricians who really know computer room wiring.  Once this
expert figures out what's wrong you can still use your existing
electrical contractors to rewire things under his supervision.

Regards,

David Mathog
mathog at caltech.edu
Manager, Sequence Analysis Facility, Biology Division, Caltech