[Beowulf] delayed savings time crashes

[David Kewley]

On Wednesday 12 April 2006 12:12, Florent Calvayrac wrote:
> David Kewley wrote:
> >David,
> >
> >The reboots were due to a City of Pasadena power glitch at 9:17 that
> >morning. :)  It was raining, and a 34kV city feeder line that runs
> > between the generating plant at the entrance of the 110 and a
> > substation at Del Mar & Los Robles faulted.  The responsible breaker
> > took 13 cycles to break, during which time the single-phase voltage
> > seen at Caltech dropped to about 75V.
> >induce massive 12Hz oscillations on the room's power lines.
> >
> >As for the time glitch, that is probably induced by the fact that
> > Daylight Savings Time changes only take place on the "system" clock,
> > and in a standard Red Hat system those changes only get synced to the
> > hardware clock upon a clean shutdown.  So if your machine crashes after
> > a DST change, then upon bootup syslogd gets its time from the hardware
> > clock, which is wrong. The system clock is only corrected later in the
> > bootup sequence, when ntpd starts.  The best solution is probably to
> > set the hardware clock to UCT rather than local time.  UCT doesn't
> > undergo step changes like most timezones in the U.S. do, so the
> > compensation for DST happens dynamically in software, rather than
> > requiring a hardware clock change.
>
> Why don't you use a line conditioner ? It's much cheaper than a similar
> powered UPS ($1000 for 10kW),  many UPS dont guarantee voltage and
> waveform excepted during clean power cuts, anyway.  A line conditioner is
> very handy in time of brownouts like during August thunderstorms.  We have
> a Salicru one on our cluster  and have a much better MTBF on our compute
> nodes in comparison with the ones of a computing  room nearby.   

I wasn't here when the room was designed.  But I understand that at the 
time, it was unknown what the machine load in the room would be -- it was 
quite possible that we'd get several professor-specific clusters rather 
than the monolith that we have now.  So rather than try to make only 
certain outlets in the room UPS-backed, or make individual users supply 
their own UPSes, they decided to get a big UPS to feed all the outlets.

If I were involved in the design of another room like this, with a 
monolithic cluster, I'd advocate for getting a UPS big enough to handle the 
servers and network gear, and a wiring plan to distribute UPS power to just 
those boxes.  The, as you say, put a power conditioner in front of the 
remaining outlets.

Our peak power is 320kVA -- any idea what the offerings are for a 
conditioner of that size, or a set of conditioners (vendor, price, and 
cubic footage of the box(es))?  I'm just curious.

Our UPS is a Liebert Series 600, which (although I don't actually know the 
specs) I'd expect would handle any power event very well.  Liebert makes a 
more modern large UPS (the nPower series), but I'd expect them to have all 
their best tricks at the time of design rolled into the Series 600.

Problem is, the Series 600 was designed before modern computer Power Factor 
Correction power supplies (at least before they were *mandated* by the 
NEC), and it doesn't handle a large load of that type very well.  That's 
the issue that Liebert is still working on, and the reason we're running in 
bypass.

These PFC power supplies' feedback circuits lock up if the power source 
impedance at ~10-40 Hz isn't low enough (frequency depending on the 
specific power supplies, and perhaps on details of the room's power line 
diagram).  Our PDUs see major phase current oscillation at ~12Hz when the 
UPS is powering a full load of 320kVA, and there's even a tiny bit of 
oscillation when we're in bypass (powered directly by our own substation).

> Besides, I was confronted with the same problem about daylight saving
> time. Just added a
>
> clock -w
>
> after the NTP synchronization in cron.daily so that the time would be
> corrected automatically on the hardware clock.

Are you saying that you're both running ntpd and doing a daily ntpdate?  If 
so, why?  On my systems, there is no NTP syncronization in cron.daily, and 
I've never seen a need to set the system clock outside the ntp initscript 
(and /etc/rc.d/rc.sysinit).

On distributions by Red Hat (RHL, RHEL, Fedora), here is how the system 
(software-tracked) and hardware (CMOS on x86) clocks get set and 
maintained:

Early in boot, the system clock is set from the hardware clock using this 
code snippet in /etc/rc.d/rc.sysinit (taken from RHEL3):

--------------------------

# Set the system clock.
update_boot_stage RCclock
ARC=0
SRM=0
UTC=0

if [ -f /etc/sysconfig/clock ]; then
   . /etc/sysconfig/clock

   # convert old style clock config to new values
   if [ "${CLOCKMODE}" = "GMT" ]; then
      UTC=true
   elif [ "${CLOCKMODE}" = "ARC" ]; then
      ARC=true
   fi
fi

CLOCKDEF=""
CLOCKFLAGS="$CLOCKFLAGS --hctosys"

case "$UTC" in
    yes|true)   CLOCKFLAGS="$CLOCKFLAGS --utc"
                CLOCKDEF="$CLOCKDEF (utc)" ;;
    no|false)   CLOCKFLAGS="$CLOCKFLAGS --localtime"
                CLOCKDEF="$CLOCKDEF (localtime)" ;;
esac
case "$ARC" in
    yes|true)   CLOCKFLAGS="$CLOCKFLAGS --arc"
                CLOCKDEF="$CLOCKDEF (arc)" ;;
esac
case "$SRM" in
    yes|true)   CLOCKFLAGS="$CLOCKFLAGS --srm"
                CLOCKDEF="$CLOCKDEF (srm)" ;;
esac

/sbin/hwclock $CLOCKFLAGS

action $"Setting clock $CLOCKDEF: `date`" date

--------------------------

Next, the ntpd initscript (if enabled in the present runlevel) slews the 
system clock to the correct time using ntpdate, then starts ntpd.  After 
that, ntpd should keep the system clock well-synced.

At system reboot or halt, the hardware clock setting is done by this code 
snippet in /etc/init.d/halt (taken here from RHEL3), which you'll notice is 
almost but not identical to the /etc/rc.d/rc.sysinit code; the main 
difference is --systohc versus --hctosys:

--------------------------

# Sync the system clock.
ARC=0
SRM=0
UTC=0

if [ -f /etc/sysconfig/clock ]; then
   . /etc/sysconfig/clock

   # convert old style clock config to new values
   if [ "${CLOCKMODE}" = "GMT" ]; then
      UTC=true
   elif [ "${CLOCKMODE}" = "ARC" ]; then
      ARC=true
   fi
fi

CLOCKDEF=""
CLOCKFLAGS="$CLOCKFLAGS --systohc"

case "$UTC" in
   yes|true)
        CLOCKFLAGS="$CLOCKFLAGS -u";
        CLOCKDEF="$CLOCKDEF (utc)";
        ;;
   no|false)
        CLOCKFLAGS="$CLOCKFLAGS --localtime";
        CLOCKDEF="$CLOCKDEF (localtime)";
        ;;
esac

case "$ARC" in
   yes|true)
        CLOCKFLAGS="$CLOCKFLAGS -A";
        CLOCKDEF="$CLOCKDEF (arc)";
        ;;
esac
case "$SRM" in
   yes|true)
        CLOCKFLAGS="$CLOCKFLAGS -S";
        CLOCKDEF="$CLOCKDEF (srm)";
        ;;
esac

runcmd $"Syncing hardware clock to system time" /sbin/hwclock $CLOCKFLAGS

---------------

If I were to recommend changes to Red Hat (or to sysadmins of Red Hat 
systems), I'd say do these three things:

* In the ntpd initscript, after slewing the system clock, set the hardware 
clock from the system clock.

* Add a script to /etc/cron.hourly/ that sets the hardware clock from the 
system clock.  Because the scripts in cron.hourly are by default run at 1 
minute past every hour (see /etc/crontab), this will take care of Daylight 
Saving Time changes that happen on the hour (do any timezones have shifts 
on the half-hour?), with only 1 minute's window for the system to go down & 
miss the setting of the hardware clock.  It also serves generally to keep 
the hardware clock well-synced to true time.

* Just make UTC the default for the hardware clock, avoiding Daylight 
Savings Time issues altogether.

Now that I've laid all that out, I will strongly consider doing all that on 
my own systems. :)

David