[Beowulf] extreme dynamic underclocking and undervolting

[David Mathog]

Long ago I started keeping my notes and a bit of editorial content
on idle power consumption for various computers and related here:

  http://saf.bio.caltech.edu/saving_power.html

A few days ago I realized that there was no Intel Core information
in there.  Since I don't have any myself, I hunted down an
iMac and a PC and found much to my surprise, that while the
Core processors were quite efficient when idling, there was
apparently no way to adjust the power consumption downward any
further via Enhanced Speed Step (or whatever Intel calls it
today.)  I assume that the CPUs in these two boxes supported this
capability, but the BIOS (or it's equivalent on a Mac) apparently
didn't enable this feature.  Sure it's a small sample, but in this
day and age I really expected it to be enabled by default pretty
much everywhere.

Anyway, that got me thinking about idle power consumption on clusters. 
Many of you have machines that run at 100% CPU 24/7, and for
those systems the following discussion is irrelevant.  But there
are other clusters around that tend to sit for long periods of
time between jobs, and whatever power they are using while waiting
for a job is pretty close to a total waste.  This is even more
common on regular PCs, where CPU usage is extremely "bursty".
The thing is, on pretty much every machine I've seen (exception:
some laptops) there is a gaping hole between the lowest power
level on a running machine, and the power level when it
goes to sleep.  Putting idle nodes all the way to sleep would
save the most power, but it is a nightmare in terms
of waking them back up again.  Besides the issue of disks that
might not spin back up, there is the problem of the (many)
network protocols which are going to time out and break 
connections.  Also returning from sleep nodes tends to be relatively
slow, taking many seconds to many minutes, depending on a whole
lot of variables.

So it would be nice if the range of underclocking / undervolting
adjustments  provided on compute nodes extended quite a bit further
towards the lower end than it currently does.  Typically
idle is something like 70-80W at the lowest clock speed and sleep
is 2-4W.  There's a lot of room in there to work with.  Why is there
not a system that can slow down far enough to use only 15W and
still run, albeit very slowly?  On a diskless node 7-10W might
even be possible.  Machines running in these nodes would be
alive enough to keep network connections open, and would be
a whole lot easier to get back up to full speed than the
equivalent machine in a sleep state.  Assuming the transition
speed is similar to Cool'N Quiet we're talking much less than a second
to speed back up again.  

There are a lot of articles around about statically underclocked
machines, which proves that running modern hardware slowly is
possible, but the statically underclocked machines cannot be
sped up again - they start slow, and stay slow.  Via sells some
processors like the C7 which will operate over a very wide power range,
but unfortunately the fastest those will crunch isn't anywhere
near the speed of an Opteron or Core.

Big iron SMP machines often the ability to shut off CPUs
while the machine is running, well, except for the last one
obviously.  With quad cores pretty much here, and octo cores
on the horizon, one might imagine large power savings at idle 
could be achieved the same way on these chips.  Can any of the
high core number Opterons or Core CPUs power down unused
cores now?

In closing, does anybody currently make a rack mountable 
compute node with a really, really, really, low idle power mode,
and also competitive performance when running at 100%?

Regards,

David Mathog
mathog at caltech.edu
Manager, Sequence Analysis Facility, Biology Division, Caltech