[Beowulf] cpufreq, multiple cores, load

[David Mathog]

I am currently configuring a new (for us) dual Opteron 280 system. 
cpufreq works on this system, moving each pair of cores between 1000 and
2400MHz using the "ondemand" governor.  The interesting thing, at least
from my point of view, is how rapidly the power savings degrade as CPU
load increases.  Naively one might have thought it would go up in steps
corresponding to the power difference between 2400MHz full CPU load on a
core and 1000MHz idle on the same core.  Not so. Here is the data
measured with our Kill-A-Watt, view the table with a fixed width font:

Watts   CPU0 MHz      CPU1 MHz cpuburn#  governor
       core0 core1  core0 core1
115    1000  1000   1000  1000    0      ondemand
157    2400  2400   1000  1000    1      ondemand
199    2400  2400   2400  2400    2      ondemand
214    2400  2400   2400  2400    3      ondemand
228    2400  2400   2400  2400    4      ondemand
172    2400  2400   2400  2400    0      performance
186    2400  2400   2400  2400    1      performance
199    2400  2400   2400  2400    2      performance
214    2400  2400   2400  2400    3      performance
228    2400  2400   2400  2400    4      performance

Starting one cpuburn flips both of the cores on the first CPU
to the faster clock speed, resulting in a 42W (37%) increase in power
consumption.  Starting a second cpuburn apparently schedules it
on one of the cores on the unused second processor, rather than
on the equally unused, but already sped up, second core on the first
CPU.  This flips the remaining two cores also to 2400 MHz, negating any
further benefit from "ondemand" as more cpuburn processes are added.
That is, there are 5 states for increasing cpuburn load but, only the
lowest two have different power consumption for "ondemand" than for
"performance".

I have not repeated this experiment yet with the "conservative"
governor, but since cpuburn is intentionally such a cpu hog, I expect
the results would be about the same.

Anyway, this is just 4 cores total, but it makes me wonder what happens
with a system having say, two quad core processors, where if the same
sort of scheduling/cpufreq logic apply, two CPU saturating jobs (still
only 1/4 of total available CPU capacity) will effectively negate the
energy saving modes.  For instance, imagine some future 24 core behemoth
that acts the same way.  One might almost dispense with power saving
modes altogether if one CPU intensive job is going to kick the other 23
cores into a high power state.  Or do the newer CPUs, either AMD's or
Intel's, allow different frequencies on each core of a CPU?  

(Kernel 2.6.31.12, Arima HDAMAI motherboard, Mandriva 2010.0).

Regards,

David Mathog
mathog at caltech.edu
Manager, Sequence Analysis Facility, Biology Division, Caltech