[Beowulf] Academic sites: who pays for the electricity?

[Robert G. Brown]

On Wed, 16 Feb 2005, David Mathog wrote:

> In most universities services like electricity, water, and 
> A/C are paid for by the school.  To do so they take "overhead"
> out of every grant.  Partially as a consequence of this they
> typically have a very poor ability to meter usage on a room
> by room basis.
> 
> Now somewhere between the 10 node Pentium II beowulf sitting on
> a lab bench and the 1000 node dual P4 Xeon beowulf in a machine
> room that takes up half the basement the cost of the electricity
> (both for power and A/C) goes from  a minor expense to a major
> one.  Really major. For instance, in that hypothetical large machine,
> at 10 cents per kilowatt hour (a round number), assuming 100 watts
> per CPU (another round number) that's:
> 
>   1000  (nodes) *
>      2  (cpus/node) *
>      .1 (kilowatts/cpu) *
>      .1 (dollars/kilowatt-hour) *
>   365   (days /year) *
>    24   (hours/day) =
> -----------------------
>   175200 dollars/year

I usually assume $1/watt/year (including AC) which is likely to be good
within 20% or so depending on the actual cost of electricity in your
area and amount of AC required on a seasonally averaged basis.  That
yields an estimate of $200K in your example -- not really different,
just easier to do in your head as a round number.

> 
> The A/C expense is going to vary tremendously depending upon
> the outside temperature.  It's going to be much higher for us
> in Southern California than for a site in Anchorage.
> 
> "Typical" lab usage is widely variable but I'd be amazed
> if most biology or chemistry labs burn through even 1/10th this
> much for the equivalent lab area.  Some physics lab running
> a tokamak might come close.
> 
> 
> Anyway, the question is, have any of the universities said "enough
> is enough" and started charging these electricity costs directly?
> If so, what did they use for a cutover level, where usage was
> "above and beyond" overhead?

This issue has most definitely come up at Duke, although we're still
seeking a formula that will permit us to deal with it equitably.  This
is only one of several pieces of overhead associated with clusters that
go above and beyond the assumptions that went in to the original
indirect costs formulas.  For example, Duke now charges grants a
"recycling fee" for certain pieces of environmentally toxic end-of-life
hardware (e.g.  monitors, with their lead-filled screens).  Then there
are the really HUGE costs for physical space renovations as valuable and
scarce campus space is converted for use in the burgeoning clusters.

As our Dean of A&S recently remarked, if there aren't any checks and
balances or cost-equity in funding and installing clusters, they may
well continue to grow nearly exponentially, without bound (Duke's
cluster population is doubling almost according to Moore's Law -- every
couple of years).  Costs associated with those clusters from the space
to hold them, the power to run them, and the people to operate them, all
grow roughly linearly with the number of nodes.  This much is known.

What isn't known is the details of the income stream.  Each cluster (or
part of a cluster) is typically connected with a specific grant-funded
project and its associated income stream.  Indirect costs >>are<<
assessed on those grants; it may be that on average, enough income comes
from those indirect costs to easily support the clusters.  This isn't
crazy -- it is really a question of just what the ratio is of supported
people and other IC-producing expenses are to the number of cluster
nodes associated with the research.  I wish I knew this number -- it
would be very useful in a CWM column;-) -- but I don't, and last I heard
Duke still didn't know either, although they are perhaps moving slowly
towards expending the energy required to find out.  

Finding out isn't trivial -- it involves running down ALL the clusters
on campus, figuring out whom ALL those nodes "belong" to, determining
ALL the grant support associated with all those people and projects and
clusters (since even research done without a cluster by a person who
runs a cluster has to be considered as contributing, as the cluster may
be "essential" to retaining that person), figuring out what the sum of
the indirect costs are on all those grants, and finally connecting that
total to the estimated cost of running all the nodes.  By enabling more
research projects, postdocs, laboratory operations, and other
grant-funded activity to occur their presence on campus might MAKE the
university money, who knows?

Indirect cost formulas actually tend to EXCLUDE capital equipment such
as clusters.  If it didn't the University would have made something on
the order of 50% indirect costs on the roughly $2M the hardware in your
example above would cost, and out of the resulting $1M (noting that the
total grant would have had to be $3M for the hardware alone) plus
overhead on the salary of the 2-3 people likely to be hired to run the
1000 node cluster, they could have easily paid for power for 3-5 years.

So one proposal is to no longer exclude clusters from indirect cost
assessments.  Of course this "solution" creates another problem just as
big -- will granting agencies stand for this?  There is a reason
indirect costs aren't charged on capital equipment and it isn't because
Universities don't WANT to charge them, it is because many granting
agencies flatly refuse to pay them.  Some do -- IIRC, NIH is pretty
tolerant about indirect costs associated with hardware, probably because
in medical research they "expect" to have to support entire labs as
there is less likelihood of having a teaching stream of income to
partially defray the costs.  NSF does not, and I don't believe the DoD
or DOE grants like to as well.

Another is to just force clusters to budget and pay their own utility
bills.  I don't know how this would fly with grant agencies.  They might
be irritated if they had to pay for both the utilities and for indirect
costs on the utility money (basically paying 1.5x or so of the cost of
the power/AC used, so that the University would actually make another
$100K in overhead in your example above, but they might hold still for
the $200K/year for power alone.  They almost certainly WOULD pay for
utilities for clusters in places other than Universities, so this isn't
so big a jump.

> >From an economic perspective having electricity and A/C come out
> of overhead (without limit) grossly distorts the true cost
> of the project over time and can lead to choices which increase
> the total overall cost. For instance, the use of Xeons instead of
> Opterons has little effect on TCO if somebody else is picking
> up the electricity tab, but could change the power consumption
> significantly on a large project.

Absolutely.  Or, using shelved tower units vs 2U rackmounts vs 1U
rackmount nodes, when space is "scarce" and hence expensive.  Or
requiring each node to have remote management hardware, PXE network
cards, 3 year onsite service plans -- all of these choices will be very
differently made depending on how the chooser is constrained and who is
paying for what.

I don't have a really perfect solution to this dilemna, and indeed I
think it is a bit premature to expect one.  When SOME institution does a
real CBA on the total cash flow associated with grant-funded
cluster-based research projects, including the more esoteric benefits
such as "institutional prestige" (which is serious business, don't
forget -- a weight factor that affects ALL grants submitted from an
institution) perhaps we can start to think about which clever idea for
recovering costs is realistic and fair.  In the meantime, budgets of the
groups that actuall pay these costs continue to get a wee bit strained
as the number of nodes and associated costs continue to spiral upward.

Maybe I'll do a column on this soon.  I did a whole article on
infrastructure for Linux Mag a year or two ago, but the particular
aspect of infrastructure that you raise is still unresolved.  I wonder
if I could get Duke people to expedite collecting and assembling the
data required to get the big picture on this...?

   rgb

> 
> Regards,
> 
> David Mathog
> mathog at caltech.edu
> Manager, Sequence Analysis Facility, Biology Division, Caltech
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
> 

-- 
Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu