[Beowulf] Academic sites: who pays for the electricity?

[Mark Hahn]

> > room that takes up half the basement the cost of the electricity
> > (both for power and A/C) goes from  a minor expense to a major
> > one.

but it's not as if the university will fail to notice a new 1K-node cluster.
so if they choose not to install metering equipment, it's on them.
we recently built a machineroom for 1.5Kcpus, and sure enough, the U
made us buy metering crap (IMO, the U should pay for it.)  the U also
made us buy harmonic mitigating PDUs because they didn't understand 
what PFC means in a power supply.

> > "Typical" lab usage is widely variable but I'd be amazed
> > if most biology or chemistry labs burn through even 1/10th this
> > much for the equivalent lab area.  Some physics lab running
> > a tokamak might come close.

I guess, I figure around 300W/sq ft is pretty typical for a new HPC
machineroom.  that's obviously more than a bio lab, but how about 
comparing to that glassblower over in arts, or someone pouring a new
alloy in materials engineering?

> > Anyway, the question is, have any of the universities said "enough
> > is enough" and started charging these electricity costs directly?
> > If so, what did they use for a cutover level, where usage was
> > "above and beyond" overhead?
> 
> This issue has most definitely come up at Duke, although we're still
> seeking a formula that will permit us to deal with it equitably.  This

I guess I'm a little surprised - one main Canadian funding agency
(CFI) has a program that provides infrastructure operating funds
(you apply for this after being awarded a capital grant.)  it pays
for electricity as well as sysadmins.

> As our Dean of A&S recently remarked, if there aren't any checks and
> balances or cost-equity in funding and installing clusters, they may
> well continue to grow nearly exponentially, without bound (Duke's

I find that most faculty who have compute needs (and funding) will
seriously consider buying into a shared facility instead.  that's our
(SHARCnet's) usual pitch: let us help you spend your grant, and we'll
give you first cut at that resource, but otherwise take all the pain
off your hands.  most people realize that running a cluster is a pain:
heat/noise, but more importantly the fact that it soaks up tons of 
the most expensive resource, human attention.  do you want your grad
students spending even 20% of their time screwing around with cluster
maintenance?

not to mention the fact that most computer use is bursty, and therefore
very profitably pooled.  a shared resource means that a researcher 
can burst to 200 cpus, rather than just the 20 that his grant might 
have bought.  and after the burst, someone else can use them...

> to hold them, the power to run them, and the people to operate them, all
> grow roughly linearly with the number of nodes.  This much is known.

well, operator cost scales linearly, but that line certainly does not 
pass through zero, and is nearly flat (a 100p cluster takes almost the 
same effort as a 200p one.)

> Finding out isn't trivial -- it involves running down ALL the clusters
> on campus, figuring out whom ALL those nodes "belong" to, determining
> ALL the grant support associated with all those people and projects and

here at least, the office of research services sees all funding traffic,
and so is sensitized to the value of cluster pooling.  the major funding
agencies have also expressed some desire to see more facility centralization,
since the bad economics of a bunch of little clusters is so clear...

regards, mark hahn.