[Beowulf] Academic sites: who pays for the electricity?

[Robert G. Brown]

On Sat, 19 Feb 2005, Mark Hahn wrote:

> > As our Dean of A&S recently remarked, if there aren't any checks and
> > balances or cost-equity in funding and installing clusters, they may
> > well continue to grow nearly exponentially, without bound (Duke's
> 
> I find that most faculty who have compute needs (and funding) will
> seriously consider buying into a shared facility instead.  that's our
> (SHARCnet's) usual pitch: let us help you spend your grant, and we'll
> give you first cut at that resource, but otherwise take all the pain
> off your hands.  most people realize that running a cluster is a pain:
> heat/noise, but more importantly the fact that it soaks up tons of 
> the most expensive resource, human attention.  do you want your grad
> students spending even 20% of their time screwing around with cluster
> maintenance?
> 
> not to mention the fact that most computer use is bursty, and therefore
> very profitably pooled.  a shared resource means that a researcher 
> can burst to 200 cpus, rather than just the 20 that his grant might 
> have bought.  and after the burst, someone else can use them...

A model that we have emulated at Duke, actually, and I mean emulated
literally.  You'll recall that we had that offline discussion about your
share cluster operation a few years ago -- that was a fairly important
component that I took into various discussions with provosty-level folks
that led ultimately to the creation of Duke's CSEM.  So we owe you a
debt of gratitude, and possibly even a beer (standard compensation for
cluster support services on list, is it not:-).  However we, like many
institutions, are not pure anything -- the high-cycle-consuming
departments (including physics, but also CPS, chemistry, statistics,
biology, econ) still run their own clusters, although individuals with
BIG needs are encouraged to join the campus grid project and share their
unused cycles.

I feel like we're likely in a gradual and purely voluntary transition
to a point where this model will ultimately dominate and perhaps even
become universal because of the various economies of scale.  Frankly,
even where people DON'T participate in the sharing of resources there
will have to be colocation, as physical space with adequate
infrastructure is both scarce and expensive.  So physics may become part
of the campus grid, etc, although the way things look now there will
continue to be a half dozen separate server-class spaces all over campus
for the hardware.  

I actually feel that this "decentralized centralization" is a good thing
-- we have historical reasons to strongly mistrust overcentralization of
resources at Duke.  One gets economies of scale, perhaps, but at the
expense of permitting empire builders in the bureaucracy entrench and
start to dictate policy, often to groups who are a hell of a lot smarter
and cost effective when left to their own devices.  I mean, we'd still
be using mainframes if it were up to the folks that run centralized
mainframe compute centers.  Hell, at Duke I believe we still ARE running
some mainframes.  Beowulfs themselves are another example of an
innovation that could only have come about from the bottom up.  So we're
trying to operate according to a model where individual enterprise and
innovation aren't squelched and "power" (planning and control) aren't
totally centralized, but we still can centralize primary infrastructure
(the network, the main server rooms) and help coordinate the planning
and operation and sharing of cluster resources without trying to dictate
them.

This isn't as difficult as it might sound, at a University.  All the
faculty are innately mistrustful, cynical, and jealous of control, and
at the same time tend to be really bright and recognize the benefits of
cooperation on some issues.  We've also been blessed for the last 5-8
years with some really good people in Arts and Science Network
Administrators group (from Melissa Mills as the assistant dean at the
top, through the actual sysadmins down to, well, me:-), at the Provost
level (leading to the birth of CSEM), and in the Office of Information
Technology (OIT, which runs the campus network and student/academic
computing).  When you have enlightened leadership that ISN'T empire
building, things are good and even if you try something and it turns out
to be wrong it doesn't matter -- you just fix it or try something else
and move on.

> > to hold them, the power to run them, and the people to operate them, all
> > grow roughly linearly with the number of nodes.  This much is known.
> 
> well, operator cost scales linearly, but that line certainly does not 
> pass through zero, and is nearly flat (a 100p cluster takes almost the 
> same effort as a 200p one.)

Yeah, yeah, yeah -- it's an irregular scale with flat patches and a
minimum buy-in.  However, the discussion concerns the planning of a 1000
node, 2000 CPU cluster, and at that point I think you can start to talk
meaningfully about the number of nodes per support person in planning
discussions where you recognize that you CAN'T run 1000 nodes with the
same effort as 100 nodes.  The hardware support component most
definitely scales per node, and that large a cluster could eat a person
alive with hardware maintenance as the cluster ages or if you hit a bad
patch.  Hardware support is in some sense the scale limiting resource
for a well-designed cluster (whether you provide it locally or contract
it out), especially now that PXE, yum, etc have made it possible for a
single person to install and maintain 1000 nodes on the SOFTWARE side of
things.

But to avoid going into all this is why I used the word "roughly".
Perhaps I should have said "in the limit of a large number of machines".

> > Finding out isn't trivial -- it involves running down ALL the clusters
> > on campus, figuring out whom ALL those nodes "belong" to, determining
> > ALL the grant support associated with all those people and projects and
> 
> here at least, the office of research services sees all funding traffic,
> and so is sensitized to the value of cluster pooling.  the major funding
> agencies have also expressed some desire to see more facility centralization,
> since the bad economics of a bunch of little clusters is so clear...

Yes, ditto here as well, with exceptions as noted above.  Although
granting agencies can swing both ways -- they like to see resource and
cost sharing, but they are also jealous of resource "ownership" and
don't want to fund project A (including cluster) and then find that that
cluster has been hijacked by project B, possibly run by somebody else
entirely.  There's also a WIDE range of what the different agencies view
as reasonable "cost sharing".  What Duke has tried to do is use a
thoughtful model and not a one-size-fits-all plan with mandatory
participation.  The one thing that I think Duke still really needs is
the detailed CBA of existing cluster operations.  I suspect that they'd
find that they are remarkably efficient as is, but I'm not certain.

As always, a pleasure to read what you write.

  rgb

> 
> regards, mark hahn.
> 
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
> 

-- 
Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu