[Beowulf] dollars-per-teraflop : any lists like the Top500?

[Joshua mora acosta]

Now it comes the funny part:
Out of that electric bill of say 300k for 1000 nodes at a reasonable
efficiency of 30% for a real well tuned application (not Linpack which is
>80%) it basically means 90K USD is worth the work. The other 210K US is
electric bill wasted waiting for data to hit the caches.

Joshua

------ Original Message ------
Received: 10:12 AM CDT, 06/30/2010
From: Bill Rankin <Bill.Rankin at sas.com>
To: Joshua mora acosta <joshua_mora at usa.net>, Rahul Nabar <rpnabar at gmail.com>,
Beowulf Mailing List <beowulf at beowulf.org>
Subject: RE: [Beowulf] dollars-per-teraflop : any lists like the Top500?

> > I think the money part will be difficult to get (it is like a
> > politically
> > incorrect question).
> 
> Joe addressed this pretty well.  For the large systems, it's almost always
under NDA.
> 
> > Nevertheless, you can split the money in two parts: purchase (which I
> > am sure
> > you will never get) and electric bill for kipping the system up and
> > running while you run HPL and when you run stream.
> 
> Once you start looking at the power bills (for both the system as well as
all the associated infrastructure, like cooling) then you pretty much need to
start looking at approximations for the total cost of ownership (TCO). 
Depending on the organization, many of these costs are well hidden.  We went
through this exercise when I was at Duke (with due credit to Rob Brown who did
a lot of the heavy lifting).  Some of the things you have to consider are:
> 
> - Power (for both machines and cooling).  Given commercial rates at the time
(~2004) this worked out to about $1/Watt/year.  That makes for $300k/year for
a 1000 node cluster at 300W/node.
> 
> - Don't forget depreciation on all that support equipment.  While your
cluster may have a useful lifetime of around 3-5 years, all those air
handlers, power conditioners and UPS's have lifetimes too.  Figure 10-15 years
(if you can reuse them) and factor that amortized cost into your bottom line.
> 
> - Staff salaries, both for administration and operations/monitoring.  Loaded
salary for a decent cluster admin may be $100k/year or more.
> 
> Bottom line is that you could spend 30%-50% (or more) additional dollars
beyond the cost of the hardware just to cover the basic support needs for the
facility.
> 
> -b
> 
>