[Beowulf] dollars-per-teraflop : any lists like the Top500?

[Bill Rankin]

> I think the money part will be difficult to get (it is like a
> politically
> incorrect question).

Joe addressed this pretty well.  For the large systems, it's almost always under NDA.

> Nevertheless, you can split the money in two parts: purchase (which I
> am sure
> you will never get) and electric bill for kipping the system up and
> running while you run HPL and when you run stream.

Once you start looking at the power bills (for both the system as well as all the associated infrastructure, like cooling) then you pretty much need to start looking at approximations for the total cost of ownership (TCO).  Depending on the organization, many of these costs are well hidden.  We went through this exercise when I was at Duke (with due credit to Rob Brown who did a lot of the heavy lifting).  Some of the things you have to consider are:

- Power (for both machines and cooling).  Given commercial rates at the time (~2004) this worked out to about $1/Watt/year.  That makes for $300k/year for a 1000 node cluster at 300W/node.

- Don't forget depreciation on all that support equipment.  While your cluster may have a useful lifetime of around 3-5 years, all those air handlers, power conditioners and UPS's have lifetimes too.  Figure 10-15 years (if you can reuse them) and factor that amortized cost into your bottom line.

- Staff salaries, both for administration and operations/monitoring.  Loaded salary for a decent cluster admin may be $100k/year or more.

Bottom line is that you could spend 30%-50% (or more) additional dollars beyond the cost of the hardware just to cover the basic support needs for the facility.

-b