[Beowulf] $1, 279-per-hour, 30, 000-core cluster built on Amazon EC2 cloud

[Lux, Jim (337C)]

> -----Original Message-----
> From: beowulf-bounces at beowulf.org [mailto:beowulf-bounces at beowulf.org] On Behalf Of mathog
> Sent: Tuesday, October 04, 2011 1:07 PM
> To: beowulf at beowulf.org
> Subject: Re: [Beowulf] $1, 279-per-hour, 30, 000-core cluster built on Amazon EC2 cloud
> 
> > "Robert G. Brown" <rgb at phy.duke.edu> wrote:
> 
> > Often, but not always, housing, cooling, powering, and even managing
> > the
> > hardware is "free" to the researcher, absorbed into the ongoing costs
> > of
> > the server room and management staff already needed to run the
> > department LAN and servers.
> 
> Not always indeed.  My little machine room houses a half dozen machines
> from other biology division
> people, and they are not charged to keep them there.  However, putting
> a computer in the central
> campus machine rooms is not free.  And new computer rooms, at least
> those of any size, do not
> get free power.  After geology put in this monster:
> 
>    http://www.gps.caltech.edu/uploads/Image/Facilities/Beowulf.jpg
> 
http://citerra.gps.caltech.edu/wiki/Public/Technology
A mere 512 nodes, each with 8 cores.

670W power supply is standard, so let's say about 500 nodes at 700 watts each or 350kW...

HVAC will add on top of that, but I doubt they're loaded to the max.

Call it 400kW.. That's big, but not enormous.  (e.g you can rent a trailer mounted generator for that kind of power for about $1000/day.. the bigger generators one sees on a movie set might be 200-300kW))  CalTrans will only pay $123/hr for a 500kW generator (and fuel cost comes out of that)


But, if you were paying SoCalEdison for the juice..You'd be on (minimum) the TOU-GS-3 tariff.. On peak you'd be paying 0.02/kWh for delivery and 0.104/kWh for the power.  (off peak would be 0.045/kWh)

So call it 12c/kWh on peak.  At 400kW, that's $48/hr, which isn't bad, operating expenses wise.

Let's compare to the EC2.. $1300/hr for 30k cores. 23 core hours/$
            The CITerra is $50/hr for 4000 cores.  80 core hours/$

Yes, one had to go out and BUY all those cores for CITerra. $5000/node, all in, including cabling racks, etc.? What's that, about $1.25M.  Spread that out over 3 years at 2000 hrs/year (we only consider working in the daytime, etc. and you get about $210/hr for the capital cost (for all 500+ nodes..)


So, the EC2 seems like a good solution when you need rapid scalability to huge sizes and you have a big expense budget and a small capital budget.   You could call up Amazon this afternoon and run that 30,000 core job tonight.  And you'd pay substantially for that flexibility (which is how Amazon makes money, eh?)