[Beowulf] $1, 279-per-hour, 30, 000-core cluster built on Amazon EC2 cloud

[Lux, Jim (337C)]

On 10/4/11 7:55 AM, "Rayson Ho" <raysonlogin at gmail.com> wrote:

>On Mon, Oct 3, 2011 at 3:21 PM, Robert G. Brown <rgb at phy.duke.edu> wrote:
>> I would be very interested in seeing the
>> detailed scaling of "fine grained parallel" applications on cloud
>> resources -- one point that the talk made that I agree with is that
>> embarrassingly parallel applications that require minimal I/O or IPCs
>> will do well in a cloud where all that matters is how many instances you
>> can run of jobs that don't talk to each other or need much access to
>> data.  But what of jobs that require synchronous high speed
>> communications?
>
>Amazon (and I believe other cloud providers have something similar?)
>introduced Cluster Compute Instances with 10 Gb Ethernet. For
>traditional MPI workloads, the real advantage is actually from HVM
>(Hardware VM), as it cuts the communication latency by quite a lot.
>
>
>> What of jobs that require access to huge datasets?
>
>Getting data in & out of the cloud is still a big problem, and the
>highest bandwidth way of sending data to AWS is by FedEx. In fact, it
>is quite often that the fastest way to send data from one data center
>to another when the data size is big.

The classic: nothing beats a station wagon full of tapes for bandwidth.
(today, it's minivan with terabyte hard drives, but that's the idea)
>
>
>
>> Ultimately the problem comes down to this.  Your choice is to rent time
>> on somebody else's hardware or buy your own hardware.  For many people,
>> one can scale to infinity and beyond, so using "all" of the
>> time/resource you have available either way is a given.  In which case
>> no matter how you slice it, Amazon or Google have to make a profit above
>> and beyond the cost of delivering the service.  You don't (or rather,
>> your "profit" is just the ability to run your jobs and get paid as usual
>> to do your research either way).  This means that it will always be
>> cheaper to directly provision a lot of computing rather than run it in
>> the cloud, or for that matter at an HPC center.
>
>Provided that the machines are used 24x7. A lot of enterprise users do
>not have enough work to load up the machines. Eg, I worked with a
>client that has lots of data & numbers to crunch at night, and during
>day time most of the machines are idle.

In a situation where you've got an existing application and data, and you
just want to crunch numbers, and you pay either cloud or in-house, then
you make the choice based on the incremental cost.

However, even at the smallest increment on a cloud/hosted scheme, you have
to pay from CPU second #1 (plus the fixed overhead of getting the job
ready to go).

If you have a cluster in house, there is likely a way to get a test job
run essentially for free (perhaps on an older non-production cluster).
That test job provides the performance data and preliminary results that
you use in preparing the proposal to get real money to pay for real
computation.

This has been my argument for personal clusters... There's no accounting
staff or administrative person watching over you to make sure you are
effectively using the capital investment, in the same sense that most
places don't care how much idle time there is on your desktop PC.  If
you've got an idea, and you're willing to put your own time (free?) into
it, using the box that happens to be in your office or lab, nobody cares
one way or another, as long as your primary job gets done.
Notwithstanding that there ARE places that do cycle harvesting from
desktop machines, but the management and sysadmin hassles are so extreme
(I've written software to DO such harvesting, in pre-Beowulf days).. Those
kinds of places go to thin clients and hosted VM instances eventually, I
think.


Where an Amazon could do themselves a favor (maybe they do this already)
is to provide a free downloadable version of their environment for your
own computer, or some "low priority cycles" for free, to get people
hooked.  Sort of like IBM providing computers for cheap to universities in
the 60s and 70s. Razors, razor blades. Kindles, e-books. Subsidized
cellphones, 10 cent text messages. Give us your child 'til 7, and he's
ours for life.


>