[Beowulf] $1, 279-per-hour, 30, 000-core cluster built on Amazon EC2 cloud

[Rayson Ho]

On Tue, Oct 4, 2011 at 11:26 AM, Lux, Jim (337C)
<james.p.lux at jpl.nasa.gov> wrote:
> The classic: nothing beats a station wagon full of tapes for bandwidth.
> (today, it's minivan with terabyte hard drives, but that's the idea)

BTW, I've heard horror stories related to routing errors with this
method - truck drivers delivering wrong tapes or losing tapes
(hopefully the data is properly encrypted).


> Notwithstanding that there ARE places that do cycle harvesting from
> desktop machines, but the management and sysadmin hassles are so extreme
> (I've written software to DO such harvesting, in pre-Beowulf days).

The technology part of cycle harvesting is solvable, the accounting
part is (IMO) much harder.

A few years ago I talked to a University HPC lab about deploying cycle
harvesting in the libraries (it's a big University, so we are talking
about 1000+ library desktops). The technology was there (BOINC
client), but getting the software installed & maintained means extra
work, which means an extra IT guy... and means no one wants to pay for
this.

I wonder how many University labs or Biotech companies are doing
organization wide cycle harvesting these days, for example, with
technologies like BOINC:

http://boinc.berkeley.edu/


> Where an Amazon could do themselves a favor (maybe they do this already)
> is to provide a free downloadable version of their environment for your
> own computer,

AMI is not private (in the end, it is IaaS, so the VM images are open).

In fact, StarCluster has AMIs for download & install (mainly for
developers who want to code for StarCluster locally):

http://web.mit.edu/stardev/cluster/download_amis.html


And one can roll a custom StarCluster AMI and upload it to AWS, such
that the image settings are optimized to the needs:

http://web.mit.edu/stardev/cluster/docs/0.91/create_new_ami.html


> or some "low priority cycles" for free, to get people hooked.

AWS Free Usage Tier -- (most people just use the free tier as free hosting):

http://aws.amazon.com/free/

Rayson

=================================
Grid Engine / Open Grid Scheduler
http://gridscheduler.sourceforge.net



>  Sort of like IBM providing computers for cheap to universities in
> the 60s and 70s. Razors, razor blades. Kindles, e-books. Subsidized
> cellphones, 10 cent text messages. Give us your child 'til 7, and he's
> ours for life.
>
>
>>
>
>