[Beowulf] cloud: ho hum?

[Chris Dagdigian]

My $.02 from what I see in industry (life sciences)

- The ability to transform capital expense money into OpEx money alone 
is pushing some cloud interest at high levels. No joke. Possibly a very 
large cloud interest driver in the larger organizations. This is also 
attractive for tiny startups and companies just leaving the VC 
incubation phase.

- Deployment speed. We have customers who wait weeks after making an IT 
helpdesk request for a new VM to be created. Other customers take 1+ 
years to design, RFP and choose their HPC solution and another 4 months 
to deploy it.  If you can do in minutes (via good DevOps techniques) 
what the IT organization normally takes weeks or months to do then 
you've got some good arguments for targeting cloud environments for 
quick, dev, test and on-off scientific computing environments

- Quick capability gains - in some cases it's quicker and easier to get 
quick access to GPUs, servers with 10Gbe interconnects and well-built 
systems for running MapReduce style big data workflows on cloud platforms

- Data exchange. Cloud is a good place for collaborators to meet and 
work together without punching massive holes in local firewalls. It's 
also a good place to either put data or get data from an outsourced 
provider or collaborator/partner. Many Genome Sequencing outsourcing 
companies can deliver your genomes directly to an EBS or AWS S3 bucket 
these days.

- I'm a believer in the pricing and economies of scale argument in some 
cases. For pricing take AWS S3 as an example - internal IT people who 
snipe at the pricing willfully (or not) seem to ignore the inconvenient 
fact that S3 does not acknowledge a successful object PUT request until 
the data has landed in 3 datacenters.

If you want an honest cost comparison for cloud-based object storage 
then you have to start with legit fully-loaded cost estimates for 
deploying and running an internal petascale-capable system that spans 
three separate facilities. That ain't cheap.

- Truthfully though I don't use or push cloud economic arguments all 
that much these days. It's incredibly easy to distort the numbers anyway 
you want so it's rare to have a

- Ability to do work that was not considered viable at home. The 90,000 
core AWS Top500 cluster that was in the news is a good example. Some 
organizations have HPC or other problems of such scale that running them 
internally is not even on the radar. In rare cases spinning up something 
massive and exotic for a few days is a viable option.

- Cyclical needs. Some of my customers have big compute needs that come 
about only every 3-4 years; most are looking at cloud now rather than 
buying local gear and seeing it depreciate or be under-utilized most of 
the time


I agree that the cloud is overhyped and we certainly don't see a ton of 
HPC migrating entirely to the cloud. What we see in the trenches and out 
in the real world is significant interest in leveraging the cloud for  
Speed, Capability, Cost or "weird" use cases.


-Chrius



	
Tel: | Mobile: