[Beowulf] Utility Supercomputing...

[James Cuff]

On Fri, Mar 1, 2013 at 11:31 PM, Mark Hahn <hahn at mcmaster.ca> wrote:
>>
>> http://www.hpcwire.com/hpcwire/2013-02-28/utility_supercomputing_heats_up.html

Hi Mark,

Great points you raise here!  Lovely to have this discussion, let me
have a crack at some of them in line.  This is fun!

> well, it's HPC wire - I always assume their name is acknowledgement that
> their content is much like "HPC PR wire", often or mostly vendor-sponsored.
> call me ivory-tower, but this sort of thing:
>
>         Cycle has seen at least two examples of real-world MPI applications
>         that ran as much as 40 percent better on the Amazon EC2 cloud than
>         on an internal kit that used QDR InfiniBand.

> really PISSES ME OFF.  it's insulting to the reader.  let's first assume
> it's not a lie - next we should ask "how can that be"?

Exactly - should always be asking questions.  What was that old
phrase?  There are lies, damn lies and marketing? (or something like
that, I forget the phrase) ;-)

> EC2 has a smallish amount of virt overhead and weak interconnect, so why
> would it be faster?  AFAIKT, the only possible explanation is that the "internal kit"
> was just plain botched.  or else they're comparing apples/oranges (say, different
> vintage cpu/ram, or the app was sensitive to the particular cache size,
> associativity, SSE level, etc.)  in other words, these examples do not
> inform the topic of the article, which is about the viability of cloud/utility HPC. the article
> then concludes "well, you should try it (us) because it doesn't
> cost much".  instead I say: yes, gather data and when it indicates your
> "kit" is botched, you should fix your kit.

In essence all of the above.  Cycle has been a bootstrapped company
for the last 7 years, this basically reflects that they *have* to
respond and react directly to customers.  There's no time for anything
in the biz that does not directly improve customer work loads.  You
are right there are 1,000's of things that can effect performance.
Bad kit, slow kit, old kit, shabby kit, bad admins, poor networks, bad
code, bad schedulers, bad hair days etc...  The run we talked about
had some of all of this.  The cool thing about IaaS providers is they
can (not always) get a hold of more recent kit, and if you are smart
about how you use it you can see huge benefits from all of this, even
down to simple changes to CPU spec at a simple level, to much larger
wins if you fix all of the bad/old/shabby/poor stuff.

> I have to add: I've almost never seen a non-fluff quote from IDC.  the ones
> in this article are doozies.

I'll take the fifth on that one :-)

>> that are only great in straight lines ;-)  Another thing to think of is
>> total cost per unit of science. Given we can now exploit much larger
>
> people say a lot of weaselly things in the guise of TCO.  I do not really
> understand why cloud/utility is not viewed with a lot more suspicion.
> AFAIKT, people's thinking gets incredibly sloppy in this area, and they
> start accepting articles of faith like "Economies of Scale".  yes, there is
> no question that some things get cheaper at large scale.  even if we model
> that as a monotonic increase in efficiency, it's highly nonlinear.
>
> 1. capital cost of hardware.
> 2. operating costs: power, cooling, rent, connectivity, licenses.
> 3. staff operating costs.

I hear you.  Numbers have to take in to account all of these factors,
you are absolutely spot on.

> big operations probably get some economy of large-scale HW purchases.  but
> it's foolish to think this is giant: why would your HW vendor not want to
> maintain decent margins?

Yup - everyone needs to make a buck at a certain point,
price/performance is a serious issue.  In all things you get what you
pay for. $ / ABV is one of my fave optimization routines, albeit at
extreme values of $ the algorithm always resolves to shabby domestic
lager, or something random out of the well... but I digress.
*chortles*.

> power/cooling/rent are mostly strictly linear once you get past trivial
> clusters (say, few tens of racks).  certainly there is some economy
> possible, but there's isn't much room to work with.  since power is about 10% of
> purchase cost per year, mediocre PUE makes that 13%, and because we're
> talking cloud, rent is off the table.  I know Google/FB/etc manage PUEs of
> near 1.0 and site their facilities to get better power prices.  I suspect
> they do not get half-priced power, though. and besides, that's still only
> going to take the operating component of TCO down to 5%.  at the rate cpu
> speed and power is improving, they probably care more about accelerated
> amortization.

PUE is starting to be a big issue.  We built the MGHPCC as both a
responsible response to sustainable energy costs (hydro), and the fact
that on campus machine rooms were heading towards the 3.0 PUE scale,
which is just plain irresponsible.

> staff: box-monkeying is strictly linear with size of cluster, but can be
> extremely low.  (do you even bother to replace broken stuff?).  actual
> sysadmin/system programming is *not* a function of the size of the facility
> at all, or at least not directly.  diversity of nodes and/or environments
> is what costs you system-person time.  you can certainly model this, but
> it's not really part of the TCO examination, since you have to pay it either way.
> in short, scale matters, but not much.

Node diversity happens in IaaS also.  We have spun tens of thousands
of nodes up in popular IaaS environments.  At that scale you have to
get extremely clever in how you do this, it's the software equivalent
of "box-monkeying", just at a level with systems and what folks call
"devops" - the needle is just moving, it's the same concept just in
software.  Smart software costs a whole lot less than fixed
infrastructure.

> so in a very crude sense, cloud/utility computing is really just asking
> another company to make a profit from you.  if you did it yourself, you'd
> simply not be making money for Amazon - everything else could be the same.
> Amazon has no special sauce, just a fairly large amount of DIN-standard
> ketchup.  unless outsource-mania is really a reflection of doubts about
> competence: if we insource, we're vulnerable to having incompetent staff.

Re: "special sauce" - there is a fair amount of it at scale. I've seen
inside the source for what Cycle does, and have also run 25K+ proc
environments, we all need some sauce to make any of this large scale
stuff even slightly palatable!

> the one place where cloud/utility outsourcing makes the most sense is at
> small scale.  if you don't have enough work to keep tens of racks busy,
> then there are some scaling and granularity effects.  you probably can't
> hire 3% of a sysadmin, and some of your nodes will be idle at times...

Agree - I've seen significant win at both the small end and the large
end of things.  Utility being a phrase that identifies how one can
turn on and off resource at the drop of a hat.  If you are running
nodes 100% 365d/y on prem is still a serious win.  If you have the odd
monster run, or are in a small shop with no access to large scale
compute, utility is the only way to go.  Reminds me when I first came
to the USA, we never owned a car, did not really need it, but when we
did we used a company called "Zip car", they rent by the hour.  My, oh
my, if I'd have used that for a whole year we would have gone broke,
but it is totally awesome to use for a few hours, and then go park it
- the clock stops ticking, it's a great idea.  Even better to stretch
that analogy, take the Ferrari you can rent in Vegas - sure per hour
it is crazy pricing, but blasting round the strip for a few hours gets
you joy you could never afford if you wanted to buy one of Mr Enzo's
motors for yourself.

Clearly, John Hearns has a subliminal effect on all my beowulf
postings *chuckles!*

> I'm a little surprised there aren't more cloud cooperatives, where smaller
> companies pool their resources to form a non-profit entity to get past these
> dis-economies of very small scale.  fundamentally, I think it's just that
> almost anyone thrust into a management position is phobic about risk.
> I certainly see that in the organization where I work (essentialy an
> academic HPC coop.)

You guys proved this out:

http://webdocs.cs.ualberta.ca/~jonathan/PREVIOUS/Grad/Papers/ciss.pdf

It's a wonderful idea, and the basis for my prior work in .edu for
http://www.mghpcc.org

> people really like EC2.  that's great.  but they shouldn't be deluded into
> thinking it's efficient: Amazon is making a KILLING on EC2.

It's as efficient as "the bench" shows you, no more no less.

>> systems than some of us have internally, are we are starting to see
>> overhead issues of vanish due to massive scale, certainly at cost?  I know
>
>
> eh?  numbers please.  I see significant overheads only on quite small
> systems.

You are right numbers were missing.  Here's an example of a recent Cycle run:

http://blog.cyclecomputing.com/2013/02/built-to-scale-10600-instance-cyclecloud-cluster-39-core-years-of-science-4362.html

So ignoring any obvious self promotion here, 40yrs of compute for
$4.3K is a pretty awesome number, least to me.  Probably compares with
some academic rates, at least it is getting close, comparing a single
server:

http://www.rhpcs.mcmaster.ca/current-rates

I do have to say  - I absolutely LOVE that you guys put in real FTE
support for these servers - this is a very cool idea, I never did
implement charge back in my old gig, what you guys are doing here is
awesome!


>> for a fact that what we call "Pleasantly Parallel" workloads all of this
>
>
> hah.  people forget that "embarassingly parallel" is a reference to the
> "embarassment of riches" idiom.  perhaps a truer name would be
> "independent concurrency".

Nothing quite like an embarrassment of riches!  As Hannibal Smith so
famously said: "I love it when a plan comes together"

>> I personally think the game is starting to change a little bit yet again
>> here...
>
>
> I don't.  HPC clusters have been "PaaS cloud providers" from the beginning.
> outsourcing is really a statement about a organization's culture: an assertion that if
> it insources, it will do so badly.  it's interesting that some very large organizations
> like GE are in the middle of a reaction from outsourcing...

My point was more around the crazy "elastic scale" one can now pull
off with the appropriate software and engineering available from top
tier IaaS and PaaS providers.  To me this is a fascinating and very
exciting time for grand challenge, large scale awesome science.  It's
lovely to have another thing in the tool kit to be able to exploit
both on and off prem compute.  Although I'm only a few weeks into this
gig, I do see this bridging and enabling technology to help us do what
we all want to do - make bigger science happen, provide better
outcomes, design better systems, therapeutics and further understand
the insane and inordinately complicated world that we live in.

> regards, mark hahn.

Cool beans - thanks ever so for the follow up, this is a great
conversation to have!

Much appreciated!