[Beowulf] General cluster management tools - Re: Southampton engineers a Raspberry Pi Supercomputer

[Joe Landman]

Uh oh ... this is getting into the TL;DR lengths.

Executive summary:  I think having source/redistribution rights to code 
bases your business/work depend critically upon lowers your risk as it 
gives you options in the event of "issues".  My interpretation of 
Andrew's response is that he disagrees with this.

The following is one of the last in the series of my responses.  This is 
getting tiring, and I have a flight to catch in the morning, so I'll 
limit myself to this one.

On 09/16/2012 04:43 PM, Andrew Holway wrote:
>>> This is also demonstrably false. Just because cluster vendor A is
>>> using a completely open source stack does not mean that you have any
>>> less risk then Cluster Vendor B with their proprietary closed source
>>> stack.
>>
>> Risk is a function of your control over the stack against small or large
>> change of business operations of one of the suppliers.  If one of the
>> critical elements of your stack is completely closed, you have no
>> control over that aspect, and cannot change it out without incurring
>> great cost/time/effort, yes, that is, by definition, an increased risk
>> versus a functionally similar part (of similar operational level and
>> quality) which is completely open.
>
> Its part of the risk. If you sign a contract with a cluster software
> vendor that they will support those features for X amount of years
> then you have mitigated that risk. In comparison, you have no control

Really?  Do you honestly, really believe that a support contract for X 
years will be enforced after a company liquidation (for example)?   Or 
after a change in business conditions/offerings?  There are lots ... 
many ... counter examples of this, with very pissed off customers as a 
result of this.  Recently Joyent had an issue with their techdrive 
customers, as they handled the "lifetime"/forever account issue in an 
initially less than happy manner for those who thought that "forever" 
means "until the heat death of the universe".

I am thinking we are having a significant miscommunication.  Risk is to 
be managed, and that includes risk you can control and risk you cannot 
control.  Risk you cannot control needs to be managed in such a way that 
critical path business dependencies are survivable in the worst case, 
and low cost in the best case.

> over the open source community. It could be that support for a

That is correct.  But you have the source code.  And that, plus a 
small/large/whatever team of developers you HIRE give you fairly close 
to ABSOLUTE control over your stack.

So which is riskier ... a set of risks you cannot control, as you are 
not making critical business decisions for the people you are paying for 
their stack, and they have their interests to consider first and 
foremost ... or a team of people working FOR YOU whom are building and 
supporting the software?  Which is riskier?

> critical part of your infrastructure is no longer supported because
> the lead dev decides he wants to spend more time licking frogs in the
> amazon. You are powerless to prevent this.

Erp ... bad argument.  Many examples of lead devs and critical devs 
moving on and someone taking over.  RPM, X-Windows on Linx, Linux kernel 
(Alan Cox, et al), Lustre,  ...

> By your criteria I could easily argue that the risk with opensource is
> higher then with proprietary software.

... based upon a bad argument, which neglects the history of what has 
actually happened.  You are welcome to make that argument, but it won't 
change that it is wrong.

> case in point: We have based a reasonable chunk of our backend
> infrastructure on openindiana. http://lwn.net/Articles/514046/. What
> do we do now?

As indicated in my response to Greg, this shows that you may not know 
much about the OpenIndiana relationship to other Illumos based 
OpenSolaris builds.  You have several pretty good options.  All are 
trivial to move to.  And if you want to pay a company for support you 
can.  Joyent, Nexenta, ... etc. would be happy to take your money and 
provide you support for your open source software.  And if you insist 
that only OpenIndiana could possibly work for you ... you can pull down 
the source ... hire some devs ... and build it yourself.  Its only a 
risk if you are unwilling to act to mitigate the risk, that is, if you 
want support on a silver platter for free, sure, go ahead and make the 
argument you are making.

We've been looking at SmartOS and Illumian, and we currently support (as 
in deploy) the former, working on the latter.  We have this integrated 
into our Tiburon storage/computing cluster system, as we do for various 
Centos, Debian/Ubuntu flavors.

You can keep using OI forever.  You can pay for patches by hiring 
people, or you can leave it unpatched.  You can switch to another distro 
(very easy, your pools will transfer w/o issue).  There are many things 
you can do.

As I noted, this was a bad argument to make, and the example was even worse.

>> You said my thesis is demonstrably false, and I provided the simple
>> argument that supports my thesis.  Your argument is ... what?  You disagree?
>
> Your saying that open source software is somehow less risky than
> proprietary software. I dont see any evidence for this.

Ok, lets make this very simple.

Two products, one from group X, one from company Y.  They do the same 
thing.  With the same level of quality.  The mailing list of X is 
exceptionally good for support.  The support from company Y is 
wonderful, no call centers, and the CTO/VP Eng helps answer questions 
and solve problems.  The CEO has commit privs on the repository.


Company A decides to use group X's product.  They use the mailing list 
of the group X for support.

Company B decides to use company Y's product.  They contract with the 
company Y for support.

Group X's leader departs.  Decides to 'lick frogs in the Amazon' 
(seriously?)  A new leader emerges and project continues.  Company A is 
in good shape support wise.

Company Y's VP of engineering decides to 'lick frogs in the Amazon'.  A 
new VP is hired and project continues.  Company B is in good shape 
support wise.

So far they look similar, right?

Group X decides to start writing iPhone apps on advice from some random 
VC, ditching their previous work.  Company A has the source code, 
decides to hire a small group to support the code, and group X goes on 
building their free 'lick-a-frog' app.  Risk ameliorated.  Critical 
business dependencies have assured continuity because Company A has 
control over their own software stack.  They may choose to eventually 
fork it, and grow their own open source community to create a wider pool 
of users and developers.  Possibly even monetize support.  But they are 
safe.

Company Y goes out of business.  Chapter 7 liquidation here in the US, 
not sure in each locale what the rough approximation would be, but this 
is dividing up and selling off assets to satisfy creditors.  This is 
what happened to Linux Networx btw (highly oversimplified).  There is no 
one left in the company to support the product.  Support contracts are 
now effectively meaningless.  Customers with critical business 
dependencies upon this code are in deep trouble, and have to start risk 
mitigation by platform shift.

The latter paragraph is part of the reason why (at least in the 80s and 
90s), many large companies and US government agencies DEMANDED escrowed 
copies of the OS source code before they would commit any money to 
purchase.  Its all about the risk mitigation.  If you have source, you 
can always find people ... this is a "solved" problem.  You simply have 
to decide to do it.

Arguing that a defunct company with no capacity to support a product is 
somehow of lower risk to critical business pathway dependencies is ... 
well ...

I am half expecting a Monty Python-ish "is the the 5 minute argument or 
the full half hour?" to be asked any moment.

Its pretty widely accepted by the customers I speak with, that open 
systems are generally lower in risk profile than closed systems.  You 
may disagree with this, that's your right.  Disagreeing doesn't alter 
the risk profile though.

If you have source and the right to distribute/use, you can always hire 
coders/developers to help you.  If you do not have these, then you are 
signing up for a very close relationship with a company that, if they go 
out of business, may wind up leaving you with few options for critical 
systems.

This said, if you are complaining that OI went away and "what should we 
do now"; and someone hired you to do this, and did not ask you for the 
"plan B" should "plan A" go awry, then someone hasn't quite done their 
homework (as Greg suggested).

Though reading what you've been writing on the other topics, I think 
you've settled on Solaris 11.  There's nothing wrong with that choice, 
though I don't think this community is terribly friendly to Oracle these 
days for a number of reasons (MySQL, Lustre, GridEngine, ...)

>>> I have seen Rocks clusters that are an utter bag of shieße because the
>>> people deploying it had no clue and also seen Clusters based on Bright
>>> et al that were perfectly executed. And vice versa for that matter.
>>
>> We understand that you are currently engaged in a Bright Cluster Manager
>> deployment.  We are (see company info in .sig) , for the record, a
>> reseller (in the past anyway) of their tools (though we haven't sold any
>> for a number of reasons that I won't get into).  Do you have a business
>> relationship with them?  I see no problem advocating for them if you
>> disclose your interests, though Chris S/Doug E/...  are the arbiters.
>>
>
> I dont have a business relationship with them. They just let me play
> with their toys. In any case, its not about Bright. Its about your
> original statement that open source deployments are less risky than
> closed source deployments.

And I have supported that statement with some very simple arguments. 
You've opined the opposite, effectively a gainsay, without providing a 
good argument as to why the risk profile for commercial code where you 
have none of the source, redistribution control, etc. is lower.  I just 
saw anecdotes and opinions.  And a bad argument based upon OI leader 
stepping down after getting pissed off at the other Illumos developers.

>
>> And here I thought we were having a nice discussion, and you slip in a
>> little snide comment like this.  Sad.
>
> It was a joke Joe! I was making a point that blind adherence to open
> source can be a bad thing if a better supported, closed source
> alternative exists. By the same stroke, Blind adherence to proprietary
> is also a bad idea when a plethora of truly fantastic software is out
> there for free! Horses for courses etc.

I didn't read it as such.  I didn't see hints that it was a joke.  My 
apologies for not reading the clues that it was.

I look at risk very carefully.  I try to make sure our customers 
understand the issues.  We'll help them no matter what their choices 
are, but I want them to be informed choices.  We do our best to educate 
ourselves over the issues.  We are pragmatic (we have to be).

>> And for the record, our tools (Tiburon's core
>> functionality) is completely open.  Its written in Perl, and if we were
>> hit by a bus in a physical or metaphorical sense, our customers could
>> continue to get support by paying someone to do this.
>
> At what cost? My googles of Tiburon yielded no results whatsoever. If
> you dropped dead tomorrow who would be there to support your stack?

You'd have to pay someone for it, but yes, they could.

> There is nothing on sourceforge or Github...

Nope, nor will there be.  Not every open project needs to be hosted in 
that manner.

> In addition. Could we, a company of 15 people pay for the continued
> development and support of OpenIndina?

Yes, if this is what you wished to use.  Or you could do it yourself.


-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics Inc.
email: landman at scalableinformatics.com
web  : http://scalableinformatics.com
        http://scalableinformatics.com/sicluster
phone: +1 734 786 8423 x121
fax  : +1 866 888 3112
cell : +1 734 612 4615