[Beowulf] General cluster management tools - Re: Southampton engineers a Raspberry Pi Supercomputer

[Joe Landman]

On 09/24/2012 08:31 AM, Hearns, John wrote:
> Andrew Holway <andrew.holway at gmail.com> writes:
>
>> Your saying that open source software is somehow less risky than
>> proprietary software. I dont see any evidence for this.
>
> But it is!
>
> Through the years, I have been screwed n times by commercial vendors
> saying "No, we will not sell you our proprietary solution anymore. We
> are discontinuing it. We don't want your money". And I have been stuck.
>
> I have never experienced the same with open source. In the OSS world, it
> is more likely than not that somebody else will take over the project.
> And even if that doesn't happen, you can alway pay somebody to do the
> support you need.
>
>
>
> Indeed. Look what happened to Gridengine.
>
> http://gridengine.org/blog/
>
>
> Now you can get a commercial supported version from Univa.
> If Gridengine were closed source, it would have withered and died.

Univa's is closed, but there are two functional open source forks ... 
Dave Love's and Scalable Logic's.

Unfortunately, the transition from (partially) open to full on closed at 
Oracle left many high and dry.  The forks began in earnest shortly after 
this.  Sadly, they haven't been as closely cooperating in the community 
as one might wish ... that is, there are two open builds, both with 
value and support, but two different visions and directions.  With the 
third build (Univa) sadly sniping at both of these.

Basically the transition nearly killed the project/community.  Lots of 
folks grew tired of this and moved on.  We are one such ... I don't have 
time for political battles between two nearly identical projects that 
should merge.  This "sectarianism" is deadly to open source projects ... 
forks pull communities apart.  And the GE community is pretty small to 
begin with.

FWIW:  This is what I was worried would happen to Lustre last year with 
3 different organizations driving it.  Coming together and merging 
efforts made the community stronger.  The GE community needs to do that 
as well, though, I am not convinced it would happen.

That said, I stand by my thesis that open source is lower risk.  Open 
source is not "hosted on sourceforge, github, ...".  Open source is 
redistribution and changes aren't disallowed, rather encouraged, and 
there is no hiding changes.  This allows you to build communities, lower 
risk to participants of that community.  Fracturing the community does 
increase risk.

Use GridEngine (GE) as an object lesson for what happens when a code 
owner decides to take their marbles and go their own way.  If you have a 
business dependency upon this code, your risk just skyrocketed.  If you 
have an open source version that you depend upon, and the code owner 
goes away, or decides to start in on the iOS app market rather than work 
on the product you need, you are not up a proverbial creek without a 
proverbial paddle.  You can *choose* to hire someone (including 
allocation of your own, or your teams time) to support the product. 
That is, you have the choice, and you are not beholden to the business 
change decisions of another.  That is, you have decoupled your risk from 
their choices.  Which reduces vendor lockin, end user risk due to 
changes in business condition, etc.

This isn't saying that closed source is evil/bad/wrong.  I don't believe 
this.  I do believe that you have to make informed choices, and perform 
cost benefit analyses.  And more to the point, you have to have a plan 
"B" in place in case the company you depend upon does, in fact, get hit 
by a bus.

What open source does is lower the impact to you, if they (the vendor) 
gets hit by that run-away mass transit vehicle.

All this said, the OP talked about Bright Computing.  They have good kit 
(yes, we are or were a reseller, and I think Penguin, list hosters are 
as well).  For very windows-y customers, I'd have a hard time 
recommending other tools, as most windows admin blanch at the prospect 
of a command line.  But its not open source, and there are open source 
alternatives.  Not as nicely polished (really, Bright is very well built 
... worth checking it out!), but very good tools.

Warewulf is a great kit, does exactly what we want it to do for 
clusters.  And its open.

Xcat2 is pretty widely used.  Extreme configurability.  And its open.

Rocks is something akin to the grandparent to these (in a philosophical 
sense, not actual code base).  Has both commercial and open (for 
academic use) flavors.

There are many others ... Onesis, et al.

Feel free to chime in on which ones are out there if you have a favorite.


-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics Inc.
email: landman at scalableinformatics.com
web  : http://scalableinformatics.com
        http://scalableinformatics.com/sicluster
phone: +1 734 786 8423 x121
fax  : +1 866 888 3112
cell : +1 734 612 4615