[Beowulf] General cluster management tools - Re: Southampton engineers a Raspberry Pi Supercomputer
landman at scalableinformatics.com
Sun Sep 16 09:35:11 PDT 2012
On 09/16/2012 11:53 AM, Mark Hahn wrote:
> how many clusters built this year will just be a bunch of nodes bid
> from a big-name vendor? I suspect that tweaking (inherently risky,
> but also inherently valuable) is pretty much limited to tiny clusters,
> where no risk-averse personage is involved. I think this is a real
> shame, since it means there basically are no interesting clusters
> anymore: every meal is gruel (you can survive that way, but...)
I've pointed out elsewhere in the past, that freezing out the innovative
folks in the market tends to result in cookie cutter (basically
glorified web servers) clusters with specialized communication fabrics.
You can still build your own clusters with various machines, but you
have to make decisions between various forms of lock-in at some level.
Whether this is a good thing or a bad thing is best left to the
historians of industry. Car companies, to a large extent, provided a
mass market product that was "good enough", and many of the innovative
companies that sprung up during the rapid growth of that market fell by
In HPC we have 1-2 ODM/OEM suppliers in tier 2 and higher space, and 2-3
ODM/OEM suppliers in the tier 1 space, selling (relatively minor)
variations of the same computational hardware. There really is very
little differentiation for most cases ... There are a few folks who do
(significant) variations on this, and build decidedly non-traditional
units for customers. Things you really can't order off a menu, there's
a significant engineering design element to these. But these systems
(and the vendors with capability to build them) are few and far between.
The HPC Raspberry Pi cluster is one of those fairly innovative units.
> I sometimes wonder whether organizations should aim a particular
> fraction of their budget at higher-risk projects. I know companies
> like google and facebook are quite deliberate at cultivating risk,
> but there's a lot less derring-do in, say, a multi-institutional
> academic HPC consortium.
This is easier to do when you have a known (and consistent) funding
model, and your mission objectives allow for this. As often as not, the
mission objectives tend to grow more risk averse as the funding climate
deteriorates. Government funded work tends to (eventually, with some
time lag) track the economy as a whole, so upturns provide more
operational capital, with downturns providing consolidation forces.
In the private sector, companies must have profits to invest in order to
... invest ... in HPC. Their mission objectives need to include
shareholder value as a primary focus. These mission objectives are
often under pressure during the best of times, and under active assault
during down turns.
It would be nice to "mandate" some thing like what you suggest, though
its probably not well aligned with private sector companies, or publicly
funded organizations. Pretty universally, we are seeing a "do more,
spend less" mantra (have for a number of years). This perversely
negatively impacts risk taking, and risk perception.
In the US, we've not had a budget enacted into law in something like 3
years. I won't get into the politics of this, but I will point out that
this raises uncertainty for the people needing to acquire stuff for
their mission. Higher uncertainty results in a preference for lower risk.
With regards to risk perception, I am still blown away at some of the
conversations I have with prospective customers who, still to this day,
insist that "larger company == less risk". This is demonstrably false.
A company with open products (real open products), open software stacks,
... that lowers risks. With the closed, vendor lock-in stacks, you
increase risk. And this is, perversely, what is called "lowering risk"
by people looking for an excuse to go with the larger company.
Perception is at right angles to reality for many people with respect to
risk. If your systems are serviceable and supportable if the company
goes away, you have a lower risk as compared to systems that are
completely dependent upon a single company for service and support.
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics Inc.
email: landman at scalableinformatics.com
web : http://scalableinformatics.com
phone: +1 734 786 8423 x121
fax : +1 866 888 3112
cell : +1 734 612 4615
More information about the Beowulf