[Beowulf] Why I want a microsoft cluster...

[Mark Hahn]

> Ultimately though, to update the aphorism: "nobody ever got fired for 
> recommending MS"

that aphorism just reflects the strength of the policy of opposing change:
because the MS network effect is large, it's a bit of a risk to go against
the current.  which is fine, assuming you want to go where the current takes
you.

the thing you have to realize about the Windows world (WW) is that it's not a
real ecosystem, with equilibrium between predator and prey, feedback, etc.
non-MS people in the Windows world are more like competing ant colonies in
the same terrarium.

for those of us outside WW, there are two challenges:
	- do exciting new things that don't recapitulate WW.  after all,
	if what we wanted was already inside WW, we'd just be reinventing
	the same old wheel.  the goal is _better_wheels_ (or wings, etc).

	- somehow make it obvious to people stuck in the terrarium
	that there are unique (disruptive) things outside.

personally, I think the clustering world is a bit complacent, not unlike how 
unix was before Linux.  the terrarium is a barrier in both directions - 
because we are, for the most part structurally incompatible with WW, 
the competition is more tectonic, rather than teeth-at-neck.

so, getting back to the point, I don't really see from SC05, where WW will
gain much from HPC clustering.  multiple vendors said that MSFT's goal was 
to insinuate itself into the low-end cluster market first, then move up.
no real surprise there, but I have doubts about whether this makes sense.
why?  because low-end clusters are mostly a mistake.

at a university like mine, for instance, nearly _everyone_ realizes that it's
insane for each researcher to buy/use/maintain his own little $50-500K
cluster.  I see three clear reasons for this:
	- the value of a cluster is some superlinear function of its size.
	- the maintenance cost of a cluster is very sub-linear.
	- most workloads are bursty.
the first two factors encourage larger clusters; the latter means that bursts
can be overlapped in a shared resource very nicely.  but the punchline is
that this argues against MS's strategy of gaining a million tiny toeholds,
since those small clusters are a bad idea in the first place.  in Canada, at
least, funding agencies understand the three factors above, at least fuzzily,
and are quite encouraging of HPC consortia.  I don't see why business CIO's 
wouldn't realize the same things, and force some amount of coalesced
clusters.  maybe we should talk about clustering in terms of virtualized
parallel machines, or rename batch schedulers as "on-demand resource allocators".

not all clusters should be treated this way, of course.  some workloads are
steady, so you don't get any benefit out of resource-sharing (just potential 
screwups if someones burstiness crimps your constant demand.)

the other aspect of the MSFT-SC05 thing is that there doesn't appear to be
any real synergy between the WW software ethos and clustering.  WW is all
about plugging applications together - embedding activeX controls, etc.
what would you plug into in a cluster?  there's nothing wrong with MSFT's 
plan to do a cluster environment based on SQL server, visual studio and MPI,
but there's no real skeleton there, just connective tissue.  no Word/Excel
standards that give shape to the environment.

regards, mark hahn.