[Beowulf] SC05 blogs and observations

[Jim Lux]

At 06:17 AM 11/23/2005, Robert G. Brown wrote:
<snip>


>Let me be perfectly clear about this.  What we will NOT see is Microsoft
>participating in the open cluster development process supported by this
>list.  We will NOT see it contributed GPL tools written for
>standards-based compilers and libraries.



>  We will NOT see a user-level
>applications development environment.

That's actually the part you might see.. MS's IDE (Visual whatever) isn't 
bad, and over the years they've put a fair amount into tools for debugging 
multithreaded widgets, even ones that potentially run across multiple 
processors. With the increasing orientation towards the "web services" 
model and multi-tiered architectures, you need a way to develop/test/debug 
applications that are spread across multiple tiers.


>  We will NOT see free,

That's for darn sure.. (although the latest Visual tools are available free 
for downloading)

>  we will NOT
>see cheap,

Cheap is a relative term.  If they charge, say, $5K for the tools, that's 
less than 2% of the annual cost of a relatively senior developer ($250K, 
fully burdened), and if it increases the productivity by even a miniscule 
5%, you're ahead of the game.

>  except possibly where they "inherit" and deploy existing Open
>Source tools to get things started and gain market credibility.
>
>What we will see (peering into my crystal ball) is Microsoft targeting a
>very specific market -- very likely genetics research or some other
>NIH-funded, deep pocketed group of researchers who are Unix averse
>because they lack the skills and expertise to make a Unix-like system
>work and want everything to run from a Windows GUI and to dump all its
>output straight into e.g. Excel and Power Point.  Here they will go head
>to head with Apple and turnkey linux/bsd cluster builders, but they have
>a decent chance of winning the market AND MAKING MONEY FROM IT because
>the problem is actually not that complex from an IPC point of view (and
>is more closely related to e.g. HA computing than, say, CFD or Monte
>Carlo) and because the people they will sell to will value a GUI and
>ease of use as much as they will value the actual speed and efficiency
>of the computations.

Precisely that...

I don't see MS getting into areas where the problems are tightly coupled 
across processors with fine granularity. I DO see them providing tools to 
do basic EP type tasks.  Example: firing off 500 copies of a simulation 
program that takes 10 seconds to run with different parameters, and then 
collecting the data back...  This is not much different than scattering a 
database query across multiple servers, each with a piece of the 
database.  The latter is much more in keeping with the traditional business 
base for MS (SQL Server, for instance)

>The other thing they will leverage -- unfortunately quite successfully
>-- is their ability to deliver a kernel that is "happy" with
>shrink-wrapped proprietary device drivers, and their ability to offer
>lavish development support to makers of hardware devices to ensure that
>they offer drivers that work with their setup.  I wouldn't be surprised
>to see Microsoft buying out one or more makers of high end cluster
>network hardware
>Or is this too cynical a picture for even my cracked crystal ball?

I'm not sure it fits with their business model to do so.  High end 
interconnects will never be a high volume enough market to justify their 
investment.  More likely is that they'd apply pressure to a high volume mfr 
of interconnects to add "windows specific" enhancements, holding out the 
carrot of increased sales.  Realistically, this will probably be in 
connection with some  sort of Digital Rights Management and media server 
kind of application, and so, will be out of the HPC mainstream.


>The "hard" 40% of the market they'll leave to Apple (to avoid antitrust
>suits, at least for the time being) and linux in general.  In five years
>if you look at any industry magazine catering to cluster users, it will
>look like Microsoft "invented" the cluster and that any cluster that
>isn't using Microsoft's clustering toolset (MCT) is bound to be slow,
>expensive to manage and debug, and hence costly beyond measure in a CBA.

You mean Visual Cluster .NET with the MultiManagementConsole plug in?



>Mind you, this is their PLAN (I'm channelling somebody in their
>marketing department, not their engineering department). I think that it
>isn't quite half right.  I think that if they push very hard, they are
>looking at 25-30% of the cluster market in five years time, and that
>while they will make money there (by focussing on low-hanging fruit with
>the highest potential profit margins and by inducing new customers who
>wouldn't otherwise have done clustering to do a cluster) they won't make
>MUCH money there because they'll find (to their chagrin) that several of
>their assumptions above are incorrect -- cluster builders DO tend to
>want to use standards based software, DO care about portability, most
>DEFINITELY do care about stability and performance, and DO care about
>cost.

Uhhh.. current cluster builders, who are academically oriented, low 
capital, low cash, high free time, or who come from that background, care 
about such things.  Future cluster USERS, who will just call up their 
corporate IT weenies to actually create the cluster, care not a whit about 
such things, because the future Visual Cluster (.NET edition) will conceal 
all the ugliness. And while sure, there's some overhead from that, they'll 
just buy some extra nodes, and the performance is what it is.  As long as 
the cost/benefit analysis shows it's worth doing, it gets done.


>Unlike the situation with Borland International (whom they crushed in
>the software development market by a mix of underselling them and
>controlling the operating system on top of which the integrated
>compilers had to run) it is simply not possible to undersell a market
>where any cluster builder has -- among many other choices -- not one but
>MANY ways of building a cluster with zero software costs.

There may not be a lot of alternatives, if you want compatibility with all 
your other corporate IT assets.. If you've already got an infrastructure 
and staff managing 30,000 winXP desktops and 10,000 Win Whizbang servers, 
the cost to add something new could be substantial, compared to the small 
cost to manage 500 nodes in a cluster using WindowsClusterServer2010.  Such 
things as active directories, domain authentication, access to corporate 
data repositories, and all sorts of things creep into the picture.


>Because of this I expect that they'll find the market tough going, and
>it may even be a spectacular failure for them, one where they LOSE money
>for years and years.  How many cluster builders buy RHEL or SuSE for
>their cluster OS, when they can use Scientific Linux, Centos, FCX,
>Debian, Mandriva -- for free?  How many clusters use Scyld given bproc
>or OpenMosix for free, given warewulf or ROCKS?  How many clusters are
>built by turnkey vendors rather than by the owners?  How DOES the cost
>scaling work if you have to hire an MCSE from a highly competitive
>market AND pay for Microsoft rather than hire a linux/cluster SE (also
>from a highly competitive but less commercially lucrative market) but
>get to use free software with no per-node scaling of software costs, or
>if you compare turnkey clusters built on top of linux to turnkey (third
>party) clusters built on top of Microsoft products?

However, in the "big iron" world, the differential cost of a MCSE vs a 
Linux SE is determined not by salary, but by the rest of the support 
infrastructure. Adding another "family" of software into the mix which must 
now be configuration managed, disaster recovery planned, etc., could 
actually be a significant disincentive. For instance, you might wind up 
needing to hire 4(!) Linux cluster engineers rather than one, so you can 
get 24/7 3 shift coverage.  We're not talking here about people doing heavy 
development, but who are there, able to "push the reset button"  in the 
middle of the night when something goes "bump". (actually, more is 
involved.. think in terms of sitting down at a console and figuring out why 
the jobs have come to a grinding halt) For the MS case, you might have 
folks on staff already who can go in and at least figure out who needs to 
be called.    Besides, the software cost is a very tiny part of the overall 
IT budget..  Most big (>500 or 1000 employees) companies negotiate site 
licenses with MS and don't have "per copy" pricing.

>Remember, per node costs EAT INTO THE TOTAL NUMBER OF NODES and hence to
>total compute power in aggregate FLOPS.  For better or for worse,
>aggregate FLOPS "sells" in an ignorant marketplace.

Transaction processing rates and Low support costs also sells.


>  In order to "make
>money", Microsoft is going to need to be paid some percentage of the
>cost, per node, of a cluster.  That percentage will need to be LARGER
>for small clusters than for big ones.  If we assume that they'll need at
>least 5% of the per-node cost just to pay the marketing expenses of
>their products (a fair guess, I think) then a MS-based cluster will
>START OUT at 95% the maximum aggregate FLOPS of a linux or freebsd-based
>cluster using one of the free distros or cluster packages.

I'd say that a MS cluster will actually perform WORSE than that.. maybe 
60-70% of a Linux cluster in terms of "useful work done".  However, say 
you've got a 100 node cluster.  You now have to buy 30 extra nodes at, say, 
$3000 each (rolling in HVAC, room, etc).  So, you spend $100K on the extra 
nodes.  You DON'T have to spend $300,000 for the Linux Cluster Specific 
support staff, though.  The same old MS server jockeys can run your cluster 
as run the rest of the corporate data center.   When the cluster throws up 
a BSOD, they know what to do.

The real question is whether there is a significant "business scale" market 
for "non-scientific" cluster computing at all.  Or, is the market really HA 
or transaction processing, which is a whole 'nother world.  Historically, 
MS hasn't really been very interested in scientific computing anyway. (all 
those cranky and principled iconclastic scientists who have no budget but 
lots of free time)

But, have to go now.. Next installment for my devil's advocacy:
"Why windows based clusters make sense for individual users"

Jim...