[Beowulf] MS HPC... Oh dear...

[Gerry Creager N5JXS]

Joe, et al,

Unfortunately, we find, time and time again, when Microsoft comes in to 
tell us how good they're gonna do something, there's a catch.

Here, it remains simple to me: I can tweak the OS if I need to, and I 
can tweak the apps.  With a Windows solution fo rmy cluster environment, 
I certainly can't tweak the OS.  I'm half-way to dead in the water.

In general, we integrate our own hardware.  If I found the API was too 
complex, I'd complain mightily, but actually, I tend to pick hardware I 
can already talk to, and API's I'm already familiar with to program to. 
  At this point, Microsoft is claimint to save me from myself.  Hmmm. 
I'm reminded of the time IBM decided... and advertised... their intent 
to "legitimize" the minicomputer industry.  If I recall correctly, the 
Data General ad in response included the comment, "The bastards say, 
'Welcome'."  That's sort of what I feel here.

I'm reminded that Microsoft made attempts to wander into this community 
several years ago.  I further recall being simply told that there was no 
direction for Linux, no core of paid developers, and that the HPC work I 
was doing couldn't expect to continue using that paradigm. Simply, I 
should migrate my applications to Windows (NT4, at that time) and wait 
for  the Microsoft HPC tool.

I'm not sure I'm ready for Microsoft to define my timeline and my code 
requirements.

I'll stand by my position.  The bulk of the Microsoft announcement is 
marketing, and demonstrates a less than stellar acknowledgment of the 
HPC environment they're trying to break into.  If y'all think your 
corporate wagon's hitched to their star, that's great.  I'll watch and 
learn.  I'm not completely wedded to Linux, but I'mnot likely to change 
until it does something similar to a series of data losses eventually 
identified as registry corruption which encouraged me to make the swap 
to Linux.  As long as the problems I have continue to be hardware 
related (think, statistical failure rates) and not OS definable, I doubt 
I'll seriously investigate them.

gerry

Joe Landman wrote:
> (sorry in advance for the length)
> 
> Gerry Creager N5JXS wrote:
> 
>>Ooh!  ISO-9002 Buzzword Compliant marketing.
> 
> 
> Hmmm.... not astroturfing here.  I opine on http://scalability.org/?p=69
> .  A few others have linked to this, so we are getting some traffic.
> 
> Specifically, I am of the opinion that the sentence "but until now it
> has been too expensive and too difficult for many people to use
> effectively" is factually wrong.   The reasoning is very simple, and
> borne out by existing data.
> 
> If HPC has been both too expensive AND too difficult to use, then why is
> it as a market growing at 20+% per year?  Moreover, as IDC points out
> that Linux clusters are driving the growth in the HPC market, with this
> segment currently at about 1/2 of the $9B market and growing at 60+% per
> year for the past 3 years ...
> 
> Short version: it is neither too expensive, nor too hard, as people are
> doing it effectively now, and have been for the last 5+ years, with the
> growth showing up on the radar 3 years ago.
> 
> If the statement of too expensive could be applied to Linux clusters,
> then this fails to explain the observation of the growth.  Last I
> checked, real measurement trumps hypothesis.  These aren't windows
> clusters that are growing, these are Linux clusters.  These aren't unix
> boxen.  They are Linux clusters.  There must be a reason for this.
> 
> If the statement of too hard could be applied to the market, one would
> need to ask exactly what people were buying that was not too hard which
> is generating all that growth.  Since we know the answer (linux
> clusters), they must not be too hard to use.  The systems we put
> together for our customers who don't care what is under the hood looks a
> great deal like a large windows disk (or disks) and a web page.  Those
> who care about the details prefer the command line.
> 
> All this said, and not to disagree with Doug Eadline and others on the
> technical details, I do think Microsoft has something to offer here, but
> I think they need to work within the existing community, and not dismiss
> it out of hand.  The latter is the sense I am getting out of the
> marketing.  The folks from Microsoft I have spoken with, Kyril and
> Patrick, seem to be quite interested in doing the right thing, though
> with a decidedly Microsoft spin.  The problem I see is that the spin and
> some of the core assumptions are, IMO, incorrect.  One expects marketing
> to be so.  Building a go-to-market strategy upon the basis of core
> assumptions that are not in line with the market reality is dangerous,
> even for an infinitely deep pocketed corporation.  All I suggest is
> appropriate debunking of the marketing, and drilling into the
> technological core of what they are going to market.
> 
> They do have a number of very hard hills to climb, specifically pricing
> compared to competitors, technological feature lists, interoperability,
> security, and stability.  Most of these are going to work against it.
> It would be unwise to count them out of the game though.  Anyone
> remember or still use Lotus 123?  Wordperfect?  May take them a while,
> and they are persistent.  With very deep pockets, lots of patience, and
> the ability to purchase talent.
> 
> Linux was able to effectively kill Unix by presenting a single API to
> write to, a simple stack to deal with, a much larger potential installed
> base, a lower cost of acquisition.  Microsoft has learned from this.
> Assume that this is their direction.  The arguments they presented to me
> involved driving a wedge between various linux distros, and painting the
> Linux scene in a similar manner.  Their MPI argument (to many stacks)
> was not a good one, as the same problem exists on windows.  But the
> point is one that I and many others have complained about at some point
> in time or the other.  You have different MPI stacks which are binary
> incompatible.  Which means if the PathScale folks came out with a new
> hardware device to accelerate networking for folks like LSTC, then the
> LSTC folks have to relink their app against the new stack.  Which is
> exactly what happened.  While some folks here defend this, I want to
> note that end users don't give a rip about that.  They want the new
> fangled hardware to work.  Right away.  Without a rebuild of the app.
> So do the vendors.
> 
> ISVs don't want a rotating collection of MPI stacks, but one to work
> with.  One API.  Each app now needs to decide how many MPI stacks to
> support on each ABI.  You have IA64 with several, AMD64 with several,
> ia32 with several, ...  Each stack adds cost/complexity to them.  Again,
> this is something I have argued for a while.  The ISVs as well.  They
> don't want to support Joe's MPI stack, they want one MPI stack per ABI
> (even better would be one MPI stack and one ABI, but we are not there
> yet).  MPICH runs pretty much everywhere.  LAM (which I like a little
> better) runs fewer places (never been able to get it to work the right
> way under Cygwin).  I can live with MPICH.  The problem though is that
> we have to relink the application for each new networking advance.  And
> customers don't like that.
> 
> What Microsoft will do is to take away as much of this as they can.  I
> haven't seen it yet, but I believe they will offer MPICH as a DLL, so if
> PathScale wants to work along side some other device, you can select
> this at runtime, and just have it work.  This is a nice idea.
> 
> Schedulers are another area, but my impression from speaking with them
> is that they haven't looked at the market carefully enough yet, or some
> non-business reasons got in the way of them exploring whats out there.
> 
> The idea is that they have a good story in some parts, and a very weak
> story in others.  Assume they will improve the weak points.  It would
> make more sense to engage them so that they improve the weak points
> along reasonable lines.  I would rather have them fit in, than try to
> overtake, as the latter will just piss off the customers when they
> realize what they want to do is not possible with their shiny new WCC,
> or they suffer far worse performance than the folks with the Linux
> cluster due to all those corporate mandated firewalls and copies of
> Norton running.
> 
> Joe
> 

-- 
Gerry Creager -- gerry.creager at tamu.edu
Texas Mesonet -- AATLT, Texas A&M University	
Cell: 979.229.5301 Office: 979.458.4020 FAX: 979.862.3983
Office: 1700 Research Parkway Ste 160, TAMU, College Station, TX 77843