Windows HPC

[Paul Redfern]

Dear Robert,
Thank you for your point of view. To clarify, the grant is not for two
machines and software, but it is actually for multiple machines over a
four year period, development projects, staff, research, training, etc.
Microsoft software donations are not included in the grant. Our goal is
to help clients around the world who are interested in HPC on Windows to
port, parallelize, and optimize their applications and systems. Those
clients may choose to install very large clusters; our goal, however, is
to enable their success whether it's a workstation cluster, midrange
system, or a top 500 machine.

As far as Windows HPC goes, here's a few of the reasons universities and
businesses are telling us they're interested in Windows clusters.

** Reliability. We ran a 256-processor Dell cluster with Windows 2000
and collected all errors (OS, I/O, hardware) on a secure web site for 6
months. MIT analyzed and independently verified the up-time--99.9986%. 
** Existing infrastructure. Many companies (especially small and medium
size businesses), universities, and government agencies already run
Windows servers. The capability to scale an application from a Windows
desktop to Windows servers interests them because they can do it on
their existing infrastructure and on an operating system they are
familiar with (everyone doesn't want to learn LINUX).
"LINUX-only-HPC-or-no-HPC" may be a non-starter for many researchers and
businesses. It ignores the Windows installed base as well as server
growth projections (analysts predict Windows will be 1/3 of the overall
server market by 2005; the other 1/3 will be Linux/UNIX and 1/3
proprietary/legacy OS--many choices, perhaps a good thing?). Each year
attendance by business at SCXX gets smaller and smaller. The same
universities/labs tend to participate year after year. Is it possible
that HPC on Windows might help expand the HPC community? That new
classes of users might develop new applications? That businesses might
begin to think of HPC as a service that can be accessed seamlessly from
their Windows desktop on existing infrastructure rather than built out
as a separate island of performance with its own hardware, OS, staff and
inherent costs. Anyone can run Windows and some organizations want to
leverage the environment they have.
** TCO. Studies show that the most costly element of any total cost of
ownership study is people not software. Our clients find they can
dramatically reduce the number of systems staff needed to run their
Windows cluster because of the systems manageability tools provided. To
date, the HPC community has not considered TCO as a metric. It would be
interesting if people costs were truly measured in universities and
government labs like they are in businesses. "Slave labor" is often used
as a descriptor for grad students/post docs for a reason. Many
businesses, perhaps most, want to reduce people costs. They tell us they
want their people to focus on their "core" business, such as doing
structural engineering design (rather than each becoming experts in
source coding or dealing with systems administration issues). 
** Standardization. Many organizations are beginning to look at
standardization as a way to reduce TCO. Windows standardization may be a
motivation to pursue Windows HPC. A major oil company recently completed
a TCO study that found they could reduce their IT expenditures by $210
million per year if they standardized on Windows. Air Products and
Chemicals, one of our clients, is standardizing their corporation
worldwide on Windows starting with engineering and parallel HPC
applications and then moving to the glass house. 
** Manageability. We were able to dramatically reduce the size of our
systems staff with Windows clusters. More people are now available to
focus on their science. Because they are easy to manage, we add Windows
clusters without adding staff. We installed a 256-processor cluster and
had it up and running, pinging processors in less than 10 hours. We've
found that adding to existing clusters and adding new clusters as well
as storage is fast and easy with Windows. 
** Desktop savings. Rather than having 2 desktops per person (one
UNIX/Linux and one Windows) like we used to, we can now simplify things
and go with 1 laptop and dock. Employees love it. They get their laptops
refreshed more often. And the facility is wireless. Everyone can take
their work home with them too (ok, not really a plus).
** Performance. Results reported by Windows users are as good as or
better than other OS's according to the users we've talked to and
benches being run.
** Data access. We are using SQL server and our clients are using other
commercial databases as data delivery engines for HPC rather than flat
files. A lot of programming errors can be eliminated that way. There are
a wide variety of databases to choose from that operate on the Windows
OS.
** Visualization. CAVEs are now available on Windows. We are operating
ours with 3 Dell workstations with Wildcat cards. Users love the ability
to drag their data to H: drive, walk over the CAVE, and there it is, up
and running. Great usability. Less visualization staff specialists (self
service instead). And, lot's of new applications because it's easier to
use, such as architecture design classes and engineering students
designing new structural materials for the space shuttle replacement.
** Feature functionality. Some clients want Web services with HPC or
.NET cluster on the back end. They want the parallel processing accessed
seamlessly through a Windows program front end such as Excel.   
** Application availability. As Dan aptly pointed out, there are
Windows-only applications that companies want to run in parallel and do
not want to port. For example, El Paso energy recently contacted with
CTC to parallelize Estimas' Time Regression Analysis software to run
against the company's database. At Cornell we have new classes of users
who would have never considered HPC in the past. Business school
researchers are running large-scale manufacturing optimization problems
for electronics companies and finance simulations (that were previously
run on desktops for days), social scientists are doing secure data
analysis with parallel SAS on government data, and lab biologists, who
only know Windows, are now interested in HPC for the first time. A DOE
lab is planning on running VASP on a Windows cluster. One hard core UNIX
enthusiast, who first said no way he'd run on a Windows system, is now a
regular user. When asked about Windows, he now says, "I'm OS agnostic; I
just want to get my work done." Note: parallel SAS is a commercially
available application for Windows, like SQL Server and VASP. CTC is
working with a number of commercial vendors on parallel versions of
their standard applications.

I hope this provides a little insight into some people's motiations. If
anyone is interested in how Windows can help you with your research,
engineering, or business intelligence, feel free to drop me an email.
We'd be glad to work with you. CTC and our OEM/ISV team plan many
exciting things to come. 
Paul

Paul Redfern
CTC High-Performance Solutions
http://www.ctc-hpc.com
tel 607-254-8693



-----Original Message-----
From: Robert G. Brown [mailto:rgb at phy.duke.edu] 
Sent: Tuesday, August 06, 2002 1:01 PM
To: Paul Redfern
Cc: beowulf at beowulf.org
Subject: Re: Windows HPC

On Tue, 6 Aug 2002, Paul Redfern wrote:

> FYI .
> 
> Cornell Joins Forces with Dell, Intel, and Microsoft to Expand Usage
of
> High Performance Cluster Computing in the Corporate Data Center
> 
> Aug. 5, 2002 - Cornell Theory Center (CTC) today announced an
agreement
> with Dell, Intel, and Microsoft to develop and deliver CTC
> High-Performance Solutions, a suite of industry standards-based
> high-performance computing (HPC) solutions and services for business,
> government and academic clients. The agreement provides $60 million
> worth of resources over the next four years to aid in solutions
> development.

Microsoft discovers the beowulf concept?  Again?

$60 million worth of "resources" -- would that be $30 million worth of
hardware and $30 million dollars worth of full retail Microsoft software
(which costs them nothing but media to provide)?  Let's see, 425 new
Dell servers at an absolutely insane highball price of $5K each is all
of $2.5M, adding $5K in infrastructure costs including gold-plated
networking -- hard for me to see more than $5 million in hardware, $10
million if they completely replace the other 425 "servers" (with enough
slop in my estimates to likely build a building just to house all this).

So how about $10M in hardware and infrastructure, $10M in salaries
(that's enough salary money for at least 10 full professors and their
students for four years even at Cornell:-) and $40M in MS software?
Enquiring minds like to know...

Hmmm, Win2K server licenses for (say) 1000 seats -- ooo, that's a lot of
money.  Compilers and programming tools are expensive.  No compute node
is complete without Office;-).  And so forth.

Now the interesting question is -- who in their right mind would ever
buy into this?  A linux cluster can be installed completely over the
network from PXE boots in an unbelievably short period of time, and its
software maintenance is completely automatable.  A single person can run
a cluster with hundreds of nodes, and we can build a cluster with
hundreds of nodes for around $2500/node INCLUDING the cost of building
server-level infrastructure (AC and power).  Software costs scale
perfectly (ranging from marginal costs of perhaps 3-5 minutes of human
time per node per month after configuring the cluster servers and
installing the first node, all opportunity cost and $0-100 per node for
software per se, depending on what you choose to install and how many
nodes you've got).

All the source is open, so one can FIX the numerous problems encountered
trying to write and implement HPC parallel code running over a network.
Or at least understand them and work around them.

Then there is security.  WinXX security is legendary.  Really.

Finally, there are all the lovely aspects of Microsoft's rumored future
pricing models.  Not only will one have to buy the original software,
but (whether you like it or not, or need to or not) one will have to
repurchase it all over again, year after year, and the joy of being
locked in by the outrageous DMCA for anything "interesting" that you
ever develop that uses proprietary libraries or tools to implement.

Not to mention the personal satisfaction that comes from helping
perpetuate the largest and most ruthless monopoly that the world has
ever seen, which quite literally reduces your personal freedom as it
manipulates the political process to protect its "right" to squash
competition and have the use of Microsoft products mandated by law.

I'm not impressed.  I can hardly blame Cornell for taking the money --
even an inflated $60M undoubtedly makes them a lot of money, and
starving professors do have to eat (I'm being serious here, not
facetious -- I'm a starving professor myself:-).

Still, give me TWENTY million dollars and I will install a cluster with
many times the capacity implied here and run it four years and do all
sorts of cool research and development, especially if I get actual cash
and not the preconfigured servers that Intel/Dell tends to give away in
these projects, which have heavily inflated price tags and heavily
deflated price/performance.  Oh, and this includes spending $1M or two
building or remodelintg a facility for the cluster.

   rgb

> 
> CTC High-Performance Solutions will be based on Dell PowerEdge
servers;
> IntelR XeonT and ItaniumR family processors and tools; and running
> Microsoft Server software. This combination is designed to provide
> customers with the performance and availability once only achieved by
> proprietary supercomputers at a fraction of the price. CTC will double
> the size of its existing 425-server Dell, Intel and Windows-based HPC
> clusters as a result of this agreement. With the standards-based
> technologies in CTC's clusters, it can provide users with documented
> high performance, reliability and security while functioning at
> significantly reduced total cost of ownership when compared to the
> proprietary supercomputer CTC previously used. 
> 
> HPC clustering has been successfully used in university and research
> environments for years to solve complex problems, but also has many
> practical applications for business such as scalable online
transaction
> processing with Web clients, decision support systems, engineering
> design and analysis, bioinformatics and more. CTC High-Performance
> Solutions will apply its Windows HPC expertise to accelerate the
> deployment and scale out of Windows-based IT infrastructure in the
> private sector. 
> 
> CTC High-Performance Solutions will develop robust Windows HPC
solution
> stacks for broad industry deployment, and will include HPC services
such
> as UNIX to Windows code porting, optimization, and porting to parallel
> environments; systems planning and integration; systems and
applications
> training and testing; benchmarking. CTC will also offer
high-performance
> Web services based on Microsoft's .NET software and SQL Server. CTC's
> TechExchange Consortium will provide members with more immediate
access
> to IT technologies and will help drive the evolution of Windows HPC. 
> 
> In addition, CTC will establish a technology showcase for
> proof-of-concept applications for HPCC in the financial district of
New
> York City. This facility will be linked to related activities in CTC's
> Ithaca, N.Y., laboratories and will serve as the setting for customers
> to pilot projects. 
> 
> "Establishment of CTC High-Performance Solutions comes at a time when
> all sectors of the economy face increasing competition, pressure on
> margins, and the need to demonstrate strong and quick returns on
> investment," said Thomas F. Coleman, CTC director and Cornell computer
> scientist. "With our expanded relationships and combined strengths, we
> can show companies, government agencies, and academic institutions how
> to expand their technical computing environment, while reducing their
> overall IT budget. They can take their existing expensive, proprietary
> systems, which are often islands of performance requiring extra
systems
> staff, and replace them with a more flexible, scale-out clustered
> environment that is expandable and that fits in the overall
> Windows-based office environment." 
> 
> "Cornell Theory Center is playing an important leadership role in
> Windows Server-based high-performance computing," said Brian
Valentine,
> Microsoft Senior Vice President, Windows Division. "They were first to
> move completely to Windows for HPC. They have shown that it works in
the
> most demanding settings. And they will be instrumental in moving HPC
out
> of the research environment and into the mainstream industry. As we
work
> together with CTC, Dell, and Intel, the efforts coming out of this
> agreement will very clearly show Windows brings the highest value to
> high-performance computing applications and companies' business
systems
> on an industry standards-based IT platform."
> 
> "The flexibility, performance and cost-effectiveness of Dell PowerEdge
> servers with Intel technology is becoming more and more attractive to
> customers in research-intensive industries outside of the university,
> due in part to initiatives like CTC's Windows program," said Russ
Holt,
> vice president of Dell's Enterprise Systems Group. "Through Dell's own
> HPCC program, we continue to see customers replacing legacy,
proprietary
> systems with Intel-based HPC clusters." 
> 
> "Intel continues to invest in HPC to propel the industry forward and
> drive innovation using Intel's volume economics model - delivering
> absolute performance, price/performance, flexibility and choice to
> enable supercomputing for the masses," said Mike Fister, senior vice
> president and general manager, Intel Enterprise Platforms Group.
"Using
> the industry-leading floating point performance of the Intel Itanium 2
> processor and the world-class price/performance of the Intel Xeon
> processor, CTC High-Performance Solutions will help accelerate the
> migration of leading-edge computational research into the corporate
data
> center of the future."
> 
> "This tremendous investment by Dell, Intel and Microsoft in the
Cornell
> Theory Center is a true vote of confidence in the intellectual power
of
> one of our State's finest academic institutions," said Governor
Pataki.
> "Industry, university and government collaboration is critical to
> economic success in our State and throughout the nation, especially in
> the fast-paced world of information technology. This project is a
prime
> example of how expertise at New York State's top-flight universities
can
> help industry solve complex problems that will benefit all sectors,
> public and private." 
> 
> For more information about CTC High-Performance Solutions, visit
> http://www.ctc-hpc.com.
> 
> About the Cornell Theory Center
> CTC is a high-performance computing and interdisciplinary research
> center located on the Ithaca campus of Cornell University. CTC
currently
> operates a Dell/Intel/Windows cluster complex consisting of more than
> 900 processors. Scientific and engineering projects supported by CTC
> represent a vast variety of disciplines, including bioinformatics,
> behavioral and social sciences, computer science, engineering,
> geosciences, mathematics, physical sciences, and business.
> 
> About CTC Systems
> CTC's Systems are configured into general purpose, strategic
> application, and dedicated clusters. Among the dedicated research
> clusters housed at CTC are a 64-node system devoted to computational
> materials, 64 nodes for computational biology solutions, 32 nodes to
> support the USDA-ARS Center for Agricultural Bioinformatics, and 32
> nodes dedicated to social and economic research. CTC also provides a
> novel Windows/Dell/Intel 3D, stereo immersive CAVE environment for
> scientific visualization. 
> 
> Note: Intel, Itanium and Xeon are trademarks or registered trademarks
of
> Intel Corporation or its subsidiaries in the United States and other
> countries. Dell and PowerEdge are trademarks or registered trademarks
of
> Dell Corporation in the United States and/or other countries.
Microsoft,
> Windows, SQL Server, and .NET are trademarks or registered trademarks
of
> Microsoft Corp. in the United States and/or other countries. Other
names
> and brands may be claimed as the property of others.
> 
> Paul Redfern            
> 
> red at tc.cornell.edu
> 
>  
> 
> 

Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu