OFFTOPIC: Re: Gamess: Re: (fwd)

[Eugen Leitl]

Igneant emptor.

---------- Forwarded message ----------
Date: Thu, 6 Mar 2003 19:10:59 +0300
From: Alex. A. Granovsky <gran at classic.chem.msu.su>
Reply-To: Gamess-L at gl.ciw.edu
To: Gamess-L at gl.ciw.edu
Subject: Re: OFFTOPIC: Re: Gamess: Re:

Dear (PC) GAMESS users,

despite being the possible offtopic, I would like to share some my experiences
of using both Windows and Linux clusters with PC GAMESS. After all, you 
should agree that the PC GAMESS developers have rich experiences in this 
area because the PC GAMESS supports both Windows and Linux, and both 
versions are able to run in parallel on PC clusters. Another point is that 
we can more closely look at these two OSes from the programmer's/optimizers' 
perspective.

First of all, I would suggest not to consider Win95, 98 and WinMe at all - 
they are not based on the NT technology and are not true Windows OSes.
Thus I'll consider only NT, Win2K and (to lesser extent) WinXP.

In our Lab, we have our own 19-node Windows NT 4.0 cluster which is used 
to run PC GAMESS and NWCHEM jobs both in parallel and sequential modes.
At the same time, 5 days per week this cluster serves as an educational class 
and its throughoutput is more than 300 students*hours weekly. This cluster has
been working for over 3 years in non-stop mode. The average uptime is 200 days 
or so and is limited by the power failures on our department. This is the head of the 
pstat output on the master node (four-cpu 550 MHz Pentium 3 Xeon 1 MB L2 cache,
4 GB of RAM, Windows NT Server)

>pstat 

Pstat version 0.3:  memory: 3996048 kb  uptime:254 22:41:52.187

PageFile: \??\C:\pagefile.sys
        Current Size: 524288 kb  Total Used:   1548 kb   Peak Used  12944 kb
PageFile: \??\E:\pagefile.sys
        Current Size:   2048 kb  Total Used:   1304 kb   Peak Used   1928 kb

 Memory:3996048K Avail:3762384K  TotalWs: 168072K InRam Kernel: 4188K P:31112K
 Commit:  68468K/  25728K Limit:4364528K Peak:1671364K  Pool N:11648K P:31212K

During these three years, we have seen the "Blue Screen of Death" only once - 
and the reason was the "unrecoverable ECC memory error" - which means that
the stop was due to unrecoverable hardware problem (memory fault) and thus 
was unavoidable. Certainly, the equipment we are using is high quality one - 
each node has ECC memory for example.

On the other hand, we have the unlimited access to the newer 
20-node Linux cluster which is based on dual CPU Pentium 3 
850 MHz systems running old kernel 2.2.16 with tcpfix applied.
This cluster is less stable than our NT cluster, and I have never 
seen the uptime of 200 days on it. Administrators of this clusters
prefer to reboot it approximately once per month to avoid 
problems with NFS, frozen nodes, etc... This cluster was used to 
develop the parallel PC GAMESS version for Linux.

As to my personal observations, I found that the most
MPI implementations (there are lots of them to choose) for Windows 
are _faster_ than MPICH or LAM under Linux, at least for 100 Mbit Ethernet.
As to NT kernel scalability on the SMP boxes, it is also better than that of Linux - 
and the reason is that NT was built from the scratch having 
multithreading/SMP in mind, while Linux historically was not.

As to PC GAMESS performance, it is usually close under Windows 
and under Linux, with Windows version being several percents faster
than Linux one. The reason is some additional optimization for Windows 
incorporated into PC GAMESS, and the lack of the corresponding features 
in the Linux API - which prevents us from add the same optimization to the 
Linux version (because it is not possible).

>From the programmer's perspective, the Linux API is Unix one, while the 
NT being _older_ than Linux is much newer than Unix/Linux by its design and API. 
The base NT application programmer interface (API) is much more logical
and self consistent, and has more features than Unix one. The main point is 
that to get the good performance it does not worth to emulate Unix API under 
Windows like cygwin does, but to use the native Win32 API. 

Several comments regarding other points which were raised during this discussion.

> You won't find a single nontrivial sized cluster out there which runs
> anything else than Linux. There are technical reasons for that, and 
> Redmond offering free licenses and sponsoring clusters hasn't been able to 
> to make one iota of headway into that market. And not for lack of trying.
This is not true. There are many places which have large Windows clusters, 
for example NCSA (where I spent several weeks during one of my USA trips) has.

> A high-performance system takes more than a decade to get right. You 
> might have a point if Windows Server 2003 was launched in 1990. Nevermind 
> the issue of network performance, stability, support (sans source you 
> can't even diagnose, and the only way to do it right is to do it 
> yourself, as that level of support is unavailable in the industry).
If you professionally work for NT platform, you'll most likely will have 
access to MSDN and MS support which is much better than that of RedHat - 
and you should agree that RedHat is now the MS of the Linux world.

> > > An OS built around a desktop metaphor is unsuitable for a server.
> > 
> > Red Hat's KDE has 4 desktops and I don't see anyone complaining.
> You misunderstand. X with all its warts is completely decoupled from the 
> kernel. *nix is rather hardware-agnostic, allowing it be ported from an 
> embedded to a mainframe. Compare this with Redmond's kitchen sink 
> approach, where even display drivers are part of the kernel.
You might be surprising but Unix display drivers are partially work in kernel 
mode too - and this is unavoidable, at least not sacrifying the performance by the
orders of magnitude. And bad display driver can put Linux down as well.
Moreover, Windows kernel is not related with desktop at all - and it is even 
possible to replace the default logon and shell by the text-mode programs 
(although this is really tricky).

> Current clusters operate at 10^3..10^4 nodes. What's 
> the licese costs for 10000 copies of Win2k?
Most of clusters operates with at most hundreds of nodes.
When we had built our NT cluster, we got academic prices for NT - 
less than $40 per copy. Then we asked for Win2K and got it (including
several copies of Win2K Server & costly Advanced Server) from MS free
of charge. In any case, $40 is only several percents of the total price per node.

>>It's difficult to get numerical (especially parallel) work/viz done on a
>>Windows box. 
See above.

>>Besides, they don't ship with compilers, and there are just
>>no tools available, despite cygwin.
OpenWatcom is free and is now available.

> With CYGWIN you end up running LINUX within WINtendo. 
> It is even worse because you need more physical memory. 
> Which WINtendo doesn't manage too well, by the way.
Using CYGWIN exclusively under Windows is as illogical as 
using only Wine under Linux.

> Hang it up, WIN is a 32 bit emulation of UNIX. That's really DOS, a lousy
> subset of UNIX. That's what Gates used to do, steal ideas from Kildall and
> others. 
That's not true - so false so I will not even comment this.

> The new RedHat 8.0 actually works like a 64 bit system. Try it, it's free,
> unlike WINtendo
It's 32-bit OS.

> PC_Gamess is excellent for MP2, MP3, MP4 it is very fast but it run in parallel
> with REPLICATED DATA. 
In PC GAMESS, different calculation methods use both distributed and 
replicated data models as appropriate.

> It packs integral in files of 2Gb. Therefore it remedies
> the barrier of 2Gb by using splitf (split files).
PC GAMESS supports large files directly on NTFS and OS/2 JFS volumes.
We have decided not to support directly large files under Linux because
there are so many old kernel versions still in use.

> GAMESS(US) when compiled with the gcc 3.2 run perfectly well on the same
> hardware but in parallel uses DISTRIBUTED DATA INTERFASE. A totally different
> strategy to distribute work over the net. 
The DDI implementation in the GAMESS (US) is the typical sample of the bad styles - 
both conceptually and at the implementation level.
The PC GAMESS v. 6.3 will include the specially developed P2P interface to deal with 
distributed data by much more intelligent manner.

> It is a litle slower than PC_Gamess
> for integral files 50Gb and larger, but you are supposed to BE PATIENT for works
> that big.
The disk I/O in the PC GAMESS will always be faster than that in the GAMESS (US) due to
real time data packing and advanced I/O interface.

> They have high license costs and lousy network performance (both latency
> and consistent QoS). Latter doesn't bite if you don't do parallel
> applications with MPI over Ethernet.
See above.

> I think the 
> time is coming when computational chemistry is more and more platform 
> independent which can only be good.
I agree. 


To summarize, both Windows and Linux have their own strong and weak points.
It is up to cluster designers to decide which OS to use on it. Both can be used successfully. 
The "OS wars" discussions are of little use, and unfortunately often attract the people who 
does not have enough information/experience to discuss these topics.

Best regards,
Alex Granovsky

----- Original Message ----- 
From: "Ted Packwood" <malice at cray.com>
To: <Gamess-L at gl.ciw.edu>
Sent: Thursday, March 06, 2003 12:11 AM
Subject: Re: OFFTOPIC: Re: Gamess: Re:


> I hate to interrupt, but the arguments of Unix versus Windows
> has grown tiresome.  Hasn't this issue been beat to death on
> any number of other threads?
> 
> Can we stick to Gamess issues please?  I could tolerate the 
> discussion when it was merely which OS was useful for specific
> runtypes for Gamess, but it seems to have reverted to the age-
> old argument which will never be resolved.  I prefer to remain on
> this list but if people are just going to try to convince each
> other which OS is the best, then this list has become pointless.
> 
> Thanks for your consideration.
> Ted
> -- 
> ()_____________________________________________________)
>  | Ted Packwood             Cray Inc.                 |
>  | malice at cray.com          1340 Mendota Heights Rd.  |
>  | Chemistry Applications   Mendota Heights, MN 55120 |
>  | (651) 605-9037           http://www.cray.com       |
> ()_____________________________________________________)
> 




 GAMESS    : http://www.msg.ameslab.gov/GAMESS/
 PC GAMESS : http://classic.chem.msu.su/gran/gamess/
 List      : http://www.gl.ciw.edu/gamess/