[Beowulf] Opteron/Athlon Clustering

[Robert G. Brown]

On Tue, 8 Jun 2004, Sean Dilda wrote:

> On Sun, 2004-06-06 at 00:47, Gerry Creager N5JXS wrote:
> > I've been playing with my Appro 16 node dual Opteron cluster.  If you 
> > load a 64 bit OS (say, SuSE in 64 bit mode), it's a 64 bit system, 
> > unless you do some interesting things to make it operaate in 32-bit mode.
> > 
> > If you load a 32-bit kernel and OS it's a 32 bit system.  But, why?
> 
> You're missing some key functionality here.  If you load a 64-bit
> version of Linux on it, then install some 32-bit software (assuming you
> have the 32-bit libraries installed as well), then it will be able to
> run the 32-bit software right next to the 64-bit software.  What's more,
> it runs it as speeds that in some cases are faster than competing Xeon
> and Athlon chips.
> 
> Why's this so great?  Unfortunately, some proprietary non-open-source
> code is still used for serious computation.  But if you have all
> athlons, people can compile their own 64-bit code and run it on the same
> machine as someone who's running a proprietary 32-bit piece of code.

One more advantage that has occurred to me as I do work on an extended
cluster with Athlons and Opterons running side by side on the same
computation -- it is indeed a bit of a pain to organize a computation
split across two architectures.  In full production of course it is
easily worth the energy of doing a full recompile and getting the full
speed boost.  However, at this very moment I'm in an intermediary period
where I'm doing lots of trial runs to ensure that my new code is working
correctly and that there is indeed interesting physics in the results it
will take me weeks to months of CPU to compute.  In this trial period
the hassle of maintaining two distinct binaries as I make small changes
in the sources and keeping straight just where I'm running balances the
loss of speed in running the 32 bit versions quite nicely.  In fact, the
only reason I'm doing it the hard way is because there is also the work
of installing the 32 bit libraries (including a 32 bit version of e.g.
the GSL) so that everything will work transparently.  If this were done
for me kickstart ready I'd just run the 32 bit binaries at this point
and do the recompile later in full production.  Since it is NOT done for
me kickstart ready I'm maintaining the two variants and losing a bit in
human work that might or might not be quite worth the speed advantage
for these many, short runs.

To put it another way, I think that an Opteron equipped to transparently
run 32 bit binaries is a superior configuration to one that isn't.  The
benefit of the work required to so equip it is nonlinear and highly
dependent on local environment and task.  The cost of the work required
to equip it is CURRENTLY still rather high as everybody is doing
one-offs of it.  However, if this is ever encapsulated in e.g. Fedora
x64 in a package group so it can be done once and used many times, who
of us wouldn't install it?  It is a major PITA to rebuild all my local
and personal binaries and build two version of ~/bin and add the
requisite path automagic.  When I run jove (yes, I run antique editors
as I am an Old Guy_tm) or hpc (my nifty hewlett-packard calculator
emulator written by George Welch who may or may not still be at tamu) I
don't care if the are 64 bit or not -- they are already fast enough on
an 20 MHz 80486 PC.  However, rebuilding them (especially jove) is a
Real Pain In The Ass, because the code is pretty old and nobody loves it
any more.  But me, of course.  Library changes are a real problem.

   rgb

-- 
Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu