[Beowulf] Opteron/Athlon Clustering

[Joe Landman]

On Thu, 2004-06-10 at 02:31, Greg Kurtzer wrote:
> On Tue, Jun 08, 2004 at 08:19:25PM -0400, Joe Landman mentioned:
> > If you want to solve 32 bit binaries on 64 bit linux today, this is 
> > doable.  Simply requires you buy the right distribution.  No, not every 
> > distribution is there yet, and in fact some of the ones I am quite 
> > interested in (cAos), are a bit behind.  Thats ok, I expect them to 
> > mature.  If we get enough time, you may see us contribute to them.
> 
> cAos-2 is in the works as we speak and it will support x86_64 from the get go.
> As a matter of fact, we already have the self hosting buildroot (1) core
> complete for both ia32 and x86_64. At the moment we are finalizing the 
> infrastructure to open our package based SCM to the external package 
> maintainers. This should be ready in about another week.

I managed to imply negative things about one of the distributions I
prefer.  My apologies.  cAos is a very nice distribution.  I like it,
makes installing nodes quite easy.  The only issue I have run into is
the x86_64 port, and I know it is coming, so I am patient.

In another thread, we saw a discussion of a Fedora based cluster
distribution.  Given the purpose of Fedora from RedHat's perspective,
this does not seem to me to be a very good direction, either in terms of
long term maintainability, stability, or ease of attracting code ports. 

As I noted, software distribution today on linux can be done by source
RPM (one per distribution), binary RPM (one per distribution), or
tarball/similar (one).  While not the most technically sophisticated
method, certainly with no noticeable package management support, it
solves the problem of one package for multiple distributions quite
nicely.  Its the wrapper layer above that, RPM, where the problems
occur, as it has to speak to the OS, and each version of Linux is
sufficiently different, and if you speak to their assemblers, they will
happily tell you why theirs is better (usually not correct, but it is
hard to argue with such people).

I like many things about cAos, including that I can use many of the RPMs
I had built for RH previous on it.  I ran into a few gotchas, but those
were compiler issues (see the platform bits above, and remember RH's gcc
2.96 bit).  The install was painless, though I had connection issues
with some mirrors, since cleared up.  RGB would like it as it uses the
yum repository concept to install most of itself.

[...]

> > RPM is not perfect.  It does fix some things.  Some of the things that 
> > are broken in RPM IMO are config management.   Just have a look at the 
> > RedHat kernel spec file as an example.  Scripting within the RPM is 
> > broken (its just basic pre-processor stuff). 
> 
> RPM is specifically a package manager, not a build or build management tool. 

Note:  I am talking about package configuration.  That is, the options
specified on the autobuild command line (configure --enable-this
--disable-that).  This configuration is not handled well, and things
which get packaged often get packaged with default options, versus
useful options.  I haven't seen this so much in cAos, but it is
definitely the case in FC1, SuSE 9, and others.  It is very hard to
change the config options to the package in an RPM without rewriting the
RPM.  You cannot pass in options to the configure command, or to set the
config of the Perl modules you require.  This means some things need to
be built by hand or repackaged to work.  One of the things I wind up
spending (wasting) time on with new distributions are fixing some of the
critical tool packages).  Of course, some are so thoroughly  integrated
into the system with so many dependencies, that it is simply easier to
built it into a new tree (namespace).

The package configuration is not well done.  More than pointing the
config to where you want the files to go, we need a mechanism of setting
the config options properly (and storing the new ones in the RPM if
specified on the command line.

[...]

> > I'd say lets fix the config issues in RPM, make it really scriptable, so 
> > we can have *portable* RPMs at the *source* level
> 
> That would be nice. Unfortunately, I think that the changes to RPM across
> multiple distros (eg: Mandrake and SuSE) may make SRPM portability difficult. 
> Some of the changes include tag specifications IIRC. Of course this may only
> be a problem for a small subset of packages utilizing those tags.

I think it is a larger problem.  Years ago, we used to be able to build
one RPM (source RPM that is), and have it build binaries most places. 
Even the binary RPMs would largely work across distros modulo the
pathing.

Of course, a linux vendor needs to add value and differentiate.

Today, we have source RPMs that build cleanly across RH and SuSE.  As
you saw a few weeks ago on cAos list, we will be generating cAos SRPMs
as well.  Each additional distro is porting and testing time for us. 
FC-X is out in general, due to its nature of being a test-bed
distribution.  Our work would be rendered worthless in rather short
order as the system moves to the next FC-X+1. Gentoo is possible, though
we haven't seen that much demand from our customer base for it (a
little, but not much).  ROCKS is RHEL3, so this isn't a problem for us.

This means we have to maintain this many machines, or OS installs on
machines, in an up to date manner, with correct build environments, just
to make sure that our SRPMs build across them.  

Ours are not complex SRPMs.  They are really quite simple.  

My point being that RGB's statement that tarballs are not the way to go
due to their overly simplistic nature and lack of package management
tools is correct, but that the alternative which he espouses, RPM, needs
some work as well.  It (RPM) makes it hard to distribute source code due
to the differences in distributions RPM implementations (not to mention
compiler and library differences, but that is a whole other
discussion).  Of course this would be vastly simplified by there being
"one distribution that binds them all" or even distribution base specs
so that you can guarantee compiler and library levels across
distributions at particular specs.  This would be helpful to folks
attempting to develop for the platform. Your SRPM could have a
"BuildRequires:" tag or similar in there, and have a fighting chance at
working across distributions.

> (1) A self hosting buildroot is a development system that has the ability to
> build itself in entirety from source code (in our case, SRPMS). In our case, 
> it is also used as the foundation for building every other package in the 
> distribution. We distribute the buildroot as a chroot'able file system to the 
> package maintainers for doing development and testing.

cAos is good, and I am looking forward to working with it and WW2.  I
just have to fight the packaging issues for our customers, as well as
the library and support issues.  Customers get very unhappy when their
kernel/glibc are not supported by the package they want to run (binary
only commercial packages).  Not much can be done, apart from
loading/running a system that supports it.  cAos does a good job of
this, as do the earlier RH's, SuSE, and a few others.   


-- 
Joseph Landman, Ph.D
Scalable Informatics LLC,
email: landman at scalableinformatics.com
web  : http://scalableinformatics.com
phone: +1 734 612 4615