[Beowulf] best linux distribution

[Robert G. Brown]

On Mon, 8 Oct 2007, Mike Davis wrote:

> My experience is similar to Bill's. We've been using CentOs 3,4 for the past 
> few years on our larger clusters. It is a good choice for stability, good 
> performance, and since it is RH for SW compatability.

The only thing I'd comment on that is negative about it is one of its
"advantages".  There is a narrow line between stability and stagnation,
and you have to figure out which side of that line your cluster will
fall on.  Specifically, the fact that Centos/RHEL is frozen for two year
intervals has two disadvantages for some people:

   a) The hardware it supports is left behind by the real leading edge of
hardware design.  In some cases this doesn't really matter -- many
motherboards and CPUs are sufficiently "generic" that Centos 4 will
still work on them.  In others, however, it just plain will not.  This
is a chipset by chipset, motherboard by motherboard, NIC by NIC sort of
question and YMMV from "what's the fuss all about" to "showstopper".  My
own experience here is that Centos is useless for laptops for this very
reason -- laptops evolve too fast and quickly leave Centos behind both
at the device level and at the desktop/utility software level.  It is of
questionable utility on desktops -- I had showstopper problems on a
number of AMD64 machines when first they came out, for example, that FC
X eventually supported perfectly.  This could well be a problem for you
if your cluster were to be AMD64 based, right?  Centos 4 was also the
last release that required split UP and SMP kernels, which could create
certain problems.

   b) The libraries it provides are left behind by the real leading edge
of library development.  Again, this can range in impact from "no big
deal" to "showstopper", depending on just what libraries your code uses.
A specific example of this historically was the Gnu Scientific Library
-- seems like the kind of thing likely to be useful in a cluster, no?
Unfortunately, the version "frozen" in Centos was so far behind the
STABLE release version by the end of the two year cycle that a lot of
code, including mine, that ran just fine using the stable release
wouldn't run on a Centos cluster without going to the trouble of
rebuilding the library and setting up a private repo for updating at
least selected libraries more aggressively.

This isn't THAT big a deal, but it is very definitely an added "cost"
and needs to be considered when making the decision.

It is worth noting that Fedora has actually proven remarkably stable if
one simply adds a 3-6 month offset into the time you do the next
upgrade, and that more and more cluster tools have been included with
Fedora out of the box to the point where a very passable cluster node
can be installed without ANY custom software builds, straight out of the
updated repo base.  It isn't on a par with Debian (where every known
package is available somewhere, somehow) but it is a decent compromise
between the surf-the-wave aspect of Debian and the stodginess of Centos.

So a good way of putting it is that Centos is a good solution where it
is a good solution, and a really terrible one otherwise.  With
conservative, known to be supported hardware and little likelihood of
adding bleeding edge stuff to your cluster over the next Centos cycle,
with more or less frozen and acceptible library requirements you can
install it and forget it, letting it yum update for the lifetime of the
cluster nodes.  If you plan to add ever new and ever more exotic
hardware, if you KNOW that the library you depend most on is under
active development and you need to be able to track that development
closely as bugs are fixed and features added, fedora or debian might
well be better choices.

I myself agree with Gerry -- Fedora whatever with PXE/kickstart is hard
to beat for diskful nodes, and after tracking FC from 2 on I'm no longer
worried about its "stability" or update cycle's finite lifetime.  I
appreciate the fact that it has a remarkably large set of packages that
are built right into it and consistently maintained as a part of the
distribution, and the fact that it is ALMOST rapidly varying enough to
keep up with laptop hardware, maybe with a one-version delay.  But this
is very much a religious type thing, and as has already been remarked,
you can use ANY linux distro and build a cluster out of it.  The only
real difference is how hard you have to work to do so -- how much of the
work required is "done for you" in the prebuilt distribution and how
much you have to do for yourself from the (after all!) open sources.

    rgb

>
>
> Mike Davis
>
>
> Bill Rankin wrote:
>> Yes, we use it with good effect on our 500+ node cluster at Duke.  It's 
>> currently running Centos-4.  I think that the only issue is that some of 
>> our developers require newer releases of a couple packages, but it's easy 
>> enough to maintain a local yum repository with those packages.
>> 
>> It's been a good stable platform for us.
>> 
>> -bill
>> 
>> 
>> On Oct 8, 2007, at 9:48 AM, Barnet Wagman wrote:
>> 
>>> Does any one use Centos on Beowulf nodes?  Of course Centos is really just 
>>> Redhat, but many people prefer it for use on servers.
>>> _______________________________________________
>>> Beowulf mailing list, Beowulf at beowulf.org
>>> To change your subscription (digest mode or unsubscribe) visit 
>>> http://www.beowulf.org/mailman/listinfo/beowulf
>> 
>> _______________________________________________
>> Beowulf mailing list, Beowulf at beowulf.org
>> To change your subscription (digest mode or unsubscribe) visit 
>> http://www.beowulf.org/mailman/listinfo/beowulf
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit 
> http://www.beowulf.org/mailman/listinfo/beowulf
>

-- 
Robert G. Brown
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone(cell): 1-919-280-8443
Web: http://www.phy.duke.edu/~rgb
Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977