[Beowulf] Which distro for the cluster?
Andrew M.A. Cater
amacater at galactic.demon.co.uk
Thu Dec 28 16:57:49 PST 2006
On Fri, Dec 29, 2006 at 09:39:59AM +1100, Chris Samuel wrote:
> On Friday 29 December 2006 04:24, Robert G. Brown wrote:
> > I'd be interested in comments to the contrary, but I suspect that Gentoo
> > is pretty close to the worst possible choice for a cluster base. Maybe
> > slackware is worse, I don't know.
> But think of the speed you could emerge applications with a large cluster,
> distcc and ccache! :-)
> Then add on the hours of fun trying to track down a problem that's unique to
> your cluster due to combinations of compiler quirks, library versions, kernel
> bugs and application odditites..
This is a valid point. If you are not a professional sysadmin / don't
have one of those around, you don't want to spend time needlessly doing
hacker/geek/sysadmin type wizardry - you need to get on with your
Most of the academics and bright people on this list have become Beowulf
experts and admins. by default - no one else has been there to do it for
them - but that's not originally their main area of expertise.
Pick a distribution that you know that provides the maximum ease of
maintenance with the maximum number of useful applications already
packaged / readily available / easily ported. This will depend on your
problem set: simulating nuclear explosions/weather storm cells/crashing
cars or are you sequencing genomes/calculating pi/drawing ray traced
> > I personally would suggest that you go with one of the mainstream,
> > reasonably well supported, package based distributions. Centos, FC, RH,
> > SuSE, Debian/Ubuntu.
> I'd have to agree there.
Red Hat Enterprise based solutions don't cut it on the application
front / packaged libraries in my (very limited) experience. The
upgrade/maintenance path is not good - it's easier to start from scratch
and reinstall than to move from one major release to another.
Fedora Core - you _have_ to be joking :) Lots of applications - but
little more than 6 - 12 months of support.
SuSE is better than RH in some respects, worse in others. OpenSuSE - you
may be on your own. SLED 10.2 may have licence costs?
Debian (and to a lesser extent Ubuntu) has the largest set of
pre-packaged "stuff" for specialist maths that I know of and has
reasonable general purpose tools.
> > I myself favor RH derived, rpm-based,
> > yum-supported distros that can be installed by PXE/DHCP, kickstart, yum
> > from a repository server. Installation of such a cluster on diskful
> > systems proceeds as follows:
If I read the original post correctly, you're talking of an initial 8
nodes or so and a head node. Prototype it - grab a couple of desktop
machines from somewhere, a switch and some cat 5. Set up three machines:
one head and two nodes. Work your way through a toy problem. Do this for
Warewulf/Rocks/Oscar or whatever - it will give you a feel for something
of the complexity you'll get and the likely issues you'll face.
> What I'd really like is for a kickstart compatible Debian/Ubuntu (but with
> mixed 64/32 bit support for AMD64 systems). I know the Ubuntu folks started
> on this , but I don't think they managed to get very far.
dpkg --get-selections >> tempfile ; pxe boot for new node ; scp tempfile
root at newnode ; ssh newnode; dpkg --set-selections < /root/tempfile ;
apt-get update ; apt-get dselect-upgrade
goes a long way :)
> The sad fact of the matter is that often it's the ISV's and cluster management
> tools that determine what choice of distro you have. :-(
HP and IBM are distro neutral - they'll install / support whatever you
ask them to (and pay them for).
> Yes, I know all about LSB but there are a grand total of 0 applications
> certified for the current version (3.x)  and a grand total of 1 certified
> application (though on 3 platforms)  over the total life of the LSB
> standards. 
> To paraphrase the Blues Brothers:
> ISV Vendor: Oh we got both kinds of Linux here, RHEL and SLES!
> Bah humbug. :-)
More information about the Beowulf