[Beowulf] Re: distcc on Beowulf?

[scunacc]

Dear Joel,

> Does anyone have experiences using distcc on a beowulf cluster?

>From below, it seems you mean specifically a BProc-based cluster.

Yes, I've been doing that for some little while now.

> I'm in the process of attempting to setup a Scyld cluster to do
> distcc, but the BProc processing methods and lack of complete mounts
> on compute nodes is giving me some trouble.
> (gcc needs to exec cc1)

Not sure what you mean by the BProc processing methods. That's not
*really* relevant to what you need to achieve. If you mean the way
things are configured then I agree. But it's not really a hindrance, -
just need to be creative with NFS :-)

The lack of mounts, libraries, binaries, etc. is a tad more important.

By altering the config files and making sure you have the correct
binaries and libraries in a mountable place on the master, you can
kludge it to make distcc work. I'll try and fill in with details
tomorrow as I have time.

However, I wouldn't recommend leaving things that way long term owing to
the potential amount of NFS traffic you will generate. The issue is
letting distcc be in charge rather than BProc and friends for the remote
distribution. If your cluster isn't being used for anything else at the
time, it might be OK for you depending upon what you're trying to build.
I have been building MPI-based libraries, math libraries, tools,
applications etc. this way for a while but on an otherwise quiescent
cluster.

It's late right now, but I'll try and dig the details out on Tuesday.

An alternative that I've also developed for providing a "general-case"
alternative to BProc control of a cluster is to boot the cluster in a
non-BProc fashion with a custom-built diskless/NFS setup. That gives you
the ability to better regulate the "normality" of what the compute nodes
see, but again, it's only really a workable solution if a.) You already
have access to such a diskless/NFS setup (or could grow one quickly),
b.) You have administrative and "time" control over the cluster c.) You
have a spare machine with sufficient disk space able to act as a
diskless/NFS server. You wouldn't have to alter the clients at all
except to get them to remote boot from this alternate server rather than
the BProc master. 

a.) might be tackled with ClusterKnoppix or Quantian or EduOscar or some
other cluster livecd if you didn't want to develop your own diskless/NFS
master/client setup.

Kind regards

Derek Jones.