[Beowulf] Setting up a new Beowulf cluster

[Berkley Starks]

Thank you all so much for the advice so far.  This has helped me see a few
more of the things that I did not realize at first.

For a little info on the project, I developed this project as a tool to work
on my Senior Thesis in a year or so.  Doing computational nuclear physics
requires such resources.  It will also be used heavily for Monte Carlo
Simulations and just about any other form of computational physics.  The two
named are definite projects that are already on the line up for when I do
get the cluster up and functional.

I want to be able to make the cluster easily expandable, in that I will be
starting with only a few machines (about 2-8), but will be acquiring more as
time goes on.  The university that I am attending surpluses out "old"
machines every 4 years, and we have set up a program where we can get a
percentage of the surplus machines for out cluster.

So, as for size.  Initially it will be a smaller cluster, but will grow as
time goes on.

Being new to the Beowulf world, I am just mainly looking for some advice as
to what distro to use (I would never dream of setting up a cluster on
windows) and if there were any little tricks that weren't mentioned in the
setup how to guides.

Oh, and I would also like to know if there was a way to set up a task
priority where if I had only only application running it would use all the
processors on the cluster, but if I had two tasks sent to the cluster then
it would split the load between them and run both simultaneously, but still
using a maximum for the needed processors.

Thanks again so much,

Berkley

On Feb 8, 2008 9:11 AM, Robert G. Brown <rgb at phy.duke.edu> wrote:

> On Thu, 7 Feb 2008, Berkley Starks wrote:
>
> > Hello all,
> >
> > I've been a computer user for the past several years working in
> different
> > areas of the IT world.  I've recently been commissioned by my university
> to
> > set up the first operating Beowulf Cluster.
> >
> > I'm am moderately familiar with the Linux OS, having ran it for the past
> > several years using the distro's of Debian, Ubuntu, Fedora Core, and
> > Mandriva.
> >
> > With setting up this new cluster I would like any advice possible on
> what OS
> > to use, how to set it up, and any other pertinent information that I
> might
> > need.
>
> This question has been answered on-list in detail a few zillion times.
> I'd suggest consulting (in rough order):
>
>   a) The list archives (now that you're a member you can get to them,
> although they are digested and googleable for the most part anyway).
>
>   b) Google.  For example, there is a lovely howto here:
>
>     http://www.linux.org/docs/ldp/howto/Parallel-Processing-HOWTO.html
>
> that is remarkably current and a good quick place to start.
>
>   c) Feel free to browse my free online book here:
>
>     http://www.phy.duke.edu/~rgb/Beowulf/beowulf_book.php<http://www.phy.duke.edu/%7Ergb/Beowulf/beowulf_book.php>
>
> I'm working on making it paper-printable via lulu, but I need time I
> don't have and so that project languishes a bit.  You "can" get a paper
> copy there if you want, but it is pretty much what is on the free
> website including the holes.
>
> > Oh, and the cluster will be used for computational physics.  I am a
> physics
> > major making it for the physics department here.  It will need to be
> able to
> > use C++ and Fortran at a bare minimum.
>
> C, C++ and Fortran are all no problem.  The more important questions
> are:
>
>   a) How coupled are the parallel tasks?  That is, do you want a cluster
> that can run N independent jobs on N independent nodes (where the jobs
> don't communicate with each other at all), or do you want a cluster
> where the N nodes all do work on a common task as part of one massive
> parallel program?  If the former, you're in luck and cluster design is
> easy and the cluster purchase will be cheap.
>
>   b) If they are coupled, are the tasks "tightly coupled" so each
> subtask can only advance a little bit before communications are required
> in order to take the next step?  "Synchronous" so all steps have to be
> completed on all nodes before any can advance?  Are the messages really
> big (bandwidth limited) or tiny and frequent (latency limited)?
>
> If any of these latter answers are "yes", post a detailed description of
> the tasks (as best you can) to get some advice on choosing a network, as
> that's the design parameter that is largely controlled by the answers.
>
>    rgb
>
> >
> > Thanks again
> >
>
> --
> Robert G. Brown                            Phone(cell): 1-919-280-8443
> Duke University Physics Dept, Box 90305
> Durham, N.C. 27708-0305
> Web: http://www.phy.duke.edu/~rgb <http://www.phy.duke.edu/%7Ergb>
> Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php<http://www.phy.duke.edu/%7Ergb/Lilith/Lilith.php>
> Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.beowulf.org/pipermail/beowulf/attachments/20080208/de51a8f0/attachment.html>