[Beowulf] Is Beowulf the right solution for my problem ?

[Robert G. Brown]

On Fri, 22 Apr 2005, Nicolas GUIOT wrote:

> Hello everybody.
> 
> I was just looking for a cluster management solution, and found out that Beowulf could be a good solution.
> I then started reading this : http://www.phy.duke.edu/~rgb/Beowulf/beowulf_book/beowulf_book/index.html
> 
> And I'm now wondering if my applications were right designed to run on the beowulf cluster.
> 
> >From what I read so far, it seems beowulf is adapted to make ONE calculation on several nodes, but this is not my objective : 
> -I have a COW of about 20 to 40 computers, mostly differents, in terms of CPU, RAM, and so on
> - We run several applications (Theoritical Bio-chemistry), not especially designed for parallel processing

> My goal is to have a master node to which everyone supplies his jobs,
> then the master dispatches the jobs to the different computers,
> according to criterias (CPU must be Intel, CPU must be over 2 GHz, need
> much RAM, etc....)
>
> So the question is : is beowulf adapted to my needs, and if it's not,
> do you know any available (free) solution for me ? (I've already heard
> about Sun Grid Engine, but for "political" reasons....)

What's political about SGE?  It's open source, and while it is also a
bit of a PITA to build because of the way they package and distribute it
it is in widespread usage in grid environments.  RPM packagings of it
exist, as well (see e.g.

  http://www.linux.duke.edu/~agrajag/RPMS/

and maybe look at warewulf as well).  Other things to look into include
Condor and OpenPBS.  PBS (in my mind) has more "politics" associated
with it, as it has a really sucky license that requires "registration"
Condor, also, has a fairly sucky license and primary distribution site,
but perhaps less so, as they no longer limit distribution (although they
still require "registration").  Sun, on the other hand, just makes you
read their license and click "accept", and even that's only for binaries
-- the sources are in publically accessible CVS.

But anyway, to address your other remarks yes, it sounds like you are de
facto running a more generic compute cluster than a "beowulf" per se --
something closer to a distributed parallel computer or Grid.  This may
or may not matter -- a beowulf can run SGE (or other batch system) on a
front end as easily as a NOW/COW cluster.  It's just a matter of which
one is easiest for you to set up and administer. These days, with PXE
and various automated linux install tools for both rpm and deb based
linuces, installing a LAN-style cluster is very, very easy if you know
what you're doing, and there is a lot of documentation to learn from if
you don't.  OTOH things like Scyld reduce the level of expertise
required to get going still further and provide human support -- for a
fee.

One thing I'd recommend looking at is warewulf

  http://warewulf.lbl.gov/pmwiki/

as it is a cluster toolkit that fits (as packages) on top of whatever
LAN and Linux distro you're using now.  Unfortunately, I couldn't find a
current list of the software it bundles on their site, but it is very
easy to build and install (or just install depending on what you're
running) and probably contains SGE ready to run (and possibly other
batch managers).  It is supported by real cluster humans who actually
use it, and is pretty current in terms of what it provides.

   rgb

> Thanks for your help
> 
> Nicolas GUIOT
> -----------------------------------------------
> Institut de Biologie Physico Chimique - UPR 9080
> 13 rue Pierre et Marie Curie
> 75005 PARIS
> 
> Tel : 01 58 41 51 70
> Fax : 01 58 41 50 26
> ------------------------------------------------
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
> 

-- 
Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu