[Beowulf] where can i learn to build a cluster machine?

[Robert G. Brown]

On Thu, 24 Feb 2005, Starship Warrior wrote:

> I am totally new to clusters but have been a list member for some time 
> and read all the emails trying to learn more - so can anyone tell me 
> where there is a good how to guide - I have three machines that I would 
> like to use linus and cluster together just to learn more - two are 3.0 
> Pentiums and the last one maybe a 3.5 or 6 not sure yet they the two 
> have the same ASUS MB  and will use SCSI  drives the third will also 
> have a ASUS MB just not sure yet what I will get
> 
> thanks for any and all information

A "good howto guide" is a tough thing to define, because of the breadth
of the problem, but here goes.

See:

  http://www.phy.duke.edu/brahma

(for example) as a cluster resource clearinghouse, with links to other
cluster resource clearinghouses such as beowulf.org, beowulf underground
and others.  In particular there are links to the beowulf howto, the
beowulf FAQ, and an online book on cluster engineering that can probably
suffice to get you started.

See also a series of columns I wrote last year for the then brand new
Cluster World Magazine on this very topic.

See also the list archives -- this is a FAQ beyond the FAQ and has been
discussed/described on list repeatedly over years.

See also sites like the warewulf site -- there are now free
cluster-in-a-box (so to speak) distributions that should permit you to
build and configure a learning cluster very easily indeed, often without
even installing an operating system image on the nodes (so they can
continue to run whatever you like except when you boot them into a
cluster).

To give you the direct answer, it goes something like the following:

  a) Hook systems into a common switched LAN e.g. an ethernet switch.
  b) If possible use decent quality PXE-aware NICs
  c) If possible use nodes with a decent amount of installed memory (>=
192 MB) although it is possible to get by with less, with effort.
  d) Node hard disk is optional for at least some installation methods
(e.g. warewulf) but is useful and enables others.
  e) At least one system NEEDS ample hard disk and will serve as a
"server" or "head node" to your cluster.  This node will manage boot
images, the distro you wish to install, NFS or other shared filesystems,
authentication, and gives you a place to "login to the cluster".  Note
that this is a sloppy requirement -- there are many different ways to
manage this and I'm just describing one of the simplest and most
straightforward ones.

It then goes like this.  Set up linux (of your choice) on the boot
server/head node.  Learn to PXE boot (dhpc, tftp) and set up head node
as boot server.  Learn to set up an installation repository for e.g.
kickstart or the distribution and packaging scheme of your choice and do
so via a mirror on your head node.  Create accounts and NFS home
directories and so on on your head node.

Then pick a kind of cluster and install it.  This could be a bootable
remote mountable node image on your server (warewulf) or kickstarting an
installation onto all your nodes native or using FAI or something else.
What you choose here depends on what linux distro you're comfortable and
how serious a cluster you want to build.  For most beginners, I tend to
suggest either using a canned cluster (warewulf) or a simple NOW cluster
(e.g. kickstarted FC2 on all nodes).  Install "parallel clustering"
packages as desired, e.g. pvm, lam mpi or mpich, ganglion, wulfware
(xmlsysd/wulfstat), whatever.  Or not, you can actually do and learn
about clustering without them in at least some modes of operation using
just plain old compilers, modern perl, ssh, and some scripts.

That's it.  Write a parallel program (using PVM or MPI) or run a serial
program in parallel (lots of ways) and you can start to learn about
parallel speedup and scaling and everything...

   rgb

> 
> Starship Warrior
> Cluster user wantabe LOL
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
> 

-- 
Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu