[Beowulf] A start in Parallel Programming?

[Robert G. Brown]

On Tue, 13 Mar 2007, Kyle Spaans wrote:

> Do I really need MPI/OpenMPI/OpenMP/PVM? If so, does that mean I
> should suck it up and learn Fortran/C/C++? Or can I just devise my own
> message passing system to help me parallelize the workload? For
> example, TCP sockets were mentioned. In my CS class I learned Scheme
> [a dialect of LISP], and I know Scheme can work with TCP sockets. It
> has also gotten me very interested in the Functional Programming
> paradigm [but is it ideal for parallel programming?]. If possible, I'd
> like to stick with Scheme for now, as it'll make my CS class coming up
> this summer all the more interesting.

Everybody will have their own biases here, so let me freely acknowledge
that these are mine and that they are at least partly religious.

   a) Learn C.  No matter what you do, if you plan to be a "real coder"
you will sooner or later need to learn C.  Sane people don't write
operating systems in Fortran or C++ or Lisp, and there are damn good
reasons for this.  Also you can do amazing things with C and actually
understand what the computer is doing when you do them (because C has
been described on this very list as "a thin veneer of upper-level
language sensibility on top of raw assembler", a phrase that I just love
that is SO true even though I can't remember who actually said it.  I
wish it were me but it wasn't:-).  Who knows what LISP is actually doing
and how it is doing it? Not even the developers...

   b) You will need to learn to code using EITHER MPI (any flavor), PVM
(there is only one flavor), or raw sockets to become a "real parallel
programmer".  PVM is easy to learn and adequate for simple projects, but
is less "in favor" with the professional community if that matters to
you.  MPI is better maintained/advanced/optimized and is arguably more
portable, although this very list has seen a whole lot of that
"arguably" side discussed because of its lack of an ABI (so you can't
just switch MPI libraries, you have to switch MPI installations
altogether and rebuild your code and hope that there isn't too much
drift in the >>API<< let alone ABI).  Or (as you note) you can shoot for
doing real socket programming -- raw parallelism under your complete
control.

I would RECOMMEND that you start with some flavor of MPI or PVM (in that
order) and work your way through to sockets if you live long enough and
can find some good examples of working parallel applications built on
top of raw sockets to get going with.  There are many, many reasons for
this:

   * MPI is well documented and manages a lot of things such as defining
a virtual parallel machine for you.  Don't laugh -- this is NOT TRIVIAL
to do in a scalable way in compiled socket code -- see my wulfware
package (which is basically a socket-based parallel application) for at
least one XML-based solution that has its own imperfections.

   * There are many MPI users and lots of resources on the web at e.g.
www.clustermonkey.net to get you going.  Lots of books, too, some of
them even GOOD books.

   * MPI (the better variants, anyway) do a better job of doing e.g.
one-to-many communictions than you will be able to do on your own
without learning a whole lot of stuff about communication trees,
broadcasts, network latency vs bandwidth, and barriers than you want to
starting out.  Eventually you'll need to learn them, but with MPI you'll
actually end up with reasonably efficient code even when you don't
understand WHY it is efficient.

   * MPI is portable and used in many professional projects.

   * Drivers exist for MPI for advanced networking hardware, something
that won't matter to you when learning but which does matter when trying
to do serious parallel code in a scalable way on larger clusters.

   * PVM is also well doucmented, manages virtual cluster configuration
(arguably better than MPI does) has many users, has web and book
resources, and does at least a decent job in managing efficient
communications.  There are still people using it for professional
projects, and there are SOME (fewer) drivers.  I personally "like" it
better, but reluctantly concede that you'd be better off learning MPI
first and maybe only in spite of it.  (We'll see what other list users
have to say here:-).

   * Socket programming is actually one of the most difficult single
things you can tackle, no matter what the language.  In order to master
it, you will need to invest roughly $200 dollars in books right off the
top.  No, I'm not joking.  If you want to go this route let me know and
I'll dust off the titles -- two or three volumes by Stevens, plus one or
two general purpose systems programming books such as Glass and Abel,
plus finding a source of simple sample code.  That will get you to where
you can successfully write a daemon that listens on a socket, permits a
client to connect, forks off an instance of the daemon to handle the
connection while not blocking the socket, and enables both ends to read
from and write to the sockets.  A little bit of work on some
horrendously low level subroutines and you can even read and write
RELIABLY.

Then you get to do the hard part -- write all the layers of toplevel
software that permit you to define a cluster (in some reasonably
extensible way, from an input file), initiate/spawn program instances on
all of the cluster nodes (which might be daemons, might be clients
connecting back to a daemon, might be still other things), get all your
client programs to connect into some sort of network of sockets or
connect back to a toplevel "master" program that manages the
client-to-client communications or... (remember my remarks about
efficient topologies and how you don't want to learn them yet), and
then, when you're done and your application actually "works", making it
run EFFICIENTLY and ROBUSTLY.

Not for the faint of heart.  I'm one of the relatively few humans
on-list that have actually written something like this, and the others
that I know of are even beefier Mr. America He-Man weight lifter coders
(as in, I seem to recall that Don Becker likes sockets too... but then,
he DID write a large portion of the original linux network stack... ;-).
This is not to ULTIMATELY discourage this, of course.  With C and
sockets, you have absolute control.  You can do things that simply
cannot be done using PVM or MPI.  However, you'll have to work damn hard
FIRST to just get to where you can do the things that they do totally
transparently for you with a single simple subroutine call.  And for all
by a tiny fraction of tasks you might want to do, the benefits of doing
it the hard way just don't equal the extra work required.

Hope this helps.  There are a number of articles on the Monkey on
getting started with cluster, PVM, MPI, cluster programming in general
using even things like threads, ssh, and perl.  I even wrote some of
them, and others were written by people who actually know what they are
doing;-), like Jeff Layton, Doug Eadline, Jeff Squyers, and many others.

And I'm sure you CAN write parallel code in Scheme.

Just don't.  Nobody knows why CPS departments no longer teach students
to code in C (and instead teach a bizarre mix of C++, java, lisp, and
god-knows-what else first -- at one time they just LOVED pascal and
where is THAT now I aske you), but the sad truth is that to do real
computer science, sooner or later you'll almost certainly need to learn
to code in it anyway.  And maybe (shudder) fortran too.  C++ is the one
alternative that I wouldn't fight over if somebody really pushed for it,
except to note that sooner or later you'll still have to learn C, and if
you learn C++ first you'll NEVER have the patience to work WITHOUT all
the safety nets it provides and hence things like pointers will likely
remain a mystery to you.  Which is OK if you only plan to work at the
application level, maybe, but...

   HTH,

       rgb

[rgb now goes down the well-worn steps into his concrete bunker,
thoughtfully dons his mirror-reflective suit, hooks up its
self-contained refrigeration unit, closes the radiation dampers on the
slit looking out over the battlefield, makes sure that there is plenty
of food and water on the shelves, and awaits the arrival of the first
nuclear flames...]

-- 
Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu