[Beowulf] Which distro for the cluster?

[Robert G. Brown]

On Fri, 29 Dec 2006, Geoff Jacobs wrote:

> Robert G. Brown wrote:
> <snip>
>> The problem is REALLY evident for laptops -- there are really major
>> changes in the way the kernel, rootspace, and userspace manages devices,
>> changes that are absolutely necessary for us to be able to plug cameras,
>> memory sticks, MP3 players, printers, bluetooth devices, and all of that
>> right into the laptop and have it "just work".  NetworkManager simply
>> doesn't work for most laptops and wireless devices before FC5, and it
>> doesn't really work "right" until you get to FC6 AND update to at least
>> 0.6.4. On RHEL/Centos 4 (FC4 frozen, basically), well...
> Laptops. *shudder* How often do laptop manufacturers change their
> hardware configurations? Every other week?
>
> I've found the optimal way to purchase laptop hardware is with a good
> live cd for testing. No boot, no buy.

I just drink heavily, personally, and trust in various deities.

>> FC (and with a lot of synergy, of course, even though the two
>> communities in some respects resemble Sunnis vs the Shites in Iraq:-).
> RGB must now go into hiding due to the fatwa against him.

I'm just wearing my flameproof suit.

That's really what they need to introduce into Iraq -- a fireproof
kevlar birka.  Bomb goes off in the market?  No problem.  Just dust
yourself off and move on, picking bomb splinters out of your robes.
Eventually the bombers get bored and go back to hurling imprecations at
one another after drinking large amounts of extremely strong coffee.

Sigh.  I suppose I really shouldn't joke about it, but it IS so very
very very tiresome and evil.

>> pvm or mpi?  How difficult is it to take over a PVM virtual machine and
>> insert your own binary?  I suspect that it isn't that difficult, but I
>> don't really know.  Any comments, any experts out there?
> Would compromising PVM frag a user or the whole system?

I think this very much depends on the architecture and how things are
set up.  Some message libraries run as root, others as nobody, others as
users.  And compromising users isn't necessarily all that great -- it is
usually the first step in compromising a network anyway.  There are
usually small rare carefully guarded holes at the port level as nearly
everybody knows by now that leaving lots of open ports on a system is a
bad idea, so there are only a very few applications that really have to
be secure to keep a system safe from outside intruders.  In many cases,
only ssh, for example.  Once an intruder has user-level access, though,
they have the ability to use ALL of the applications on the system to
promote to root.  Suddenly there are lots of running daemons, lots of
root processes, lots of suid root binaries all of which have to be
exploit free to prevent promotion.  These are almost invariably less
well audited than primary network daemons.

Again, though, the point is clear.  Compute farm style grids can be made
"semi-hard" -- as hard as any client/server lan with a very large and
uncontrolled user base running lots of unaudited code with very nearly
arbitrary libraries (in some cases users can even upload their own
libraries to the system as part of their "package").  Still, the usual
boundaries between users and at the network layer exist and are pretty
well defended, and it is not at all easy to take over another user's
process or corrupt its data stream or monitor its data stream from an
ordinary user account or network PoP (assume a laptop snapped into a hot
port where a bad guy has root and promiscuous network access).

Inside a "true beowulf" or firewall protected not-quite-so-tight grid
cluster that runs VM parallel code this is not true, because the VM
libraries do not use encryption for obvious reasons and use nothing
beyond the network stacks themselves for validating connections.  Which
in the case of UDP is virtually nothing -- connectionless and nearly
anonymous save for the IP numbers in the packet headers, that are pretty
much just trusted.  TCP is a bit better, although I was around in the
days of the Morris worm and would never bet than an ubercracker couldn't
snag a TCP connection given enough time and effort.  rsh is a joke as
far as security is concerned (and ssh adds overhead to at least some
things...).

I just have little confidence that a cluster in active use, especially
one that uses rsh as a base and/or bproc, is in any way "secure" against
cracking.  In most cases the security layer is the head node through
which it is accessed, which is a de facto firewall.  I don't know that
people have spent a lot of time trying to figure out how hard PVM or MPI
is to exploit, but as network services they are certainly pretty high up
there on the list of possibly exploitable things...

>> Where one is welcome to argue about what constitutes a "fast-moving"
>> repository.  yum doesn't care, really.  Everything else is up to the
>> conservative versus experimental inclinations of the admin.
> How usable is the FC development repository?

I don't know.  My laptop is the only thing that I've got that is running
FC6 yet (until post server upgrade, at which time I'll flash everything
on up) and I don't screw around on a production system, which my laptop
definitely is.  As in I cannot afford for it to be down.  I no longer
use a real desk for much of anything except holding bill receipts until
I can file them.  My laptop, with internet access to e.g. wikipedia and
google, is a large fraction of my brain.  I've used it a dozen times to
access information I half remember (or half forgot) and validate it in
the discussion so far.

I've thought about turning it on in yum long enough to nail e.g. the
latest upgrade to NetworkManager, as it apparently fixes a nasty bug in
the openvpn segment that causes my access to Duke's vpn to work, but
only by crashing NM.  I really want to be able to toggle it up and down
at will, and expect that I could if I grabbed from the devel branch
and/or downloaded from CVS and rebuilt.  Time time time.  It works well
enough (and ever better).  Better than WinXX's related tools that are
just plain evil, especially on a user's laptop that has loaded three
layers of things -- microsoft's, the laptop vendor's, and an ISP's --
all trying to provide an "easy" interface to the wireless card, and all
colliding...

    rgb

-- 
Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu