[Beowulf] Building a new cluster - seeking some advice

[Mark Hahn]

> small cluster (microwulf style) with each node having a single USB
> thumbdrive instead of a disk.  I thought it might be easier than trying to
> get nodes to boot PXE style over the network.  And it seemed to me that

PXE is dead simple these days, since you'll probably be using an integrated
nic, and it almost certainly has PXE support.  5 years ago, integrated nics
and PXE were less common but then again boot-from-USB was too...

> thumbdrives might be easier than disk-per-node to keep in sync: I'd just
> unplug them from the nodes, plug them into to a USB hub on another computer
> where I build my distribution, and copy files to them, then plug them back
> into their nodes.

even for fully diskful (heavyweight) installs, I'd use still PXE to 
give me a single point of management.  I suppose for a ~8-node cluster,
fiddling with USB sticks might be acceptable, but it's the kind of thing
that gives clusters a bad name (sorry!).  that is, it's easy to make 
clusters scale very sublinearly, so the effort to do a 200 node cluster
is only marginally more than a 100 node one.

> Also the USB drives would serve for any local filesystem
> needs, e.g., for logging or whatever.

I think people mistake how much this is used - you want your syslogs
to go to a central management server, for instance.  otherwise you'll
probably never look at them.

> $12 it seemed a pretty easy and cheap and low power solution.  And no moving
> parts means the "disks" won't die for mechanical reasons (and they won't be
> written to enough to worry about flash-wear).

the best thing about nfs-root, disk-for-swap+tmp is that you almost don't 
care whether the disk fails.  certainly such a node can still be used,
though it won't be quite as robust wrt high memory use.