diskless nodes? (was Re: Xbox clusters?)

[Velocet]

On Thu, Dec 06, 2001 at 03:04:22AM -0600, W Bauske's all...
> Velocet wrote:
> > 
> > I'd go insane configuring things if I had disk-full nodes. Diskless node
> > installations require me to run one cp -r command, sed 2 files with a new ip
> > address and mac address and power the thing up - presto, node n+1 is online.
> 
> How big is that file tree you're copying?

Just the OS, 500Mb or so. Its a onetime thing to setup a node to do that.
Most nodes went up on day one before the cluster was active, but a number
were added later. The copying didnt have a major impact on other jobs
running at the time. Takes about 30-45 seconds. Editing the dhcp.conf
and unique /etc files for each node is another 5 seconds with a sed script
I wrote in 2 minutes way back when.

> > I wouldnt do with the headaches of having to install stuff on n disks.
> > Been through that where n = 18, its a big pain. Not to mention that 5
> > of the nodes were down for servicing when I had to do the work. Now
> > I gotta schedule a context-switch to do the SAME wokr over again and
> > recall how I did it for those nodes when they finally come back. If
> > I had a node down on a diskless system they'd boot up with all the current
> > changes.
> 
> Diskless is a bit simpler than disk-full but not alot.

Not having to login to n nodes to change something is quite nice. Especialy
when some are offline on the day you make your changes - better remember that
you didnt change those nodes yet.

> Using RH kickstart, a disk-full node installs in around 10 minutes.
> If one trimmed down the RPM's, it would probably be 5 minutes or less. 

Right. Initial install isnt the problem, maintenance is IMHO.

> I currently load around a GB for an image. I'd bet if I worked at it,
> I could get a disk-full node to install in 3-4 minutes at which point,
> it would look very similar to diskless. You just install everytime the
> node reboots or comes up. I read somewhere about a university that
> does exactly that and will install whatever OS you select at boot.

We flip between two versions of FreeBSD and a couple versions of linux
as desired on the cluster with diskless booting. Its really quite sweet.

> > In fact, as jobs finish we reboot nodes just for the hell of it - clears
> > memory leaks and loads all current configs, kernels, etc. Its just too
> > sweet to avoid!
> > 
> > Does NO ONE use diskless clusters?
> > 
> 
> I ran them around 8-10 years ago.

Must have been hell on 10Mbps networks!: )

> One thing I liked about diskless was I could edit the files that
> controlled the node even if it wasn't running. Can be handy sometimes.

as I said, for downed nodes, definitely. And most of my changes to the OS are
to a single image loaded by all the nodes, I forgot to add. (Hmm glaring
omission, my point before about downed nodes doenst make sense without this
statement). There are only 3 unique files per node, and we use a special
mounting setup that mounts these unique files overtop of the default
disk image into a ramdisk at boot. Just so flexible and elegant.
 
> Of course, with kickstart, you add customizations in the kickstart
> file and re-install so it's not so far off.

How often do you reboot and suffer 3-4 min of install time?

/kc
-- 
Ken Chase, math at velocet.ca  *  Velocet Communications Inc.  *  Toronto, CANADA