[Beowulf] cluster deployment and config management

[Joe Landman]

Good morning ...


On 09/05/2017 01:24 AM, Stu Midgley wrote:
> Morning everyone
>
> I am in the process of redeveloping our cluster deployment and config 
> management environment and wondered what others are doing?
>
> First, everything we currently have is basically home-grown.

Nothing wrong with this, if it adequately solves the problem.  Many of 
the frameworks people use for these things are highly opinionated, and 
often, you'll find their opinions grate on your expectations.  At 
$dayjob-1, I developed our own kit precisely because so many of the 
other toolkits did little to big things wrong; not simply from an 
opinion point of view, but actively made specific errors that the 
developers glossed over as that aspect was unimportant to them ... while 
being of critical importance to me and my customers at the time.

>
> Our cluster deployment is a system that I've developed over the years 
> and is pretty simple - if you know BASH and how pxe booting works.  It 
> has everything from setting the correct parameters in the bios, zfs 
> ram disks for the OS, lustre for state files (usually in /var) - all 
> in the initrd.
>
> We use it to boot cluster nodes, lustre servers, misc servers and 
> desktops.
>
> We basically treat everything like a cluster.

The most competent baked distro out there for this was (in the past, 
haven't used it recently) Warewulf.  See https://github.com/warewulf/ .  
Still under active development, and Greg and team do a generally great 
job.  Least opinionated distro around, most flexible, and some of the 
best tooling.

>
> However... we do have a proliferation of images... and all need to be 
> kept up-to-date and managed.  Most of the changes from one image to 
> the next are config files.

Ahhh ... One of the things we did with our toolchain (it is open source, 
I've just never pushed it to github) was to completely separate booting 
from configuration.  That is, units booted to an operational state 
before we applied configuration.  This was in part due to long 
experience with nodes hanging during bootup with incorrect 
configurations.  If you minimize the chance for this, your nodes 
(barring physical device failure) always boot.  The only specific 
opinion we had w.r.t. this system was that the nodes had to be bootable 
via PXE, and there fore a working dhcp needed to exist on the network.

Post boot configuration, we drove via a script that downloaded and 
launched other scripts.   Since we PXE booted, network addresses were 
fine.  We didn't even enforce final network address determination on PXE 
startup.

We looked at the booting process as a state machine.  Lower level was 
raw hardware, no power.  Subsequent levels were bios POST, PXE of 
kernel, configuration phase.  During configuration phase *everything* 
was on the table w.r.t. changes.  We could (and did) alter networking, 
using programmatic methods, databases, etc. to determine and configure 
final network configs.  Same for disks, and other resources.

Configuration changes could be pushed post boot by updating a script and 
either pushing (not normally recommended for clusters of reasonable 
size) or triggering a pull/run cycle for that script/dependencies.

This allowed us to update images and configuration asynchronously.

We had to manage images, but this turned out to be generally simple.  I 
was in the midst of putting image mappings into a distributed object 
store when the company died.  Config store is similarly simple, again 
using the same mechanisms, and could be driven entirely programmatically.

Of course, for the chef/puppet/ansible/salt/cloudformation/... people, 
we could drive their process as well.


>
> We don't have a good config management (which might, hopefully, reduce 
> the number of images we need).  We tried puppet, but it seems everyone 
> hates it.  Its too complicated?  Not the right tool?

Highly opinionated config management is IMO (and yes, I am aware this is 
redundant humor) generally a bad idea.  Config management that gets out 
of your way until you need it is the right approach. Which is why we 
never tried to dictate what config management our users would use.  We 
simply handled getting the system up to an operational state, and they 
could use ours, theirs, or Frankensteinian kludges.

>
> I was thinking of using git for config files, dumping a list of rpm's, 
> dumping the active services from systemd and somehow munging all that 
> together in the initrd.  ie. git checkout the server to get config 
> files and systemctl enable/start the appropriate services etc.
>
> It started to get complicated.
>
> Any feedback/experiences appreciated.  What works well?  What doesn't?

IMO things that tie together config and booting are problematic at 
scale.  Leads to nearly unmanageable piles of images, as you've 
experienced.  Booting to an operational state, and applying all config 
post boot (ask me about my fstab replacement some day), makes for a very 
nice operational solution that scales wonderfully .... you can replicate 
images to local image servers if you wish, replicate config servers, 
load balance the whole thing to whatever scale you need.


>
> Thanks.
>
>
>
> -- 
> Dr Stuart Midgley
> sdm900 at gmail.com <mailto:sdm900 at gmail.com>
>
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf

-- 
Joe Landman
e: joe.landman at gmail.com
t: @hpcjoe
w: https://scalability.org
g: https://github.com/joelandman
l: https://www.linkedin.com/in/joelandman