<div dir="ltr"><div>Regarding Rocks clusters, permit me to vent a little.</div><div>In my last employ we provided Rocks clusters.... Rocks is firmly embedded in the Redhat 6 era with out of date 2.6 kernels.</div><div>It uses kickstart for installations (which is OK). However with modern generations of Intel processors you get a warning from Kickstart that the architecture is unsupported... so you must add the flag to accept unsupported architectures to your kickstart.cfg</div><div>If at this point the Forbidden Planet "Danger Will Robinson!" soundbite is not playing on your head then I have a bridge to sell you in London...</div><div>Rocks version 7 is supposed to be on the horizon, but I doubt it. It may have been launched, but I left the Rocks list five months ago and I do not feel the lack. Sorry Rocks types if you are on here... no intention to give you a kicking.</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On 5 September 2017 at 20:19, psc <span dir="ltr"><<a href="mailto:pscadmin@avalon.umaryland.edu" target="_blank">pscadmin@avalon.umaryland.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hey everyone, .. any idea what happened with perceus?<br>
<a href="http://www.linux-mag.com/id/6386/" target="_blank" rel="noreferrer">http://www.linux-mag.com/id/63<wbr>86/</a><br>
<a href="https://github.com/perceus/perceus" target="_blank" rel="noreferrer">https://github.com/perceus/per<wbr>ceus</a><br>
<br>
.. yeah; what happened with Arthur Stevens (Perceus, GravityFS/OS Green Provisioning, etc. ) where he is now; who is maintain, if anyone, perceus ?<br>
<br>
.. and come on Greg K. ... we know you are luring there somewhere being busy with singularity<br>
<a href="http://singularity.lbl.gov/" target="_blank" rel="noreferrer">http://singularity.lbl.gov/</a> (kudos .. great job as always !!!)<br>
.. wasn't perceus yours original baby?<br>
<a href="https://gmkurtzer.github.io/" target="_blank" rel="noreferrer">https://gmkurtzer.github.io/</a><br>
.. can you bring some light what happened with the perceus project? .. I'd love to see it integrated with singularity -- that would made my day/month/year !!!!!!<br>
<br>
thanks!<br>
cheers,<br>
psc<br>
<br>
p.s. .. there there used to be rocks clusters (not sure about it's status these days)<br>
<a href="http://www.rocksclusters.org/wordpress/" target="_blank" rel="noreferrer">http://www.rocksclusters.org/w<wbr>ordpress/</a><br>
<br>
p.s.s. .. I'd say Warewulf is the "best" bet in most cases .. why keep reinventing the wheel ?<br>
<br>
<br>
On 09/05/2017 01:43 PM, <a href="mailto:beowulf-request@beowulf.org" target="_blank">beowulf-request@beowulf.org</a> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Send Beowulf mailing list submissions to<br>
<a href="mailto:beowulf@beowulf.org" target="_blank">beowulf@beowulf.org</a><br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="http://www.beowulf.org/mailman/listinfo/beowulf" target="_blank" rel="noreferrer">http://www.beowulf.org/mailman<wbr>/listinfo/beowulf</a><br>
or, via email, send a message with subject or body 'help' to<br>
<a href="mailto:beowulf-request@beowulf.org" target="_blank">beowulf-request@beowulf.org</a><br>
<br>
You can reach the person managing the list at<br>
<a href="mailto:beowulf-owner@beowulf.org" target="_blank">beowulf-owner@beowulf.org</a><br>
<br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of Beowulf digest..."<br>
<br>
<br>
Today's Topics:<br>
<br>
1. Re: cluster deployment and config management (Joe Landman)<br>
2. Re: cluster deployment and config management (Arif Ali)<br>
3. RAID5 rebuild, remount with write without reboot? (mathog)<br>
4. Re: RAID5 rebuild, remount with write without reboot?<br>
(John Hearns)<br>
<br>
<br>
------------------------------<wbr>------------------------------<wbr>----------<br>
<br>
Message: 1<br>
Date: Tue, 5 Sep 2017 08:20:03 -0400<br>
From: Joe Landman <<a href="mailto:joe.landman@gmail.com" target="_blank">joe.landman@gmail.com</a>><br>
To: <a href="mailto:beowulf@beowulf.org" target="_blank">beowulf@beowulf.org</a><br>
Subject: Re: [Beowulf] cluster deployment and config management<br>
Message-ID: <<a href="mailto:2da2937a-9055-6514-3d24-a739aee12845@gmail.com" target="_blank">2da2937a-9055-6514-3d24-a739a<wbr>ee12845@gmail.com</a>><br>
Content-Type: text/plain; charset=utf-8; format=flowed<div><div class="h5"><br>
<br>
Good morning ...<br>
<br>
<br>
On 09/05/2017 01:24 AM, Stu Midgley wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Morning everyone<br>
<br>
I am in the process of redeveloping our cluster deployment and config<br>
management environment and wondered what others are doing?<br>
<br>
First, everything we currently have is basically home-grown.<br>
</blockquote>
Nothing wrong with this, if it adequately solves the problem. Many of<br>
the frameworks people use for these things are highly opinionated, and<br>
often, you'll find their opinions grate on your expectations. At<br>
$dayjob-1, I developed our own kit precisely because so many of the<br>
other toolkits did little to big things wrong; not simply from an<br>
opinion point of view, but actively made specific errors that the<br>
developers glossed over as that aspect was unimportant to them ... while<br>
being of critical importance to me and my customers at the time.<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Our cluster deployment is a system that I've developed over the years<br>
and is pretty simple - if you know BASH and how pxe booting works. It<br>
has everything from setting the correct parameters in the bios, zfs<br>
ram disks for the OS, lustre for state files (usually in /var) - all<br>
in the initrd.<br>
<br>
We use it to boot cluster nodes, lustre servers, misc servers and<br>
desktops.<br>
<br>
We basically treat everything like a cluster.<br>
</blockquote>
The most competent baked distro out there for this was (in the past,<br>
haven't used it recently) Warewulf. See <a href="https://github.com/warewulf/" target="_blank" rel="noreferrer">https://github.com/warewulf/</a> .<br>
Still under active development, and Greg and team do a generally great<br>
job. Least opinionated distro around, most flexible, and some of the<br>
best tooling.<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
However... we do have a proliferation of images... and all need to be<br>
kept up-to-date and managed. Most of the changes from one image to<br>
the next are config files.<br>
</blockquote></div></div>
Ahhh ... One of the things we did with our toolchain (it is open source,<span><br>
I've just never pushed it to github) was to completely separate booting<br>
from configuration. That is, units booted to an operational state<br>
before we applied configuration. This was in part due to long<br>
experience with nodes hanging during bootup with incorrect<br>
configurations. If you minimize the chance for this, your nodes<br>
(barring physical device failure) always boot. The only specific<br>
opinion we had w.r.t. this system was that the nodes had to be bootable<br>
via PXE, and there fore a working dhcp needed to exist on the network.<br>
<br></span><div><div class="h5">
Post boot configuration, we drove via a script that downloaded and<br>
launched other scripts. Since we PXE booted, network addresses were<br>
fine. We didn't even enforce final network address determination on PXE<br>
startup.<br>
<br>
We looked at the booting process as a state machine. Lower level was<br>
raw hardware, no power. Subsequent levels were bios POST, PXE of<br>
kernel, configuration phase. During configuration phase *everything*<br>
was on the table w.r.t. changes. We could (and did) alter networking,<br>
using programmatic methods, databases, etc. to determine and configure<br>
final network configs. Same for disks, and other resources.<br>
<br>
Configuration changes could be pushed post boot by updating a script and<br>
either pushing (not normally recommended for clusters of reasonable<br>
size) or triggering a pull/run cycle for that script/dependencies.<br>
<br>
This allowed us to update images and configuration asynchronously.<br>
<br>
We had to manage images, but this turned out to be generally simple. I<br>
was in the midst of putting image mappings into a distributed object<br>
store when the company died. Config store is similarly simple, again<br>
using the same mechanisms, and could be driven entirely programmatically.<br>
<br>
Of course, for the chef/puppet/ansible/salt/cloud<wbr>formation/... people,<br>
we could drive their process as well.<br>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
We don't have a good config management (which might, hopefully, reduce<br>
the number of images we need). We tried puppet, but it seems everyone<br>
hates it. Its too complicated? Not the right tool?<br>
</blockquote>
Highly opinionated config management is IMO (and yes, I am aware this is<br>
redundant humor) generally a bad idea. Config management that gets out<br>
of your way until you need it is the right approach. Which is why we<br>
never tried to dictate what config management our users would use. We<br>
simply handled getting the system up to an operational state, and they<br>
could use ours, theirs, or Frankensteinian kludges.<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I was thinking of using git for config files, dumping a list of rpm's,<br>
dumping the active services from systemd and somehow munging all that<br>
together in the initrd. ie. git checkout the server to get config<br>
files and systemctl enable/start the appropriate services etc.<br>
<br>
It started to get complicated.<br>
<br>
Any feedback/experiences appreciated. What works well? What doesn't?<br>
</blockquote>
IMO things that tie together config and booting are problematic at<br>
scale. Leads to nearly unmanageable piles of images, as you've<br>
experienced. Booting to an operational state, and applying all config<br>
post boot (ask me about my fstab replacement some day), makes for a very<br>
nice operational solution that scales wonderfully .... you can replicate<br>
images to local image servers if you wish, replicate config servers,<br>
load balance the whole thing to whatever scale you need.<br>
<br>
<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5">
Thanks.<br>
<br>
<br>
<br>
-- <br>
Dr Stuart Midgley<br>
<a href="mailto:sdm900@gmail.com" target="_blank">sdm900@gmail.com</a> <mailto:<a href="mailto:sdm900@gmail.com" target="_blank">sdm900@gmail.com</a>><br>
<br>
<br></div></div><span>
______________________________<wbr>_________________<br>
Beowulf mailing list, <a href="mailto:Beowulf@beowulf.org" target="_blank">Beowulf@beowulf.org</a> sponsored by Penguin Computing<br>
To change your subscription (digest mode or unsubscribe) visit <a href="http://www.beowulf.org/mailman/listinfo/beowulf" target="_blank" rel="noreferrer">http://www.beowulf.org/mailman<wbr>/listinfo/beowulf</a><br>
</span></blockquote></blockquote><div class="HOEnZb"><div class="h5">
<br>
<br>
______________________________<wbr>_________________<br>
Beowulf mailing list, <a href="mailto:Beowulf@beowulf.org" target="_blank">Beowulf@beowulf.org</a> sponsored by Penguin Computing<br>
To change your subscription (digest mode or unsubscribe) visit <a href="http://www.beowulf.org/mailman/listinfo/beowulf" target="_blank" rel="noreferrer">http://www.beowulf.org/mailman<wbr>/listinfo/beowulf</a><br>
</div></div></blockquote></div><br></div>