<div dir="ltr">
<p class="MsoNormal" style="margin:0in 0in 8pt;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif">Dear colleagues,<span></span></p>
<div><br></div>
<p class="MsoNormal" style="margin:0in 0in 8pt;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif">We are working on a new full Ansible based cluster stack,
and because we are reaching a stable state, we think it would be a good idea to
present it to the Beowulf community.<span></span></p><span> </span>
<p class="MsoNormal" style="margin:0in 0in 8pt;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif">I tried to provide as much as possible relevant information in this mail.<span></span></p>
<p class="MsoNormal" style="margin:0in 0in 8pt;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif"><b><u><span><br></span><span></span></u></b></p>
<p class="MsoNormal" style="margin:0in 0in 8pt;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif"><b><u>Origin of the need:<span></span></u></b></p>
<p class="MsoNormal" style="margin:0in 0in 8pt;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif">We are building a small HPC center in a FabLab, at Auray –
France (<a href="https://en.wikipedia.org/wiki/Auray">https://en.wikipedia.org/wiki/Auray</a>). We needed something simple to
manage our cluster and our workstations, but also very flexible as we are
working with enterprises and universities (some wants Centos, others Ubuntu,
etc).<span></span></p>
<p class="MsoNormal" style="margin:0in 0in 8pt;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif">We also wanted something with less scripts as possible, to
easily maintain the product and spend less time managing the cluster to have
spare time doing interesting things.<span></span></p>
<p class="MsoNormal" style="margin:0in 0in 8pt;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif">After a deep search, Ansible was chosen for its simplicity,
and we iterated over and over to converge to this stack.<span></span></p>
<p class="MsoNormal" style="margin:0in 0in 8pt;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif">Stack is fully open, in MIT. We do not sell anything, just
free best effort help :-)<span></span></p>
<p class="MsoNormal" style="margin:0in 0in 8pt;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif"><span> </span></p>
<p class="MsoNormal" style="margin:0in 0in 8pt;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif"><b><u>What it can do:<span></span></u></b></p>
<p class="MsoNormal" style="margin:0in 0in 8pt;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif">In its current state, stack can deploy on Centos 7.6 and
RHEL 8.0 (and nearly on Ubuntu 18.04):<span></span></p>
<p class="gmail-MsoListParagraphCxSpFirst" style="margin:0in 0in 0.0001pt 0.5in;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif"><span><span>-<span style="font:7pt "Times New Roman"">
</span></span></span>An /etc/hosts file<span></span></p>
<p class="gmail-MsoListParagraphCxSpMiddle" style="margin:0in 0in 0.0001pt 0.5in;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif"><span><span>-<span style="font:7pt "Times New Roman"">
</span></span></span>A dhcp configuration with optional shared
networks<span></span></p>
<p class="gmail-MsoListParagraphCxSpMiddle" style="margin:0in 0in 0.0001pt 0.5in;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif"><span><span>-<span style="font:7pt "Times New Roman"">
</span></span></span>A DNS configuration based on Bind
(server/client)<span></span></p>
<p class="gmail-MsoListParagraphCxSpMiddle" style="margin:0in 0in 0.0001pt 0.5in;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif"><span><span>-<span style="font:7pt "Times New Roman"">
</span></span></span>A time configuration based on Chrony
(server/client)<span></span></p>
<p class="gmail-MsoListParagraphCxSpMiddle" style="margin:0in 0in 0.0001pt 0.5in;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif"><span><span>-<span style="font:7pt "Times New Roman"">
</span></span></span>A full PXE stack, based on simplicity and
verbosity<span></span></p>
<p class="gmail-MsoListParagraphCxSpMiddle" style="margin:0in 0in 0.0001pt 1in;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-family:"Courier New""><span>o<span style="font:7pt "Times New Roman"">
</span></span></span>Tftp based on atftp<span></span></p>
<p class="gmail-MsoListParagraphCxSpMiddle" style="margin:0in 0in 0.0001pt 1in;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-family:"Courier New""><span>o<span style="font:7pt "Times New Roman"">
</span></span></span>Apache for all repos/files<span></span></p>
<p class="gmail-MsoListParagraphCxSpMiddle" style="margin:0in 0in 0.0001pt 1in;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-family:"Courier New""><span>o<span style="font:7pt "Times New Roman"">
</span></span></span>iPXE advanced stack with menu to handle all
exotic hardware<span></span></p>
<p class="gmail-MsoListParagraphCxSpMiddle" style="margin:0in 0in 0.0001pt 1.5in;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-family:Wingdings"><span>§<span style="font:7pt "Times New Roman"">
</span></span></span>EFI / Legacy<span></span></p>
<p class="gmail-MsoListParagraphCxSpMiddle" style="margin:0in 0in 0.0001pt 1.5in;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-family:Wingdings"><span>§<span style="font:7pt "Times New Roman"">
</span></span></span>PXE / iPXE native ROM<span></span></p>
<p class="gmail-MsoListParagraphCxSpMiddle" style="margin:0in 0in 0.0001pt 1.5in;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-family:Wingdings"><span>§<span style="font:7pt "Times New Roman"">
</span></span></span>CD or USB boot to PXE when no native PXE
available (or stupidly made BIOS/EFI)<span></span></p>
<p class="gmail-MsoListParagraphCxSpMiddle" style="margin:0in 0in 0.0001pt 0.5in;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif"><span><span>-<span style="font:7pt "Times New Roman"">
</span></span></span>Repositories server/client<span></span></p>
<p class="gmail-MsoListParagraphCxSpMiddle" style="margin:0in 0in 0.0001pt 0.5in;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif"><span><span>-<span style="font:7pt "Times New Roman"">
</span></span></span>NIC configuration (basic for now, based on NMCLI
Ansible module)<span></span></p>
<p class="gmail-MsoListParagraphCxSpMiddle" style="margin:0in 0in 0.0001pt 0.5in;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif"><span><span>-<span style="font:7pt "Times New Roman"">
</span></span></span>Rsyslog (server/client, systemd split files)<span></span></p>
<p class="gmail-MsoListParagraphCxSpMiddle" style="margin:0in 0in 0.0001pt 0.5in;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif"><span><span>-<span style="font:7pt "Times New Roman"">
</span></span></span>Conman<span></span></p>
<p class="gmail-MsoListParagraphCxSpLast" style="margin:0in 0in 8pt 0.5in;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif"><span><span>-<span style="font:7pt "Times New Roman"">
</span></span></span>NFS (server/client)<span></span></p>
<p class="MsoNormal" style="margin:0in 0in 8pt;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif">And as addons:<span></span></p>
<p class="gmail-MsoListParagraphCxSpFirst" style="margin:0in 0in 0.0001pt 0.5in;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif"><span><span>-<span style="font:7pt "Times New Roman"">
</span></span></span>Slurm (basic configuration, master/nodes)<span></span></p>
<p class="gmail-MsoListParagraphCxSpMiddle" style="margin:0in 0in 0.0001pt 0.5in;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif"><span><span>-<span style="font:7pt "Times New Roman"">
</span></span></span>Clustershell groups<span></span></p>
<p class="gmail-MsoListParagraphCxSpMiddle" style="margin:0in 0in 0.0001pt 0.5in;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif"><span><span>-<span style="font:7pt "Times New Roman"">
</span></span></span>Basic OpenLdap with phpldapadmin (currently unsecure)<span></span></p>
<p class="gmail-MsoListParagraphCxSpMiddle" style="margin:0in 0in 0.0001pt 0.5in;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif"><span><span>-<span style="font:7pt "Times New Roman"">
</span></span></span>Very basic user’s management for very small
clusters <u>with a single login node</u> to replace Ldap<span></span></p>
<p class="gmail-MsoListParagraphCxSpLast" style="margin:0in 0in 8pt 0.5in;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif"><span><span>-<span style="font:7pt "Times New Roman"">
</span></span></span>Prometheus configuration (Prometheus,
Alertmanager, NodeExporter, with basic configuration and already few alerts)<span></span></p>
<p class="MsoNormal" style="margin:0in 0in 8pt;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif">Also, stack can deploy Ubuntu 18.04 and OpenSuse Leap 15.1
via PXE, but still not all Ansible roles can be deploy on it after OS
deployment (we are implementing Ubuntu right now for our workstations).<span></span></p>
<p class="MsoNormal" style="margin:0in 0in 8pt;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif"><b>The stack is fully modular</b>. Any new role can be
created, and any new data can be added to the inventory. We worked hard so that
roles are fully independent (you can replace DHCP/DNS/PXE by Cobbler, others
roles do not care. You can replace Slurm by another JobScheduler, same. Etc).<span></span></p>
<p class="MsoNormal" style="margin:0in 0in 8pt;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif"><br></p><p class="MsoNormal" style="margin:0in 0in 8pt;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif">Note to Ansible users: we are using “merge” hash_behaviour.
Using this, stack can cover simple but also very complex clusters (parameters
can be targeted to cover specific host(s) configuration or simply experiment).<span></span></p>
<p class="MsoNormal" style="margin:0in 0in 8pt;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif"><span> </span></p>
<p class="MsoNormal" style="margin:0in 0in 8pt;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif">The stack also as few native mechanisms under the hood ready
and tested as POC, that we are implementing. These features are a need for us,
so we added them:<span></span></p>
<p class="gmail-MsoListParagraphCxSpFirst" style="margin:0in 0in 0.0001pt 0.5in;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif"><span><span>-<span style="font:7pt "Times New Roman"">
</span></span></span>Multi Icebergs (sometime called multi islands in
HPC)<span></span></p>
<p class="gmail-MsoListParagraphCxSpMiddle" style="margin:0in 0in 0.0001pt 1in;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-family:"Courier New""><span>o<span style="font:7pt "Times New Roman"">
</span></span></span>When there is a need to separated parts of the
cluster (our case, to provide dedicated small sub-clusters to some
enterprises), stack is able to split hosts into icebergs, each iceberg being
managed by a group of management nodes and isolated from the others (but can be
reached through interconnect if exist and asked).<span></span></p>
<p class="gmail-MsoListParagraphCxSpMiddle" style="margin:0in 0in 0.0001pt 1in;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-family:"Courier New""><span>o<span style="font:7pt "Times New Roman"">
</span></span></span>Because Ansible rely on groups and ssh, we can
simply achieve that.<span></span></p>
<p class="gmail-MsoListParagraphCxSpMiddle" style="margin:0in 0in 0.0001pt 1in;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-family:"Courier New""><span>o<span style="font:7pt "Times New Roman"">
</span></span></span>This feature would allow the stack to handle an
old cluster aside a new one keeping both in production, or maybe scale to very
large configurations.<span></span></p>
<p class="gmail-MsoListParagraphCxSpMiddle" style="margin:0in 0in 0.0001pt 1in;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif"><span> </span></p>
<p class="gmail-MsoListParagraphCxSpMiddle" style="margin:0in 0in 0.0001pt 0.5in;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif"><span><span>-<span style="font:7pt "Times New Roman"">
</span></span></span>Accelerated modes<span></span></p>
<p class="gmail-MsoListParagraphCxSpMiddle" style="margin:0in 0in 0.0001pt 1in;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-family:"Courier New""><span>o<span style="font:7pt "Times New Roman"">
</span></span></span>Ansible is famous for its simplicity, but also
for being slow, even with Mitogen.<span></span></p>
<p class="gmail-MsoListParagraphCxSpMiddle" style="margin:0in 0in 0.0001pt 1in;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-family:"Courier New""><span>o<span style="font:7pt "Times New Roman"">
</span></span></span>Accelerated mode is a POC we designed to heavily
accelerate critical templates rendering. It is based on some analysis we made
on Ansible. It also reduces memory usage.<span></span></p>
<p class="gmail-MsoListParagraphCxSpLast" style="margin:0in 0in 8pt 1in;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-family:"Courier New""><span>o<span style="font:7pt "Times New Roman"">
</span></span></span>Accelerated mode is just a basic trick we found
in the inventory, no Ansible modifications.<span></span></p>
<p class="MsoNormal" style="margin:0in 0in 8pt;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif">Note that for us, it is mandatory that the stack stay simple.
These features are by default not activated, keeping Ansible inventory very
simple for basic usage.<span></span></p>
<p class="MsoNormal" style="margin:0in 0in 8pt;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif"><span> </span></p>
<p class="MsoNormal" style="margin:0in 0in 8pt;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif">Last point: stack do not aim HPC in particular, it is
generic and can adapt to HPC clusters, but also enterprise or university IT
networks (workstations, laptops, etc.).<span></span></p>
<p class="MsoNormal" style="margin:0in 0in 8pt;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif"><span> </span></p>
<p class="MsoNormal" style="margin:0in 0in 8pt;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif"><b><u>What we are doing with it:<span></span></u></b></p>
<p class="MsoNormal" style="margin:0in 0in 8pt;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif">We are managing our 14 thin (32GbRam/16cores) + 1 fat
(1TbRam/64cores) supermicro servers, and also few workstations.<span></span></p>
<p class="MsoNormal" style="margin:0in 0in 8pt;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif">We are a FabLab made of science and technology fans, so not
a lot of money, which means we are gathering “old” equipment (we are Sandy
Bridge right now) for our cluster. This is why we needed a so flexible stack:
to be able to handle exotic hardware.<span></span></p>
<p class="MsoNormal" style="margin:0in 0in 8pt;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif"><span> </span></p>
<p class="MsoNormal" style="margin:0in 0in 8pt;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif"><b><u>The future of the stack:<span></span></u></b></p>
<p class="MsoNormal" style="margin:0in 0in 8pt;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif">Main future objective would be to create a base for a <b>modular</b>
and <b>simple</b> stack. For simple clusters or just testing/dev on part of
large clusters.<span></span></p>
<p class="MsoNormal" style="margin:0in 0in 8pt;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif">This stack has (we think) a nice PXE mechanism, and all the
basis needed. It could be used as a base for other things. Like a skeleton
missing flesh. Anyone could add new roles/modules/tools to it, keeping in mind
simplicity and roles independency.<span></span></p>
<p class="MsoNormal" style="margin:0in 0in 8pt;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif">We are right now working on Ubuntu 18.04 implementation, and
we hope to release version 1.0 soon.<span></span></p>
<p class="MsoNormal" style="margin:0in 0in 8pt;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif"><span> </span></p>
<p class="MsoNormal" style="margin:0in 0in 8pt;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif"><b><u>Where to find:<span></span></u></b></p>
<p class="MsoNormal" style="margin:0in 0in 8pt;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif">You can find the stack on github here:<span></span></p>
<p class="MsoNormal" style="margin:0in 0in 8pt;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif"><a href="https://github.com/oxedions/bluebanquise" style="color:rgb(5,99,193);text-decoration:underline">https://github.com/oxedions/bluebanquise</a><span></span></p>
<p class="MsoNormal" style="margin:0in 0in 8pt;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif">Documentation is in resources/documentation/_build/html (the
_build/html directory will soon be removed from the repository and hosted
somewhere, as only documentation sources should be here).<span></span></p>
<p class="MsoNormal" style="margin:0in 0in 8pt;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif">The few packages are still not online, but we can provide
them if someone which to test the stack.<span></span></p>
<p class="MsoNormal" style="margin:0in 0in 8pt;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif"><span> </span></p>
<p class="MsoNormal" style="margin:0in 0in 8pt;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif">Stack is young, so we are looking for any feedbacks
(positive and/or negative). Feel free to have a look. For any detail, please do
not hesitate to contact us :-)<span></span><span> </span>
</p><p class="MsoNormal" style="margin:0in 0in 8pt;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif">Thank you for reading this very long and boring mail.<span></span></p>
<p class="MsoNormal" style="margin:0in 0in 8pt;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif"><span> </span></p>
<p class="MsoNormal" style="margin:0in 0in 8pt;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif">With our best regards<span></span></p>
<p class="MsoNormal" style="margin:0in 0in 8pt;line-height:107%;font-size:11pt;font-family:"Calibri",sans-serif">Oxedions and Johnny Keats<span></span></p>
</div>