<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">Beowulf <beowulf-bounces@beowulf.org> on behalf of "beowulf@beowulf.org" <beowulf@beowulf.org><br>
<b>Reply-To: </b>John Hearns <hearnsj@googlemail.com><br>
<b>Date: </b>Thursday, May 3, 2018 at 6:54 AM<br>
<b>To: </b>"beowulf@beowulf.org" <beowulf@beowulf.org><br>
<b>Subject: </b>Re: [Beowulf] Bright Cluster Manager<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><a name="_MailOriginalBody">I agree with Doug. The way forward is a lightweight OS with containers for the applications.<o:p></o:p></a></p>
</div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody">I think we need to learn from the new kids on the block - the webscale generation.<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody">They did not go out and look at how massive supercomputer clusters are put together.<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody">No, they went out and build scale out applications built on public clouds.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody">We see 'applications designed to fail' and 'serverless'<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody">Yes, I KNOW that scale out applications like these are Web type applications, and all application examples you
<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody">see are based on the load balancer/web server/database (or whatever style) paradigm<o:p></o:p></span></p>
<div>
<div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody">The art of this will be deploying the more tightly coupled applications with HPC has,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="mso-bookmark:_MailOriginalBody">which depend upon MPI communications over a reliable fabric, which depend upon GPUs etc.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody">The other hat I will toss into the ring is separating parallel tasks which require computation on several<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody">servers and MPI communication between them versus 'embarrassingly parallel' operations which may run on many, many cores<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="mso-bookmark:_MailOriginalBody">but do not particularly need communication between them.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody">The best successes I have seen on clusters is where the heavy parallel applications get exclusive compute nodes.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody">Cleaner, you get all the memory and storage bandwidth and easy to clean up. Hell, reboot the things after each job. You got an exclusive node.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody">I think many designs of HPC clusters still try to cater for all workloads - Oh Yes, we can run an MPI weather forecasting/ocean simulation<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="mso-bookmark:_MailOriginalBody">But at the same time we have this really fast IO system and we can run your Hadoop jobs.
<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody">I wonder if we are going to see a fork in HPC. With the massively parallel applications being deployed, as Doug says, on specialised
<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody">lightweight OSes which have dedicated high speed, reliable fabrics and with containers.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody">You won't really be able to manage those systems like individual Linux servers. Will you be able to ssh in for instance?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody">ssh assumes there is an ssh daemon running. Does a lightweight OS have ssh? Authentication Services? The kitchen sink?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody">The less parallel applications being run more and more on cloud type installations, either on-premise clouds or public clouds.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody">I confound myself here, as I cant say what the actual difference between those two types of machines is, as you always needs<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody">an interconnect fabric and storage, so why not have the same for both types of tasks.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody">Maybe one further quip to stimulate some conversation. Silicon is cheap. No, really it is.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody">Your friendly Intel salesman may wince when you say that. After all those lovely Xeon CPUs cost north of 1000 dollars each.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="mso-bookmark:_MailOriginalBody">But again I throw in some talking points:<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="mso-bookmark:_MailOriginalBody">power and cooling costs the same if not more than your purchase cost over several years<o:p></o:p></span></p>
</div>
<div>
<div style="mso-element:para-border-div;border:none;border-bottom:solid windowtext 1.0pt;padding:0in 0in 1.0pt 0in">
<p class="MsoNormal" style="margin-bottom:12.0pt;border:none;padding:0in"><span style="mso-bookmark:_MailOriginalBody">are we exploiting all the capabilities of those Xeon CPUs<br>
<br>
<o:p></o:p></span></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="mso-bookmark:_MailOriginalBody">And another aspect of this - I’ve been doing stuff with “loose clusters” of low capability processors (Arduino, Rpi, Beagle) doing distributed sensing kinds of tasks
– leaving aside the Arduino (no OS) – the other two wind up with some flavor of Debian but often with lots of stuff you don’t need (i.e. Apache). Once you’ve fiddled with one node to get the configuration right, you want to replicate it across a bunch of nodes
– right now that means sneakernet of SD cards - although in theory, one should be able to push an image out to the local file system (typically 4GB eMMC in the case of beagles), and tell it to write that to the “boot area” – but I’ve not tried it.
<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="mso-bookmark:_MailOriginalBody">While I’d never claim my pack of beagles is HPC, it does share some aspects – there’s parallel work going on, the nodes need to be aware of each other and synchronize
their behavior (that is, it’s not an embarrassingly parallel task that’s farmed out from a queue), and most importantly, the management has to be scalable. While I might have 4 beagles on the bench right now – the idea is to scale the approach to hundreds.
Typing “sudo apt-get install tbd-package” on 4 nodes sequentially might be ok (although pdsh and csshx help a lot) it’s not viable for 100 nodes.<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="mso-bookmark:_MailOriginalBody">The other aspect of my application that’s interesting, and applicable to exascale kinds of problems, is tolerance to failures – if I have a low data rate link among
nodes (with not necessarily all to all connectivity), one can certainly distribute a new OS image (or container) with time. There’s some ways to deal with errors in the transfers (other than just retransmit all – which doesn’t work if the error rate is high
enough that you can guarantee at least one error will occur in a long transfer). But how do you *<b>manage</b>* a cluster with hundreds or thousands of nodes where some fail randomly, reset randomly, etc.<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="mso-bookmark:_MailOriginalBody">All of a sudden simple “send the same command to all nodes” just doesn’t work. And that’s what will inevitably be the case as we scale up in the HPC world – there
will always be dead or malfunctioning nodes.<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="mso-bookmark:_MailOriginalBody"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="mso-bookmark:_MailOriginalBody"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="mso-bookmark:_MailOriginalBody"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="mso-bookmark:_MailOriginalBody"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="mso-bookmark:_MailOriginalBody"><br>
<br>
<br>
<br>
<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="mso-bookmark:_MailOriginalBody"><br>
<br>
<br>
<br>
<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="mso-bookmark:_MailOriginalBody"><br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="mso-bookmark:_MailOriginalBody"><br>
<br>
<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="mso-bookmark:_MailOriginalBody"><br>
<br>
<br>
<br>
<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="mso-bookmark:_MailOriginalBody"><br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="mso-bookmark:_MailOriginalBody"><br>
<br>
<br>
<br>
<br>
<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="mso-bookmark:_MailOriginalBody"><o:p> </o:p></span></p>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="mso-bookmark:_MailOriginalBody"><br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
<div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody">On 3 May 2018 at 15:04, Douglas Eadline <</span><a href="mailto:deadline@eadline.org" target="_blank"><span style="mso-bookmark:_MailOriginalBody">deadline@eadline.org</span><span style="mso-bookmark:_MailOriginalBody"></span></a><span style="mso-bookmark:_MailOriginalBody">>
wrote:<o:p></o:p></span></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody"><br>
<br>
Here is where I see it going<br>
<br>
1. Computer nodes with a base minimal generic Linux OS<br>
(with PR_SET_NO_NEW_PRIVS in kernel, added in 3.5)<br>
<br>
2. A Scheduler (that supports containers)<br>
<br>
3. Containers (Singularity mostly)<br>
<br>
All "provisioning" is moved to the container. There will be edge cases of<br>
course, but applications will be pulled down from<br>
a container repos and "just run"<br>
<br>
--<br>
Doug<br>
<br>
<br>
> I never used Bright. Touched it and talked to a salesperson at a<br>
> conference but I wasn't impressed.<br>
><br>
> Unpopular opinion: I don't see a point in using "cluster managers"<br>
> unless you have a very tiny cluster and zero Linux experience. These<br>
> are just Linux boxes with a couple applications (e.g. Slurm) running on<br>
> them. Nothing special. xcat/Warewulf/Scyld/Rocks just get in the way<br>
> more than they help IMO. They are mostly crappy wrappers around free<br>
> software (e.g. ISC's dhcpd) anyway. When they aren't it's proprietary<br>
> trash.<br>
><br>
> I install CentOS nodes and use<br>
> Salt/Chef/Puppet/Ansible/WhoCares/Whatever to plop down my configs and<br>
> software. This also means I'm not suck with "node images" and can<br>
> instead build everything as plain old text files (read: write SaltStack<br>
> states), update them at will, and push changes any time. My "base<br>
> image" is CentOS and I need no "baby's first cluster" HPC software to<br>
> install/PXEboot it. YMMV<br>
><br>
><br>
> Jeff White<br>
><br>
> On 05/01/2018 01:57 PM, Robert Taylor wrote:<br>
>> Hi Beowulfers.<br>
>> Does anyone have any experience with Bright Cluster Manager?<br>
>> My boss has been looking into it, so I wanted to tap into the<br>
>> collective HPC consciousness and see<br>
>> what people think about it.<br>
>> It appears to do node management, monitoring, and provisioning, so we<br>
>> would still need a job scheduler like lsf, slurm,etc, as well. Is that<br>
>> correct?<br>
>><br>
>> If you have experience with Bright, let me know. Feel free to contact<br>
>> me off list or on.<br>
>><br>
>><br>
>><br>
>> _______________________________________________<br>
>> Beowulf mailing list, </span><a href="mailto:Beowulf@beowulf.org"><span style="mso-bookmark:_MailOriginalBody">Beowulf@beowulf.org</span><span style="mso-bookmark:_MailOriginalBody"></span></a><span style="mso-bookmark:_MailOriginalBody"> sponsored by Penguin
Computing<br>
>> To change your subscription (digest mode or unsubscribe) visit<br>
>> </span><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__www.beowulf.org_mailman_listinfo_beowulf&d=DwIGaQ&c=C3yme8gMkxg_ihJNXS06ZyWk4EJm8LdrrvxQb-Je7sw&r=DhM5WMgdrH-xWhI5BzkRTzoTvz8C-BRZ05t9kW9SXZk&m=2km_EqLvNf2v9rNf8LphAYkJ-Sc_azfEyHqyDIzpLOc&s=kq0wdhy80VqcBCwcQAAQa0RbsgWIekhd0qU0zC81g1Q&e=" target="_blank"><span style="mso-bookmark:_MailOriginalBody">https://urldefense.proofpoint.com/v2/url?u=http-3A__www.beowulf.org_mailman_listinfo_beowulf&d=DwIGaQ&c=C3yme8gMkxg_ihJNXS06ZyWk4EJm8LdrrvxQb-Je7sw&r=DhM5WMgdrH-xWhI5BzkRTzoTvz8C-BRZ05t9kW9SXZk&m=2km_EqLvNf2v9rNf8LphAYkJ-Sc_azfEyHqyDIzpLOc&s=kq0wdhy80VqcBCwcQAAQa0RbsgWIekhd0qU0zC81g1Q&e=</span><span style="mso-bookmark:_MailOriginalBody"></span></a><span style="mso-bookmark:_MailOriginalBody"><br>
><br>
><br>
> --<br>
> MailScanner: Clean<br>
><br>
> _______________________________________________<br>
> Beowulf mailing list, </span><a href="mailto:Beowulf@beowulf.org"><span style="mso-bookmark:_MailOriginalBody">Beowulf@beowulf.org</span><span style="mso-bookmark:_MailOriginalBody"></span></a><span style="mso-bookmark:_MailOriginalBody"> sponsored by Penguin
Computing<br>
> To change your subscription (digest mode or unsubscribe) visit<br>
> </span><a href="http://www.beowulf.org/mailman/listinfo/beowulf" target="_blank"><span style="mso-bookmark:_MailOriginalBody">http://www.beowulf.org/mailman/listinfo/beowulf</span><span style="mso-bookmark:_MailOriginalBody"></span></a><span style="mso-bookmark:_MailOriginalBody"><br>
><br>
<br>
<br>
-- <br>
Doug<br>
<br>
-- <br>
MailScanner: Clean<br>
<br>
_______________________________________________<br>
Beowulf mailing list, </span><a href="mailto:Beowulf@beowulf.org"><span style="mso-bookmark:_MailOriginalBody">Beowulf@beowulf.org</span><span style="mso-bookmark:_MailOriginalBody"></span></a><span style="mso-bookmark:_MailOriginalBody"> sponsored by Penguin
Computing<br>
To change your subscription (digest mode or unsubscribe) visit </span><a href="http://www.beowulf.org/mailman/listinfo/beowulf" target="_blank"><span style="mso-bookmark:_MailOriginalBody">http://www.beowulf.org/mailman/listinfo/beowulf</span><span style="mso-bookmark:_MailOriginalBody"></span></a><span style="mso-bookmark:_MailOriginalBody"><o:p></o:p></span></p>
</blockquote>
</div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody"><o:p> </o:p></span></p>
</div>
</div>
</body>
</html>