<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Isn’t this what the whole Byzantine Generals thing is about: you have unreliable processes, and need a cumulatively reliable result.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Jim Lux<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">(818)354-2075 (office)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">(818)395-2714 (cell)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Justin Y. Shi [mailto:shi@temple.edu]
<br>
<b>Sent:</b> Thursday, March 10, 2016 5:54 AM<br>
<b>To:</b> Lux, Jim (337C) <james.p.lux@jpl.nasa.gov><br>
<b>Cc:</b> beowulf@beowulf.org<br>
<b>Subject:</b> Re: [Beowulf] MPI, fault handling, etc.<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">I will supper C's "hater" listing effort just to keep a spot light on the important subject.<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">The question is not MPI is efficient or not. Fundamentally, all electronics will fail in unexpected ways. Bare metal computing was important decades ago but detrimental to large scale computing. It is simply flawed for extreme scale computing.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">The Alan Fekete, Nancy Lynch, John Spinneli's impossible proof is the fundamental "line in the sand" that cannot be crossed.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">The corollary of that proof is that it is impossible to detect failure reliably either. Therefore, those efforts for for runtime detection/repair/reschedule are also flawed for extreme scale computing.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Justin <o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Thu, Mar 10, 2016 at 8:44 AM, Lux, Jim (337C) <<a href="mailto:james.p.lux@jpl.nasa.gov" target="_blank">james.p.lux@jpl.nasa.gov</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black">This is interesting stuff.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black">Think back a few years when we were talking about checkpoint/restart issues: as the scale of your problem gets bigger, the time to checkpoint becomes bigger than
the time actually doing useful work.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black">And, of course, the reason we do checkpoint/restart is because it’s bare-metal and easy. Just like simple message passing is “close to the metal” and “straightforward”.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black">Similarly, there’s “fine grained” error detection and correction: ECC codes in memory; redundant comm links or retries. Each of them imposes some speed/performance
penalty (it takes some non-zero time to compute the syndrome bits in a ECC, and some non-zero time to fix the errored bits… in a lot of systems these days, that might be buried in a pipeline, but the delay is there, and affects performance)<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black">I think of ECC as a sort of diffuse fault management: it’s pervasive, uniform, and the performance penalty is applied evenly through the system. Redundant (in
the TMR sense) links are the same way.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black">Retries are a bit different. The “detecting” a fault is diffuse and pervasive (e.g. CRC checks occur on each message), but the correction of the fault is discrete
and consumes resources at that time. In a system with tight time coupling (a pipelined systolic array would be the sort of worst case), many nodes have to wait to fix the one that failed.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black">A lot depends on the application: tighter time coupling is worse than embarrassingly parallel (which is what a lot of the “big data” stuff is: fundamentally EP,
scatter the requests, run in parallel, gather the results).<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black">The challenge is doing stuff in between: You may have a flock with excess capacity (just as ECC memory might have 1.5N physical storage bits to be used to store
N bits), but how do you <b>automatically</b> distribute the resources to be failure tolerant. The original post in the thread points out that MPI is not a particularly facile tool for doing this. But I’m not sure that there is a tool, and I’m not sure that
MPI is the root of the lack of tools. I think it’s that moving from close to the metal is a “hard problem” to do in a generic way. (The issues about 32 bit counts are valid, though)<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">James Lux, P.E.</span><span style="font-size:11.0pt;color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Task Manager, DHFR Space Testbed</span><span style="font-size:11.0pt;color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Jet Propulsion Laboratory</span><span style="font-size:11.0pt;color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">4800 Oak Grove Drive, MS 161-213</span><span style="font-size:11.0pt;color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Pasadena CA 91109</span><span style="font-size:11.0pt;color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"><a href="tel:%2B1%28818%29354-2075" target="_blank">+1(818)354-2075</a></span><span style="font-size:11.0pt;color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"><a href="tel:%2B1%28818%29395-2714" target="_blank">+1(818)395-2714</a> (cell)</span><span style="font-size:11.0pt;color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black"> <o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
_______________________________________________<br>
Beowulf mailing list, <a href="mailto:Beowulf@beowulf.org">Beowulf@beowulf.org</a> sponsored by Penguin Computing<br>
To change your subscription (digest mode or unsubscribe) visit <a href="http://www.beowulf.org/mailman/listinfo/beowulf" target="_blank">
http://www.beowulf.org/mailman/listinfo/beowulf</a><o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</body>
</html>