<html><body><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; color: #000000"><div>Thanks for all the feedback. I'll incorporate the other suggestion over the long weekend as well. </div><div><br data-mce-bogus="1"></div><div>My method for looking at outliers have been mostly "pipe through sort and look for the obvious problem nodes". The outliers tend to be pretty clear as they deviate 20-50% from the worst and typically there's just a handful. Based on experience I'd say ~0.1-1% of the node population on a fresh system. </div><div><br data-mce-bogus="1"></div><div>I don't have them at hand but I could try to dig out some of the reference runs from our systems and it would be interesting if some others have similar results as well. </div><div><br></div><div>Best regards,</div><div>O-P</div><div data-marker="__SIG_PRE__">-- <br>Olli-Pekka Lehto<br>Development Manager<br>Computing Platforms<br>CSC - IT Center for Science Ltd.<br>E-Mail: olli-pekka.lehto@csc.fi<br>Tel: +358 50 381 8604<br>skype: oplehto // twitter: ople</div><br><hr id="zwchr" data-marker="__DIVIDER__"><div data-marker="__HEADERS__"><blockquote style="border-left: 2px solid #1010FF; margin-left: 5px; padding-left: 5px; color: #000; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica,Arial,sans-serif; font-size: 12pt;" data-mce-style="border-left: 2px solid #1010FF; margin-left: 5px; padding-left: 5px; color: #000; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica,Arial,sans-serif; font-size: 12pt;"><b>From: </b>"Jeffrey Layton" <laytonjb@gmail.com><br><b>To: </b>"Olli-Pekka Lehto" <olli-pekka.lehto@csc.fi>, beowulf@beowulf.org<br><b>Sent: </b>Friday, 25 March, 2016 20:39:12<br><b>Subject: </b>Re: [Beowulf] Cluster consistency checks<br></blockquote></div><div data-marker="__QUOTED_TEXT__"><blockquote style="border-left: 2px solid #1010FF; margin-left: 5px; padding-left: 5px; color: #000; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica,Arial,sans-serif; font-size: 12pt;" data-mce-style="border-left: 2px solid #1010FF; margin-left: 5px; padding-left: 5px; color: #000; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica,Arial,sans-serif; font-size: 12pt;"><div dir="ltr"><div><div><div><div><div>Olli-Pekka, et al,<br><br></div>I took a look at your updated website - it looks very good. One thing I wanted to ask, and this question is probably one for the entire list, when you run a test across all of the nodes in the cluster, what process do you use to determine if nodes are "outliers" and need attention?<br></div><br>For example, one test you mention is to run stream and look at the TRIAD results for all of the nodes. If you run it across an entire cluster you end up with a collection of results. What do you do with those results? Do you look for nodes that are a certain percentage outside of the mean? Or do you look for nodes that are outside one standard deviation from the mean?<br><br></div>Thanks!<br><br></div>Jeff<br><br></div>P.S. I have my own ideas but I'm really curious what other people do.<br><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Mar 23, 2016 at 9:55 AM, Douglas Eadline <span dir="ltr"><<a href="mailto:deadline@eadline.org" target="_blank">deadline@eadline.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin: 0 0 0 .8ex; border-left: 1px #ccc solid; padding-left: 1ex;" data-mce-style="margin: 0 0 0 .8ex; border-left: 1px #ccc solid; padding-left: 1ex;"><span class=""><br>
> Thanks for the kind words and comments! Good catch with HPL. It's<br>
> definitely part of the test regime. I typically run 3 tests for<br>
> consistency:<br>
><br>
> - Separate instance of STREAM2 on each node<br>
> - Separate instance of HPL on each node<br>
> - Simple MPI latency / bandwidth test called mpisweep that tests every<br>
> link (I'll put this up on github later as well)<br>
><br>
> I now made the changes to the document.<br>
><br>
> After this set of tests I'm not completely sure if NPB will add any<br>
> further information. Those 3 benchmarks combined with the other checks<br>
> should pretty much expose all the possible issues. However, I could be<br>
> missing something again :)<br>
<br>
</span>NAS will verify the results. On several occasion I have<br>
found NAS gave good numbers but the results did not verify.<br>
This allowed me to look at lower level issues until I found<br>
the problem (in one case a cable IIRC)<br>
<br>
BTW, I run NAS all the time to test performance and make sure<br>
things are running properly on my deskside clusters. I have done<br>
it so often I can tell which test is running by watching wwtop<br>
(Warewulf cluster based top that shows loads, net, memory but no<br>
application names).<br>
<br>
--<br>
Doug<br>
<span class=""><br>
><br>
> Best regards,<br>
> O-P<br>
> --<br>
> Olli-Pekka Lehto<br>
> Development Manager<br>
> Computing Platforms<br>
> CSC - IT Center for Science Ltd.<br>
> E-Mail: <a href="mailto:olli-pekka.lehto@csc.fi" target="_blank">olli-pekka.lehto@csc.fi</a><br>
> Tel: <a href="tel:%2B358%2050%20381%208604" target="_blank">+358 50 381 8604</a><br>
> skype: oplehto // twitter: ople<br>
><br>
</span><div><div class="h5">>> From: "Jeffrey Layton" <<a href="mailto:laytonjb@gmail.com" target="_blank">laytonjb@gmail.com</a>><br>
>> To: "Olli-Pekka Lehto" <<a href="mailto:olli-pekka.lehto@csc.fi" target="_blank">olli-pekka.lehto@csc.fi</a>><br>
>> Cc: <a href="mailto:beowulf@beowulf.org" target="_blank">beowulf@beowulf.org</a><br>
>> Sent: Tuesday, 22 March, 2016 16:45:20<br>
>> Subject: Re: [Beowulf] Cluster consistency checks<br>
><br>
>> Olli-Pekka,<br>
><br>
>> Very nice - I'm glad you put a list down. Many of the things that I do<br>
>> are based<br>
>> on experience.<br>
><br>
>> A long time ago, in one of my previous jobs, we used to run NAS Parallel<br>
>> Benchmark (NPB) on single nodes to get a baseline of performance. We<br>
>> would look<br>
>> for outliers and triage and debug them based on these results. We're not<br>
>> running the test for performance but to make sure the cluster was a<br>
>> homogeneous<br>
>> as possible. Have you done this before?<br>
><br>
>> I've also seen people run HPL on single nodes and look for outliers.<br>
>> After<br>
>> triaging these, HPL is run on smaller groups of nodes within a single<br>
>> switch,<br>
>> look for outliers and triage them. This continues up to the entire<br>
>> system. The<br>
>> point is not to get a great HPL number to submit to the Top500 but<br>
>> rather to<br>
>> find potential network issues, particularly network links.<br>
><br>
>> Thanks for the good work!<br>
><br>
>> Jeff<br>
><br>
>> On Tue, Mar 22, 2016 at 11:32 AM, Olli-Pekka Lehto <<br>
>> <a href="mailto:olli-pekka.lehto@csc.fi" target="_blank">olli-pekka.lehto@csc.fi</a> ><br>
>> wrote:<br>
><br>
>>> Hi,<br>
><br>
>>> I finally got around to writing down my cluster-consistency checklist<br>
>>> that I've<br>
>>> been planning for a long time:<br>
><br>
>>> <a href="https://github.com/oplehto/cluster-checks/" rel="noreferrer" target="_blank">https://github.com/oplehto/cluster-checks/</a><br>
>>> The goal is to try to make the baseline installation of a cluster as<br>
>>> consistent<br>
>>> as possible and make vendors work for their money. :) Of course<br>
>>> hopefully<br>
>>> publishing this will help vendors capture some of the issues that slip<br>
>>> through<br>
>>> the cracks even before clusters are handed over. It's also a good idea<br>
>>> to run<br>
>>> these types of checks during the lifetime of the system as there's<br>
>>> always some<br>
>>> consistency creep as hardware gets replaced.<br>
><br>
>>> If someone is interested in contributing, pull requests or comments on<br>
>>> the list<br>
>>> are welcome. I'm sure that there's something missing as well. Right now<br>
>>> it's<br>
>>> just a text-file but making some nicer scripts and postprocessing for<br>
>>> the<br>
>>> output might happen as well at some point. All the examples are very HP<br>
>>> oriented as well at this point.<br>
><br>
>>> Best regards,<br>
>>> Olli-Pekka<br>
>>> --<br>
>>> Olli-Pekka Lehto<br>
>>> Development Manager<br>
>>> Computing Platforms<br>
>>> CSC - IT Center for Science Ltd.<br>
>>> E-Mail: <a href="mailto:olli-pekka.lehto@csc.fi" target="_blank">olli-pekka.lehto@csc.fi</a><br>
>>> Tel: <a href="tel:%2B358%2050%20381%208604" target="_blank">+358 50 381 8604</a><br>
>>> skype: oplehto // twitter: ople<br>
><br>
>>> _______________________________________________<br>
>>> Beowulf mailing list, <a href="mailto:Beowulf@beowulf.org" target="_blank">Beowulf@beowulf.org</a> sponsored by Penguin<br>
>>> Computing<br>
>>> To change your subscription (digest mode or unsubscribe) visit<br>
>>> <a href="http://www.beowulf.org/mailman/listinfo/beowulf" rel="noreferrer" target="_blank">http://www.beowulf.org/mailman/listinfo/beowulf</a><br>
><br>
</div></div>> --<br>
> Mailscanner: Clean<br>
<span class="">><br>
> _______________________________________________<br>
> Beowulf mailing list, <a href="mailto:Beowulf@beowulf.org" target="_blank">Beowulf@beowulf.org</a> sponsored by Penguin Computing<br>
> To change your subscription (digest mode or unsubscribe) visit<br>
> <a href="http://www.beowulf.org/mailman/listinfo/beowulf" rel="noreferrer" target="_blank">http://www.beowulf.org/mailman/listinfo/beowulf</a><br>
><br>
<br>
<br>
</span>--<br>
Doug<br>
<span class="HOEnZb"><span color="#888888" data-mce-style="color: #888888;" style="color: #888888;"><br>
--<br>
Mailscanner: Clean<br>
<br>
</span></span></blockquote></div></div><br></blockquote></div></div></body></html>