<html><body><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; color: #000000"><div>Thanks for the kind words and comments! Good catch with HPL. It's definitely part of the test regime. I typically run 3 tests for consistency: </div><div><br data-mce-bogus="1"></div><div>- Separate instance of STREAM2 on each node</div><div>- Separate instance of HPL on each node</div><div>- Simple MPI latency / bandwidth test called mpisweep that tests every link (I'll put this up on github later as well)</div><div><br data-mce-bogus="1"></div><div>I now made the changes to the document. </div><div><br data-mce-bogus="1"></div><div>After this set of tests I'm not completely sure if NPB will add any further information. Those 3 benchmarks combined with the other checks should pretty much expose all the possible issues. However, I could be missing something again :)</div><div><br></div><div>Best regards,</div><div>O-P</div><div data-marker="__SIG_PRE__">-- <br>Olli-Pekka Lehto<br>Development Manager<br>Computing Platforms<br>CSC - IT Center for Science Ltd.<br>E-Mail: olli-pekka.lehto@csc.fi<br>Tel: +358 50 381 8604<br>skype: oplehto // twitter: ople</div><br><hr id="zwchr" data-marker="__DIVIDER__"><div data-marker="__HEADERS__"><blockquote style="border-left: 2px solid #1010FF; margin-left: 5px; padding-left: 5px; color: #000; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica,Arial,sans-serif; font-size: 12pt;" data-mce-style="border-left: 2px solid #1010FF; margin-left: 5px; padding-left: 5px; color: #000; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica,Arial,sans-serif; font-size: 12pt;"><b>From: </b>"Jeffrey Layton" <laytonjb@gmail.com><br><b>To: </b>"Olli-Pekka Lehto" <olli-pekka.lehto@csc.fi><br><b>Cc: </b>beowulf@beowulf.org<br><b>Sent: </b>Tuesday, 22 March, 2016 16:45:20<br><b>Subject: </b>Re: [Beowulf] Cluster consistency checks<br></blockquote></div><div data-marker="__QUOTED_TEXT__"><blockquote style="border-left: 2px solid #1010FF; margin-left: 5px; padding-left: 5px; color: #000; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica,Arial,sans-serif; font-size: 12pt;" data-mce-style="border-left: 2px solid #1010FF; margin-left: 5px; padding-left: 5px; color: #000; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica,Arial,sans-serif; font-size: 12pt;"><div dir="ltr"><div><div><div><div><div>Olli-Pekka,<br><br></div>Very nice - I'm glad you put a list down. Many of the things that I do are based on experience.<br><br></div>A long time ago, in one of my previous jobs, we used to run NAS Parallel Benchmark (NPB) on single nodes to get a baseline of performance. We would look for outliers and triage and debug them based on these results. We're not running the test for performance but to make sure the cluster was a homogeneous as possible. Have you done this before?<br><br></div>I've also seen people run HPL on single nodes and look for outliers. After triaging these, HPL is run on smaller groups of nodes within a single switch, look for outliers and triage them. This continues up to the entire system. The point is not to get a great HPL number to submit to the Top500 but rather to find potential network issues, particularly network links.<br><br></div>Thanks for the good work! <br><br></div>Jeff<br><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Mar 22, 2016 at 11:32 AM, Olli-Pekka Lehto <span dir="ltr"><<a href="mailto:olli-pekka.lehto@csc.fi" target="_blank">olli-pekka.lehto@csc.fi</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin: 0 0 0 .8ex; border-left: 1px #ccc solid; padding-left: 1ex;" data-mce-style="margin: 0 0 0 .8ex; border-left: 1px #ccc solid; padding-left: 1ex;"><div><div style="font-family: arial,helvetica,sans-serif; font-size: 12pt; color: #000000;" data-mce-style="font-family: arial,helvetica,sans-serif; font-size: 12pt; color: #000000;"><div>Hi,</div><br><div>I finally got around to writing down my cluster-consistency checklist that I've been planning for a long time: </div><br><div><a href="https://github.com/oplehto/cluster-checks/" target="_blank">https://github.com/oplehto/cluster-checks/</a> <br></div><div>The goal is to try to make the baseline installation of a cluster as consistent as possible and make vendors work for their money. :) Of course hopefully publishing this will help vendors capture some of the issues that slip through the cracks even before clusters are handed over. It's also a good idea to run these types of checks during the lifetime of the system as there's always some consistency creep as hardware gets replaced. </div><br><div>If someone is interested in contributing, pull requests or comments on the list are welcome. I'm sure that there's something missing as well. Right now it's just a text-file but making some nicer scripts and postprocessing for the output might happen as well at some point. All the examples are very HP oriented as well at this point.</div><br><div>Best regards,</div><div>Olli-Pekka</div><span class="HOEnZb"><span color="#888888" data-mce-style="color: #888888;" style="color: #888888;"><div>-- <br>Olli-Pekka Lehto<br>Development Manager<br>Computing Platforms<br>CSC - IT Center for Science Ltd.<br>E-Mail: <a href="mailto:olli-pekka.lehto@csc.fi" target="_blank">olli-pekka.lehto@csc.fi</a><br>Tel: <a href="tel:%2B358%2050%20381%208604" target="_blank">+358 50 381 8604</a><br>skype: oplehto // twitter: ople</div></span></span></div></div><br>_______________________________________________<br>
Beowulf mailing list, <a href="mailto:Beowulf@beowulf.org" target="_blank">Beowulf@beowulf.org</a> sponsored by Penguin Computing<br>
To change your subscription (digest mode or unsubscribe) visit <a href="http://www.beowulf.org/mailman/listinfo/beowulf" rel="noreferrer" target="_blank">http://www.beowulf.org/mailman/listinfo/beowulf</a><br>
<br></blockquote></div></div><br></blockquote></div></div></body></html>