<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.hoenzb
{mso-style-name:hoenzb;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Do you have a temperature probe? One of those IR thermometers?
<br>
A FLIR One camera for your phone?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Then you can quickly check things like heat sink temperatures and surroundings. Air temp is hard to measure quickly and accurately.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Jim Lux<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">(818)354-2075 (office)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">(818)395-2714 (cell)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Beowulf [mailto:beowulf-bounces@beowulf.org]
<b>On Behalf Of </b>Andrew Latham<br>
<b>Sent:</b> Friday, September 08, 2017 11:56 AM<br>
<b>To:</b> Prentice Bisbal <pbisbal@pppl.gov><br>
<b>Cc:</b> Beowulf List <beowulf@beowulf.org><br>
<b>Subject:</b> Re: [Beowulf] Varying performance across identical cluster nodes.<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">Shooting from hip<o:p></o:p></p>
<div>
<p class="MsoNormal">1. BIOS identical version and settings<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2. Firmware on device (I assume nothing just thinking out loud)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">3. Re-seat fans/replace (oxidized contacts - silly but why not)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">4. Verify the power supplies are identical (various watts etc... maybe swap out and test)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">5. Memory cooling heat-sinks? (have seen identical orders with different memory some with heatsinks)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">6. Thermal paste<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">7. Blank panels on empty drive bays<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">8. Location in rack/room<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">9. Blanking on rack<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Shared to promote thought<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Fri, Sep 8, 2017 at 1:41 PM, Prentice Bisbal <<a href="mailto:pbisbal@pppl.gov" target="_blank">pbisbal@pppl.gov</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<p class="MsoNormal">Beowulfers,<br>
<br>
I need your assistance debugging a problem:<br>
<br>
I have a dozen servers that are all identical hardware: SuperMicro servers with AMD Opteron 6320 processors. Every since we upgraded to CentOS 6, the users have been complaining of wildly inconsistent performance across these 12 nodes. I ran LINPACK on these
nodes, and was able to duplicate the problem, with performance varying from ~14 GFLOPS to 64 GFLOPS.<br>
<br>
I've identified that performance on the slower nodes starts off fine, and then slowly degrades throughout the LINPACK run. For example, on a node with this problem, during first LINPACK test, I can see the performance drop from 115 GFLOPS down to 11.3 GFLOPS.
That constant, downward trend continues throughout the remaining tests. At the start of subsequent tests, performance will jump up to about 9-10 GFLOPS, but then drop to 5-6 GLOPS at the end of the test.<br>
<br>
Because of the nature of this problem, I suspect this might be a thermal issue. My guess is that the processor speed is being throttled to prevent overheating on the "bad" nodes.<br>
<br>
But here's the thing: this wasn't a problem until we upgraded to CentOS 6. Where I work, we use a read-only NFSroot filesystem for our cluster nodes, so all nodes are mounting and using the same exact read-only image of the operating system. This only happens
with these SuperMicro nodes, and only with the CentOS 6 on NFSroot. RHEL5 on NFSroot worked fine, and when I installed CentOS 6 on a local disk, the nodes worked fine.<br>
<br>
Any ideas where to look or what to tweak to fix this? Any idea why this is only occuring with RHEL 6 w/ NFS root OS?<span style="color:#888888"><br>
<br>
<span class="hoenzb">-- </span><br>
<span class="hoenzb">Prentice</span><br>
<br>
<span class="hoenzb">_______________________________________________</span><br>
<span class="hoenzb">Beowulf mailing list, <a href="mailto:Beowulf@beowulf.org" target="_blank">
Beowulf@beowulf.org</a> sponsored by Penguin Computing</span><br>
<span class="hoenzb">To change your subscription (digest mode or unsubscribe) visit
<a href="http://www.beowulf.org/mailman/listinfo/beowulf" target="_blank">http://www.beowulf.org/mailman/listinfo/beowulf</a></span></span><o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal"><br>
<br clear="all">
<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal">-- <o:p></o:p></p>
<div>
<div>
<div>
<div>
<p class="MsoNormal">- Andrew "lathama" Latham <a href="mailto:lathama@gmail.com" target="_blank">
lathama@gmail.com</a> <a href="http://lathama.org" target="_blank">http://lathama.com</a> -<o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>