<div dir="ltr">Problem solved.<div>I have changed the QPI Snoop Mode on these servers from </div><div>ClusterOnDIe Enabled to Disabled and they display what I take to be correct behaviour - ie </div><div><br></div><div><div>[root@comp006 ~]# numactl --hardware</div><div>available: 2 nodes (0-1)</div><div>node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11</div><div>node 0 size: 32673 MB</div><div>node 0 free: 31541 MB</div><div>node 1 cpus: 12 13 14 15 16 17 18 19 20 21 22 23</div><div>node 1 size: 32768 MB</div><div>node 1 free: 31860 MB</div><div>node distances:</div><div>node 0 1</div><div> 0: 10 21</div><div> 1: 21 10</div></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On 16 December 2016 at 14:51, John Hearns <span dir="ltr"><<a href="mailto:hearnsj@googlemail.com" target="_blank">hearnsj@googlemail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">hwloc is finding weirdness also.<div>I am going to find I have done something stupid, right?</div><div><br></div><div><br><div><br></div><div><div>[johnh@comp006 ~]$ lstopo</div><div>******************************<wbr>******************************<wbr>****************</div><div>* hwloc 1.11.3 has encountered what looks like an error from the operating system.</div><div>*</div><div>* Package (P#1 cpuset 0x00fff000) intersects with NUMANode (P#0 cpuset 0x00fc0fff) without inclusion!</div><div>* Error occurred in topology.c line 1046</div><div>*</div><div>* The following FAQ entry in the hwloc documentation may help:</div><div>* What should I do when hwloc reports "operating system" warnings?</div><div>* Otherwise please report this error message to the hwloc user's mailing list,</div><div>* along with the output+tarball generated by the hwloc-gather-topology script.</div><div>******************************<wbr>******************************<wbr>****************</div><div>Machine (64GB total)</div><div> NUMANode L#0 (P#0 32GB)</div><div> Package L#0</div><div> L3 L#0 (15MB)</div><div> L2 L#0 (256KB) + L1d L#0 (32KB) + L1i L#0 (32KB) + Core L#0 + PU L#0 (P#0)</div><div> L2 L#1 (256KB) + L1d L#1 (32KB) + L1i L#1 (32KB) + Core L#1 + PU L#1 (P#1)</div><div> L2 L#2 (256KB) + L1d L#2 (32KB) + L1i L#2 (32KB) + Core L#2 + PU L#2 (P#2)</div><div> L2 L#3 (256KB) + L1d L#3 (32KB) + L1i L#3 (32KB) + Core L#3 + PU L#3 (P#3)</div><div> L2 L#4 (256KB) + L1d L#4 (32KB) + L1i L#4 (32KB) + Core L#4 + PU L#4 (P#4)</div><div> L2 L#5 (256KB) + L1d L#5 (32KB) + L1i L#5 (32KB) + Core L#5 + PU L#5 (P#5)</div><div> L3 L#1 (15MB)</div><div> L2 L#6 (256KB) + L1d L#6 (32KB) + L1i L#6 (32KB) + Core L#6 + PU L#6 (P#6)</div><div> L2 L#7 (256KB) + L1d L#7 (32KB) + L1i L#7 (32KB) + Core L#7 + PU L#7 (P#7)</div><div> L2 L#8 (256KB) + L1d L#8 (32KB) + L1i L#8 (32KB) + Core L#8 + PU L#8 (P#8)</div><div> L2 L#9 (256KB) + L1d L#9 (32KB) + L1i L#9 (32KB) + Core L#9 + PU L#9 (P#9)</div><div> L2 L#10 (256KB) + L1d L#10 (32KB) + L1i L#10 (32KB) + Core L#10 + PU L#10 (P#10)</div><div> L2 L#11 (256KB) + L1d L#11 (32KB) + L1i L#11 (32KB) + Core L#11 + PU L#11 (P#11)</div><div> L3 L#2 (15MB)</div><div> L2 L#12 (256KB) + L1d L#12 (32KB) + L1i L#12 (32KB) + Core L#12 + PU L#12 (P#18)</div><div> L2 L#13 (256KB) + L1d L#13 (32KB) + L1i L#13 (32KB) + Core L#13 + PU L#13 (P#19)</div><div> L2 L#14 (256KB) + L1d L#14 (32KB) + L1i L#14 (32KB) + Core L#14 + PU L#14 (P#20)</div><div> L2 L#15 (256KB) + L1d L#15 (32KB) + L1i L#15 (32KB) + Core L#15 + PU L#15 (P#21)</div><div> L2 L#16 (256KB) + L1d L#16 (32KB) + L1i L#16 (32KB) + Core L#16 + PU L#16 (P#22)</div><div> L2 L#17 (256KB) + L1d L#17 (32KB) + L1i L#17 (32KB) + Core L#17 + PU L#17 (P#23)</div><div> HostBridge L#0</div><div> PCIBridge</div><div> PCI 8086:24f0</div><div> PCI 8086:8d62</div><div> PCIBridge</div><div> PCI 102b:0522</div><div> GPU L#0 "card0"</div><div> GPU L#1 "controlD64"</div><div> PCIBridge</div><div> PCI 8086:1521</div><div> Net L#2 "enp7s0f0"</div><div> PCI 8086:1521</div><div> Net L#3 "enp7s0f1"</div><div> PCI 8086:8d02</div><div> Block(Disk) L#4 "sda"</div><div> NUMANode L#1 (P#2 32GB) + L3 L#3 (15MB)</div><div> L2 L#18 (256KB) + L1d L#18 (32KB) + L1i L#18 (32KB) + Core L#18 + PU L#18 (P#12)</div><div> L2 L#19 (256KB) + L1d L#19 (32KB) + L1i L#19 (32KB) + Core L#19 + PU L#19 (P#13)</div><div> L2 L#20 (256KB) + L1d L#20 (32KB) + L1i L#20 (32KB) + Core L#20 + PU L#20 (P#14)</div><div> L2 L#21 (256KB) + L1d L#21 (32KB) + L1i L#21 (32KB) + Core L#21 + PU L#21 (P#15)</div><div> L2 L#22 (256KB) + L1d L#22 (32KB) + L1i L#22 (32KB) + Core L#22 + PU L#22 (P#16)</div><div> L2 L#23 (256KB) + L1d L#23 (32KB) + L1i L#23 (32KB) + Core L#23 + PU L#23 (P#17)</div></div><div><br></div></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On 16 December 2016 at 14:36, John Hearns <span dir="ltr"><<a href="mailto:hearnsj@googlemail.com" target="_blank">hearnsj@googlemail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">This is in the context of Ominpath cards and the hfi1 driver.<div>In the file pio.c there is a check on the NUMA zones being online</div><div><br></div><div><p class="MsoNormal"><span style="color:rgb(31,73,125)"> </span></p>
<p class="MsoNormal" style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><b><span style="font-size:11.5pt;font-family:"courier new";color:rgb(120,120,120)">
num_numa = <a href="http://lxr.free-electrons.com/ident?v=4.4;i=num_online_nodes" target="_blank"><span style="color:black;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;text-decoration:none">num_online_nodes</span></a>();<span></span></span></b></p>
<p class="MsoNormal" style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><a name="m_-2112328074132684035_m_9001818904416462702_L1711"></a><a href="http://lxr.free-electrons.com/source/drivers/staging/rdma/hfi1/pio.c?v=4.4#L1711" target="_blank"><b><span style="font-size:11.5pt;font-family:"courier new";color:black;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;text-decoration:none">1711</span></b></a><b><span style="font-size:11.5pt;font-family:"courier new";color:rgb(120,120,120)"> <i>/* enforce the
expectation that the numas are compact */</i><span></span></span></b></p>
<p class="MsoNormal" style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><a name="m_-2112328074132684035_m_9001818904416462702_L1712"></a><a href="http://lxr.free-electrons.com/source/drivers/staging/rdma/hfi1/pio.c?v=4.4#L1712" target="_blank"><b><span style="font-size:11.5pt;font-family:"courier new";color:black;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;text-decoration:none">1712</span></b></a><b><span style="font-size:11.5pt;font-family:"courier new";color:rgb(120,120,120)"> for (<a href="http://lxr.free-electrons.com/ident?v=4.4;i=i" target="_blank"><span style="color:black;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;text-decoration:none">i</span></a> = 0; <a href="http://lxr.free-electrons.com/ident?v=4.4;i=i" target="_blank"><span style="color:black;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;text-decoration:none">i</span></a> <
num_numa; <a href="http://lxr.free-electrons.com/ident?v=4.4;i=i" target="_blank"><span style="color:black;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;text-decoration:none">i</span></a>++)
{<span></span></span></b></p>
<p class="MsoNormal" style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><a name="m_-2112328074132684035_m_9001818904416462702_L1713"></a><a href="http://lxr.free-electrons.com/source/drivers/staging/rdma/hfi1/pio.c?v=4.4#L1713" target="_blank"><b><span style="font-size:11.5pt;font-family:"courier new";color:black;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;text-decoration:none">1713</span></b></a><b><span style="font-size:11.5pt;font-family:"courier new";color:rgb(120,120,120)">
if (!<a href="http://lxr.free-electrons.com/ident?v=4.4;i=node_online" target="_blank"><span style="color:black;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;text-decoration:none">node_online</span></a>(<a href="http://lxr.free-electrons.com/ident?v=4.4;i=i" target="_blank"><span style="color:black;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;text-decoration:none">i</span></a>)) {<span></span></span></b></p>
<p class="MsoNormal" style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><a name="m_-2112328074132684035_m_9001818904416462702_L1714"></a><a href="http://lxr.free-electrons.com/source/drivers/staging/rdma/hfi1/pio.c?v=4.4#L1714" target="_blank"><b><span style="font-size:11.5pt;font-family:"courier new";color:black;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;text-decoration:none">1714</span></b></a><b><span style="font-size:11.5pt;font-family:"courier new";color:rgb(120,120,120)">
<a href="http://lxr.free-electrons.com/ident?v=4.4;i=dd_dev_err" target="_blank"><span style="color:black;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;text-decoration:none">dd_dev_err</span></a>(<a href="http://lxr.free-electrons.com/ident?v=4.4;i=dd" target="_blank"><span style="color:black;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;text-decoration:none">dd</span></a>, <i>"NUMA
nodes are not compact\n"</i>);<span></span></span></b></p>
<p class="MsoNormal" style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><a name="m_-2112328074132684035_m_9001818904416462702_L1715"></a><a href="http://lxr.free-electrons.com/source/drivers/staging/rdma/hfi1/pio.c?v=4.4#L1715" target="_blank"><b><span style="font-size:11.5pt;font-family:"courier new";color:black;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;text-decoration:none">1715</span></b></a><b><span style="font-size:11.5pt;font-family:"courier new";color:rgb(120,120,120)">
<a href="http://lxr.free-electrons.com/ident?v=4.4;i=ret" target="_blank"><span style="color:black;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;text-decoration:none">ret</span></a>
= -<a href="http://lxr.free-electrons.com/ident?v=4.4;i=EINVAL" target="_blank"><span style="color:black;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;text-decoration:none">EINVAL</span></a>;<span></span></span></b></p>
<p class="MsoNormal" style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><a name="m_-2112328074132684035_m_9001818904416462702_L1716"></a><a href="http://lxr.free-electrons.com/source/drivers/staging/rdma/hfi1/pio.c?v=4.4#L1716" target="_blank"><b><span style="font-size:11.5pt;font-family:"courier new";color:black;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;text-decoration:none">1716</span></b></a><b><span style="font-size:11.5pt;font-family:"courier new";color:rgb(120,120,120)">
goto <a href="http://lxr.free-electrons.com/ident?v=4.4;i=done" target="_blank"><span style="color:black;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;text-decoration:none">done</span></a>;<span></span></span></b></p>
<p class="MsoNormal" style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><a name="m_-2112328074132684035_m_9001818904416462702_L1717"></a><a href="http://lxr.free-electrons.com/source/drivers/staging/rdma/hfi1/pio.c?v=4.4#L1717" target="_blank"><b><span style="font-size:11.5pt;font-family:"courier new";color:black;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;text-decoration:none">1717</span></b></a><b><span style="font-size:11.5pt;font-family:"courier new";color:rgb(120,120,120)">
}<span></span></span></b></p>
<p class="MsoNormal" style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><a name="m_-2112328074132684035_m_9001818904416462702_L1718"></a><a href="http://lxr.free-electrons.com/source/drivers/staging/rdma/hfi1/pio.c?v=4.4#L1718" target="_blank"><b><span style="font-size:11.5pt;font-family:"courier new";color:black;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;text-decoration:none">1718</span></b></a><b><span style="font-size:11.5pt;font-family:"courier new";color:rgb(120,120,120)"> }<span></span></span></b></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)"> </span></p><p class="MsoNormal"><span style="color:rgb(31,73,125)"><br></span></p><p class="MsoNormal"><span style="color:rgb(31,73,125)">On some servers I have I see this weirdness with the NUMA zones:</span></p><p class="MsoNormal"><span style="color:rgb(31,73,125)">(2650-v4 processors, HT is off)</span></p><p class="MsoNormal"><font color="#1f497d">[root@comp006 ~]# numactl --hardware</font></p><p class="MsoNormal"><font color="#1f497d">available: 2 nodes (0,2)</font></p><p class="MsoNormal"><font color="#1f497d">node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 18 19 20 21 22 23</font></p><p class="MsoNormal"><font color="#1f497d">node 0 size: 32673 MB</font></p><p class="MsoNormal"><font color="#1f497d">node 0 free: 29840 MB</font></p><p class="MsoNormal"><font color="#1f497d">node 2 cpus: 12 13 14 15 16 17</font></p><p class="MsoNormal"><font color="#1f497d">node 2 size: 32768 MB</font></p><p class="MsoNormal"><font color="#1f497d">node 2 free: 31753 MB</font></p><p class="MsoNormal"><font color="#1f497d">node distances:</font></p><p class="MsoNormal"><font color="#1f497d">node 0 2</font></p><p class="MsoNormal"><font color="#1f497d"> 0: 10 20</font></p><p class="MsoNormal"></p><p class="MsoNormal"><font color="#1f497d"> 2: 20 10</font></p><p class="MsoNormal"><font color="#1f497d"><br></font></p><p class="MsoNormal"><font color="#1f497d"><br></font></p><p class="MsoNormal"><font color="#1f497d">Someone will be along in a minute to explain why.</font></p><p class="MsoNormal"><font color="#1f497d">I am sure this is a BISO Setting, but which oen is not makign itself clear to me.</font></p><p class="MsoNormal"><font color="#1f497d"><br></font></p><p class="MsoNormal"><font color="#1f497d"><br></font></p><p class="MsoNormal"><font color="#1f497d"><br></font></p><div><br></div><p class="MsoNormal"><span style="color:rgb(31,73,125)"><br></span></p></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>