<div dir="ltr">This is in the context of Ominpath cards and the hfi1 driver.<div>In the file pio.c there is a check on the NUMA zones being online</div><div><br></div><div><p class="MsoNormal"><span style="color:rgb(31,73,125)"> </span></p>
<p class="MsoNormal" style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><b><span style="font-size:11.5pt;font-family:"courier new";color:rgb(120,120,120)">
num_numa = <a href="http://lxr.free-electrons.com/ident?v=4.4;i=num_online_nodes"><span style="color:black;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;text-decoration:none">num_online_nodes</span></a>();<span></span></span></b></p>
<p class="MsoNormal" style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><a name="L1711"></a><a href="http://lxr.free-electrons.com/source/drivers/staging/rdma/hfi1/pio.c?v=4.4#L1711"><b><span style="font-size:11.5pt;font-family:"courier new";color:black;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;text-decoration:none">1711</span></b></a><b><span style="font-size:11.5pt;font-family:"courier new";color:rgb(120,120,120)"> <i>/* enforce the
expectation that the numas are compact */</i><span></span></span></b></p>
<p class="MsoNormal" style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><a name="L1712"></a><a href="http://lxr.free-electrons.com/source/drivers/staging/rdma/hfi1/pio.c?v=4.4#L1712"><b><span style="font-size:11.5pt;font-family:"courier new";color:black;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;text-decoration:none">1712</span></b></a><b><span style="font-size:11.5pt;font-family:"courier new";color:rgb(120,120,120)"> for (<a href="http://lxr.free-electrons.com/ident?v=4.4;i=i"><span style="color:black;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;text-decoration:none">i</span></a> = 0; <a href="http://lxr.free-electrons.com/ident?v=4.4;i=i"><span style="color:black;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;text-decoration:none">i</span></a> <
num_numa; <a href="http://lxr.free-electrons.com/ident?v=4.4;i=i"><span style="color:black;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;text-decoration:none">i</span></a>++)
{<span></span></span></b></p>
<p class="MsoNormal" style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><a name="L1713"></a><a href="http://lxr.free-electrons.com/source/drivers/staging/rdma/hfi1/pio.c?v=4.4#L1713"><b><span style="font-size:11.5pt;font-family:"courier new";color:black;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;text-decoration:none">1713</span></b></a><b><span style="font-size:11.5pt;font-family:"courier new";color:rgb(120,120,120)">
if (!<a href="http://lxr.free-electrons.com/ident?v=4.4;i=node_online"><span style="color:black;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;text-decoration:none">node_online</span></a>(<a href="http://lxr.free-electrons.com/ident?v=4.4;i=i"><span style="color:black;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;text-decoration:none">i</span></a>)) {<span></span></span></b></p>
<p class="MsoNormal" style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><a name="L1714"></a><a href="http://lxr.free-electrons.com/source/drivers/staging/rdma/hfi1/pio.c?v=4.4#L1714"><b><span style="font-size:11.5pt;font-family:"courier new";color:black;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;text-decoration:none">1714</span></b></a><b><span style="font-size:11.5pt;font-family:"courier new";color:rgb(120,120,120)">
<a href="http://lxr.free-electrons.com/ident?v=4.4;i=dd_dev_err"><span style="color:black;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;text-decoration:none">dd_dev_err</span></a>(<a href="http://lxr.free-electrons.com/ident?v=4.4;i=dd"><span style="color:black;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;text-decoration:none">dd</span></a>, <i>"NUMA
nodes are not compact\n"</i>);<span></span></span></b></p>
<p class="MsoNormal" style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><a name="L1715"></a><a href="http://lxr.free-electrons.com/source/drivers/staging/rdma/hfi1/pio.c?v=4.4#L1715"><b><span style="font-size:11.5pt;font-family:"courier new";color:black;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;text-decoration:none">1715</span></b></a><b><span style="font-size:11.5pt;font-family:"courier new";color:rgb(120,120,120)">
<a href="http://lxr.free-electrons.com/ident?v=4.4;i=ret"><span style="color:black;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;text-decoration:none">ret</span></a>
= -<a href="http://lxr.free-electrons.com/ident?v=4.4;i=EINVAL"><span style="color:black;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;text-decoration:none">EINVAL</span></a>;<span></span></span></b></p>
<p class="MsoNormal" style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><a name="L1716"></a><a href="http://lxr.free-electrons.com/source/drivers/staging/rdma/hfi1/pio.c?v=4.4#L1716"><b><span style="font-size:11.5pt;font-family:"courier new";color:black;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;text-decoration:none">1716</span></b></a><b><span style="font-size:11.5pt;font-family:"courier new";color:rgb(120,120,120)">
goto <a href="http://lxr.free-electrons.com/ident?v=4.4;i=done"><span style="color:black;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;text-decoration:none">done</span></a>;<span></span></span></b></p>
<p class="MsoNormal" style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><a name="L1717"></a><a href="http://lxr.free-electrons.com/source/drivers/staging/rdma/hfi1/pio.c?v=4.4#L1717"><b><span style="font-size:11.5pt;font-family:"courier new";color:black;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;text-decoration:none">1717</span></b></a><b><span style="font-size:11.5pt;font-family:"courier new";color:rgb(120,120,120)">
}<span></span></span></b></p>
<p class="MsoNormal" style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><a name="L1718"></a><a href="http://lxr.free-electrons.com/source/drivers/staging/rdma/hfi1/pio.c?v=4.4#L1718"><b><span style="font-size:11.5pt;font-family:"courier new";color:black;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;text-decoration:none">1718</span></b></a><b><span style="font-size:11.5pt;font-family:"courier new";color:rgb(120,120,120)"> }<span></span></span></b></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)"> </span></p><p class="MsoNormal"><span style="color:rgb(31,73,125)"><br></span></p><p class="MsoNormal"><span style="color:rgb(31,73,125)">On some servers I have I see this weirdness with the NUMA zones:</span></p><p class="MsoNormal"><span style="color:rgb(31,73,125)">(2650-v4 processors, HT is off)</span></p><p class="MsoNormal"><font color="#1f497d">[root@comp006 ~]# numactl --hardware</font></p><p class="MsoNormal"><font color="#1f497d">available: 2 nodes (0,2)</font></p><p class="MsoNormal"><font color="#1f497d">node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 18 19 20 21 22 23</font></p><p class="MsoNormal"><font color="#1f497d">node 0 size: 32673 MB</font></p><p class="MsoNormal"><font color="#1f497d">node 0 free: 29840 MB</font></p><p class="MsoNormal"><font color="#1f497d">node 2 cpus: 12 13 14 15 16 17</font></p><p class="MsoNormal"><font color="#1f497d">node 2 size: 32768 MB</font></p><p class="MsoNormal"><font color="#1f497d">node 2 free: 31753 MB</font></p><p class="MsoNormal"><font color="#1f497d">node distances:</font></p><p class="MsoNormal"><font color="#1f497d">node 0 2</font></p><p class="MsoNormal"><font color="#1f497d"> 0: 10 20</font></p><p class="MsoNormal"></p><p class="MsoNormal"><font color="#1f497d"> 2: 20 10</font></p><p class="MsoNormal"><font color="#1f497d"><br></font></p><p class="MsoNormal"><font color="#1f497d"><br></font></p><p class="MsoNormal"><font color="#1f497d">Someone will be along in a minute to explain why.</font></p><p class="MsoNormal"><font color="#1f497d">I am sure this is a BISO Setting, but which oen is not makign itself clear to me.</font></p><p class="MsoNormal"><font color="#1f497d"><br></font></p><p class="MsoNormal"><font color="#1f497d"><br></font></p><p class="MsoNormal"><font color="#1f497d"><br></font></p><div><br></div><p class="MsoNormal"><span style="color:rgb(31,73,125)"><br></span></p></div></div>