<div dir="ltr">I come from the math side, not the electronics side, so this may be an ill-posed question, but could you try running the job with 12 cores on just one node, then with 6 cores on each of two nodes? I'm thinking, the 24 core version may get assigned to more nodes than your 12 core, and it's communication between nodes that slows it down, e.g. for a job that doesn't require so much memory as inter-process communication.<div>Peter</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Feb 28, 2016 at 11:01 AM, Jon Tegner <span dir="ltr"><<a href="mailto:tegner@renget.se" target="_blank">tegner@renget.se</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Thanks!<br>
<br>
we are using openmpi-1.10.2, and I believe bind-to core and socket are on by default here. Openmpi is built using<br>
<br>
./configure \<br>
--build=x86_64-redhat-linux-gnu \<br>
--host=x86_64-redhat-linux-gnu \<br>
--disable-dependency-tracking \<br>
--prefix=/remote/soft/OpenMPI/1.10.2 \<br>
--disable-mpi-profile \<br>
--enable-shared \<br>
--with-tm \<br>
--with-sge \<br>
--with-verbs<br>
<br>
gcc is used throughout.<br>
<br>
Code should be using RDMA. I have verified that infiniband performance (using openmpi-1.10.2) is what is to be expected (through mpitests-osu_bw and mpitest_osu_latency).<br>
<br>
Hugepagesize haven't been touched ("grep Hugepagesize /proc/meminfo" gives 2048 kB). Is this something worth changing?<br>
<br>
numastat gives<br>
<br>
node0 node1<br>
numa_hit 6288694 1193663<br>
numa_miss 0 0<br>
numa_foreign 0 0<br>
interleave_hit 66539 66287<br>
local_node 6288301 1126078<br>
other_node 393 67585<br>
<br>
on one of the nodes - but these numbers seems to be fairly representative for the nodes. However, haven't used numastat before, and maybe it should be used simultaneously as the job is running (which it wasn't now)?<br>
<br>
Thanks!<span class="HOEnZb"><font color="#888888"><br>
<br>
/jon</font></span><div class="HOEnZb"><div class="h5"><br>
<br>
On 02/28/2016 04:30 PM, <a href="mailto:cbergstrom@pathscale.com" target="_blank">cbergstrom@pathscale.com</a> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Do you have CPU affinity enabled? Is this omp + MPI? Which compilers and flags. Give more details about the software stack<br>
<br>
Original Message<br>
From: Jon Tegner<br>
Sent: Sunday, February 28, 2016 22:28<br>
To: <a href="mailto:beowulf@beowulf.org" target="_blank">beowulf@beowulf.org</a><br>
Subject: [Beowulf] Scaling issues on Xeon E5-2680<br>
<br>
Hi,<br>
<br>
have issues with performance on E5-2680. Each of the nodes have 2 of<br>
these 12 core CPUs on SuperMicro SuperServer 1028R-WMR (i.e., 24 cores<br>
on each node).<br>
<br>
For one of our applications (CFD/OpenFOAM) we have noticed that the<br>
calculation runs faster using 12 cores on 4 nodes compared to when using<br>
24 cores on 4 nodes.<br>
<br>
In our environment we also have older AMD hardware (nodes with 4 CPUs<br>
with 12 cores each), and here we don't see these strange scaling issues.<br>
<br>
System is CentOS-7, and communication is over FDR Infiniband. BIOS is<br>
recently updated, and hyperthreading is disabled.<br>
<br>
Feel a bit lost here, and any hints on how to proceed with this are<br>
greatly appreciated!<br>
<br>
Thanks,<br>
<br>
/jon<br>
_______________________________________________<br>
Beowulf mailing list, <a href="mailto:Beowulf@beowulf.org" target="_blank">Beowulf@beowulf.org</a> sponsored by Penguin Computing<br>
To change your subscription (digest mode or unsubscribe) visit <a href="http://www.beowulf.org/mailman/listinfo/beowulf" rel="noreferrer" target="_blank">http://www.beowulf.org/mailman/listinfo/beowulf</a><br>
</blockquote>
<br>
_______________________________________________<br>
Beowulf mailing list, <a href="mailto:Beowulf@beowulf.org" target="_blank">Beowulf@beowulf.org</a> sponsored by Penguin Computing<br>
To change your subscription (digest mode or unsubscribe) visit <a href="http://www.beowulf.org/mailman/listinfo/beowulf" rel="noreferrer" target="_blank">http://www.beowulf.org/mailman/listinfo/beowulf</a><br>
</div></div></blockquote></div><br></div>