[Beowulf] Scaling issues on Xeon E5-2680

[Jon Tegner]

Hi,

have issues with performance on E5-2680. Each of the nodes have 2 of 
these 12 core CPUs on SuperMicro SuperServer 1028R-WMR (i.e., 24 cores 
on each node).

For one of our applications (CFD/OpenFOAM) we have noticed that the 
calculation runs faster using 12 cores on 4 nodes compared to when using 
24 cores on 4 nodes.

In our environment we also have older AMD hardware (nodes with 4 CPUs 
with 12 cores each), and here we don't see these strange scaling issues.

System is CentOS-7, and communication is over FDR Infiniband. BIOS is 
recently updated, and hyperthreading is disabled.

Feel a bit lost here, and any hints on how to proceed with this are 
greatly appreciated!

Thanks,

/jon