[Beowulf] Scaling issues on Xeon E5-2680
pbisbal at oit.rutgers.edu
Mon Feb 29 07:55:47 PST 2016
As others have said, there's a million things that could be going on here.
What InfiniBand software stack are you using? Are you using the RPMs
that come with CentOS 7, or are you using the lated version of OFED
downloaded directly from Mellanox. For the past year or so, I've been
hearing that the distro-provided RPMS perform much worse than the
Mellanox-provided packages, but I haven't had the opportunity to test
When you set up the InfiniBand stack, there's usually ulimits you need
to tune and I think some kernel parameters. Have you done that on the
new system? I think OpenMPI will report an error if these changes aren't
made, but I'm not 100% sure. If you use the distro RPMS, I don't think
these changes are made automatically.
Also, did you configure OpenMPI so that it uses IB for the BTL instead
of TCP? That would be an easy step to overlook when setting up a new
system. I just checked the OpenMPI FAQ, and it says OpenMPI should now
detect IB automatically and use that instead of TCP, but I would
explicitly telling OpenMPI to not use TCP as a BTL, and see if that
changes anything. If it's not automatically detecting IB correctly, that
should cause it to through an error.
On 02/28/2016 10:27 AM, Jon Tegner wrote:
> have issues with performance on E5-2680. Each of the nodes have 2 of
> these 12 core CPUs on SuperMicro SuperServer 1028R-WMR (i.e., 24 cores
> on each node).
> For one of our applications (CFD/OpenFOAM) we have noticed that the
> calculation runs faster using 12 cores on 4 nodes compared to when
> using 24 cores on 4 nodes.
> In our environment we also have older AMD hardware (nodes with 4 CPUs
> with 12 cores each), and here we don't see these strange scaling issues.
> System is CentOS-7, and communication is over FDR Infiniband. BIOS is
> recently updated, and hyperthreading is disabled.
> Feel a bit lost here, and any hints on how to proceed with this are
> greatly appreciated!
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
More information about the Beowulf