[Beowulf] Scaling issues on Xeon E5-2680

[Prentice Bisbal]

As others have said, there's a million things that could be going on here.

What InfiniBand software stack are you using? Are you using the RPMs 
that come with CentOS 7, or are you using the lated version of OFED 
downloaded directly from Mellanox.  For the past year or so, I've been 
hearing that the distro-provided RPMS perform much worse than the 
Mellanox-provided packages, but I haven't had the opportunity to test 
that myself.

When you set up the InfiniBand stack, there's usually ulimits you need 
to tune and I think some kernel parameters. Have you done that on the 
new system? I think OpenMPI will report an error if these changes aren't 
made, but I'm not 100% sure. If you use the distro RPMS, I don't think 
these changes are made automatically.

Also, did you configure OpenMPI so that it uses IB for the BTL instead 
of TCP? That would be an easy step to overlook when setting up a new 
system. I just checked the OpenMPI FAQ, and it says OpenMPI should now 
detect IB automatically and use that instead of TCP, but I would 
explicitly telling OpenMPI to not use TCP as a BTL, and see if that 
changes anything. If it's not automatically detecting IB correctly, that 
should cause it to through an error.

Prentice

On 02/28/2016 10:27 AM, Jon Tegner wrote:
> Hi,
>
> have issues with performance on E5-2680. Each of the nodes have 2 of 
> these 12 core CPUs on SuperMicro SuperServer 1028R-WMR (i.e., 24 cores 
> on each node).
>
> For one of our applications (CFD/OpenFOAM) we have noticed that the 
> calculation runs faster using 12 cores on 4 nodes compared to when 
> using 24 cores on 4 nodes.
>
> In our environment we also have older AMD hardware (nodes with 4 CPUs 
> with 12 cores each), and here we don't see these strange scaling issues.
>
> System is CentOS-7, and communication is over FDR Infiniband. BIOS is 
> recently updated, and hyperthreading is disabled.
>
> Feel a bit lost here, and any hints on how to proceed with this are 
> greatly appreciated!
>
> Thanks,
>
> /jon
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit 
> http://www.beowulf.org/mailman/listinfo/beowulf