[Beowulf] Scaling issues on Xeon E5-2680

[John Hearns]

Backing up what everyone else says.  DO you know how the 24 core jobs are being distributed between sockets and cores.

Also as a very naive diagnostic, just run 'top' or even better 'htop' while the jobs are running.
I KNOW this isnt a great diagnostic for parallel programs, but sometimes you can notice patterns.
Select the option in top to show you the cores which are running.

Also keep an eye on the memory assignments at the top - you are not going into swap?
Are any processes going into D sttes for extended amounts of time?


________________________________________
From: Beowulf [beowulf-bounces at beowulf.org] on behalf of Peter St. John [peter.st.john at gmail.com]
Sent: 29 February 2016 00:50
To: Jon Tegner
Cc: Beowulf List
Subject: Re: [Beowulf] Scaling issues on Xeon E5-2680

I come from the math side, not the electronics side, so this may be an ill-posed question, but could you try running the job with 12 cores on just one node, then with 6 cores on each of two nodes? I'm thinking, the 24 core version may get assigned to more nodes than your 12 core, and it's communication between nodes that slows it down, e.g. for a job that doesn't require so much memory as inter-process communication.
Peter

On Sun, Feb 28, 2016 at 11:01 AM, Jon Tegner <tegner at renget.se<mailto:tegner at renget.se>> wrote:
Thanks!

we are using openmpi-1.10.2, and I believe bind-to core and socket are on by default here. Openmpi is built using

./configure \
        --build=x86_64-redhat-linux-gnu \
        --host=x86_64-redhat-linux-gnu \
        --disable-dependency-tracking \
        --prefix=/remote/soft/OpenMPI/1.10.2 \
        --disable-mpi-profile \
        --enable-shared \
        --with-tm \
        --with-sge \
        --with-verbs

gcc is used throughout.

Code should be using RDMA. I have verified that infiniband performance (using openmpi-1.10.2) is what is to be expected (through mpitests-osu_bw and mpitest_osu_latency).

Hugepagesize haven't been touched ("grep Hugepagesize /proc/meminfo" gives 2048 kB). Is this something worth changing?

numastat gives

                           node0           node1
numa_hit                 6288694         1193663
numa_miss                      0               0
numa_foreign                   0               0
interleave_hit             66539           66287
local_node               6288301         1126078
other_node                   393           67585

on one of the nodes - but these numbers seems to be fairly representative for the nodes. However, haven't used numastat before, and maybe it should be used simultaneously as the job is running (which it wasn't now)?

Thanks!

/jon


On 02/28/2016 04:30 PM, cbergstrom at pathscale.com<mailto:cbergstrom at pathscale.com> wrote:
Do you have CPU affinity enabled? Is this omp + MPI? Which compilers and flags. Give more details about the software stack

   Original Message
From: Jon Tegner
Sent: Sunday, February 28, 2016 22:28
To: beowulf at beowulf.org<mailto:beowulf at beowulf.org>
Subject: [Beowulf] Scaling issues on Xeon E5-2680

Hi,

have issues with performance on E5-2680. Each of the nodes have 2 of
these 12 core CPUs on SuperMicro SuperServer 1028R-WMR (i.e., 24 cores
on each node).

For one of our applications (CFD/OpenFOAM) we have noticed that the
calculation runs faster using 12 cores on 4 nodes compared to when using
24 cores on 4 nodes.

In our environment we also have older AMD hardware (nodes with 4 CPUs
with 12 cores each), and here we don't see these strange scaling issues.

System is CentOS-7, and communication is over FDR Infiniband. BIOS is
recently updated, and hyperthreading is disabled.

Feel a bit lost here, and any hints on how to proceed with this are
greatly appreciated!

Thanks,

/jon
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org<mailto:Beowulf at beowulf.org> sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf

_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org<mailto:Beowulf at beowulf.org> sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf

#####################################################################################
Scanned by MailMarshal - M86 Security's comprehensive email content security solution.
#####################################################################################
Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of the company. Employees of XMA Ltd are expressly required not to make defamatory statements and not to infringe or authorise any infringement of copyright or any other legal right by email communications. Any such communication is contrary to company policy and outside the scope of the employment of the individual concerned. The company will not accept any liability in respect of such communication, and the employee responsible will be personally liable for any damages or other liability arising. XMA Limited is registered in England and Wales (registered no. 2051703). Registered Office: Wilford Industrial Estate, Ruddington Lane, Wilford, Nottingham, NG11 7EP