[Beowulf] NUMA zone weirdness

Mon Dec 19 08:33:37 PST 2016

All, thankyou very much for looking at this and Happy Christmas when it
comes.
And a Guid New Year tae ye a.

Number of DIMMS doubled, all channels now populated.  Et voila:

(HT is on, COD is enabled)

[root at comp007 ~]# numactl --hardware
available: 4 nodes (0-3)
node 0 cpus: 0 1 2 3 4 5 24 25 26 27 28 29
node 0 size: 32673 MB
node 0 free: 31712 MB
node 1 cpus: 6 7 8 9 10 11 30 31 32 33 34 35
node 1 size: 32768 MB
node 1 free: 31926 MB
node 2 cpus: 12 13 14 15 16 17 36 37 38 39 40 41
node 2 size: 32768 MB
node 2 free: 31972 MB
node 3 cpus: 18 19 20 21 22 23 42 43 44 45 46 47
node 3 size: 32768 MB
node 3 free: 31953 MB
node distances:
node   0   1   2   3
  0:  10  11  21  21
  1:  11  10  21  21
  2:  21  21  10  11
  3:  21  21  11  10

[root at comp007 ~]# lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                48
On-line CPU(s) list:   0-47
Thread(s) per core:    2
Core(s) per socket:    12
Socket(s):             2
NUMA node(s):          4
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 79
Model name:            Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
Stepping:              1
CPU MHz:               1200.289
BogoMIPS:              4396.61
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              15360K
NUMA node0 CPU(s):     0-5,24-29
NUMA node1 CPU(s):     6-11,30-35
NUMA node2 CPU(s):     12-17,36-41
NUMA node3 CPU(s):     18-23,42-47

On 19 December 2016 at 03:26, John Hearns <hearnsj at googlemail.com> wrote:

> Brice, thankyou for the reply. You have the answer - these systems have
> two DIMMS per socket, channels 0 and, so all channels are not populated.
>
> I had the lstopo output and the tarball all ready for the OpenMPI list
> too! Shoudl have sent it over there.
>
>
> On 18 December 2016 at 21:28, Brice Goglin <brice.goglin at gmail.com> wrote:
>
>> Hello
>> Do you know if all your CPU memory channels are populated? CoD requires
>> that each half of the CPU has some memory DIMMs (so that each NUMA node
>> actually contains some memory). If both channels of one half are empty,
>> the NUMA node might somehow disappear.
>> Brice
>>
>>
>>
>>
>> Le 16/12/2016 23:26, Elken, Tom a écrit :
>> > Hi John and Greg,
>> >
>> > You showed  Nodes 0 & 2 (no node 1) and a strange CPU assignment to
>> nodes!
>> > Even though you had Cluster On Die (CoD) Endabled in your BIOS, I have
>> never seen that arrangement of Numa nodes and CPUs.  You may have a bug in
>> your BIOS or OS ?
>> > With CoD enabled, I would have expected 4 NUMA nodes, 0-3, and 6 cores
>> assigned to each one.
>> >
>> > The Omni-Path Performance Tuning User Guide
>> > http://www.intel.com/content/dam/support/us/en/documents/net
>> work-and-i-o/fabric-products/Intel_OP_Performance_Tuning_
>> UG_H93143_v6_0.pdf
>> > does recommend Disabling CoD in Xeon BIOSes  (Table 2 on P. 12), but
>> it's not considered a hard prohibition.
>> > Disabling improves some fabric performance benchmarks, but Enabling
>> helps some single-node applications performance, which could outweigh the
>> fabric performance aspects.
>> >
>> > -Tom
>> >
>> >> -----Original Message-----
>> >> From: Beowulf [mailto:beowulf-bounces at beowulf.org] On Behalf Of Greg
>> >> Lindahl
>> >> Sent: Friday, December 16, 2016 2:00 PM
>> >> To: John Hearns
>> >> Cc: Beowulf Mailing List
>> >> Subject: Re: [Beowulf] NUMA zone weirdness
>> >>
>> >> Wow, that's pretty obscure!
>> >>
>> >> I'd recommend reporting it to Intel so that they can add it to the
>> >> descendants of ipath_checkout / ipath_debug. It's exactly the kind of
>> >> hidden gotcha that leads to unhappy systems!
>> >>
>> >> -- greg
>> >>
>> >> On Fri, Dec 16, 2016 at 03:52:34PM +0000, John Hearns wrote:
>> >>> Problem solved.
>> >>> I have changed the QPI Snoop Mode on these servers from
>> >>> ClusterOnDIe Enabled to Disabled and they display what I take to be
>> correct
>> >>> behaviour - ie
>> >>>
>> >>> [root at comp006 ~]# numactl --hardware
>> >>> available: 2 nodes (0-1)
>> >>> node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11
>> >>> node 0 size: 32673 MB
>> >>> node 0 free: 31541 MB
>> >>> node 1 cpus: 12 13 14 15 16 17 18 19 20 21 22 23
>> >>> node 1 size: 32768 MB
>> >>> node 1 free: 31860 MB
>> >>> node distances:
>> >>> node   0   1
>> >>>   0:  10  21
>> >>>   1:  21  10
>> >> _______________________________________________
>> >> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin
>> Computing
>> >> To change your subscription (digest mode or unsubscribe) visit
>> >> http://www.beowulf.org/mailman/listinfo/beowulf
>> > _______________________________________________
>> > Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin
>> Computing
>> > To change your subscription (digest mode or unsubscribe) visit
>> http://www.beowulf.org/mailman/listinfo/beowulf
>>
>> _______________________________________________
>> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
>> To change your subscription (digest mode or unsubscribe) visit
>> http://www.beowulf.org/mailman/listinfo/beowulf
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.beowulf.org/pipermail/beowulf/attachments/20161219/ddcdc4b1/attachment-0001.html>