[Beowulf] NUMA zone weirdness

[Brice Goglin]

Hello
We could easily warn when NUMA nodes have non-consecutive numbers, but
I'll need to clarify when this is really invalid (all x86?), if this
could ever occur dynamically because of memory hotplug, etc.
If it's about the BIOS failing to disable CoD when too many channels
have no DIMMs, I don't think we detect that level of detail.
Brice



Le 19/12/2016 20:42, James Cownie a écrit :
> Donc, Brice,
>
> Can hwloc detect this and give a suitably large complaint?
>
> --
> -- Jim
> --
> James Cownie <jcownie at cantab.net> (and intel, as I’m sure most of you know…)
>
>> On 18 Dec 2016, at 21:28, Brice Goglin <brice.goglin at gmail.com> wrote:
>>
>> Hello
>> Do you know if all your CPU memory channels are populated? CoD requires
>> that each half of the CPU has some memory DIMMs (so that each NUMA node
>> actually contains some memory). If both channels of one half are empty,
>> the NUMA node might somehow disappear.
>> Brice
>>
>>
>>
>>
>> Le 16/12/2016 23:26, Elken, Tom a écrit :
>>> Hi John and Greg,
>>>
>>> You showed  Nodes 0 & 2 (no node 1) and a strange CPU assignment to nodes!  
>>> Even though you had Cluster On Die (CoD) Endabled in your BIOS, I have never seen that arrangement of Numa nodes and CPUs.  You may have a bug in your BIOS or OS ?  
>>> With CoD enabled, I would have expected 4 NUMA nodes, 0-3, and 6 cores assigned to each one.
>>>
>>> The Omni-Path Performance Tuning User Guide 
>>> http://www.intel.com/content/dam/support/us/en/documents/network-and-i-o/fabric-products/Intel_OP_Performance_Tuning_UG_H93143_v6_0.pdf 
>>> does recommend Disabling CoD in Xeon BIOSes  (Table 2 on P. 12), but it's not considered a hard prohibition.
>>> Disabling improves some fabric performance benchmarks, but Enabling helps some single-node applications performance, which could outweigh the fabric performance aspects.
>>>
>>> -Tom
>>>
>>>> -----Original Message-----
>>>> From: Beowulf [mailto:beowulf-bounces at beowulf.org] On Behalf Of Greg
>>>> Lindahl
>>>> Sent: Friday, December 16, 2016 2:00 PM
>>>> To: John Hearns
>>>> Cc: Beowulf Mailing List
>>>> Subject: Re: [Beowulf] NUMA zone weirdness
>>>>
>>>> Wow, that's pretty obscure!
>>>>
>>>> I'd recommend reporting it to Intel so that they can add it to the
>>>> descendants of ipath_checkout / ipath_debug. It's exactly the kind of
>>>> hidden gotcha that leads to unhappy systems!
>>>>
>>>> -- greg
>>>>
>>>> On Fri, Dec 16, 2016 at 03:52:34PM +0000, John Hearns wrote:
>>>>> Problem solved.
>>>>> I have changed the QPI Snoop Mode on these servers from
>>>>> ClusterOnDIe Enabled to Disabled and they display what I take to be correct
>>>>> behaviour - ie
>>>>>
>>>>> [root at comp006 ~]# numactl --hardware
>>>>> available: 2 nodes (0-1)
>>>>> node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11
>>>>> node 0 size: 32673 MB
>>>>> node 0 free: 31541 MB
>>>>> node 1 cpus: 12 13 14 15 16 17 18 19 20 21 22 23
>>>>> node 1 size: 32768 MB
>>>>> node 1 free: 31860 MB
>>>>> node distances:
>>>>> node   0   1
>>>>>  0:  10  21
>>>>>  1:  21  10
>>>> _______________________________________________
>>>> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
>>>> To change your subscription (digest mode or unsubscribe) visit
>>>> http://www.beowulf.org/mailman/listinfo/beowulf
>>> _______________________________________________
>>> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
>>> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
>> _______________________________________________
>> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
>> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
>
>