[Beowulf] numactl & SuSE11.1

[Mikhail Kuzminsky]

It's interesting, that for this hard&software configuration disabling 
of NUMA in BIOS gives more high STREAM results in comparison w/"NUMA 
enabled".

I.e. for NUMA "off": 8723/8232/10388/10317 MB/s
for NUMA "on": 5620/5217/6795/6767 MB/s
(both for OMP_NUM_THREADS=1 and ifort 11.1 compiler).

The situation for Opteron's is opposite: NUMA mode gives more high 
throughput.


In message from "Mikhail Kuzminsky" <kus at free.net> (Mon, 10 Aug 2009 
21:43:56 +0400):
>I'm sorry for my mistake:
>the problem is on Nehalem Xeon under SuSE -11.1, but w/kernel 
>2.6.27.7-9 (w/Supermicro X8DT mobo). For Opteron 2350 w/SuSE 10.3 (w/ 
>more old 2.6.22.5-31 -I erroneously inserted this string in my 
>previous message) numactl works OK (w/Tyan mobo).
>
>NUMA is enabled in BIOS. Of course, CONFIG_NUMA (and CONFIG_NUMA_EMU) 
>are setted to "y" in both kernels.
>
>Unfortunately I (i.e. root) can't change files in 
>/sys/devices/system/node (or rename directory node2 to node1) :-( - as 
>it's possible w/some files in /proc filesystem. It's interesting, that 
>extraction from dmesg show, that IT WAS NODE1, but then node2 is 
>appear !
>
>ACPI: SRAT BF79A4B0, 0150 (r1 041409 OEMSRAT         1 INTL        1)
>ACPI: SSDT BF79FAC0, 249F (r1 DpgPmm    CpuPm       12 INTL 20051117)
>ACPI: Local APIC address 0xfee00000
>SRAT: PXM 0 -> APIC 0 -> Node 0
>SRAT: PXM 0 -> APIC 2 -> Node 0
>SRAT: PXM 0 -> APIC 4 -> Node 0
>SRAT: PXM 0 -> APIC 6 -> Node 0
>SRAT: PXM 1 -> APIC 16 -> Node 1
>SRAT: PXM 1 -> APIC 18 -> Node 1
>SRAT: PXM 1 -> APIC 20 -> Node 1
>SRAT: PXM 1 -> APIC 22 -> Node 1
>SRAT: Node 0 PXM 0 0-a0000
>SRAT: Node 0 PXM 0 100000-c0000000
>SRAT: Node 0 PXM 0 100000000-1c0000000
>SRAT: Node 2 PXM 257 1c0000000-340000000
>(here !!)
>
>NUMA: Allocated memnodemap from 1c000 - 22880
>NUMA: Using 20 for the hash shift.
>Bootmem setup node 0 0000000000000000-00000001c0000000
>   NODE_DATA [0000000000022880 - 000000000003a87f]
>   bootmap [000000000003b000 -  0000000000072fff] pages 38
>(8 early reservations) ==> bootmem [0000000000 - 01c0000000]
>   #0 [0000000000 - 0000001000]   BIOS data page ==> [0000000000 - 
>0000001000]
>   #1 [0000006000 - 0000008000]       TRAMPOLINE ==> [0000006000 - 
>0000008000]
>   #2 [0000200000 - 0000bf27b8]    TEXT DATA BSS ==> [0000200000 - 
>0000bf27b8]
>   #3 [0037a3b000 - 0037fef104]          RAMDISK ==> [0037a3b000 - 
>0037fef104]
>   #4 [000009cc00 - 0000100000]    BIOS reserved ==> [000009cc00 - 
>0000100000]
>   #5 [0000010000 - 0000013000]          PGTABLE ==> [0000010000 - 
>0000013000]
>   #6 [0000013000 - 000001c000]          PGTABLE ==> [0000013000 - 
>000001c000]
>   #7 [000001c000 - 0000022880]       MEMNODEMAP ==> [000001c000 - 
>0000022880]
>Bootmem setup node 2 00000001c0000000-0000000340000000
>   NODE_DATA [00000001c0000000 - 00000001c0017fff]
>   bootmap [00000001c0018000 -  00000001c0047fff] pages 30
>(8 early reservations) ==> bootmem [01c0000000 - 0340000000]
>   #0 [0000000000 - 0000001000]   BIOS data page
>   #1 [0000006000 - 0000008000]       TRAMPOLINE
>   #2 [0000200000 - 0000bf27b8]    TEXT DATA BSS
>   #3 [0037a3b000 - 0037fef104]          RAMDISK
>   #4 [000009cc00 - 0000100000]    BIOS reserved
>   #5 [0000010000 - 0000013000]          PGTABLE
>   #6 [0000013000 - 000001c000]          PGTABLE
>   #7 [000001c000 - 0000022880]       MEMNODEMAP
>found SMP MP-table at [ffff8800000ff780] 000ff780
>  [ffffe20000000000-ffffe20006ffffff] PMD -> 
>[ffff880028200000-ffff88002e1fffff] on node 0
>  [ffffe20007000000-ffffe2000cffffff] PMD -> 
>[ffff8801c0200000-ffff8801c61fffff] on node 2
  
Mikhail Kuzminsky
Computer Assistance to Chemical Research Center
Zelinsky Institute of Organic Chemistry RAS
Moscow