IBM Power Linux Flags file

Commands and Options Used to Submit Benchmark Runs

Each process was bound to a cpu using submit= with the numactl command
```
 
submit = numactl -l -C $BIND $command
```

numactl : Control NUMA policy for processes or shared memory

      -l
       Allocates memory from the local node of the cpu.

     -C
       Only execute process on cpus.  This accepts physical cpu numbers
       as shown in the processor fields of /proc/cpuinfo.

Shell, Environment, and Other Software Settings

Environment variables that can be set before the run:

XLFRTEOPTS=intrinthrds=1 : Causes the Fortran runtime to only use a single thread.
LD_PRELOAD=/opt/at11.0/lib64/libhugetlbfs.so : By preloading the Hugepage library, it can back malloc() and shared memory, and text and data segments can be partially backed if they are large enough.
TCMALLOC_MEMFS_MALLOC_PATH=/dev/hugepages/ : If set, specify a path where hugetlbfs or tmpfs is mounted. This may allow for speedier allocations.
MALLOC_MMAP_MAX_=0 : when combined, MALLOC_TRIM_THRESHOLD and MALLOC_MMAP_MAX force MALLOC to use SBRK() rather than MMAP() to allocate memory. This improves performance, but it may reduce the total amount of memory available to your user processes (to no more than 1 Gbyte/process).

To reserve 11520 huge pages out of the physical memory pool, issue the following command,

echo 11520 > /proc/sys/vm/nr_hugepages

You can also use the environment variables below to manage huge pages behavior:

HUGETLB_VERBOSE=0 : Turn off any debugging message from libhugetlbfs
HUGETLB_MORECORE=yes:  Instructs libhugetlbfs to override libc's normal morecore() function with a hugepage version and use it for malloc(). 
HUGETLB_MORECORE_HEAPBASE=0x50000000: Specifies that the hugepage heap address to start at 0x50000000. 
HUGETLB_ELFMAP=R ; Instructs libhugetlbfs to place text segment in hugepages.
HUGETLB_ELFMAP=W ; Instructs libhugetlbfs to place data and BSS segments in hugepages.
HUGETLB_ELFMAP=RW ; Instructs libhugetlbfs to place all segments in hugepages.
HUGETLB_ELFMAP=no ; Instructs libhugetlbfs not to place any segment in hugepages.

Operating System Tuning Parameters

ulimit -s unlimited
Sets the maximum stack size to "unlimited".

Firmware / BIOS / Microcode Settings

Power Mode is settable at the Advanced System Management menu that controls the trade-offs between power efficiency, frequency, and consistency. Four modes are available:

Disable all modes :
The processor clock frequency will be set to its fixed, nominal value.
Enable Static Power Saver mode :
Enabling this feature reduces power consumption by lowering the processor clock frequency and voltage to fixed values. This reduces the power consumption of the system while delivering predictable performance.
Enable Dynamic Performance mode :
Enabling this feature causes the processor frequency to vary based on workload and active core count. As the workload/active core count decreases, the processor uses less power, which enables the frequency to be increased above nominal. During periods of very low utilization, the processor frequency will be reduced in order to save energy. This mode provides consistent performance across all environmental operating conditions.
Enable Maximum Performance mode :
Enabling this feature causes the processor frequency to vary based on workload and active core count. As the workload/active core count decreases, the processor uses less power, which enables the frequency to be increased above nominal. In this mode, the allowed socket power is increased to the maximum value, which results in top performance along with increased fan noise and higher power consumption. In more stressful environmental conditions, performance may vary. This is the default mode.

Idle Power Saver is an option that can be combined with Maximum Performance Mode, Dynamic Performance Mode, and Disable All Modes to allow the system to drop to a frequency level below nominal frequency under programmable idle circumstances.

Three Speculative Execution Control knobs are available:

Speculative execution controls to mitigate user-to-kernel and user-to-user side-channel attacks
Speculative execution controls to mitigate user-to-kernel side-channel attacks
Speculative execution fully enabled

Default selection is "Speculative execution controls to mitigate user-to-kernel and user-to-user side-channel attacks".