CPU2017 Flag Description
Hewlett Packard Enterprise ProLiant DL325 Gen10 Plus v2 (2.80 GHz, AMD EPYC 7543P)
Test sponsored by HPE
Compilers: AMD Optimizing C/C++ Compiler Suite
Base Compiler Invocation
-
- clang
![[user]](https://www.spec.org/auto/cpu2017/images/user.png)
- CC, LD
-
clang is a C compiler which encompasses preprocessing, parsing, optimization, code generation, assembly, and linking.
Depending on which high-level mode setting is passed, Clang will stop before doing a full link.
-
- clang++
- CXX, LD
-
clang++ C++ compiler which encompasses preprocessing, parsing, optimization, code generation, assembly, and linking.
Depending on which high-level mode setting is passed, Clang will stop before doing a full link.
-
- flang
- FC, LD
-
flang is a Fortran compiler which encompasses parsing, optimization, code generation, assembly, and linking. Depending on
which high-level mode setting is passed, Flang will stop before doing a full link.
Peak Compiler Invocation
-
- clang
- CC, LD
-
clang is a C compiler which encompasses preprocessing, parsing, optimization, code generation, assembly, and linking.
Depending on which high-level mode setting is passed, Clang will stop before doing a full link.
-
- clang++
- CXX, LD
-
clang++ C++ compiler which encompasses preprocessing, parsing, optimization, code generation, assembly, and linking.
Depending on which high-level mode setting is passed, Clang will stop before doing a full link.
-
- flang
- FC, LD
-
flang is a Fortran compiler which encompasses parsing, optimization, code generation, assembly, and linking. Depending on
which high-level mode setting is passed, Flang will stop before doing a full link.
Base Portability Flags
-
![[suite]](https://www.spec.org/auto/cpu2017/images/suite.png)
- EXTRA_PORTABILITY
-
This option is used to indicate that the host system's integers are 32-bits
wide, and longs and pointers are 64-bits wide. Not all benchmarks
recognize this macro, but the preferred practice for data model selection
applies the flags to all benchmarks; this flag description is a placeholder
for those benchmarks that do not recognize this macro.
-
- EXTRA_PORTABILITY
-
This option is used to indicate that the host system's integers are 32-bits
wide, and longs and pointers are 64-bits wide. Not all benchmarks
recognize this macro, but the preferred practice for data model selection
applies the flags to all benchmarks; this flag description is a placeholder
for those benchmarks that do not recognize this macro.
-
- EXTRA_PORTABILITY
-
This option is used to indicate that the host system's integers are 32-bits
wide, and longs and pointers are 64-bits wide. Not all benchmarks
recognize this macro, but the preferred practice for data model selection
applies the flags to all benchmarks; this flag description is a placeholder
for those benchmarks that do not recognize this macro.
-
-
- EXTRA_PORTABILITY
-
This option is used to indicate that the host system's integers are 32-bits
wide, and longs and pointers are 64-bits wide. Not all benchmarks
recognize this macro, but the preferred practice for data model selection
applies the flags to all benchmarks; this flag description is a placeholder
for those benchmarks that do not recognize this macro.
-
- EXTRA_PORTABILITY
-
This option is used to indicate that the host system's integers are 32-bits
wide, and longs and pointers are 64-bits wide. Not all benchmarks
recognize this macro, but the preferred practice for data model selection
applies the flags to all benchmarks; this flag description is a placeholder
for those benchmarks that do not recognize this macro.
-
- EXTRA_PORTABILITY
-
This option is used to indicate that the host system's integers are 32-bits
wide, and longs and pointers are 64-bits wide. Not all benchmarks
recognize this macro, but the preferred practice for data model selection
applies the flags to all benchmarks; this flag description is a placeholder
for those benchmarks that do not recognize this macro.
-
- EXTRA_PORTABILITY
-
This option is used to indicate that the host system's integers are 32-bits
wide, and longs and pointers are 64-bits wide. Not all benchmarks
recognize this macro, but the preferred practice for data model selection
applies the flags to all benchmarks; this flag description is a placeholder
for those benchmarks that do not recognize this macro.
-
- EXTRA_PORTABILITY
-
This option is used to indicate that the host system's integers are 32-bits
wide, and longs and pointers are 64-bits wide. Not all benchmarks
recognize this macro, but the preferred practice for data model selection
applies the flags to all benchmarks; this flag description is a placeholder
for those benchmarks that do not recognize this macro.
-
- EXTRA_PORTABILITY
-
This option is used to indicate that the host system's integers are 32-bits
wide, and longs and pointers are 64-bits wide. Not all benchmarks
recognize this macro, but the preferred practice for data model selection
applies the flags to all benchmarks; this flag description is a placeholder
for those benchmarks that do not recognize this macro.
Peak Portability Flags
-
- EXTRA_PORTABILITY
-
This option is used to indicate that the host system's integers are 32-bits
wide, and longs and pointers are 64-bits wide. Not all benchmarks
recognize this macro, but the preferred practice for data model selection
applies the flags to all benchmarks; this flag description is a placeholder
for those benchmarks that do not recognize this macro.
-
- EXTRA_PORTABILITY
-
This option is used to indicate that the host system's integers are 32-bits
wide, and longs and pointers are 64-bits wide. Not all benchmarks
recognize this macro, but the preferred practice for data model selection
applies the flags to all benchmarks; this flag description is a placeholder
for those benchmarks that do not recognize this macro.
-
- EXTRA_PORTABILITY
-
This option is used to indicate that the host system's integers are 32-bits
wide, and longs and pointers are 64-bits wide. Not all benchmarks
recognize this macro, but the preferred practice for data model selection
applies the flags to all benchmarks; this flag description is a placeholder
for those benchmarks that do not recognize this macro.
-
-
- EXTRA_PORTABILITY
-
This option is used to indicate that the host system's integers are 32-bits
wide, and longs and pointers are 64-bits wide. Not all benchmarks
recognize this macro, but the preferred practice for data model selection
applies the flags to all benchmarks; this flag description is a placeholder
for those benchmarks that do not recognize this macro.
-
- EXTRA_PORTABILITY
-
This option is used to indicate that the host system's integers are 32-bits
wide, and longs and pointers are 64-bits wide. Not all benchmarks
recognize this macro, but the preferred practice for data model selection
applies the flags to all benchmarks; this flag description is a placeholder
for those benchmarks that do not recognize this macro.
-
- EXTRA_PORTABILITY
-
This option is used to indicate that the host system's integers are 32-bits
wide, and longs and pointers are 64-bits wide. Not all benchmarks
recognize this macro, but the preferred practice for data model selection
applies the flags to all benchmarks; this flag description is a placeholder
for those benchmarks that do not recognize this macro.
-
- EXTRA_PORTABILITY
-
This option is used to indicate that the host system's integers are 32-bits
wide, and longs and pointers are 64-bits wide. Not all benchmarks
recognize this macro, but the preferred practice for data model selection
applies the flags to all benchmarks; this flag description is a placeholder
for those benchmarks that do not recognize this macro.
-
- EXTRA_PORTABILITY
-
This option is used to indicate that the host system's integers are 32-bits
wide, and longs and pointers are 64-bits wide. Not all benchmarks
recognize this macro, but the preferred practice for data model selection
applies the flags to all benchmarks; this flag description is a placeholder
for those benchmarks that do not recognize this macro.
-
- EXTRA_PORTABILITY
-
This option is used to indicate that the host system's integers are 32-bits
wide, and longs and pointers are 64-bits wide. Not all benchmarks
recognize this macro, but the preferred practice for data model selection
applies the flags to all benchmarks; this flag description is a placeholder
for those benchmarks that do not recognize this macro.
Base Optimization Flags
-
- -m64
- CC, LD
-
Generate code for a 64-bit environment. The 64-bit environment sets int to 32 bits and long and
pointer to 64 bits and generates code for AMD's x86-64 architecture. The compiler generates AMD64, INTEL64,
x86-64 64-bit ABI. The default on a 32-bit host is 32-bit ABI. The default on a 64-bit host is 64-bit ABI if the target
platform specified is 64-bit, otherwise the default is 32-bit.
-
-
-
-
-
-
-
-
-
- -O3
- COPTIMIZE
-
Like -O2, except that it enables optimizations that take longer to perform or that may generate larger code (in
an attempt to make the program run faster).
If multiple "O" options are used, with or without level numbers, the last such option is the one that is effective.
- Includes:
-
- -march=znver3
- COPTIMIZE
-
Specify that Clang should generate code for a specific processor family member and later. For example, if you specify
-march=znver1, the compiler is allowed to generate instructions that are valid on AMD Zen processors, but
which may not exist on earlier products.
-
-
- -ffast-math
- COPTIMIZE
-
Enables a range of optimizations that provide faster, though sometimes less precise, mathematical operations that may
not conform to the IEEE-754 specifications. When this option is specified, the __STDC_IEC_559__ macro is
ignored even if set by the system headers.
-
-
- -fstruct-layout=5
- COPTIMIZE
-
Analyzes the whole program to determine if the structures in the code can be peeled and if pointer or integer fields in
the structure can be compressed. If feasible, this optimization transforms the code to enable these improvements. This
transformation is likely to improve cache utilization and memory bandwidth. This, in turn, is expected to improve the
scalability of programs executed on multiple cores.
This is effective only under -flto as whole program analysis is required to perform this optimization. You can choose
different levels of aggressiveness with which this optimization can be applied to your application with 1 being the least
aggressive and 7 being the most aggressive level.
Possible values:
- fstruct-layout=1: enables structure peeling.
- fstruct-layout=2: enables structure peeling and selectively compresses self-referential pointers in these
structures to 32-bit pointers wherever safe.
- fstruct-layout=3: enables structure peeling and selectively compresses self-referential pointers in these
structures to 16-bit pointers wherever safe.
- fstruct-layout=4: enables structure peeling, pointer compression as in level 2 and further enables
compression of structure fields which are of integer type. This is performed under a strict safety check.
- fstruct-layout=5: enables structure peeling, pointer compression as in level 3 and further enables compression
of structure fields which are of integer type. This is performed under a strict safety check.
- fstruct-layout=6: enables structure peeling, pointer compression as in level 2 and further enables compression
of structure fields which are of type 64-bit 'signed int' or 'unsigned Int'. The user needs to ensure that the values
assigned to 64-bit 'signed int' fields are in range -(2^31 - 1) to +(2^31 - 1) and 64-bit 'unsigned int' fields are in
range 0 to +(2^31 - 1), otherwise, incorrect results may be obtained. This compression is performed without considering
any safety analysis and so the user needs to ensure the safety based on the program compiled.
- fstruct-layout=7: enables structure peeling, pointer compression as in level 3 and further enables compression
of structure fields which are of type 64-bit 'signed int' or 'unsigned Int'. The user needs to ensure that the values
assigned to 64-bit 'signed int' fields are in range -(2^31 - 1) to +(2^31 - 1) and 64-bit 'unsigned int' field are in
range 0 to +(2^31 - 1) , otherwise, incorrect results may be obtained. This compression is performed without considering
any safety analysis and so the user needs to ensure the safety based on the program compiled.
Note:
fstruct-layout=4 and fstruct-layout=5 are derived from fstruct-layout=2 and fstruct-layout=3 respectively with the added
feature of safe compression of integer fields in structures. Going from fstruct-layout=4 to fstruct-layout=5 may result
in higher performance if the pointer values are such that the pointers can be compressed to 16-bits.
fstruct-layout=6 and fstruct-layout=7 are derived from fstruct-layout=2 and fstruct-layout=3 respectively with the added
feature of compression of integer fields in structures. These are similar to fstruct-layout=4 and fstruct-layout=5, but
here, the integer fields of the structures are always compressed from 64-bits to 32-bits without any safety guarantee.
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- -m64
- CXX, LD
-
Generate code for a 64-bit environment. The 64-bit environment sets int to 32 bits and long and
pointer to 64 bits and generates code for AMD's x86-64 architecture. The compiler generates AMD64, INTEL64,
x86-64 64-bit ABI. The default on a 32-bit host is 32-bit ABI. The default on a 64-bit host is 64-bit ABI if the target
platform specified is 64-bit, otherwise the default is 32-bit.
-
-
-
-
-
-
-
-
-
- -O3
- CXXOPTIMIZE
-
Like -O2, except that it enables optimizations that take longer to perform or that may generate larger code (in
an attempt to make the program run faster).
If multiple "O" options are used, with or without level numbers, the last such option is the one that is effective.
- Includes:
-
- -march=znver3
- CXXOPTIMIZE
-
Specify that Clang should generate code for a specific processor family member and later. For example, if you specify
-march=znver1, the compiler is allowed to generate instructions that are valid on AMD Zen processors, but
which may not exist on earlier products.
-
-
- -ffast-math
- CXXOPTIMIZE
-
Enables a range of optimizations that provide faster, though sometimes less precise, mathematical operations that may
not conform to the IEEE-754 specifications. When this option is specified, the __STDC_IEC_559__ macro is
ignored even if set by the system headers.
-
-
-
-
-
-
-
-
- -mllvm -aggressive-loop-unswitch
- CXXOPTIMIZE
-
This option enables aggressive loop unswitching heuristic (including -enable-partial-unswitch)
based on the usage of the branch conditional values. Loop unswitching leads to code-bloat. Code-bloat can be
minimized if the hoisted condition is executed more often. This heuristic prioritizes the conditions based
on the number of times they are used within the loop. The heuristic can be controlled with the following options:
- -unswitch-identical-branches-min-count=<n>
Enables unswitching of a loop with respect to a branch conditional value (B), where B appears in at least
<n> compares in the loop. This option is enabled with -aggressive-loop-unswitch. Default value is 3.
Usage: -mllvm -aggressive-loop-unswitch -mllvm -unswitch-identical-branches-min-count=<n>
where n is a positive integer and lower value of <n> facilitates more unswitching
- -unswitch-identical-branches-max-count=<n>
Enables unswitching of a loop with respect to a branch conditional value (B), where B appears in at most
<n> compares in the loop. This option is enabled with -aggressive-loop-unswitch. Default value is 6.
Usage: -mllvm -aggressive-loop-unswitch -mllvm -unswitch-identical-branches-max-count=<n>
where n is a positive integer and higher value of <n> facilitates more unswitching
Note: These options may facilitate more unswitching in some of the workloads. Since loop-unswitching inherently leads
to code bloat, facilitating more unswitching may significantly increase the code size and hence may also lead to longer
compilation times.
- Includes:
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- -m64
- FC, LD
-
Generate code for a 64-bit environment. The 64-bit environment sets int to 32 bits and long and
pointer to 64 bits and generates code for AMD's x86-64 architecture. The compiler generates AMD64, INTEL64,
x86-64 64-bit ABI. The default on a 32-bit host is 32-bit ABI. The default on a 64-bit host is 64-bit ABI if the target
platform specified is 64-bit, otherwise the default is 32-bit.
-
-
-
-
-
-
-
-
-
-
- -O3
- FOPTIMIZE
-
Like -O2, except that it enables optimizations that take longer to perform or that may generate larger code (in
an attempt to make the program run faster).
If multiple "O" options are used, with or without level numbers, the last such option is the one that is effective.
- Includes:
-
- -march=znver3
- FOPTIMIZE
-
Specify that Clang should generate code for a specific processor family member and later. For example, if you specify
-march=znver1, the compiler is allowed to generate instructions that are valid on AMD Zen processors, but
which may not exist on earlier products.
-
-
- -ffast-math
- FOPTIMIZE
-
Enables a range of optimizations that provide faster, though sometimes less precise, mathematical operations that may
not conform to the IEEE-754 specifications. When this option is specified, the __STDC_IEC_559__ macro is
ignored even if set by the system headers.
-
-
-
-
-
-
-
-
-
-
-
-
Peak Optimization Flags
-
- -m64
- CC, LD
-
Generate code for a 64-bit environment. The 64-bit environment sets int to 32 bits and long and
pointer to 64 bits and generates code for AMD's x86-64 architecture. The compiler generates AMD64, INTEL64,
x86-64 64-bit ABI. The default on a 32-bit host is 32-bit ABI. The default on a 64-bit host is 64-bit ABI if the target
platform specified is 64-bit, otherwise the default is 32-bit.
-
-
-
-
-
-
-
-
-
- -march=znver3
- COPTIMIZE
-
Specify that Clang should generate code for a specific processor family member and later. For example, if you specify
-march=znver1, the compiler is allowed to generate instructions that are valid on AMD Zen processors, but
which may not exist on earlier products.
-
-
- -ffast-math
- COPTIMIZE
-
Enables a range of optimizations that provide faster, though sometimes less precise, mathematical operations that may
not conform to the IEEE-754 specifications. When this option is specified, the __STDC_IEC_559__ macro is
ignored even if set by the system headers.
-
-
- -fstruct-layout=5
- COPTIMIZE
-
Analyzes the whole program to determine if the structures in the code can be peeled and if pointer or integer fields in
the structure can be compressed. If feasible, this optimization transforms the code to enable these improvements. This
transformation is likely to improve cache utilization and memory bandwidth. This, in turn, is expected to improve the
scalability of programs executed on multiple cores.
This is effective only under -flto as whole program analysis is required to perform this optimization. You can choose
different levels of aggressiveness with which this optimization can be applied to your application with 1 being the least
aggressive and 7 being the most aggressive level.
Possible values:
- fstruct-layout=1: enables structure peeling.
- fstruct-layout=2: enables structure peeling and selectively compresses self-referential pointers in these
structures to 32-bit pointers wherever safe.
- fstruct-layout=3: enables structure peeling and selectively compresses self-referential pointers in these
structures to 16-bit pointers wherever safe.
- fstruct-layout=4: enables structure peeling, pointer compression as in level 2 and further enables
compression of structure fields which are of integer type. This is performed under a strict safety check.
- fstruct-layout=5: enables structure peeling, pointer compression as in level 3 and further enables compression
of structure fields which are of integer type. This is performed under a strict safety check.
- fstruct-layout=6: enables structure peeling, pointer compression as in level 2 and further enables compression
of structure fields which are of type 64-bit 'signed int' or 'unsigned Int'. The user needs to ensure that the values
assigned to 64-bit 'signed int' fields are in range -(2^31 - 1) to +(2^31 - 1) and 64-bit 'unsigned int' fields are in
range 0 to +(2^31 - 1), otherwise, incorrect results may be obtained. This compression is performed without considering
any safety analysis and so the user needs to ensure the safety based on the program compiled.
- fstruct-layout=7: enables structure peeling, pointer compression as in level 3 and further enables compression
of structure fields which are of type 64-bit 'signed int' or 'unsigned Int'. The user needs to ensure that the values
assigned to 64-bit 'signed int' fields are in range -(2^31 - 1) to +(2^31 - 1) and 64-bit 'unsigned int' field are in
range 0 to +(2^31 - 1) , otherwise, incorrect results may be obtained. This compression is performed without considering
any safety analysis and so the user needs to ensure the safety based on the program compiled.
Note:
fstruct-layout=4 and fstruct-layout=5 are derived from fstruct-layout=2 and fstruct-layout=3 respectively with the added
feature of safe compression of integer fields in structures. Going from fstruct-layout=4 to fstruct-layout=5 may result
in higher performance if the pointer values are such that the pointers can be compressed to 16-bits.
fstruct-layout=6 and fstruct-layout=7 are derived from fstruct-layout=2 and fstruct-layout=3 respectively with the added
feature of compression of integer fields in structures. These are similar to fstruct-layout=4 and fstruct-layout=5, but
here, the integer fields of the structures are always compressed from 64-bits to 32-bits without any safety guarantee.
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
Implicitly Included Flags
This section contains descriptions of flags that were included implicitly
by other flags, but which do not have a permanent home at SPEC.
-
- -O2
-
Moderate level of optimization which enables most optimizations. This is the default when no "-O" option is
specified, or if no value is specified (i.e. "-O").
If multiple "O" options are used, with or without level numbers, the last such option is the one that is effective.
- Includes:
-
- -O1
-
Somewhere between -O0 and -O2.
If multiple "O" options are used, with or without level numbers, the last such option is the one that is effective.
-
-
- -O3
-
Like -O2, except that it enables optimizations that take longer to perform or that may generate larger code (in
an attempt to make the program run faster).
If multiple "O" options are used, with or without level numbers, the last such option is the one that is effective.
- Includes:
Using numactl to bind processes and memory to cores
For multi-copy runs or single copy runs on systems with multiple sockets, it is advantageous to bind a process to a
particular core. Otherwise, the OS may arbitrarily move your process from one core to another. This can affect
performance. To help, SPEC allows the use of a "submit" command where users can specify a utility to use to bind
processes. We have found the utility 'numactl' to be the best choice.
numactl runs processes with a specific NUMA scheduling or memory placement policy. The policy is set for a
command and inherited by all of its children. The numactl flag "--physcpubind" specifies
which core(s) to bind the process. "-l" instructs numactl to keep a process's memory on the
local node while "-m" specifies which node(s) to place a process's memory. For full details on using
numactl, please refer to your Linux documentation, 'man numactl'
Note that some older versions of numactl incorrectly interpret application arguments as its own. For
example, with the command "numactl --physcpubind=0 -l a.out -m a", numactl will interpret
a.out's "-m" option as its own "-m" option. To work around this problem, we put
the command to be run in a shell script and then run the shell script using numactl. For example:
"echo 'a.out -m a' > run.sh ; numactl --physcpubind=0 bash run.sh"
Transparent Huge Pages (THP)
THP is an abstraction layer that automates most aspects of creating, managing, and using huge pages. THP is designed
to hide much of the complexity in using huge pages from system administrators and developers, as normal huge pages
must be assigned at boot time, can be difficult to manage manually, and often require significant changes to code in
order to be used effectively. Most recent Linux OS releases have THP enabled by default.
ulimit -s <n>
Sets the stack size to n kbytes, or unlimited to allow the stack size to grow without limit.
ulimit -l <n>
Sets the maximum size of memory that may be locked into physical memory.
powersave -f (on SuSE)
Makes the powersave daemon set the CPUs to the highest supported frequency.
/etc/init.d/cpuspeed stop (on Red Hat)
Disables the cpu frequency scaling program in order to set the CPUs to the highest supported frequency.
LD_LIBRARY_PATH
An environment variable that indicates the location in the filesystem of bundled libraries to use when running the
benchmark binaries.
kernel/randomize_va_space
This option can be used to select the type of process address space randomization that is used in the system, for
architectures that support this feature.
- 0 - Turn the process address space randomization off. This is the default for architectures that do not support
this feature anyway, and kernels that are booted with the "
norandmaps" parameter.
- 1 - Make the addresses of mmap base, stack and VDSO page randomized. This, among other things, implies that
shared libraries will be loaded to random addresses. Also for PIE-linked binaries, the location of code start is
randomized. This is the default if the
CONFIG_COMPAT_BRK option is enabled.
- 2 - Additionally enable heap randomization. This is the default if
CONFIG_COMPAT_BRK is
disabled.
MALLOC_CONF
An environment variable set to tune the jemalloc allocation strategy during the execution of the binaries. This
environment variable setting is not needed when building the binaries on the system under test.
PGHPF_ZMEM
An environment variable used to initialize the allocated memory. Setting PGHPF_ZMEM to "Yes" has the effect of
initializing all allocated memory to zero.
GOMP_CPU_AFFINITY
This environment variable is used to set the thread affinity for threads spawned by OpenMP.
OMP_DYNAMIC
This environment variable is defined as part of the OpenMP standard.
Setting it to "false" prevents the OpenMP runtime from dynamically adjusting the number of threads to use for parallel
execution.
For more information, see chapter 4 ("Environment Variables") in the
OpenMP 4.5 Specification.
OMP_SCHEDULE
This environment variable is defined as part of the OpenMP standard.
Setting it to "static" causes loop iterations to be assigned to threads in round-robin fashion in the order of the thread
number.
For more information, see chapter 4 ("Environment Variables") in the
OpenMP 4.5 Specification.
OMP_STACKSIZE
This environment variable is defined as part of the OpenMP standard and controls the size of the stack for threads created
by OpenMP.
For more information, see chapter 4 ("Environment Variables") in the
OpenMP 4.5 Specification.
OMP_THREAD_LIMIT
This environment variable is defined as part of the OpenMP standard and limits the maximum number of OpenMP threads that
can be created.
For more information, see chapter 4 ("Environment Variables") in the
OpenMP 4.5 Specification.
Operating System (OS) Application/Service Tuning:
The following OS tunes could've been applied to better optimize performance of some areas of the system:
- ulimit: Used to set user limits of system-wide resources. Provides control over resources available to the shell and processes started by it. Some common ulimit commands may include:
- ulimit -s [n | unlimited]: Set the stack size to n kbytes, or unlimited to allow the stack size to grow without limit.
- ulimit -l (number): Set the maximum size that can be locked into memory.
- Performance Governors (Linux): In-kernel CPU frequency governors are pre-configured power schemes for the CPU. The CPUfreq governors use P-states to change frequencies and lower power consumption. The dynamic governors can switch between CPU frequencies, based on CPU utilization to allow for power savings while not sacrificing performance. To set the governor, use the following commmand: "cpupower frequency-set -r -g {desired_governor}"
- Disabling Linux services: Certain Linux services may be disabled to minimize tasks that may consume CPU cycles.
- irqbalance: Disabled through "service irqbalance stop". Depending on the workload involved, the irqbalance service reassigns various IRQ's to system CPUs. Though this service might help in some situations, disabling it can also help environments which need to minimize or eliminate latency to more quickly respond to events.
- tuned-adm: The tuned-adm tool is a commandline interface for switching between different tuning profiles available to the tuned tuning daemon available in supported Linux distros. The default configuration file is located in /etc/tuned.conf and the supported profiles can be found in /etc/tune-profiles. Some profiles that may be available by default include: default, desktop-powersave, server-powersave, laptop-ac-powersave, laptop-battery-powersave, spindown-disk, throughput-performance, latency-performance, enterprise-storage. To set a profile, one can issue the command "tuned-adm profile (profile_name)". Here are details about relevant profiles:
- throughput-performance: Server profile for typical throughput tuning. This profile disables tuned and ktune power saving features, enables sysctl settings that may improve disk and network IO throughput performance, switches to the deadline scheduler, and sets the CPU governor to performance.
- latency-performance: Server profile for typical latency tuning. This profile disables tuned and ktune power saving features, enables the deadline IO scheduler, and sets the CPU governor to performance.
- enterprise-storage: Server profile to high disk throughput tuning. This profile disables tuned and ktune power saving features, enables the deadline IO scheduler, enables hugepages and disables disk barriers, increases disk readahead values, and sets the CPU governor to performance
OS Kernel Parameter Tuning:
The following Linux Kernel parameters were tuned to better optimize performance of some areas of the system:
- dirty_background_ratio: Set through "echo 40 > /proc/sys/vm/dirty_background_ratio". This setting can help Linux disk caching and performance by setting the percentage of system memory that can be filled with dirty pages.
- dirty_ratio: Set through "echo 8 > /proc/sys/vm/dirty_ratio". This setting is the absolute maximum amount of system memory that can be filled with dirty pages before everything must get committed to disk.
- ksm/sleep_millisecs: Set through "echo 200 > /sys/kernel/mm/ksm/sleep_millisecs". This setting controls how many milliseconds the ksmd (KSM daemon) should sleep before the next scan.
- khugepaged/scan_sleep_millisecs: Set through "echo 50000 > /sys/kernel/mm/transparent_hugepage/khugepaged/scan_sleep_millisecs". This setting controls how many milliseconds to wait in khugepaged is there is a hugepage allocation failure to throttle the next allocation attempt.
- swappiness: The swappiness value can range from 1 to 100. A value of 100 will cause the kernel to swap out inactive processes frequently in favor of file system performance, resulting in large disk cache sizes. A value of 1 tells the kernel to only swap processes to disk if absolutely necessary. This can be set through a command like "echo 1 > /proc/sys/vm/swappiness"
- numa_balancing: Disabled through "echo 0 > /proc/sys/kernel/numa_balancing". This feature will automatically migrate data on demand so memory nodes are aligned to the local CPU that is accessing data. Depending on the workload involved, enabling this can boost the performance if the workload performs well on NUMA hardware. If the workload is statically set to balance between nodes, then this service may not provide a benefit.
- Zone Reclaim Mode: Zone reclaim allows the reclaiming of pages from a zone if the number of free pages falls below a watermark even if other zones still have enough pages available. Reclaiming a page can be more beneficial than taking the performance penalties that are associated with allocating a page on a remote zone, especially for NUMA machines. To tell the kernel to free local node memory rather than grabbing free memory from remote nodes, use a command like "echo 1 > /proc/sys/vm/zone_reclaim_mode"
- Free the file system page cache: The command "echo 1> /proc/sys/vm/drop_caches" is used to free up the filesystem page cache.
- kernel/randomize_va_space, also known as ASLR (Address Space Layout Randomization): This setting can be used to select the type of process address space randomization. Defaults differ based on whether the architecture supports ASLR, whether the kernel was built with the CONFIG_COMPAT_BRK option or not, or the kernel boot options used. Possible settings:
- 0: Turn process address space randomization off.
- 1: Randomize addresses of mmap base, stack, and VDSO pages.
- 2: Additionally randomize the heap. (This is probably the default.)
- Disabling ASLR can make process execution more deterministic and runtimes more consistent. For more information see the randomize_va_space entry in the Linux sysctl documentation.
- Transparent Hugepages (THP): THP is an abstraction layer that automates most aspects of creating, managing, and using huge pages. It is designed to hide much of the complexity in using huge pages from system administrators and developers. Huge pages increase the memory page size from 4 kilobytes to 2 megabytes. This provides significant performance advantages on systems with highly contended resources and large memory workloads. If memory utilization is too high or memory is badly fragmented which prevents hugepages being allocated, the kernel will assign smaller 4k pages instead. Most recent Linux OS releases have THP enabled by default. THP usage is controlled by the sysfs setting /sys/kernel/mm/transparent_hugepage/enabled. Possible values:
- never: entirely disable THP usage.
- madvise: enable THP usage only inside regions marked MADV_HUGEPAGE using madvise(3).
- always: enable THP usage system-wide. This is the default.
- THP creation is controlled by the sysfs setting /sys/kernel/mm/transparent_hugepage/defrag. Possible values:
- never: if no THP are available to satisfy a request, do not attempt to make any.
- defer: an allocation requesting THP when none are available get normal pages while requesting THP creation in the background.
- defer+madvise: acts like "always", but only for allocations in regions marked MADV_HUGEPAGE using madvise(3); for all other regions it's like "defer".
- madvise: acts like "always", but only for allocations in regions marked MADV_HUGEPAGE using madvise(3). This is the default.
- always: an allocation requesting THP when none are available will stall until some are made.
- An application that "always" requests THP often can benefit from waiting for an allocation until those huge pages can be assembled. For more information see the Linux transparent hugepage documentation.
Linux Huge Page settings:
If one prefers not to use Transparent Hugepages, one can always setup Huge Pages by following the below steps:
- Create a mount point for the huge pages: "mkdir /mnt/hugepages"
- The huge page file system needs to be mounted when the systems reboots. Add the following to a system boot configuration file before any services are started: "mount -t hugetlbfs nodev /mnt/hugepages"
- Set vm/nr_hugepages=N in your /etc/sysctl.conf file where N is the maximum number of pages the system may allocate.
- Reboot to have the changes take effect.
Note that further information about huge pages may be found in your Linux documentation file: /usr/src/linux/Documentation/vm/hugetlbpage.txt
Environment Variables:
The following Linux envrionment variables that could've possibly been tuned to better optimize performance of some areas of the system:
- GOMP_CPU_AFFINITY: Used to bind threads to specific CPUs. The variable should contain a space-separated or comma-separated list of CPUs. This list may contain different kinds of entries: either single CPU numbers in any order, a range of CPUs (M-N) or a range with some stride (M-N:S). CPU numbers are zero based. For example, GOMP_CPU_AFFINITY="0 3 1-2 4-15:2" will bind the initial thread to CPU 0, the second to CPU 3, the third to CPU 1, the fourth to CPU 2, the fifth to CPU 4, the sixth through tenth to CPUs 6, 8, 10, 12, and 14 respectively and then start assigning back from the beginning of the list. GOMP_CPU_AFFINITY=0 binds all threads to CPU 0. There is no libgomp library routine to determine whether a CPU affinity specification is in effect. As a workaround, language-specific library functions, e.g., getenv in C or GET_ENVIRONMENT_VARIABLE in Fortran, may be used to query the setting of the GOMP_CPU_AFFINITY environment variable. A defined CPU affinity on startup cannot be changed or disabled during the runtime of the application. If both GOMP_CPU_AFFINITY and OMP_PROC_BIND are set, OMP_PROC_BIND has a higher precedence. If neither has been set and OMP_PROC_BIND is unset, or when OMP_PROC_BIND is set to FALSE, the host system will handle the assignment of threads to CPUs.
- OMP_DYNAMIC: Dynamic adjustment of threads. Enable or disable the dynamic adjustment of the number of threads within a team. The value of this environment variable shall be TRUE or FALSE. If undefined, dynamic adjustment is disabled by default.
- OMP_SCHEDULE: How threads are scheduled. Allows to specify schedule type and chunk size. The value of the variable shall have the form: type[,chunk] where type is one of static, dynamic or guided. The optional chunk size shall be a positive integer. If undefined, dynamic scheduling and a chunk size of 1 is used.
- OMP_THREAD_LIMIT: Set the maximum number of threads. Specifies the number of threads to use for the whole program. The value of this variable shall be a positive integer. If undefined, the number of threads is not limited.
- MALLOC_CONF: This environment variable affects the execution of the allocation functions. If the environment variable MALLOC_CONF is set, the characters it contains will be interpreted as options.
Firmware Settings:
One or more of the following settings may have been set. If so, the "Platform Notes" section of the report will say so; and you can read below to find out more about what these settings mean.
- AMD SMT Option (Default = Enabled): This feature allows enabling or disabling of logical processor cores on processors supporting AMD SMT. When enabled, each physical processor core operates as two logical processor cores. When disabled, each physical core operates as only one logical processor core. Enabling this option can improve overall performance for applications that benefit from a higher processor core count.
- Thermal Configuration (Default = Optimal Cooling): This feature allows the user to select the fan cooling solution for the system. Values for this BIOS option can be:
- Optimal Cooling: Provides the most efficient solution by configuring fan speeds to the minimum required to provide adequate cooling.
- Increased Cooling: Will run fans at higher speeds to provide additional cooling. Increased Cooling should be selected when non-HPE storage controllers are cabled to the embedded hard drive cage, or if the system is experiencing thermal issues that cannot be resolved in another manner.
- Maximum Cooling: Will provide the maximum cooling available by this platform.
- Determinism Control (Default = Auto): This option allows the user to choose between an Auto and Manual mode for Determinism Control. Values for this BIOS option can be:
- Auto: The system will decide what Performance Determinism setting to use. Auto also uses the processor fused values for Determinism.
- Manual: Allows the user to select either Power Deterministic or Performance Deterministic as described below.
- Performance Determinism (Default = Performance Deterministic): This option allows the user to configure the AMD processor Determinism setting for AGESA ("AMD Generic Encapsulated Software Architecture", a bootstrap protocol by which system devices on AMD64-architecture mainboards are initialized) control or BIOS control. Values for this BIOS option can be:
- Performance Deterministic: This option allows for AGESA control and provides predictable (capped) performance across all processors of the same type in the system. This could cause the system to run at or report lower than expected power values.
- Power Deterministic: This option will not enable performance deterministic control and allows for BIOS control. This option will maximize performance with power limits defined by the system design and operate as close to the thermal design point (TDP) as possible.
- Package Power Limit Control Mode (Default = Auto): This is a per Processor Power Limit value applicable for all populated processors in the system. This can be set to limit the processor power to a certain value. Values for this BIOS option can be:
- Auto: Uses the default processor value.
- Manual: Lets the user set a power limit and exposes Package Power Limit value field that a user can enter a number into. If a number is entered higher than what the processor and/or system supports, the BIOS will limit the power to the max possible value.
- Memory Patrol Scrubbing (Default = Enabled): This option allows for correction of soft memory errors. Over the length of system runtime, the risk of producing multi-bit and uncorrected errors is reduced with this option. Values for this BIOS setting can be:
- Enabled: Correction of soft memory errors can occur during runtime.
- Disabled: Soft memory error correction is turned off during runtime.
- Workload Profile (Default = General Power Efficient Compute): This option allows a user to choose a workload profile that best fits the user`s needs. The workload profiles control many power and performance settings that are relevant to general workload areas. Values for this BIOS option can be:
- General Power Efficient Compute, General Peak Frequency Compute, General Throughput Compute, Virtualization - Power Efficient, Virtualization - Max Performance, Low Latency, Mission Critical, Transaction Application Processing, High Performance Compute (HPC), Decision Support, Graphic Processing, I/O Throughput, and Custom.
- Power Regulator (Default = Static High Performance Mode): This option can only be configured if the Workload Profile is set to Custom. This feature allows the user to select the following Power Regulator support:
- Dynamic Power Savings Mode: This mode allows the automatic variation of the processor speed and power usage based on processor utilization, resulting a reduction in overall power consumption with little or no impact on performance. It does not require OS support.
- Static Low Power Mode: This mode reduces the processor speed and power usage and guarantees a lower maximum power usage for the system.
- Static High Performance Mode: This mode allows the processors to run in their maximum power/performance state at all times, regardless of the OS power management policy.
- OS Control Mode: This mode allows the processors to run in their maximum power/performance state at all times unless the OS enables a power management policy.
- Minimum Processor Idle Power Core C-State (Default = C6 State): This option can only be configured if the Workload Profile is set to Custom. This feature selects the processor's lowest idle power state (C-state) that the operating system uses. The higher the C-state, the lower the power usage of that idle state (C6 is the lowest power idle state supported by the processor). Values for this setting can be:
- C6 State: While in C6, the core PLLs are turned off, the core caches are flushed and the core state is saved to the Last Level Cache. Power Gates are used to reduce power consumption to close to zero. C6 is considered an inactive core.
- C1E State: C1E is defined as the enhanced halt state. While in C1E no instructions are being executed. C1E considered an active core.
- No C-states: No C-states is defined as C0, which is defined as the active state. While in C0, instructions are being executed by the core.
- C-State Efficiency mode (Default = Enabled): This option allows to adjust the frequency along with change in C-State. Enabling will monitor the workload and modulate the frequency of the core to maintain a high C0 residency. It also has latency and power benefits when the CPU is not 100% utilized. Values for this BIOS setting can be:
- XGMI Force Link Width (Default = Auto): This option forces the value of XGMI link width to a value set by the user. Performance improvement should be observed while setting the correct value. Values for this BIOS setting can be:
- Memory Patrol Scrubbing (Default = Enabled): This option allows for correction of soft memory errors. Over the length of system runtime, the risk of producing multi-bit and uncorrected errors is reduced with this option. Values for this BIOS setting can be:
- Enabled: Correction of soft memory errors can occur during runtime.
- Disabled: Soft memory error correction is turned off during runtime.
- NUMA memory domains per socket (Default = Auto): This option allows the user to divide the memory domains that each socket has into a certain number of NUMA memory domains for better memory bandwidth. Values for this BIOS setting can be:
- One memory domain per socket: Each processor socket will have one memory domain.
- Two memory domains per socket: Each processor socket will have two memory domains.
- Four memory domains per socket: Each processor socket will have four memory domains.
- Auto: The system will default to One memory domain per socket.
- Last-Level Cache (LLC) as NUMA Node (Default = Disabled): When enabled, this option allows the user to divide processor's cores into additional NUMA Nodes based on the L3 cache. Enabling this feature can increase performance for workloads that are NUMA aware and optimized. Values for this BIOS setting can be:
- Memory PStates (Default = Auto): This setting controls the power state support of the memory controllers. Values for this BIOS setting can be:
- Data Fabric C-State Enable (Default = Auto): Allows the Infinity Fabric to go into lower power states when idle, similar to CPU core C-States. There can be a delay changing back to full-power mode, causing latency jitter. In a low latency workload, or one core bursty I/O, one could disable this feature to achieve more performance with the tradeoff of higher power consumption. Values for this BIOS setting can be:
- Auto: Dynamically allow the Infinity Fabric to go to a lower-power state when the processor has entered C-States.
- Force Enabled: Force the Infinity Fabric to go to a lower-power state when the processor has entered C-States.
- Disabled: Do not allow the Infinity Fabric to go to a lower-power state when the processor has entered C-States.
- Infinity Fabric Power Management (Default = Enabled): When this feature is enabled, the EPYC processor will dynamically vary the clock frequency of the Infinity Fabric based on activity level. For NUMA optimized workloads, allowing the Infinity Fabric to run slower can lead to increased overall performance due to an increase in CPU boost. Disabling this feature may be necessary for latency sensitive workloads. Values for this BIOS setting can be:
- Disabled: Enable fixed Infinity Fabric P-State control
- Enabled: Dynamically switch Infinity Fabric P-State based on link usage.
- Infinity Fabric Performance State (Default = P0): This feature allows for customizing the performance state (P-State) of the Infinity Fabric when Infinity Fabric Power Management is disabled. The default is Auto when Infinity Fabric Power Management is Enabled. P0 is ideal for latency sensitive or I/O centric workloads. P3 is ideal for achieving best core boost frequencies. Values for this BIOS setting can be:
- P0: Will force the Infinity Fabric and memory controllers into full-power mode, eliminating latency jitter. Highest performing Infinity Fabric P-State.
- P1: Next highest performing Infinity Fabric P-State, after P0.
- P2: Next highest performing Infinity Fabric P-State, after P1.
- P3: Minimum Infinity Fabric P-State.
- L1 HW Prefetcher (Default = Enabled): Use this option to disable L1 Stream HW prefetch feature. In some cases (e.g. workloads that are random in nature) setting this option to disabled can improve performance. However, most workloads will benefit from this setting being enabled as it gathers data and keeps the core pipeline busy. Values for this BIOS setting can be:
- L2 HW Prefetcher (Default = Enabled): Use this option to disable L2 Stream HW prefetch feature. In some cases (e.g. workloads that are random in nature) setting this option to disabled can improve performance. However, most workloads will benefit from this setting being enabled as it gathers data and keeps the core pipeline busy. Values for this BIOS setting can be:
Last updated March 15, 2021.
For questions about the meanings of these flags, please contact the tester.
For other inquiries, please contact info@spec.org
Copyright 2017-2021 Standard Performance Evaluation Corporation
Tested with SPEC CPU2017 v1.1.5.
Report generated on 2021-04-27 16:20:30 by SPEC CPU2017 flags formatter v5178.
![[benchmark]](https://www.spec.org/auto/cpu2017/images/benchmark.png)