CPU2017 Flag Description
Hewlett Packard Enterprise ProLiant DL380 Gen10 (2.60 GHz, Intel Xeon Gold 6240M)
Test sponsored by HPE
Copyright © 2016 Intel Corporation. All Rights Reserved.
Base Portability Flags
-
- -DSPEC_LP64
![[suite]](https://www.spec.org/auto/cpu2017/images/suite.png)
- PORTABILITY
-
This option is used to indicate that the host system's integers are 32-bits
wide, and longs and pointers are 64-bits wide. Not all benchmarks
recognize this macro, but the preferred practice for data model selection
applies the flags to all benchmarks; this flag description is a placeholder
for those benchmarks that do not recognize this macro.
-
- -DSPEC_LP64
- PORTABILITY
-
This option is used to indicate that the host system's integers are 32-bits
wide, and longs and pointers are 64-bits wide. Not all benchmarks
recognize this macro, but the preferred practice for data model selection
applies the flags to all benchmarks; this flag description is a placeholder
for those benchmarks that do not recognize this macro.
-
- -DSPEC_LP64
- PORTABILITY
-
This option is used to indicate that the host system's integers are 32-bits
wide, and longs and pointers are 64-bits wide. Not all benchmarks
recognize this macro, but the preferred practice for data model selection
applies the flags to all benchmarks; this flag description is a placeholder
for those benchmarks that do not recognize this macro.
-
- -DSPEC_LP64
- PORTABILITY
-
This option is used to indicate that the host system's integers are 32-bits
wide, and longs and pointers are 64-bits wide. Not all benchmarks
recognize this macro, but the preferred practice for data model selection
applies the flags to all benchmarks; this flag description is a placeholder
for those benchmarks that do not recognize this macro.
-
-
- -convert big_endian
![[user]](https://www.spec.org/auto/cpu2017/images/user.png)
- FPORTABILITY
-
Specifies that the format will be big endian for INTEGER*1, INTEGER*2, INTEGER*4, or INTEGER*8, and big endian IEEE floating-point for REAL*4, REAL*8, REAL*16, COMPLEX*8,
COMPLEX*16, or COMPLEX*32.
-
- -DSPEC_LP64
- PORTABILITY
-
This option is used to indicate that the host system's integers are 32-bits
wide, and longs and pointers are 64-bits wide. Not all benchmarks
recognize this macro, but the preferred practice for data model selection
applies the flags to all benchmarks; this flag description is a placeholder
for those benchmarks that do not recognize this macro.
-
-
- -DSPEC_LP64
- PORTABILITY
-
This option is used to indicate that the host system's integers are 32-bits
wide, and longs and pointers are 64-bits wide. Not all benchmarks
recognize this macro, but the preferred practice for data model selection
applies the flags to all benchmarks; this flag description is a placeholder
for those benchmarks that do not recognize this macro.
-
-
- -convert big_endian
- FPORTABILITY
-
Specifies that the format will be big endian for INTEGER*1, INTEGER*2, INTEGER*4, or INTEGER*8, and big endian IEEE floating-point for REAL*4, REAL*8, REAL*16, COMPLEX*8,
COMPLEX*16, or COMPLEX*32.
-
- -assume byterecl
- FPORTABILITY
-
Specifies that the units for the OPEN statement RECL specifier (record length) value are in bytes for unformatted data files, not longwords (four-byte units). For formatted files, the RECL value is
always in bytes.
-
- -DSPEC_LP64
- PORTABILITY
-
This option is used to indicate that the host system's integers are 32-bits
wide, and longs and pointers are 64-bits wide. Not all benchmarks
recognize this macro, but the preferred practice for data model selection
applies the flags to all benchmarks; this flag description is a placeholder
for those benchmarks that do not recognize this macro.
-
- -DSPEC_LP64
- PORTABILITY
-
This option is used to indicate that the host system's integers are 32-bits
wide, and longs and pointers are 64-bits wide. Not all benchmarks
recognize this macro, but the preferred practice for data model selection
applies the flags to all benchmarks; this flag description is a placeholder
for those benchmarks that do not recognize this macro.
-
- -DSPEC_LP64
- PORTABILITY
-
This option is used to indicate that the host system's integers are 32-bits
wide, and longs and pointers are 64-bits wide. Not all benchmarks
recognize this macro, but the preferred practice for data model selection
applies the flags to all benchmarks; this flag description is a placeholder
for those benchmarks that do not recognize this macro.
-
- -DSPEC_LP64
- PORTABILITY
-
This option is used to indicate that the host system's integers are 32-bits
wide, and longs and pointers are 64-bits wide. Not all benchmarks
recognize this macro, but the preferred practice for data model selection
applies the flags to all benchmarks; this flag description is a placeholder
for those benchmarks that do not recognize this macro.
Base Optimization Flags
-
- -xCORE-AVX512
- COPTIMIZE
-
Code is optimized for Intel(R) processors with support for CORE-AVX512 instructions.
The resulting code may contain unconditional use of features that are not supported
on other processors. This option also enables new optimizations in addition to
Intel processor-specific optimizations including advanced data layout and code
restructuring optimizations to improve memory accesses for Intel processors.
Do not use this option if you are executing a program on a processor that
is not an Intel processor. If you use this option on a non-compatible processor
to compile the main program (in Fortran) or the function main() in C/C++, the
program will display a fatal run-time error if they are executed on unsupported
processors.
-
-
- -O3
- COPTIMIZE
-
Enables O2 optimizations plus more aggressive optimizations,
such as prefetching, scalar replacement, and loop and memory
access transformations. Enables optimizations for maximum speed,
such as:
- Loop unrolling, including instruction scheduling
- Code replication to eliminate branches
- Padding the size of certain power-of-two arrays to allow
more efficient cache use.
On IA-32 and Intel EM64T processors, when O3 is used with options
-ax or -x (Linux) or with options /Qax or /Qx (Windows), the compiler
performs more aggressive data dependency analysis than for O2, which
may result in longer compilation times.
The O3 optimizations may not cause higher performance unless loop and
memory access transformations take place. The optimizations may slow
down code in some cases compared to O2 optimizations.
The O3 option is recommended for applications that have loops that heavily
use floating-point calculations and process large data sets.
- Includes:
-
- -no-prec-div
- COPTIMIZE
-
(disable/enable[default] -prec-div)
-no-prec-div enables optimizations that give slightly less precise results
than full IEEE division.
When you specify -no-prec-div along with some optimizations, such as
-xN and -xB (Linux) or /QxN and /QxB (Windows),
the compiler may change floating-point division computations into
multiplication by the reciprocal of the denominator.
For example, A/B is computed as A * (1/B) to improve the speed of the
computation.
However, sometimes the value produced by this transformation is
not as accurate as full IEEE division. When it is important to have fully
precise IEEE division, do not use -no-prec-div.
This will enable the default -prec-div and the result will be more accurate,
with some loss of performance.
-
-
-
-
-
-
-
- -xCORE-AVX512
- FOPTIMIZE
-
Code is optimized for Intel(R) processors with support for CORE-AVX512 instructions.
The resulting code may contain unconditional use of features that are not supported
on other processors. This option also enables new optimizations in addition to
Intel processor-specific optimizations including advanced data layout and code
restructuring optimizations to improve memory accesses for Intel processors.
Do not use this option if you are executing a program on a processor that
is not an Intel processor. If you use this option on a non-compatible processor
to compile the main program (in Fortran) or the function main() in C/C++, the
program will display a fatal run-time error if they are executed on unsupported
processors.
-
-
- -O3
- FOPTIMIZE
-
Enables O2 optimizations plus more aggressive optimizations,
such as prefetching, scalar replacement, and loop and memory
access transformations. Enables optimizations for maximum speed,
such as:
- Loop unrolling, including instruction scheduling
- Code replication to eliminate branches
- Padding the size of certain power-of-two arrays to allow
more efficient cache use.
On IA-32 and Intel EM64T processors, when O3 is used with options
-ax or -x (Linux) or with options /Qax or /Qx (Windows), the compiler
performs more aggressive data dependency analysis than for O2, which
may result in longer compilation times.
The O3 optimizations may not cause higher performance unless loop and
memory access transformations take place. The optimizations may slow
down code in some cases compared to O2 optimizations.
The O3 option is recommended for applications that have loops that heavily
use floating-point calculations and process large data sets.
- Includes:
-
- -no-prec-div
- FOPTIMIZE
-
(disable/enable[default] -prec-div)
-no-prec-div enables optimizations that give slightly less precise results
than full IEEE division.
When you specify -no-prec-div along with some optimizations, such as
-xN and -xB (Linux) or /QxN and /QxB (Windows),
the compiler may change floating-point division computations into
multiplication by the reciprocal of the denominator.
For example, A/B is computed as A * (1/B) to improve the speed of the
computation.
However, sometimes the value produced by this transformation is
not as accurate as full IEEE division. When it is important to have fully
precise IEEE division, do not use -no-prec-div.
This will enable the default -prec-div and the result will be more accurate,
with some loss of performance.
-
-
-
-
-
- -nostandard-realloc-lhs
- EXTRA_FOPTIMIZE
-
Option standard-realloc-lhs (the default), tells the compiler that when the left-hand side of an assignment is an allocatable object, it should be reallocated to the shape of the
right-hand side of the assignment before the assignment occurs. This is the current Fortran Standard definition. This feature may cause extra overhead at run time. This option has
the same effect as option assume realloc_lhs.
If you specify nostandard-realloc-lhs, the compiler uses the old Fortran 2003 rules when interpreting assignment statements. The left-hand side is assumed to be allocated with the
correct shape to hold the right-hand side. If it is not, incorrect behavior will occur. This option has the same effect as option assume norealloc_lhs.
-
- -xCORE-AVX512
- COPTIMIZE, FOPTIMIZE
-
Code is optimized for Intel(R) processors with support for CORE-AVX512 instructions.
The resulting code may contain unconditional use of features that are not supported
on other processors. This option also enables new optimizations in addition to
Intel processor-specific optimizations including advanced data layout and code
restructuring optimizations to improve memory accesses for Intel processors.
Do not use this option if you are executing a program on a processor that
is not an Intel processor. If you use this option on a non-compatible processor
to compile the main program (in Fortran) or the function main() in C/C++, the
program will display a fatal run-time error if they are executed on unsupported
processors.
-
-
- -O3
- COPTIMIZE, FOPTIMIZE
-
Enables O2 optimizations plus more aggressive optimizations,
such as prefetching, scalar replacement, and loop and memory
access transformations. Enables optimizations for maximum speed,
such as:
- Loop unrolling, including instruction scheduling
- Code replication to eliminate branches
- Padding the size of certain power-of-two arrays to allow
more efficient cache use.
On IA-32 and Intel EM64T processors, when O3 is used with options
-ax or -x (Linux) or with options /Qax or /Qx (Windows), the compiler
performs more aggressive data dependency analysis than for O2, which
may result in longer compilation times.
The O3 optimizations may not cause higher performance unless loop and
memory access transformations take place. The optimizations may slow
down code in some cases compared to O2 optimizations.
The O3 option is recommended for applications that have loops that heavily
use floating-point calculations and process large data sets.
- Includes:
-
- -no-prec-div
- COPTIMIZE, FOPTIMIZE
-
(disable/enable[default] -prec-div)
-no-prec-div enables optimizations that give slightly less precise results
than full IEEE division.
When you specify -no-prec-div along with some optimizations, such as
-xN and -xB (Linux) or /QxN and /QxB (Windows),
the compiler may change floating-point division computations into
multiplication by the reciprocal of the denominator.
For example, A/B is computed as A * (1/B) to improve the speed of the
computation.
However, sometimes the value produced by this transformation is
not as accurate as full IEEE division. When it is important to have fully
precise IEEE division, do not use -no-prec-div.
This will enable the default -prec-div and the result will be more accurate,
with some loss of performance.
-
-
-
-
-
-
- -nostandard-realloc-lhs
- EXTRA_FOPTIMIZE
-
Option standard-realloc-lhs (the default), tells the compiler that when the left-hand side of an assignment is an allocatable object, it should be reallocated to the shape of the
right-hand side of the assignment before the assignment occurs. This is the current Fortran Standard definition. This feature may cause extra overhead at run time. This option has
the same effect as option assume realloc_lhs.
If you specify nostandard-realloc-lhs, the compiler uses the old Fortran 2003 rules when interpreting assignment statements. The left-hand side is assumed to be allocated with the
correct shape to hold the right-hand side. If it is not, incorrect behavior will occur. This option has the same effect as option assume norealloc_lhs.
-
- -xCORE-AVX512
- COPTIMIZE, CXXOPTIMIZE, FOPTIMIZE
-
Code is optimized for Intel(R) processors with support for CORE-AVX512 instructions.
The resulting code may contain unconditional use of features that are not supported
on other processors. This option also enables new optimizations in addition to
Intel processor-specific optimizations including advanced data layout and code
restructuring optimizations to improve memory accesses for Intel processors.
Do not use this option if you are executing a program on a processor that
is not an Intel processor. If you use this option on a non-compatible processor
to compile the main program (in Fortran) or the function main() in C/C++, the
program will display a fatal run-time error if they are executed on unsupported
processors.
-
-
- -O3
- COPTIMIZE, CXXOPTIMIZE, FOPTIMIZE
-
Enables O2 optimizations plus more aggressive optimizations,
such as prefetching, scalar replacement, and loop and memory
access transformations. Enables optimizations for maximum speed,
such as:
- Loop unrolling, including instruction scheduling
- Code replication to eliminate branches
- Padding the size of certain power-of-two arrays to allow
more efficient cache use.
On IA-32 and Intel EM64T processors, when O3 is used with options
-ax or -x (Linux) or with options /Qax or /Qx (Windows), the compiler
performs more aggressive data dependency analysis than for O2, which
may result in longer compilation times.
The O3 optimizations may not cause higher performance unless loop and
memory access transformations take place. The optimizations may slow
down code in some cases compared to O2 optimizations.
The O3 option is recommended for applications that have loops that heavily
use floating-point calculations and process large data sets.
- Includes:
-
- -no-prec-div
- COPTIMIZE, CXXOPTIMIZE, FOPTIMIZE
-
(disable/enable[default] -prec-div)
-no-prec-div enables optimizations that give slightly less precise results
than full IEEE division.
When you specify -no-prec-div along with some optimizations, such as
-xN and -xB (Linux) or /QxN and /QxB (Windows),
the compiler may change floating-point division computations into
multiplication by the reciprocal of the denominator.
For example, A/B is computed as A * (1/B) to improve the speed of the
computation.
However, sometimes the value produced by this transformation is
not as accurate as full IEEE division. When it is important to have fully
precise IEEE division, do not use -no-prec-div.
This will enable the default -prec-div and the result will be more accurate,
with some loss of performance.
-
-
-
-
- -qopenmp
- Yes
- COPTIMIZE, CXXOPTIMIZE, FOPTIMIZE
-
-
-
- -nostandard-realloc-lhs
- EXTRA_FOPTIMIZE
-
Option standard-realloc-lhs (the default), tells the compiler that when the left-hand side of an assignment is an allocatable object, it should be reallocated to the shape of the
right-hand side of the assignment before the assignment occurs. This is the current Fortran Standard definition. This feature may cause extra overhead at run time. This option has
the same effect as option assume realloc_lhs.
If you specify nostandard-realloc-lhs, the compiler uses the old Fortran 2003 rules when interpreting assignment statements. The left-hand side is assumed to be allocated with the
correct shape to hold the right-hand side. If it is not, incorrect behavior will occur. This option has the same effect as option assume norealloc_lhs.
Implicitly Included Flags
This section contains descriptions of flags that were included implicitly
by other flags, but which do not have a permanent home at SPEC.
-
- -O2
-
Enables optimizations for speed. This is the generally recommended
optimization level. This option also enables:
- Inlining of intrinsics
- Intra-file interprocedural optimizations, which include:
- inlining
- constant propagation
- forward substitution
- routine attribute propagation
- variable address-taken analysis
- dead static function elimination
- removal of unreferenced variables
- The following capabilities for performance gain:
- constant propagation
- copy propagation
- dead-code elimination
- global register allocation
- global instruction scheduling and control speculation
- loop unrolling
- optimized code selection
- partial redundancy elimination
- strength reduction/induction variable simplification
- variable renaming
- exception handling optimizations
- tail recursions
- peephole optimizations
- structure assignment lowering and optimizations
- dead store elimination
- Includes:
-
- -O1
-
Enables optimizations for speed and disables some optimizations that increase code size and affect speed.
To limit code size, this option:
- Enables global optimization; this includes data-flow analysis,
code motion, strength reduction and test replacement, split-lifetime
analysis, and instruction scheduling.
- Disables intrinsic recognition and intrinsics inlining.
The O1 option may improve performance for applications with very large
code size, many branches, and execution time not dominated by code within loops.
-O1 sets the following options:
-funroll-loops0, -fno-builtin, -mno-ieee-fp, -fomit-framepointer, -ffunction-sections, -ftz
- Includes:
-
-
-
-
-
-
- submit= MYMASK=`printf '0x%x' $((1<<$SPECCOPYNUM))`; /usr/bin/taskset $MYMASK $command
- When running multiple copies of benchmarks, the SPEC config file feature submit is used to cause individual jobs to be bound to
specific processors. This specific submit command, using taskset, is used for Linux64 systems without numactl.
Here is a brief guide to understanding the specific command which will be found in the config file:
- /usr/bin/taskset [options] [mask] [pid | command [arg] ... ]:
taskset is used to set or retreive the CPU affinity of a running
process given its PID or to launch a new COMMAND with a given CPU
affinity. The CPU affinity is represented as a bitmask, with the
lowest order bit corresponding to the first logical CPU and highest
order bit corresponding to the last logical CPU. When the taskset
returns, it is guaranteed that the given program has been scheduled
to a specific, legal CPU, as defined by the mask setting.
- [mask]: The bitmask (in hexadecimal) corresponding to a specific
SPECCOPYNUM. The specific example above, computes this mask value in the variable $MYMASK.
The value of this mask for the first copy of a
rate run will be 0x00000001, for the second copy of the rate will
be 0x00000002 etc. Thus, the first copy of the rate run will have a
CPU affinity of CPU0, the second copy will have the affinity CPU1
etc.
- $command: Program to be started, in this case, the benchmark instance to be started.
- submit= numactl --localalloc --physcpubind=$SPECCOPYNUM $command
- When running multiple copies of benchmarks, the SPEC config file feature submit is used to cause individual jobs to be bound to
specific processors. This specific submit command is used for Linux64 systems with support for numactl.
Here is a brief guide to understanding the specific command which will be found in the config file:
- syntax: numactl [--interleave=nodes] [--preferred=node] [--physcpubind=cpus] [--cpunodebind=nodes] [--membind=nodes] [--localalloc] command args ...
- numactl runs processes with a specific NUMA scheduling or memory placement policy. The policy is set for a command and inherited by all of its children.
- "--localalloc" instructs numactl to keep a process memory on the local node while "-m" specifies which node(s) to place a process memory.
- "--physcpubind" specifies which core(s) to bind the process. In this case, copy 0 is bound to processor 0 etc.
- For full details on using numactl, please refer to your Linux documentation, 'man numactl'
- numactl --interleave=all "runspec command"
- Launching a process with numactl --interleave=all sets the memory interleave policy so that memory will be allocated using round robin on nodes.
When memory cannot be allocated on the current interleave target fall back to other nodes.
- KMP_STACKSIZE
- Specify stack size to be allocated for each thread.
- KMP_AFFINITY
- Syntax: KMP_AFFINITY=[<modifier>,...]<type>[,<permute>][,<offset>]
The value for the environment variable KMP_AFFINITY affects how the threads from an auto-parallelized program are scheduled across processors.
It applies to binaries built with -qopenmp and -parallel (Linux and Mac OS X) or /Qopenmp and /Qparallel (Windows).
modifier:
granularity=fine Causes each OpenMP thread to be bound to a single thread context.
type:
compact Specifying compact assigns the OpenMP thread <n>+1 to a free thread context as close as possible to the thread context where the <n> OpenMP thread was placed.
scatter Specifying scatter distributes the threads as evenly as possible across the entire system.
permute: The permute specifier is an integer value controls which levels are most significant when sorting the machine topology map. A value for permute forces the mappings to make the specified number of most significant levels of the sort the least significant, and it inverts the order of significance.
offset: The offset specifier indicates the starting position for thread assignment.
Please see the Thread Affinity Interface article in the Intel Composer XE Documentation for more details.
- Example: KMP_AFFINITY=granularity=fine,scatter
Specifying granularity=fine selects the finest granularity level and causes each OpenMP or auto-par thread to be bound to a single thread context.
This ensures that there is only one thread per core on cores supporting HyperThreading Technology
Specifying scatter distributes the threads as evenly as possible across the entire system.
Hence a combination of these two options, will spread the threads evenly across sockets, with one thread per physical core.
- Example: KMP_AFFINITY=compact,1,0
Specifying compact will assign the n+1 thread to a free thread context as close as possible to thread n.
A default granularity=core is implied if no granularity is explicitly specified.
Specifying 1,0 sets permute and offset values of the thread assignment.
With a permute value of 1, thread n+1 is assigned to a consecutive core. With an offset of 0, the process's first thread 0 will be assigned to thread 0.
The same behavior is exhibited in a multisocket system.
- OMP_NUM_THREADS
- Sets the maximum number of threads to use for OpenMP* parallel regions if no
other value is specified in the application. This environment variable
applies to both -qopenmp and -parallel (Linux and Mac OS X) or /Qopenmp and /Qparallel (Windows).
Example syntax on a Linux system with 8 cores:
export OMP_NUM_THREADS=8
- Set stack size to unlimited
- The command "ulimit -s unlimited" is used to set the stack size limit to unlimited.
- Free the file system page cache
- The command "echo 1> /proc/sys/vm/drop_caches" is used to free up the filesystem page cache.
Red Hat Specific features
- Transparent Huge Pages
- On RedHat EL 6 and later, Transparent Hugepages increase the memory page size from 4 kilobytes to 2 megabytes. Transparent Hugepages provide significant performance advantages on systems with highly contended resources and large memory workloads.
If memory utilization is too high or memory is badly fragmented which prevents hugepages being allocated, the kernel will assign smaller 4k pages instead.
Hugepages are used by default unless the /sys/kernel/mm/redhat_transparent_hugepage/enabled field is changed from its RedHat EL6 default of 'always'.
OS Tuning
ulimit:
Used to set user limits of system-wide resources. Provides control over resources available to the shell and processes started by it. Some common ulimit commands may include:
- ulimit -s [n | unlimited]: Set the stack size to n kbytes, or unlimited to allow the stack size to grow without limit.
- ulimit -l (number): Set the maximum size that can be locked into memory.
Disabling Linux services:
Certain Linux services may be disabled to minimize tasks that may consume CPU cycles.
irqbalance:
Disabled through "service irqbalance stop". Depending on the workload involved, the irqbalance service reassigns various IRQ's to system CPUs. Though this service might help in some situations, disabling it can also help environments which need to minimize or eliminate latency to more quickly respond to events.
Performance Governors (Linux):
In-kernel CPU frequency governors are pre-configured power schemes for the CPU. The CPUfreq governors use P-states to change frequencies and lower power consumption. The dynamic governors can switch between CPU frequencies, based on CPU utilization to allow for power savings while not sacrificing performance.
Other options beside a generic performance governor can be set, such as the perf-bias:
--perf-bias, -b
On supported Intel processors, this option sets a register which allows the cpupower utility (or other software/firmware) to set a policy that controls the relative importance of performance versus energy savings to the processor. The range of valid numbers is 0-15, where 0 is maximum performance and 15 is maximum energy efficiency.
The processor uses this information in model-specific ways when it must select trade-offs between performance and energy efficiency. This policy hint does not supersede Processor Performance states (P-states) or CPU Idle power states (C-states), but allows software to have influence where it would otherwise be unable to express a preference.
On many Linux systems one can set the perf-bias for all CPUs through the cpupower utility with one of the following commands:
- "cpupower -c all set -b 0"
- "cpupower -c all set --perf-bias 0"
- "cpupower set -b 0"
Tuning Kernel parameters:
The following Linux Kernel parameters were tuned to better optimize performance of some areas of the system:
- dirty_background_ratio: Set through "echo 40 > /proc/sys/vm/dirty_background_ratio". This setting can help Linux disk caching and performance by setting the percentage of system memory that can be filled with dirty pages.
- dirty_ratio: Set through "echo 40 > /proc/sys/vm/dirty_ratio". This setting is the absolute maximum amount of system memory that can be filled with dirty pages before everything must get committed to disk.
- swappiness: The swappiness value can range from 1 to 100. A value of 100 will cause the kernel to swap out inactive processes frequently in favor of file system performance, resulting in large disk cache sizes. A value of 1 tells the kernel to only swap processes to disk if absolutely necessary. This can be set through a command like "echo 1 > /proc/sys/vm/swappiness"
- ksm/sleep_millisecs: Set through "echo 200 > /sys/kernel/mm/ksm/sleep_millisecs". This setting controls how many milliseconds the ksmd (KSM daeomn) should sleep before the next scan.
- khugepaged/scan_sleep_millisecs: Set through "echo 50000 > /sys/kernel/mm/transparent_hugepage/khugepaged/scan_sleep_millisecs". This setting controls how many milliseconds to wait in khugepaged is there is a hugepage allocation failure to throttle the next allocation attempt.
- numa_balancing: Disabled through "echo 0 > /proc/sys/kernel/numa_balancing". This feature will automatically migrate data on demand so memory nodes are aligned to the local CPU that is accessing data. Depending on the workload involved, enabling this can boost the performance if the workload performs well on NUMA hardware. If the workload is statically set to balance between nodes, then this service may not provide a benefit.
- Zone Reclaim Mode: Zone reclaim allows the reclaiming of pages from a zone if the number of free pages falls below a watermark even if other zones still have enough pages available. Reclaiming a page can be more beneficial than taking the performance penalties that are associated with allocating a page on a remote zone, especially for NUMA machines. To tell the kernel to free local node memory rather than grabbing free memory from remote nodes, use a command like "echo 1 > /proc/sys/vm/zone_reclaim_mode"
- max_map_count-n: The maximum number of memory map areas a process may have. Memory map areas are used as a side-effect of calling malloc, directly by mmap and mprotect, and also when loading shared libraries.
tuned-adm:
The tuned-adm tool is a commandline interface for switching between different tuning profiles available to the tuned tuning daeomn available in supported Linux distros. The default configuration file is located in /etc/tuned.conf and the supported profiles can be found in /etc/tune-profiles.
Some profiles that may be available by default include: default, desktop-powersave, server-powersave, laptop-ac-powersave, laptop-battery-powersave, spindown-disk, throughput-performance, latency-performance, enterprise-storage
To set a profile, one can issue the command "tuned-adm profile (profile_name)". Here are details about relevant profiles.
- throughput-performance: Server profile for typical throughput tuning. This profile disables tuned and ktune power saving features, enables sysctl settings that may improve disk and network IO throughphut performance, switches to the deadline scheduler, and sets the CPU governor to performance.
- latency-performance: Server profile for typical latency tuning. This profile disables tuned and ktune power saving features, enables the deadline IO scheduler, and sets the CPU governor to performance.
- enterprise-storage: Server profile to high disk throughput tuning. This profile disables tuned and ktune power saving features, enables the deadline IO scheduler, enables hugepages and disables disk barriers, increases disk readahead values, and sets the CPU governor to performance
Transparent Huge Pages (THP):
THP is an abstraction layer that automates most aspects of creating, managing, and using huge pages. THP is designed to hide much of the complexity in using huge pages from system administrators and developers, as normal huge pages must be assigned at boot time, can be difficult to manage manually, and often require significant changes to code in order to be used effectively. Transparent Hugepages increase the memory page size from 4 kilobytes to 2 megabytes. Transparent Hugepages provide significant performance advantages on systems with highly contended resources and large memory workloads. If memory utilization is too high or memory is badly fragmented which prevents hugepages being allocated, the kernel will assign smaller 4k pages instead. Most recent Linux OS releases have THP enabled by default.
Linux Huge Page settings:
If you need finer control and manually set the Huge Pages you can follow the below steps:
- Create a mount point for the huge pages: "mkdir /mnt/hugepages"
- The huge page file system needs to be mounted when the systems reboots. Add the following to a system boot configuration file before any services are started: "mount -t hugetlbfs nodev /mnt/hugepages"
- Set vm/nr_hugepages=N in your /etc/sysctl.conf file where N is the maximum number of pages the system may allocate.
- Reboot to have the changes take effect.
Note that further information about huge pages may be found in your Linux documentation file: /usr/src/linux/Documentation/vm/hugetlbpage.txt
Firmware Settings
One or more of the following settings may have been set. If so, the "Platform Notes" section of the report will say so; and you can read below to find out more about what these settings mean.
Intel Hyper-Threading (Default = Enabled):
This feature allows enabling or disabling of logical processor cores on processors supporting Intel Hyper-Threading (HT). When enabled, each physical processor core operates as two logical processor cores. When disabled, each physical core operates as only one logical processor core. Enabling this option can improve overall performance for applications that benefit from a higher processor core count.
Intel Virtualization Technology (Intel VT, VT-x) (Default = Enabled):
When enabled, a hypervisor or operating system supporting this option can use hardware capabilities provided by Intel VT. Some hypervisors require that you enable Intel VT. You can leave this set to enabled even if you are not using a hypervisor or an operating system that uses this option. With default BIOS settings as shipped with most systems, the default state for this setting is Enabled. However, this setting can change it's default setting depending on the Workload Profile that is selected, or what Workload Profile is default for the a certain system.
VT-d (Intel VT-d) (Default = Enabled):
If enabled, a hypervisor or operating system supporting this option can use hardware capabilities provided by Intel VT for Directed I/O. You can leave this set to enabled even if you are not using a hypervisor or an operating system that uses this option. With default BIOS settings as shipped with most systems, the default state for this setting is Enabled. However, this setting can change it's default setting depending on the Workload Profile that is selected, or what Workload Profile is default for the a certain system.
Processor x2APIC Support (Default = Enabled):
If enabled, x2APIC support enables operating system to run more efficiently on high core count configurations. It also optimizes interrupt distribution in virtualized environments. Setting this option to enables is recommended for most cases. When enabled, the operating system can optionally enable x2APCI support when it loads. Older hypervisors and operating systems might have issues with optional x2APIC support, therefore disabling x2APIC could be necessary to address these issues. Setting this option to enabled also forces Intel VT-D to be enabled.
SR-IOV (Default = Enabled):
If enabled, SR-IOV support enables a hypervisor to create virtual instances of PCI-express device, potentially increasing performance. If enabled, the BIOS allocates additional resources to PCI-express devices. You can leave this option set to enabled even if you are not using a hypervisor. With default BIOS settings as shipped with most systems, the default state for this setting is Enabled. However, this setting can change it's default setting depending on the Workload Profile that is selected, or what Workload Profile is default for the a certain system.
Thermal Configuration (Default = Optimal Cooling):
This feature allows the user to select the fan cooling solution for the system. Values for this BIOS option can be:
- Optimal Cooling: Provides the most efficient solution by configuring fan speeds to the minimum required to provide adequate cooling.
- Increased Cooling: Will run fans at higher speeds to provide additional cooling. Increased Cooling should be selected when non-HPE storage controllers are cabled to the embedded hard drive cage, or if the system is experiencing thermal issues that cannot be resolved in another manner.
- Maximum Cooling: Will provide the maximum cooling available by this platform.
Last Level Cache (LLC) Dead Line Allocation (Default = Enabled):
In the Skylake cache scheme, mid-level cache (MLC) evictions are filled into the last level cache (LLC). If a line is evicted from the MLC to the LLC, the Skylake core can flag the evicted MLC lines as "dead". This means that the lines are not likely to be read again. This option allows dead lines to be dropped and never fill the LLC if the option is disabled. Values for this BIOS option can be:
- Disabled: Disabling this option can save space in the LLC by never filling dead lines into the LLC. This can and prevent useful data from being evicted.
- Enabled: Opportunistically fill dead lines in LLC, if space is available.
Enhanced Processor Performance (Default = Disabled):
Use this option to enable the Enhanced Processor Performance setting. When enabled, this option will adjust the processor settings to a more aggressive setting that can result in improved performance, but may result in higher power consumption. Values for this BIOS option can be either disabled or enabled.
Stale A to S (Default = Disabled):
The in-memory directory has three states: invalid (I), snoopAll (A), and shared (S). Invalid (I) state means the data is clean and does not exist in any other socket`s cache. The snoopAll (A) state means the data may exist in another socket in exclusive or modified state. Shared (S) state means the data is clean and may be shared across one or more socket`s caches. When doing a read to memory, if the directory line is in the A state we must snoop all the other sockets because another socket may have the line in modified state. If this is the case, the snoop will return the modified data. However, it may be the case that a line is read in A state and all the snoops come back a miss. This can happen if another socket read the line earlier and then silently dropped it from its cache without modifying it. Values for this BIOS option can be:
- Disabled: Disabling this option allows the feature to process memory directories as described above.
- Enabled: In the situation where a line in A state returns only snoop misses, the line will transition to S state. That way, subsequent reads to the line will encounter it in S state and not have to snoop, saving latency and snoop bandwidth.
Stale A to S may be beneficial in a workload where there are many cross-socket reads.
Last Level Cache (LLC) Prefetch (Default = Disabled):
This option configures the processor Last Level Cache (LLC) prefetch feature as a result of the non-inclusive cache architecture. The LLC prefetcher exists on top of other prefetchers that that can prefetch data in the core data cache unit (DCU) and mid-level cache(MLC). In some cases, setting this option to disabled can improve performance. Typically, setting this option to enable provides better performance. Values for this BIOS option can be:
- Disabled: Disabling this option can forces data to fill the MLC before prefetching data to the LLC.
- Enabled: Giving the core prefetcher the ability to prefetch data directly to the LLC without filling the MLC.
NUMA Group Size Optimization (Default = Clustered):
This feature allows the user to configure how the BIOS reports the size of a NUMA node (number of logical processors), which assists the Operating System in grouping processors for application use (referred to as Kgroups). Values for this BIOS option can be:
- Clustered: Might provide better performance for some workloads due to optimizing the resulting groups along NUMA boundaries.
- Flat: Might provide better performance for some workloads that cannot take advantage of processors spanning multiple groups. This setting would be necessary to help this class of applications utilize more logical processors.
Sub-NUMA Clustering (SNC) (Default = Enabled):
SNC breaks up the last level cache (LLC) into disjoint clusters based on address range, with each cluster bound to a subset of the memory controllers in the system. SNC improves average latency to the LLC and memory. SNC is a replacement for the cluster on die (COD) feature found in previous processor families. For a multi-socketed system, all SNC clusters are mapped to unique NUMA domains. (See also IMC interleaving.) Values for this BIOS option can be:
- Disabled: The LLC is treated as one cluster when this option is disabled
- Enabled: Utilizes LLC capacity more efficiently and reduces latency due to core/IMC proximity. This may provide performance improvement on NUMA-aware operating systems
Xtended Prediciton Table (XPT) Prefetch (Default = Enabled):
This option configures the processor Xtended Prediciton Table (XPT) prefetch feature. The XPT prefetcher exists on top of other prefetchers that that can prefetch data in the core DCU, MLC, and LLC. The XPT prefetcher will issue a speculative DRAM read request in parallel to an LLC lookup. This prefetch bypasses the LLC, saving latency. In some cases, setting this option to disabled can improve performance. In some cases, setting this option to disabled can improve performance. Typically, setting this option to enable provides better performance. This option must be enabled when Sub-NUMA Clustering is enabled. Values for this BIOS option can be:
- Enabled: Allows a read request sent to the LLC to speculatively issue a copy of the read to the memory controller requesting the prefetch.
- Disabled: Does not allow the LLC to speculatively issue copies of reads. Disabling this will also disables Sub-NUMA Cluster (SNC).
Uncore Frequency Scaling (Default = Auto):
This option controls the frequency scaling of the processor`s internal buses (the uncore). Values for this BIOS option can be:
- Auto: Enabled the processor to dynamically change frequencies based on the workload.
- Maximum: Enables tuning for latency.
- Minimum: Enables tuning for power consumption.
Workload Profile (Default = General Power Efficient Compute):
This option allows a user to choose one workload profile that best fits the user`s needs. The workload profiles control many power and performance settings that are relevant to general workload areas. Values for this BIOS option can be:
- General Power Efficient Compute, General Peak Frequency Compute, General Throughput Compute, Virtualization - Power Efficient, Virtualization - Max Performance, Low Latency, Mission Critical, Transaction Application Processing, High Performance Compute (HPC), Decision Support, Graphic Processing, I/O Throughput, or Custom.
- Setting the Workload Profile to any option not named Custom allows the server to automatically configure various BIOS settings. These BIOS settings control many power and performance settings that are relevant to general workload areas that fit the profile name.
- Setting the Workload Profile to Custom allows a user to set any BIOS setting to any supported setting. Choosing Custom after selecting an initial profile does not change the settings controlled by the profile previously selected without user intervention.
- Further technical description about what settings a Workload Profile changes and the types of workloads that a profile may be suitable for can be found through the HPE UEFI Workload-based Performance and Tuning Guide - https://support.hpe.com/hpsc/doc/public/display?docId=a00016408en_us
Minimum Processor Idle Power Core C-State (Default = C6 State):
This option can only be configured if the Workload Profile is set to Custom, or this option is not a dependent value for the Workload Profile. This feature selects the processor's lowest idle power state (C-state) that the operating system uses. The higher the C-state, the lower the power usage of that idle state (C6 is the lowest power idle state supported by the processor). Values for this setting can be:
- C6 State: While in C6, the core PLLs are turned off, the core caches are flushed and the core state is saved to the Last Level Cache. Power Gates are used to reduce power consumption to close to zero. C6 is considered an inactive core.
- C1E State: C1E is defined as the enhanced halt state. While in C1E no instructions are being executed. C1E considered an active core.
- No C-states: No C-states is defined as C0, which is defined as the active state. While in C0, instructions are being executed by the core.
Minimum Processor Idle Power Package C-State (Default = Package C6 (retention) State):
This option can only be configured if the Workload Profile is set to Custom, or this option is not a dependent value for the Workload Profile. This feature selects the processor's lowest idle package power state (C-state) that is enabled. The processor will automatically transition into the package C-states based on the Core C-states, in which cores on the processor have transitioned. The higher the package C-state, the lower the power usage of that idle package state. Package C6 (retention) is the lowest power idle package state supported by the processor). Values for this setting can be:
- Package C6 (retention) State: All cores have saved their architectural state and have had their core voltages reduced to zero volts. The LLC retains context, but no accesses can be made to the LLC in this state, the cores must break out to the internal state package C2 for snoops to occur.
- Package C6 (non-retention) State: All cores have saved their architectural state and have had their core voltages reduced to zero volts. The LLC does not retain context, and no accesses can be made to the LLC in this state, the cores must break out to the internal state package C2 for snoops to occur.
- No Package State: All cores are in an active state and have not entered any power saving state.
Energy/Performance Bias (Default = Balanced Performance):
This option can only be configured if the Workload Profile is set to Custom, or this option is not a dependent value for the Workload Profile. This option configures several processor subsystems to optimize the processor's performance and power usage. Values for this BIOS setting can be:
- Balanced Performance: Provides optimum performance efficiency and is recommended for most environments.
- Maximum Performance: Should be used for environments that require the highest performance and lowest latency but are not sensitive to power consumption.
- Balanced Power: Similar to Balanced Performance but this option prioritizes more power savings at the sacrifice of performance.
- Power Savings Mode: Should only be used in environments that are power sensitive and are willing to accept reduced performance.
AHS PCI Logging Level (Default = Verbose Logging):
This option allows the AHS PCI Logging size to be changed. This is a boot time option that should have no effect on run time performance. Values for this BIOS setting can be:
- Verbose Logging: Allows 960 bytes to be logged.
- Minimal Logging: Allows 64 bytes to be logged.
Memory Patrol Scrubbing (Default = Enabled):
This option allows for correction of soft memory errors. Over the length of system runtime, the risk of producing multi-bit and uncorrected errors is reduced with this option. Values for this BIOS setting can be:
- Enabled: Correction of soft memory errors can occur during runtime.
- Disabled: Soft memory error correction is turned off during runtime.
HW Prefetcher (Default = Enabled):
Use this option to disable the processor HW Prefetch feature. In some cases, setting this option to disabled can improve performance. Typically, setting this option to enabled provides better performance. Only disable this option after performing application benchmarking to verify improved performance in the environment. The HW Prefetcher fetches streams of data and instruction from the memory into the second-level (L2) cache if it determins this data is likely to be required in the near future. The prefetcher is capable of handling multiple streams in either the forward or backward direction. The HW Prefetcher is triggered when successive cache misses occur in the last-level cache and a stride in the access pattern is detected, such as in the case of loop iterations that access array elements. The prefetching occurs up to a page boundary. This option can reduce the latency associated with memory reads. Values for this BIOS setting can be enabled or disabled.
Intel UPI Link Power Management (Default = Enabled):
Use this option to place the Quick Path Interconnect (UPI) links into a low power state when the links are not being used. This lowers power usage with minimal effect on performance. You can only configure this option if two or more CPUs are present and the Workload Profile is set to Customer. Values for this BIOS setting can be: enabled and disabled.
Advanced Memory Protection (Default value varies):
Use this option to configure additional memory protection with ECC (Error Checking and Correcting). Options and support vary per system and configuration. Values for this BIOS setting can be:
- Advanced ECC: Keeps all installed memory available for use while still protecting the system against all single-bit failures and certain multi-bit failures.
- Online Spare with Advanced ECC Support: Enables a system to automatically map out a group of memory that is detected to be at an increased risk of receiving uncorrected memory errors based on an advanced analysis of corrected memory errors. The mapped out memory is automatically replaced by a spare group of memory without interrupting the system.
- Mirrored Memory with Advanced ECC Support: Provides the maximum protection against uncorrected memory errors that might otherwise result in a system failure.
- Fast Fault Tolerant Advanced Double Device Data Correction (ADDDC): Enabled the system to correct memory errors and continue to operate in cases of multiple DRAM device failures on a DIMM. This provides protection against uncorrectable memory erros beyond what is available with Advanced ECC.
Last updated July 3, 2019.
For questions about the meanings of these flags, please contact the tester.
For other inquiries, please contact info@spec.org
Copyright 2017-2019 Standard Performance Evaluation Corporation
Tested with SPEC CPU2017 v1.0.5.
Report generated on 2019-08-21 12:05:38 by SPEC CPU2017 flags formatter v5178.
![[benchmark]](https://www.spec.org/auto/cpu2017/images/benchmark.png)