CPU2017 Flag Description
Lenovo Global Technology ThinkSystem SN550 (1.90 GHz, Intel Xeon Gold 6238T)

Copyright © 2016 Intel Corporation. All Rights Reserved.

Implicitly Included Flags

This section contains descriptions of flags that were included implicitly by other flags, but which do not have a permanent home at SPEC.

• • -O2
  • 

  • Enables optimizations for speed. This is the generally recommended optimization level. This option also enables:
    - Inlining of intrinsics
    - Intra-file interprocedural optimizations, which include:
    - inlining
    - constant propagation
    - forward substitution
    - routine attribute propagation
    - variable address-taken analysis
    - dead static function elimination
    - removal of unreferenced variables
    - The following capabilities for performance gain:
    - constant propagation
    - copy propagation
    - dead-code elimination
    - global register allocation
    - global instruction scheduling and control speculation
    - loop unrolling
    - optimized code selection
    - partial redundancy elimination
    - strength reduction/induction variable simplification
    - variable renaming
    - exception handling optimizations
    - tail recursions
    - peephole optimizations
    - structure assignment lowering and optimizations
    - dead store elimination
    

  • Includes:
    • -O1
      • -funroll-loops
      • -fno-builtin
      • -mno-ieee-fp
      • -fomit-framepointer
      • -ffunction-sections
      • -ftz
• • -O1
  • 

  • Enables optimizations for speed and disables some optimizations that increase code size and affect speed.
    To limit code size, this option:

    • Enables global optimization; this includes data-flow analysis, code motion, strength reduction and test replacement, split-lifetime analysis, and instruction scheduling.
    • Disables intrinsic recognition and intrinsics inlining.

    The O1 option may improve performance for applications with very large code size, many branches, and execution time not dominated by code within loops.

    -O1 sets the following options:
    -funroll-loops0, -fno-builtin, -mno-ieee-fp, -fomit-framepointer, -ffunction-sections, -ftz
  • Includes:
    • -funroll-loops
    • -fno-builtin
    • -mno-ieee-fp
    • -fomit-framepointer
    • -ffunction-sections
    • -ftz
• • -funroll-loops
  • 

  • Tells the compiler the maximum number of times to unroll loops. For example -funroll-loops0 would disable unrolling of loops.

• • -fno-builtin
  • 

  • -fno-builtin disables inline expansion for all intrinsic functions.

• • -mno-ieee-fp
  • 

  • This option trades off floating-point precision for speed by removing the restriction to conform to the IEEE standard.

• • -fomit-framepointer
  • 

  • EBP is used as a general-purpose register in optimizations.

• • -ffunction-sections
  • 

  • Places each function in its own COMDAT section.

• • -ftz
  • 

  • Flushes denormal results to zero.

Commands and Options Used to Submit Benchmark Runs

submit= MYMASK=`printf '0x%x' $((1<<$SPECCOPYNUM))`; /usr/bin/taskset $MYMASK $command

When running multiple copies of benchmarks, the SPEC config file feature submit is used to cause individual jobs to be bound to specific processors. This specific submit command, using taskset, is used for Linux64 systems without numactl.
Here is a brief guide to understanding the specific command which will be found in the config file:

• /usr/bin/taskset [options] [mask] [pid | command [arg] ... ]:
  taskset is used to set or retreive the CPU affinity of a running process given its PID or to launch a new COMMAND with a given CPU affinity. The CPU affinity is represented as a bitmask, with the lowest order bit corresponding to the first logical CPU and highest order bit corresponding to the last logical CPU. When the taskset returns, it is guaranteed that the given program has been scheduled to a specific, legal CPU, as defined by the mask setting.
• [mask]: The bitmask (in hexadecimal) corresponding to a specific SPECCOPYNUM. The specific example above, computes this mask value in the variable $MYMASK. The value of this mask for the first copy of a rate run will be 0x00000001, for the second copy of the rate will be 0x00000002 etc. Thus, the first copy of the rate run will have a CPU affinity of CPU0, the second copy will have the affinity CPU1 etc.
• $command: Program to be started, in this case, the benchmark instance to be started.

submit= numactl --localalloc --physcpubind=$SPECCOPYNUM $command

When running multiple copies of benchmarks, the SPEC config file feature submit is used to cause individual jobs to be bound to specific processors. This specific submit command is used for Linux64 systems with support for numactl.
Here is a brief guide to understanding the specific command which will be found in the config file:

• syntax: numactl [--interleave=nodes] [--preferred=node] [--physcpubind=cpus] [--cpunodebind=nodes] [--membind=nodes] [--localalloc] command args ...
• numactl runs processes with a specific NUMA scheduling or memory placement policy. The policy is set for a command and inherited by all of its children.
• "--localalloc" instructs numactl to keep a process memory on the local node while "-m" specifies which node(s) to place a process memory.
• "--physcpubind" specifies which core(s) to bind the process. In this case, copy 0 is bound to processor 0 etc.
• For full details on using numactl, please refer to your Linux documentation, 'man numactl'

Shell, Environment, and Other Software Settings

numactl --interleave=all "runspec command"

Launching a process with numactl --interleave=all sets the memory interleave policy so that memory will be allocated using round robin on nodes. When memory cannot be allocated on the current interleave target fall back to other nodes.

KMP_STACKSIZE

Specify stack size to be allocated for each thread.

KMP_AFFINITY

Syntax: KMP_AFFINITY=[<modifier>,...]<type>[,<permute>][,<offset>]
The value for the environment variable KMP_AFFINITY affects how the threads from an auto-parallelized program are scheduled across processors.
It applies to binaries built with -qopenmp and -parallel (Linux and Mac OS X) or /Qopenmp and /Qparallel (Windows).
modifier:
    granularity=fine Causes each OpenMP thread to be bound to a single thread context.
type:
    compact Specifying compact assigns the OpenMP thread <n>+1 to a free thread context as close as possible to the thread context where the <n> OpenMP thread was placed.
    scatter Specifying scatter distributes the threads as evenly as possible across the entire system.
permute: The permute specifier is an integer value controls which levels are most significant when sorting the machine topology map. A value for permute forces the mappings to make the specified number of most significant levels of the sort the least significant, and it inverts the order of significance.
offset: The offset specifier indicates the starting position for thread assignment.

Please see the Thread Affinity Interface article in the Intel Composer XE Documentation for more details.

Example: KMP_AFFINITY=granularity=fine,scatter
Specifying granularity=fine selects the finest granularity level and causes each OpenMP or auto-par thread to be bound to a single thread context.
This ensures that there is only one thread per core on cores supporting HyperThreading Technology
Specifying scatter distributes the threads as evenly as possible across the entire system.
Hence a combination of these two options, will spread the threads evenly across sockets, with one thread per physical core.

Example: KMP_AFFINITY=compact,1,0
Specifying compact will assign the n+1 thread to a free thread context as close as possible to thread n.
A default granularity=core is implied if no granularity is explicitly specified.
Specifying 1,0 sets permute and offset values of the thread assignment.
With a permute value of 1, thread n+1 is assigned to a consecutive core. With an offset of 0, the process's first thread 0 will be assigned to thread 0.
The same behavior is exhibited in a multisocket system.

OMP_NUM_THREADS

Sets the maximum number of threads to use for OpenMP* parallel regions if no other value is specified in the application. This environment variable applies to both -qopenmp and -parallel (Linux and Mac OS X) or /Qopenmp and /Qparallel (Windows). Example syntax on a Linux system with 8 cores: export OMP_NUM_THREADS=8

Set stack size to unlimited

The command "ulimit -s unlimited" is used to set the stack size limit to unlimited.

Free the file system page cache

The command "echo 1> /proc/sys/vm/drop_caches" is used to free up the filesystem page cache.

Red Hat Specific features

Transparent Huge Pages

On RedHat EL 6 and later, Transparent Hugepages increase the memory page size from 4 kilobytes to 2 megabytes. Transparent Hugepages provide significant performance advantages on systems with highly contended resources and large memory workloads. If memory utilization is too high or memory is badly fragmented which prevents hugepages being allocated, the kernel will assign smaller 4k pages instead.
Hugepages are used by default unless the /sys/kernel/mm/redhat_transparent_hugepage/enabled field is changed from its RedHat EL6 default of 'always'.

Operating System Tuning Parameters

sched_cfs_bandwidth_slice_us

This OS setting controls the amount of run-time(bandwidth) transferred to a run queue from the task's control group bandwidth pool. Small values allow the global bandwidth to be shared in a fine-grained manner among tasks, larger values reduce transfer overhead. The default value is 5000 (ns).

sched_latency_ns

This OS setting configures targeted preemption latency for CPU bound tasks. The default value is 24000000 (ns).

sched_migration_cost_ns

Amount of time after the last execution that a task is considered to be "cache hot" in migration decisions. A "hot" task is less likely to be migrated to another CPU, so increasing this variable reduces task migrations. The default value is 500000 (ns).

sched_min_granularity_ns

This OS setting controls the minimal preemption granularity for CPU bound tasks. As the number of runnable tasks increases, CFS(Complete Fair Scheduler), the scheduler of the Linux kernel, decreases the timeslices of tasks. If the number of runnable tasks exceeds sched_latency_ns/sched_min_granularity_ns, the timeslice becomes number_of_running_tasks * sched_min_granularity_ns. The default value is 8000000 (ns).

sched_wakeup_granularity_ns

This OS setting controls the wake-up preemption granularity. Increasing this variable reduces wake-up preemption, reducing disturbance of compute bound tasks. Lowering it improves wake-up latency and throughput for latency critical tasks, particularly when a short duty cycle load component must compete with CPU bound components. The default value is 10000000 (ns).

numa_balancing

This OS setting controls automatic NUMA balancing on memory mapping and process placement. Setting 0 disables this feature. It is enabled by default (1).

Firmware / BIOS / Microcode Settings

Choose Operating Mode: (Default="Efficiency -Favor Performance")

The average customer doesn't know the best way to set each individual power/performance feature for their specific environment. Because of this, a menu option is provided that can help a customer optimize the system for things such as minimum power usage/acoustic levels, maximum efficiency, Energy Star optimization, or maximum performance.

• "Minimal Power" mode strives to minimize the absolute power consumption of the system while it is operating. The tradeoff is that performance may be reduced in this mode depending on the application that is running.
• "Efficiency -Favor Power" mode maximizes the performance/watt efficiency with a bias towards power savings. It provides the best features for reducing power and increasing performance in applications where maximum bus speeds are not critical. It is expected that this will be the favored mode for SPECpower testing. "Efficiency -Favor Power" mode maintains backwards compatibility with systems that included the preset operating modes before Energy Star for servers was released.
• "Efficiency -Favor Performance" mode optimizes the performance/watt efficiency with a bias towards performance. It is the favored mode for Energy Star. Note that this mode is slightly different than "Efficiency -Favor Power" mode. In "Efficiency - Favor Performance" mode, no bus speeds are derated as they are in "Efficiency -Favor Power" mode. "Efficiency -Favor Performance" mode is the default mode.
• "Maximum Performance" mode will maximize the absolute performance of the system without regard for power. In this mode, power consumption is a don't care. Things like fan speed and heat output of the system may increase in addition to power consumption. Efficiency of the system may go down in this mode, but the absolute performance may increase depending on the workload that is running.
• A fifth setting, "Custom Mode", may be selected after any of the other 4 presets allowing low-level settings, which otherwise are preset and unchangeable, to be individually modified

Platform Controlled Type:

"Maximum Performance" allows the most aggressive use of turbo and power management functions are disabled, thereby increasing power consumption. "Minimal Power" disables turbo and maximizes the use of power management features. When a preset mode is selected, the low-level settings are not changeable and will be grayed out. If user would like to change the settings, please choose "Custom Mode" in "Operating Mode" located under "System Setting" submenu.

Page Policy:

Adaptive Open Page Policy can improve performance for applications with a highly localized memory access pattern; Closed Page Policy can benifit applications that access memory more randomly.

CPU P-state Control:

Select the method used to control CPU P-states (performance states). "None" disables all P-states and the CPUs run at either their rated frequency or in turbo mode (if turbo is enabled). When "Legacy" is selected, the CPU P-states will be presented to the operating system (OS) and the OS power management (OSPM) will directly control which P-state is selected. With "Autonomous", the P-states are controlled fully by system hardware. No P-state support is required in the OS or VM. "Cooperative" is a combination of Legacy and Autonomous. The P-states are still controlled in hardware but the OS can provide hints to the hardware for P-state limits and the desired setting. When a preset mode is selected, the low-level settings are not changeable and will be grayed out. If user would like to change the settings, please choose "Custom Mode" in "Operating Mode" located under "System Setting" submenu.

C-States:

C-states reduce CPU idle power. There are three options in this mode: Legacy, Autonomous, Disable.

• Legacy: When "Legacy" is selected, the operating system initiates the C-state transitions. For E5/E7 CPUs, ACPI C1/C2/C3 map to Intel C1/C3/C6. For 6500/7500 CPUs, ACPI C1/C3 map to Intel C1/C3 (ACPI C2 is not available). Some OS SW may defeat the ACPI mapping (e.g. intel_idle driver).
• Autonomous: When "Autonomous" is selected, HALT and C1 request get converted to C6 requests in hardware.
• Disable: When "Disable" is selected, only C0 and C1 are used by the OS. C1 gets enabled automatically when an OS autohalts.

C1 Enhanced Mode:

Enabling C1E (C1 enhanced) state can save power by halting CPU cores that are idle.

Turbo Mode:

Enabling turbo mode can boost the overall CPU performance when all CPU cores are not being fully utilized. A CPU core can run above its rated frequency for a short perios of time when it is in turbo mode.

Hyper-Threading:

Enabling Hyper-Threading let operating system addresses two virtual or logical cores for a physical presented core. Workloads can be shared between virtual or logical cores when possible. The main function of hyper-threading is to increase the number of independent instructions in the pipeline for using the processor resources more efficiently.

DCA:

DCA capable I/O devices such as network controllers can place data directly into the CPU cache, which improves response time.

Power/Performance Bias:

Power/Performance bias determines how aggressively the CPU will be power managed and placed into turbo. With "Platform Controlled", the system controls the setting. Selecting "OS Controlled" allows the operating system to control it.

CPU Frequency Limits:

The maximum frequency (turbo, AVX, and non turbo) can be restricted to a frequency that is between the maximum turbo frequency for the CPU installed and 1.2GHz. This can be useful for synchronizing CPU task. Note, the max frequency for N+1 cores cannot be higher than N cores. If an illegal frequency is entered, it will automatically be limited to a legal value. If the CPU frequency limits are being controlled through application software, leave this menu item at the default ([Full turbo uplift]), please choose [Custom Mode] in "Operating Mode" and [Enable] in "Turbo Mode" located under "System Setting" submenu.

Energy Efficient Turbo:

When energy efficient turbo is enabled, the CPU's optimal turbo frequency will be tuned dynamically based on CPU utilization.

Uncore Frequency Scaling:

When enabled, the CPU uncore will dynamically change speed based on the workload.

MONITOR/MWAIT:

MONITOR/MWAIT instructions are used to engage C-states.

Sub-NUMA Cluster (SNC):

SNC breaks up the last level cache (LLC) into disjoint clusters based on address range, with each cluster bound to a subset of the memory controllers in the system. SNC improves average latency to the LLC and memory. SNC is a replacement for the cluster on die (COD) feature found in previous processor families. For a multi-socketed system, all SNC clusters are mapped to unique NUMA domains. (See also IMC interleaving.) Values for this BIOS option can be:

• Disabled: The LLC is treated as one cluster when this option is disabled
• Enabled: Utilizes LLC capacity more efficiently and reduces latency due to core/IMC proximity. This may provide performance improvement on NUMA-aware operating systems

Snoop Preference:

Select the appropriate snoop mode based on the workload. There are two snoop modes: "Home Snoop Plus" and "Home Snoop". Default is "Home Snoop Plus".

• Home Snoop Plus: Best overall for most workloads.
• Home Snoop: Best for BW sensive workloads.

XPT Prefetcher

When enabled, enables a read request that is sent to the processor last level cache to speculatively issue a copy of that read request to the processor memory controller prefetcher to reduce latency. The performance benefit can vary so the end user must determine whether or not enabling the XPT prefetcher benefits their specific application environment. Default is Enable.

UPI Prefetcher

When enabled, memory reads on the memory bus can be started earlier by the processor memory controller to reduce latency. The performance benefit can vary so the end user must determine whether or not enabling the UPI prefetcher benefits their specific application environment. Default is Enable.

Patrol Scrub:

Patrol Scrub is a memory RAS feature which runs a background memory scrub against all DIMMs. Can negatively impact performance.

DCU Streamer Prefetcher:

DCU (Level 1 Data Cache) streamer prefetcher is an L1 data cache prefetcher. Lightly threaded applications and some benchmarks can benefit from having the DCU streamer prefetcher enabled. Default setting is Enable.

Stale AtoS

The in-memory directory has three states: invalid (I), snoopAll (A), and shared (S). Invalid (I) state means the data is clean and does not exist in any other socket`s cache. The snoopAll (A) state means the data may exist in another socket in exclusive or modified state. Shared (S) state means the data is clean and may be shared across one or more socket`s caches. When doing a read to memory, if the directory line is in the A state we must snoop all the other sockets because another socket may have the line in modified state. If this is the case, the snoop will return the modified data. However, it may be the case that a line is read in A state and all the snoops come back a miss. This can happen if another socket read the line earlier and then silently dropped it from its cache without modifying it. Values for this BIOS option can be:

• Disable: Disabling this option allows the feature to process memory directory states as described above.
• Enable: In the situation where a line in A state returns only snoop misses, the line will transition to S state. That way, subsequent reads to the line will encounter it in S state and not have to snoop, saving latency and snoop bandwidth.

Stale A to S may be beneficial in a workload where there are many cross-socket reads.

LLC dead line alloc

In some Intel CPU caching schemes, mid-level cache (MLC) evictions are filled into the last level cache (LLC). If a line is evicted from the MLC to the LLC, the core can flag the evicted MLC lines as "dead." This means that the lines are not likely to be read again. This option allows dead lines to be dropped and never fill the LLC if the option is disabled. Values for this BIOS option can be:

• Disabled: Disabling this option can save space in the LLC by never filling MLC dead lines into the LLC.
• Enabled: Opportunistically fill MLC dead lines in LLC, if space is available.

Adjacent Cache Prefetch:

When enabled, fetches both cache lines that make up a 128 byte cache line pair even if the requested data is only in the first cache line. Lightly threaded applications and some benchmarks can benefit from having the adjcent cache line prefetch enabled. Default is enable.

Intel Virtualization Technology:

Intel Virtualization Technology allows a platform to run multiple operating systems and applications in independent partitions, so that one computer system can function as multiple virtual system. Default is enable.

Hardware Prefetcher:

When enabled, fetches the next cache line into the processor L2 cache if two consecutive cache lines were read. The next cache line is fetched from memory. Lightly threaded applications can benefit from having the hardware prefetcher enabled. But, the end user must determine whether or not enabling the hardware prefetcher benefits their specific application environment. Default is Enable.

Trusted Execution Technology:

Enable Intel Trusted Execution Technology (Intel TXT). Default is disable.

DCU IP Prefetcher:

DCU IP Prefetcher is typically best left enabled for most environments. Some environments may benefit from having it disabled(e.g. Java). Default is enable.

Acoustic Mode

By using acoustic mode, the user has some control over the fan speeds and airflow (and noise) that is produced by the system fans. This mode can be used for noise or airflow concerns in the user environment. As a result, Mode1,2,3,4,5 increase the possibility that the node might have to be throttled to maintain cooling within the fan speed limitation. If there is power or thermal demanding PCI card installed in the chassis, acoustic mode is automatically disabled.

• None: Fan speeds change as required for optimal cooling
• Mode1: Highest acoustics attenuation (lowest cooling)
• Mode2: Higher acoustics attenuation.
• Mode3: Intermediate acoustics attenuation.
• Mode4: Low acoustics attenuation (higher cooling).
• Mode5: Aggressive cooling mode.

Workload Configuration:

I/O sensitive should be used with expansion cards that require high I/O bandwidth when the CPU cores are idle to allow enough frequency for the workload. Default is Balanced.

Memory Power Management:

[Disable] provides maximum performance but minmum power savings. [Automatic] is suitable for most applications. When a preset mode is selected, the low-level settings are not changeable and will grayed out. If user would like to change the settings, please choose [Custom Mode] in "Operating Mode" located under "Syetem Settings" submenu.

Flag description origin markings:

	Indicates that the flag description came from the user flags file.
	Indicates that the flag description came from the suite-wide flags file.
	Indicates that the flag description came from a per-benchmark flags file.

For questions about the meanings of these flags, please contact the tester.
For other inquiries, please contact info@spec.org
Copyright 2017-2019 Standard Performance Evaluation Corporation
Tested with SPEC CPU2017 v1.0.5.
Report generated on 2019-10-15 14:35:24 by SPEC CPU2017 flags formatter v5178.