CPU2017 Flag Description
New H3C Technologies Co., Ltd. H3C UniServer R4900 G6 Ultra (Intel Xeon Gold 6538Y+)

Copyright © 2016 Intel Corporation. All Rights Reserved.

Implicitly Included Flags

This section contains descriptions of flags that were included implicitly by other flags, but which do not have a permanent home at SPEC.

• • -O3
  • 

  • Enable O2 optimizations plus more aggressive optimizations, such as prefetching, scalar replacement, and loop and memory access transformations. Enable optimizations for maximum speed, such as:

    • Loop unrolling, including instruction scheduling
    • Code replication to eliminate branches
    • Padding the size of certain power-of-two arrays to allow more efficient cache use.
    
    

    On IA-32 and Intel EM64T processors, when O3 is used with options -ax or -x (Linux) or with options /Qax or /Qx (Windows), the compiler performs more aggressive data dependency analysis than for O2, which may result in longer compilation times. The O3 optimizations may not cause higher performance unless loop and memory access transformations take place. The optimizations may slow down code in some cases compared to O2 optimizations. The O3 option is recommended for applications that have loops that heavily use floating-point calculations and process large data sets.

  • Includes:
    • -O2
      • -O1
        • -funroll-loops
        • -fno-builtin
        • -mno-ieee-fp
        • -fomit-framepointer
        • -ffunction-sections
        • -ftz
• • -O2
  • 

  • Enable optimizations for speed. This is the generally recommended optimization level. This option also enables:
    - Inlining of intrinsics
    - Intra-file interprocedural optimizations, which include:
    - inlining
    - constant propagation
    - forward substitution
    - routine attribute propagation
    - variable address-taken analysis
    - dead static function elimination
    - removal of unreferenced variables
    - The following capabilities for performance gain:
    - constant propagation
    - copy propagation
    - dead-code elimination
    - global register allocation
    - global instruction scheduling and control speculation
    - loop unrolling
    - optimized code selection
    - partial redundancy elimination
    - strength reduction/induction variable simplification
    - variable renaming
    - exception handling optimizations
    - tail recursions
    - peephole optimizations
    - structure assignment lowering and optimizations
    - dead store elimination
    

  • Includes:
    • -O1
      • -funroll-loops
      • -fno-builtin
      • -mno-ieee-fp
      • -fomit-framepointer
      • -ffunction-sections
      • -ftz
• • -O1
  • 

  • Enable optimizations for speed and disables some optimizations that increase code size and affect speed.
    To limit code size, this option:

    • Enable global optimization; this includes data-flow analysis, code motion, strength reduction and test replacement, split-lifetime analysis, and instruction scheduling.
    • Disables intrinsic recognition and intrinsics inlining.

    The O1 option may improve performance for applications with very large code size, many branches, and execution time not dominated by code within loops.

    -O1 sets the following options:
    -funroll-loops0, -fno-builtin, -mno-ieee-fp, -fomit-framepointer, -ffunction-sections, -ftz
  • Includes:
    • -funroll-loops
    • -fno-builtin
    • -mno-ieee-fp
    • -fomit-framepointer
    • -ffunction-sections
    • -ftz
• • -funroll-loops
  • 

  • Tells the compiler the maximum number of times to unroll loops. For example -funroll-loops0 would disable unrolling of loops.

• • -fno-builtin
  • 

  • -fno-builtin disables inline expansion for all intrinsic functions.

• • -mno-ieee-fp
  • 

  • This option trades off floating-point precision for speed by removing the restriction to conform to the IEEE standard.

• • -fomit-framepointer
  • 

  • EBP is used as a general-purpose register in optimizations.

• • -ffunction-sections
  • 

  • Places each function in its own COMDAT section.

• • -ftz
  • 

  • Flushes denormal results to zero.

Commands and Options Used to Submit Benchmark Runs

submit= MYMASK=`printf '0x%x' $((1<<$SPECCOPYNUM))`; /usr/bin/taskset $MYMASK $command

When running multiple copies of benchmarks, the SPEC config file feature submit is used to cause individual jobs to be bound to specific processors. This specific submit command, using taskset, is used for Linux64 systems without numactl.
Here is a brief guide to understanding the specific command which will be found in the config file:

• /usr/bin/taskset [options] [mask] [pid | command [arg] ... ]:
  taskset is used to set or retreive the CPU affinity of a running process given its PID or to launch a new COMMAND with a given CPU affinity. The CPU affinity is represented as a bitmask, with the lowest order bit corresponding to the first logical CPU and highest order bit corresponding to the last logical CPU. When the taskset returns, it is guaranteed that the given program has been scheduled to a specific, legal CPU, as defined by the mask setting.
• [mask]: The bitmask (in hexadecimal) corresponding to a specific SPECCOPYNUM. The specific example above, computes this mask value in the variable $MYMASK. The value of this mask for the first copy of a rate run will be 0x00000001, for the second copy of the rate will be 0x00000002 etc. Thus, the first copy of the rate run will have a CPU affinity of CPU0, the second copy will have the affinity CPU1 etc.
• $command: Program to be started, in this case, the benchmark instance to be started.

submit= numactl --localalloc --physcpubind=$SPECCOPYNUM $command

When running multiple copies of benchmarks, the SPEC config file feature submit is used to cause individual jobs to be bound to specific processors. This specific submit command is used for Linux64 systems with support for numactl.
Here is a brief guide to understanding the specific command which will be found in the config file:

• syntax: numactl [--interleave=nodes] [--preferred=node] [--physcpubind=cpus] [--cpunodebind=nodes] [--membind=nodes] [--localalloc] command args ...
• numactl runs processes with a specific NUMA scheduling or memory placement policy. The policy is set for a command and inherited by all of its children.
• "--localalloc" instructs numactl to keep a process memory on the local node while "-m" specifies which node(s) to place a process memory.
• "--physcpubind" specifies which core(s) to bind the process. In this case, copy 0 is bound to processor 0 etc.
• For full details on using numactl, please refer to your Linux documentation, 'man numactl'

Shell, Environment, and Other Software Settings

numactl --interleave=all "runspec command"

Launching a process with numactl --interleave=all sets the memory interleave policy so that memory will be allocated using round robin on nodes. When memory cannot be allocated on the current interleave target fall back to other nodes.

KMP_STACKSIZE

Specify stack size to be allocated for each thread.

KMP_AFFINITY

Syntax: KMP_AFFINITY=[<modifier>,...]<type>[,<permute>][,<offset>]
The value for the environment variable KMP_AFFINITY affects how the threads from an auto-parallelized program are scheduled across processors.
It applies to binaries built with -qopenmp and -parallel (Linux and Mac OS X) or /Qopenmp and /Qparallel (Windows).
modifier:
    granularity=fine Causes each OpenMP thread to be bound to a single thread context.
type:
    compact Specifying compact assigns the OpenMP thread <n>+1 to a free thread context as close as possible to the thread context where the <n> OpenMP thread was placed.
    scatter Specifying scatter distributes the threads as evenly as possible across the entire system.
permute: The permute specifier is an integer value controls which levels are most significant when sorting the machine topology map. A value for permute forces the mappings to make the specified number of most significant levels of the sort the least significant, and it inverts the order of significance.
offset: The offset specifier indicates the starting position for thread assignment.

Please see the Thread Affinity Interface article in the Intel Composer XE Documentation for more details.

Example: KMP_AFFINITY=granularity=fine,scatter
Specifying granularity=fine selects the finest granularity level and causes each OpenMP or auto-par thread to be bound to a single thread context.
This ensures that there is only one thread per core on cores supporting HyperThreading Technology
Specifying scatter distributes the threads as evenly as possible across the entire system.
Hence a combination of these two options, will spread the threads evenly across sockets, with one thread per physical core.

Example: KMP_AFFINITY=compact,1,0
Specifying compact will assign the n+1 thread to a free thread context as close as possible to thread n.
A default granularity=core is implied if no granularity is explicitly specified.
Specifying 1,0 sets permute and offset values of the thread assignment.
With a permute value of 1, thread n+1 is assigned to a consecutive core. With an offset of 0, the process's first thread 0 will be assigned to thread 0.
The same behavior is exhibited in a multisocket system.

OMP_NUM_THREADS

Sets the maximum number of threads to use for OpenMP* parallel regions if no other value is specified in the application. This environment variable applies to both -qopenmp and -parallel (Linux and Mac OS X) or /Qopenmp and /Qparallel (Windows). Example syntax on a Linux system with 8 cores: export OMP_NUM_THREADS=8

OMP_STACKSIZE

The OMP_STACKSIZE environment variable controls the size of the stack for threads created by the OpenMP implementation

Set stack size to unlimited

The command "ulimit -s unlimited" is used to set the stack size limit to unlimited.

Free the file system page cache

The command "echo 3> /proc/sys/vm/drop_caches" is used to free up the filesystem page cache as well as reclaimable slab objects like dentries and inodes.

MALLOC_CONF

Used for Jemalloc tuning at runtime. MALLOC_CONF=retain:true will retain unused virtual memory for later resue rather than discarding it.

Red Hat Specific features

Transparent Huge Pages

On RedHat EL 6 and later, Transparent Hugepages increase the memory page size from 4 kilobytes to 2 megabytes. Transparent Hugepages provide significant performance advantages on systems with highly contended resources and large memory workloads. If memory utilization is too high or memory is badly fragmented which prevents hugepages being allocated, the kernel will assign smaller 4k pages instead.
Hugepages are used by default unless the /sys/kernel/mm/redhat_transparent_hugepage/enabled field is changed from its RedHat EL6 default of 'always'.

Operating System Tuning Parameters

cpupower frequency-set

cpupower utility is a collection of tools for power efficiency of processor. frequency-set sub-command controls settings for processor frequency. "-g [governor]" specifies a policy to select processor frequency. The performance governor statically sets frequency of the processor cores specified by "-c" option to the highest possible for maximum performance.

nohz_full

This kernel option sets adaptive tick mode (NOHZ_FULL) to specified processors. Since the number of interrupts is reduced to ones per second, latency-sensitive applications can take advantage of it.

rcu_nocbs

This kernel option offloads RCU (Read-Copy-Update) callback processing from specified processors, aiming to reduce the processing load and improve efficiency. This can be advantageous for latency-sensitive applications and real-time systems.

Firmware / BIOS / Microcode Settings

Enhanced Halt State (C1E)

Enabling this option which is the default allows the processor to transmit to its minimum frequency when entering the power state C1. If the switch is disabled the CPU stays at its maximum frequency in C1. Because of the increase of power consumption users should only select this option after performing application benchmarking to verify improved performance in their environment. The default is "Enabled".

DCU Streamer Prefetcher

This BIOS switch allows 2 options: "Enabled" and "Disabled". The default is "Enabled".

This prefetcher is a L1 data cache prefetcher, which detects multiple loads from the same cache line done within a time limit, in order to then prefetch the next line from the L2 cache or the main memory into the L1 cache based on the assumption that the next cache line will also be needed.

Hardware Prefetcher

The hardware prefetcher operates transparently, without programmer intervention, to fetch streams of data and instruction from memory into the unified second-level cache. The prefetcher is capable of handling multiple streams in either the forward or backward direction. It is triggered when successive cache misses occur in the last-level cache and a stride in the access pattern is detected, such as in the case of loop iterations that access array elements. The prefetching occurs up to a page boundary. This feature can be disabled through the BIOS. Default is Enable.

Enforce DDR Memory Frequency POR

This BIOS switch allows 2 options: "POR" and "Disabled". Enable to enforce POR restriction for DDR5 frequency and voltage programming. Memory speeds will be capped accordingly. Disabling allows user selection of additional supported memory speeds.

Isoc Mode

Enabling the isochronous (ISOC) mode option reduces the credits available for memory traffic. For memory requests, this option reduces latency at the expense of throughput under heavy loads. Select Enable to enable Isochronous support to meet QoS (Quality of Service) requirements. This feature is especially important for Virtualization Technology.

Adjacent Cache Prefetch

The Adjacent Cache-Line Prefetch mechanism, like automatic hardware prefetch, operates without programmer intervention. When enabled through the BIOS, two 64-byte cache lines are fetched into a 128-byte sector, regardless of whether the additional cache line has been requested or not. In applications with relatively poor spatial locality, the cache miss ratio is higher. A cache miss on an Intel Pentium 4 processor-based system (with adjacent sector prefetch enabled) brings in 128 byte, leading to higher bus utilization (assuming that the application did not need the other 64 bytes). When adjacent sector prefetch is disabled, an Intel Pentium4 processor-based system only fetches 64 bytes. The other 64 bytes of the sector in the last-level cache are not used unless the application explicitly issues a load to that address. Disabling adjacent sector prefetch on Intel Pentium4 processor-based systems can reduce bus traffic. Default is Enable.

Hardware P-States

This BIOS switch allows 4 options: "Native Mode", "Disabled", "Out of Band Mode" and "Native Mode with No legacy Support". The default is "Native Mode".

With Hardware Power Management(HWPM) the processors provides a flexible interface between Hardware and Platform for performance management and improving energy efficiency.

In Native Mode the HWPM operates cooperatively with the OS via a software interface to provide constraints and hints.

When disabled, system does not use HWPM.

EIST PSD Function

Function of the Enhanced Intel SpeedStep Technology (EIST).

EIST reduces the latency inherent with changing the voltage-frequency pair (P-state), thus allowing those transitions to occur more frequently. This allows for more granular, demand-based switching and can optimize the power-to-performance balance, based on the demands of the applications.

ENERGY_PERF_BIAS_CFG mode

This BIOS switch allows 4 options: "Balanced Performance", "Performance", "Balanced Power" and "Power". The default is "Balanced Performance" optimized to maximum power savings with minimal impact on performance. "Performance" disables all power management options with any impact on performance. "Balanced Power" is optimized for power efficiency and "Power" for power savings. The BIOS switch is only selectable if the BIOS switch "Power Performance Tuning" is set to "BIOS Controls EPB".

The two options "Balanced Performance" and "Balanced Power" should always be the first choice as both options optimize the efficiency of the system. In cases where the performance is not sufficient or the power consumption is too high the two options "Performance" or "Power" could be an alternative.

Enable LP [Global]

This BIOS option enables or disables additional hardware thread which shares same physical core. Generally "ALL LPs" is recommended but disabling it makes sense for the application which requires the shortest possible response times. Default setting is "ALL LPs".

LLC dead line alloc

This BIOS switch allows 3 options: "Disabled","Enabled" and "Auto". The default is "Enabled". In the Sapphire non-inclusive cache scheme, the mid-level cache (MLC) evictions are filled into the last-level cache (LLC). When lines are evicted from the MLC, the core can flag them as "dead". The LLC has the option to drop dead lines and not fill them in the LLC. If the Dead Line LLC Alloc feature is disabled, dead lines will always be dropped and will never fill into the LLC. This can help save space in the LLC and prevent the LLC from evicting useful data. However, if the Dead Line LLC Alloc feature is enabled, the LLC can opportunistically fill dead lines into the LLC if there is free space available.

Package C State

This BIOS option allows 6 options: "C0/C1 state", "C2 state", "C6(non Retention) state", "C6(Retention) state", "No Limit" and "Auto". The default setting is "Auto". Package C-states is one of energy-saving options of the processor, which not only allow the individual cores of a processor, but the entire processor chip to be put into a type of sleep state. As a result, power consumption is even further reduced. But the "waking-up time" that is required to change from the lower package C-states to the active (C0) state is even longer in comparison with the CPU or core C-states. If the "C0" setting is made in the BIOS, the processor chip always remains active. It can improve the performance of latency sensitive workloads.

Patrol Scrub

This BIOS option enables or disables the so-called memory scrubbing, which cyclically accesses the main memory of the system in the background regardless of the operating system in order to detect and correct memory errors in a preventive way. The time of this memory test cannot be influenced and can under certain circumstances result in losses in performance. The disabling of the Patrol Scrub option increases the probability of discovering memory errors in case of active accesses by the operating system. Until these errors are correctable, the ECC technology of the memory modules ensures that the system continues to run in a stable way. However, too many correctable memory errors increase the risk of discovering non-correctable errors, which then result in a system standstill.

Stale AtoS

This BIOS switch allows 3 options: "Disabled","Enabled" and "Auto". The default is "Auto".

The in-memory directory has three states: I, A, and S. I (invalid) state means the data is clean and does not exist in any other socket's cache. A (snoopAll) state means the data may exist in another socket in exclusive or modified state. S (Shared) state means the data is clean and may be shared across one or more socket's caches.

When doing a read to memory, if the directory line is in the A state we must snoop all the other sockets because another socket may have the line in modified state. If this is the case, the snoop will return the modified data. However, it may be the case that a line is read in A state and all the snoops come back a miss. This can happen if another socket read the line earlier and then silently dropped it from its cache without modifying it.

If Stale AtoS feature is enabled, in the situation where a line in A state returns only snoop misses, the line will transition to S state. That way, subsequent reads to the line will encounter it in S state and not have to snoop, saving latency and snoop bandwidth. Stale AtoS may be beneficial in a workload where there are many cross-socket reads.

SNC

Sub NUMA Cluster (SNC) breaks up the last-level cache (LLC) into two disjoint clusters based on address range, with each cluster bound to one memory controller. SNC improves average latency to the LLC/memory and is a replacement for the "Cluster On Die" (COD) feature found in previous processor families. For a multi-socketed system, all SNC clusters are mapped to unique NUMA domains. The BIOS switch "Sub NUMA Clustering" allows 4 options: "Auto", "Disabled" ,"Enable SNC2 (2-clusters)" and "Enable SNC4 (4-clusters)". The default setting is "Auto".

Intel VT for Directed I/O

This BIOS option enables or disables I/O virtualization functions of the CPU. If the server is not used for virtualization, this option should be set to "Disabled". Default setting is "Enabled".

VMX

This BIOS option enables or disables additional virtualization functions of the CPU. If the server is not used for virtualization, this option should be set to "Disabled". This can result in energy savings. Default setting is "Enabled".

XPT Prefetch

This option configures the processor Xtended Prediciton Table (XPT) prefetch feature. The XPT prefetch exists on top of other prefetchers that that can prefetch data in the core DCU, MLC, and LLC. The XPT prefetcher will issue a speculative DRAM read request in parallel to an LLC lookup. This prefetch bypasses the LLC, saving latency. In some cases, setting this option to disabled can improve performance. In some cases, setting this option to disabled can improve performance. Typically, setting this option to enable provides better performance. This option must be enabled when Sub-NUMA Clustering is enabled. Values for this BIOS option can be: Enabled: Allows a read request sent to the LLC to speculatively issue a copy of the read to the memory controller requesting the prefetch.Default setting is "Enabled". Disabled: Does not allow the LLC to speculatively issue copies of reads. Disabling this will also disables Sub-NUMA Cluster (SNC).

Power Performance Tuning

This BIOS switch allows 3 options: "BIOS Controls EPB", "OS Controls EPB" and "PECI Controls EPB". The default is "BIOS Controls EPB".

This BIOS option decides who Controls EPB.In OS mode: IA32_ENERGY_PERF_BIAS is used, In BIOS mode: ENERGY_PERF_BIAS_CONFIG is used, In PECI mode: PCS53 is used.

FB Thread Slicing

This BIOS switch allows 2 options: "Disabled" and "Enabled". The default is "Disabled".

This BIOS option enables or disables FB (Fill Buffer) Thread Slicing per thread.Disable FB slicing per thread, When set, FB are sliced according to bit 6 in MT mode.

LLC Prefetch

This BIOS switch allows 2 options: "Disabled" and "Enabled". The default is "Disabled".

Last Level Cache (LLC) Prefetch This option configures the processor last level cache (LLC) prefetch feature as a result of the non-inclusive cache architecture. The LLC prefetcher exists on top of other prefetchers that can prefetch data into the core data cache unit (DCU) and mid-level cache (MLC). In some cases, setting this option to disabled can improve performance. Typically, setting this option to enable provides better performance.

AMP Prefetch

This BIOS switch allows 2 options: "Disabled" and "Enabled". The default is "Disabled".

a simple implementation called AMP (Adaptive Multi-stream Prefetching), which adapts accordingly leading to near optimal performance for any kind of sequential workload and cache size.

CPU C6 Report

This BIOS switch allows 3 options: "Disabled","Enabled" and "Auto". The default is "Disabled".

Enable or disable reporting of the CPU C6 State (ACPI C3) to the OS. During the CPU C6 State, the power to all cache is turned off.

Flag description origin markings:

	Indicates that the flag description came from the user flags file.
	Indicates that the flag description came from the suite-wide flags file.
	Indicates that the flag description came from a per-benchmark flags file.

For questions about the meanings of these flags, please contact the tester.
For other inquiries, please contact info@spec.org
Copyright 2017-2024 Standard Performance Evaluation Corporation
Tested with SPEC CPU2017 v1.1.9.
Report generated on 2024-11-20 11:11:28 by SPEC CPU2017 flags formatter v5178.