OS Tuning
ulimit:
Used to set user limits of system-wide resources. Provides control over resources available to the shell and processes started by it. Some common ulimit commands may include:
Disabling Linux services:
Certain Linux services may be disabled to minimize tasks that may consume CPU cycles.
irqbalance:
Disabled through "service irqbalance stop". Depending on the workload involved, the irqbalance service reassigns various IRQ's to system CPUs. Though this service might help in some situations, disabling it can also help environments which need to minimize or eliminate latency to more quickly respond to events.
Performance Governors (Linux):
In-kernel CPU frequency governors are pre-configured power schemes for the CPU. The CPUfreq governors use P-states to change frequencies and lower power consumption. The dynamic governors can switch between CPU frequencies, based on CPU utilization to allow for power savings while not sacrificing performance.
Other options beside a generic performance governor can be set, such as the perf-bias:
--perf-bias, -b
On supported Intel processors, this option sets a register which allows the cpupower utility (or other software/firmware) to set a policy that controls the relative importance of performance versus energy savings to the processor. The range of valid numbers is 0-15, where 0 is maximum performance and 15 is maximum energy efficiency.
The processor uses this information in model-specific ways when it must select trade-offs between performance and energy efficiency. This policy hint does not supersede Processor Performance states (P-states) or CPU Idle power states (C-states), but allows software to have influence where it would otherwise be unable to express a preference.
On many Linux systems one can set the perf-bias for all CPUs through the cpupower utility with one of the following commands:
Tuning Kernel parameters:
The following Linux Kernel parameters were tuned to better optimize performance of some areas of the system:
tuned-adm:
The tuned-adm tool is a commandline interface for switching between different tuning profiles available to the tuned tuning daeomn available in supported Linux distros. The default configuration file is located in /etc/tuned.conf and the supported profiles can be found in /etc/tune-profiles.
Some profiles that may be available by default include: default, desktop-powersave, server-powersave, laptop-ac-powersave, laptop-battery-powersave, spindown-disk, throughput-performance, latency-performance, enterprise-storage
To set a profile, one can issue the command "tuned-adm profile (profile_name)". Here are details about relevant profiles.
Transparent Huge Pages (THP):
THP is an abstraction layer that automates most aspects of creating, managing, and using huge pages. THP is designed to hide much of the complexity in using huge pages from system administrators and developers, as normal huge pages must be assigned at boot time, can be difficult to manage manually, and often require significant changes to code in order to be used effectively. Transparent Hugepages increase the memory page size from 4 kilobytes to 2 megabytes. Transparent Hugepages provide significant performance advantages on systems with highly contended resources and large memory workloads. If memory utilization is too high or memory is badly fragmented which prevents hugepages being allocated, the kernel will assign smaller 4k pages instead. Most recent Linux OS releases have THP enabled by default.
Linux Huge Page settings:
If you need finer control and manually set the Huge Pages you can follow the below steps:
Note that further information about huge pages may be found in your Linux documentation file: /usr/src/linux/Documentation/vm/hugetlbpage.txt
Firmware Settings
One or more of the following settings may have been set. If so, the "Platform Notes" section of the report will say so; and you can read below to find out more about what these settings mean.
Intel Hyper-Threading (Default = Enabled):
This feature allows enabling or disabling of logical processor cores on processors supporting Intel Hyper-Threading (HT). When enabled, each physical processor core operates as two logical processor cores. When disabled, each physical core operates as only one logical processor core. Enabling this option can improve overall performance for applications that benefit from a higher processor core count.
Intel Virtualization Technology (Intel VT, VT-x) (Default = Enabled):
When enabled, a hypervisor or operating system supporting this option can use hardware capabilities provided by Intel VT. Some hypervisors require that you enable Intel VT. You can leave this set to enabled even if you are not using a hypervisor or an operating system that uses this option. With default BIOS settings as shipped with most systems, the default state for this setting is Enabled. However, this setting can change it's default setting depending on the Workload Profile that is selected, or what Workload Profile is default for the a certain system.
VT-d (Intel VT-d) (Default = Enabled):
If enabled, a hypervisor or operating system supporting this option can use hardware capabilities provided by Intel VT for Directed I/O. You can leave this set to enabled even if you are not using a hypervisor or an operating system that uses this option. With default BIOS settings as shipped with most systems, the default state for this setting is Enabled. However, this setting can change it's default setting depending on the Workload Profile that is selected, or what Workload Profile is default for the a certain system.
Processor x2APIC Support (Default = Enabled):
If enabled, x2APIC support enables operating system to run more efficiently on high core count configurations. It also optimizes interrupt distribution in virtualized environments. Setting this option to enables is recommended for most cases. When enabled, the operating system can optionally enable x2APCI support when it loads. Older hypervisors and operating systems might have issues with optional x2APIC support, therefore disabling x2APIC could be necessary to address these issues. Setting this option to enabled also forces Intel VT-D to be enabled.
SR-IOV (Default = Enabled):
If enabled, SR-IOV support enables a hypervisor to create virtual instances of PCI-express device, potentially increasing performance. If enabled, the BIOS allocates additional resources to PCI-express devices. You can leave this option set to enabled even if you are not using a hypervisor. With default BIOS settings as shipped with most systems, the default state for this setting is Enabled. However, this setting can change it's default setting depending on the Workload Profile that is selected, or what Workload Profile is default for the a certain system.
Thermal Configuration (Default = Optimal Cooling):
This feature allows the user to select the fan cooling solution for the system. Values for this BIOS option can be:
Last Level Cache (LLC) Dead Line Allocation (Default = Enabled):
In the Xeon Scalable processor cache scheme, mid-level cache (MLC) evictions are filled into the last level cache (LLC). If a line is evicted from the MLC to the LLC, the core can flag the evicted MLC lines as "dead". This means that the lines are not likely to be read again. This option allows dead lines to be dropped and never fill the LLC if the option is disabled. Values for this BIOS option can be:
Enhanced Processor Performance (Default = Disabled):
Use this option to enable the Enhanced Processor Performance setting. When enabled, this option will adjust the processor settings to a more aggressive setting that can result in improved performance, but may result in higher power consumption. Values for this BIOS option can be either disabled or enabled.
Enhanced Processor Performance Profile (Default = Moderate):
Use this option to enable the Enhanced Processor Performance Profile setting. In order to set this option, the Enhanced Processor Performance option must be set to Enabled. This allows a user to choose between 3 profiles: conservative, moderate, and aggressive.
Stale A to S (Default = Disabled):
The in-memory directory has three states: invalid (I), snoopAll (A), and shared (S). Invalid (I) state means the data is clean and does not exist in any other socket`s cache. The snoopAll (A) state means the data may exist in another socket in exclusive or modified state. Shared (S) state means the data is clean and may be shared across one or more socket`s caches. When doing a read to memory, if the directory line is in the A state we must snoop all the other sockets because another socket may have the line in modified state. If this is the case, the snoop will return the modified data. However, it may be the case that a line is read in A state and all the snoops come back a miss. This can happen if another socket read the line earlier and then silently dropped it from its cache without modifying it. Values for this BIOS option can be:
Stale A to S may be beneficial in a workload where there are many cross-socket reads.
Last Level Cache (LLC) Prefetch (Default = Disabled):
This option configures the processor Last Level Cache (LLC) prefetch feature as a result of the non-inclusive cache architecture. The LLC prefetcher exists on top of other prefetchers that that can prefetch data in the core data cache unit (DCU) and mid-level cache(MLC). In some cases, setting this option to disabled can improve performance. Typically, setting this option to enable provides better performance. Values for this BIOS option can be:
NUMA Group Size Optimization (Default = Clustered):
This feature allows the user to configure how the BIOS reports the size of a NUMA node (number of logical processors), which assists the Operating System in grouping processors for application use (referred to as Kgroups). Values for this BIOS option can be:
Sub-NUMA Clustering (SNC) (Default = Enabled):
SNC breaks up the last level cache (LLC) into disjoint clusters based on address range, with each cluster bound to a subset of the memory controllers in the system. SNC improves average latency to the LLC and memory. SNC is a replacement for the cluster on die (COD) feature found in previous processor families. For a multi-socketed system, all SNC clusters are mapped to unique NUMA domains. (See also IMC interleaving.) Values for this BIOS option can be:
Xtended Prediction Table (XPT) Prefetch (Default = Enabled):
This option configures the processor Xtended Prediciton Table (XPT) prefetch feature. The XPT prefetcher exists on top of other prefetchers that that can prefetch data in the core DCU, MLC, and LLC. The XPT prefetcher will issue a speculative DRAM read request in parallel to an LLC lookup. This prefetch bypasses the LLC, saving latency. In some cases, setting this option to disabled can improve performance. In some cases, setting this option to disabled can improve performance. Typically, setting this option to enable provides better performance. This option must be enabled when Sub-NUMA Clustering is enabled. Values for this BIOS option can be:
DCU Stream Prefetcher (Default = Enabled):
In most environments, leave the option enabled for optimal performance. With certain workloads, disabling it might provide a performance benefit. Do so only after performing application benchmarking to verify improved performance in a particular environment. Values for this BIOS option can be:
Uncore Frequency Scaling (Default = Auto):
This option controls the frequency scaling of the processor`s internal buses (the uncore). Values for this BIOS option can be:
Workload Profile (Default = General Power Efficient Compute):
This option allows a user to choose one workload profile that best fits the user`s needs. The workload profiles control many power and performance settings that are relevant to general workload areas. Values for this BIOS option can be:
Power Regulator (Default = Dynamic Power Savings Mode):
This option can be manually configured if the Power Profile is set to Custom. The default value is associated with the default value of the Workload Profile - General Power Efficient Compute. If the Workload Profile changes, the default value of this setting may change. Values for this BIOS setting can be:
Minimum Processor Idle Power Core C-State (Default = C6 State):
This option can only be configured if the Workload Profile is set to Custom, or this option is not a dependent value for the Workload Profile. This feature selects the processor's lowest idle power state (C-state) that the operating system uses. The higher the C-state, the lower the power usage of that idle state (C6 is the lowest power idle state supported by the processor). Values for this setting can be:
Minimum Processor Idle Power Package C-State (Default = Package C6 (retention) State):
This option can only be configured if the Workload Profile is set to Custom, or this option is not a dependent value for the Workload Profile. This feature selects the processor's lowest idle package power state (C-state) that is enabled. The processor will automatically transition into the package C-states based on the Core C-states, in which cores on the processor have transitioned. The higher the package C-state, the lower the power usage of that idle package state. Package C6 (retention) is the lowest power idle package state supported by the processor). Values for this setting can be:
Collaborative Power Control (Default = Enabled):
This BIOS option allows the enabling/disabling of the Processor Clocking Control (PCC) Interface. This option can be manually configured if the Power Profile is set to Custom. The default value is associated with the default value of the Workload Profile - General Power Efficient Compute. If the Workload Profile changes, the default value of this setting may change.
For operating systems which support this feature, enabling this option allows the Operating System to request processor frequency changes even when the server has the Power Regulator option configured for Dynamic Power Savings Mode.
For Operating Systems that do not support the PCC Interface or when the Power Regulator Mode is not configured for Dynamic Power Savings Mode, this option has no impact on system operation.
Energy/Performance Bias (Default = Balanced Performance):
This option can only be configured if the Workload Profile is set to Custom, or this option is not a dependent value for the Workload Profile. This option configures several processor subsystems to optimize the processor's performance and power usage. Values for this BIOS setting can be:
Energy Efficient Turbo (Default = Enabled):
This option controls whether the processor uses an energy efficiency based policy when engagin turbo range frequencies. This option is only applicable when Turbo Mode is enabled. Values for this BIOS setting can be: Enabled or Disabled.
AHS PCI Logging Level (Default = Verbose Logging):
This option allows the AHS PCI Logging size to be changed. This is a boot time option that should have no effect on run time performance. Values for this BIOS setting can be:
Memory Patrol Scrubbing (Default = Enabled):
This option allows for correction of soft memory errors. Over the length of system runtime, the risk of producing multi-bit and uncorrected errors is reduced with this option. Values for this BIOS setting can be:
HW Prefetcher (Default = Enabled):
Use this option to disable the processor HW Prefetch feature. In some cases, setting this option to disabled can improve performance. Typically, setting this option to enabled provides better performance. Only disable this option after performing application benchmarking to verify improved performance in the environment. The HW Prefetcher fetches streams of data and instruction from the memory into the second-level (L2) cache if it determins this data is likely to be required in the near future. The prefetcher is capable of handling multiple streams in either the forward or backward direction. The HW Prefetcher is triggered when successive cache misses occur in the last-level cache and a stride in the access pattern is detected, such as in the case of loop iterations that access array elements. The prefetching occurs up to a page boundary. This option can reduce the latency associated with memory reads. Values for this BIOS setting can be enabled or disabled.
Adjacent Sector Prefetch (Default = Enabled):
Use this option to disable the processor Adjacent Sector Prefetch feature. In some cases, setting this option to disabled can improve performance. Typically, setting this option to enabled provides better performance. Only disable this option after performing application benchmarking to verify improved performance in the environment. The Adjacent Sector Prefetcher retrieves both sectors of a cache line when it requires data that isn't currently in the cache. When disabled, the processor will only fetch the sector of the cache line that includes the requested data. Values for this BIOS setting can be enabled or disabled.
XPT Remote Prefetcher (Default = Auto):
Use this option to configure the Remote XPT Prefetcher processor performance option. When enabled, this feature can improve remote read request latency from a processor core by directly accessing the UPI. Values for this BIOS setting can be auto, enabled, or disabled.
Intel UPI Link Enablement (Default = Auto):
Use this option to configure the UPI topology to use fewer links between processors, when available. Changing from the default can reduce UPI bandwidth performance in exchange for less power consumption. Values for this BIOS setting can be: Auto and Single Link Operation.
Intel UPI Link Power Management (Default = Enabled):
Use this option to place the Quick Path Interconnect (UPI) links into a low power state when the links are not being used. This lowers power usage with minimal effect on performance. You can only configure this option if two or more CPUs are present and the Workload Profile is set to Custom. Values for this BIOS setting can be: enabled and disabled.
Intel UPI Link Frequency (Default = Auto):
Use this option to set the UPI Link frequency to a lower speed. Running at a lower frequency can reduce power consupmption, but can also affect system performance. You can only configure this option if two or more CPUs are present and the Workload Profile is set to Custom. Values for this BIOS setting can be: Auto and Min UPI Speed.
Direct to UPI (D2K) (Default = Auto):
Allows for enabling/disabling of this feature. This option can have an effect on reducing LLC miss latency. Values for this BIOS setting can be:
Advanced Memory Protection (Default = Advanced ECC Support):
Use this option to configure additional memory protection with ECC (Error Checking and Correcting). Options and support vary per system and configuration. Values for this BIOS setting can be:
Intel Speed Select Technology - Base Frequency (Default value = Disabled):
Intel Speed Select Techonology - Base Frequency support is available only on select processor models. Processors with Prioritized Base Frequency support a higher base frequency for a select number of cores (high priority cores) while the remaining cores will have a lower base frequency (low priority cores). Enabling this setting will result in increasing the CPU base frequency for the high priority cores and decreasing the CPU base frequency for the low priority cores. Consult processor documentation for more information on priority core counts and frequency adjustments. Values for this BIOS setting can be:
Last modified January 11, 2022.