SPEC HPC 2021 Flag Description - Platform settings

Firmware / BIOS / Microcode Settings

HBM Memory Modes

HBM can be exposed to software using three different memory modes. These memory modes are selected through the BIOS menu when the system boots up. These modes are: HBM-only mode, Flat mode, Cache mode.

HBM-only Mode:

If the system does not have any DDR modules installed, it boots up in the HBM-only mode. In this mode, HBM appears to software as a single flat-memory address-space, no differently than how DDR is used on DDR-only-based systems. Existing software does not have to take any additional steps to use HBM memory in this mode.

Flat Mode:

When DDR modules are installed on a system, the user can select flat mode. In this mode, both DDR and HBM address spaces are visible to software. On a socket, each address space is exposed as a separate Non-uniform Memory Access (NUMA) node. By default, all allocations go to NUMA node 0, which is usually DDR. As a result, OS, and other services allocate memory from DDR, and therefore memory on HBM node is completely available for applications. This behavior is the primary advantage of flat mode as compared to the HBM-only mode, which uses HBM memory for non-application tasks. Exposing DDR and HBM as two separate NUMA nodes allows the use of standard Linux NUMA utilities and interfaces (for example, numactl and libnuma) to place an application in the desired address space.

Cache Mode:

On systems with both HBM and DDR, cache mode allows HBM to function as a memory-side cache, which caches the contents of DDR. In this mode, HBM is transparent to all software because the HBM cache is managed by hardware memory controllers. The entire DDR space is visible to software. Since the HBM cache is transparent to software, cache mode does not require any software modifications to take advantage of HBM. A symmetric population of DIMMs among the four memory controllers is required for cache mode. For best performance, all eight DDR channels should be populated. In cache mode, HBM is organized as a direct-mapped cache.