SPEC SFS®2014_swbuild Result

The Spectrum Scale configuration parameter was set using the mmchconfig command on one of the nodes in the cluster. The verbs settings in the table above allow for efficient use of the RoCE infrastructure. The settings determine when data are transferred over IP and when they are transferred using the verbs protocol.

The configuration parameters for the Spectrum Scale file system were set using the mmchconfig command on one of the nodes in the cluster. The nodes used mostly default tuning parameters. A discussion of Spectrum Scale tuning can be found in the official documentation for the mmchconfig command and on the IBM developerWorks wiki at https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/fa32927c-e904-49cc-a4cc-870bcc8e307c/page/Tuning%20Parameters (for additional information on tuning parameters see http://files.gpfsug.org/presentations/2014/UG10_GPFS_Performance_Session_v10.pdf).

The E8 Storage controller used default parameters for all volumes.

There were no opaque services in use

A single Spectrum Scale file system was created with a 1 MiB block size for data and metadata, and 4 KiB inode size. The 14TB file system has 1 mixed data and metadata volume (aka pool). Each client node mounted the file system.

The nodes each had an ext4 file system that hosted the operating system.

The E8 Storage appliance has 24 750GB Intel Optane drives configured as a 22+2 RAID group for data protection, with a 16+2 stripe size. The mixed data / metadata volume was provisioned from this single RAID group, with the volume spanning all drives in the RAID group. All client nodes had shared read / write access to the single volume.

The cluster used a single-tier architecture. The Spectrum Scale client nodes performed both file and block level operations. Each node had access to shared volumes, so any file operation on a node was translated to a block operation and serviced on the same node.

The 1GbE network was used for administrative purposes and for Spectrum Scale inter-node communication. All benchmark traffic flowed through the Mellanox SN2700 100Gb Ethernet switch. 15 client nodes had a single active 50Gb Ethernet port which was connected to the switch with a split cable (2 50GbE clients per 100GbE switch port), with the remaining 15 clients direct connect at 100Gb Ethernet to the switch.

Each of the Spectrum Scale client nodes had 2 physical processors. On 15 clients, each processor had 6 physical cores with one thread per core by default. On the other 15 clients, each processor had 16 physical cores with one thread per core by default. The client nodes were running the following kernel, which contains some but not all of the Spectre/Meltdown patches, and no other patches were applied: 3.10.0-862.el7.x86_64

The E8-X24 is a dual controller appliance, each controller has 2 physical processors. Each processor had 14 cores with one thread per core. All available Spectre/Meltdown patches have been applied to the controller CPUs. (more info here: https://access.redhat.com/security/vulnerabilities/speculativeexecution)

In the client nodes, Spectrum Scale reserves a portion of the physical memory (per the pagepool setting above) for file data and metadata caching. Some additional memory is dynamically allocated for buffers used for node to node communication and up to 10 million inode stat informations per node.

In the E8 Storage controller, a portion of the physical memory is reserved for block write data and system metadata caching.