SPEC SFS®2014_swbuild Result

Copyright © 2016-2019 Standard Performance Evaluation Corporation

NetApp, Inc. SPEC SFS2014_swbuild = 6200 Builds
NetApp 12-node AFF A800 with FlexGroup Overall Response Time = 0.83 msec


Performance Graph

Product and Test Information

NetApp 12-node AFF A800 with FlexGroup
Tested byNetApp, Inc.
Hardware AvailableMay 2018
Software AvailableJanuary 2019
Date TestedNovember 2018
License Number33
Licensee LocationsSunnyvale, CA USA

Combined with the industry's first support of NVMe inside and out and NetApp ONTAP data management software, AFF A800 all-flash systems accelerate, manage, and protect your business-critical data with the industry's highest performance, superior flexibility, and best-in-class data management and cloud integration. By combining low-latency NVMe solid-state drives (SSDs) and the first NVMe over Fibre Channel (NVMe/FC) connectivity, the AFF A800 delivers ultra-low latency and massive throughput, and scales up to 24 nodes in a cluster.

The FlexGroup feature of ONTAP 9 enables massive scaling in a single namespace to over 20PB with over 400 billion files while evenly spreading the performance across the cluster. This makes the AFF A800 a great system for engineering and design applications as well as DevOps. It is particularly well-suited for chip development and software builds that are typically high file-count environments with high meta-data traffic.

Solution Under Test Bill of Materials

Item NoQtyTypeVendorModel/NameDescription
16Storage SystemNetAppAFF A800 Flash System (HA Pair, Active-Active Dual Controller)A single NetApp AFF A800 system is a single chassis with 2 controllers and 48 drive slots. Each set of 2 controllers comprises a High-Availability (HA) Pair. The words "controller" and "node" are used interchangeably in this document. Each AFF A800 HA Pair includes 1280GB of ECC memory, 128GB of NVRAM, 8 PCIe expansion slots and a set of included I/O ports: * 4x 40/100 GbE ports, in Slots numbered 1 in the controllers, configured as 100 GbE, used for cluster interconnect and HA connections; * 4x 40/100 GbE ports, in the mezzanine location in the controllers, configured as 100 GbE, used for cluster interconnect and HA connections; Included Premium Bundle which includes All Protocols, SnapRestore, SnapMirror, SnapVault, FlexClone, SnapManager Suite, Single Mailbox Recovery (SMBR), SnapCenter Foundation. Only the NFS protocol license is active in the test, also available in the BASE bundle.
212Network Interface CardNetApp2-Port 40/100 GbE QSFP28 X1146A1 card in Slot 3 of each controller; 2 cards per HA pair; each has 2 ports; used for data connections
3288Solid-State DriveNetApp3.84TB NVMe SSD X4002ANVMe Solid-State Drives (NVMe SSDs) installed in chassis, 48 per HA pair
472Network Interface CardIntelXXV710-DA22-port 25 GbE NIC, one installed per client. Only one 25GbE port per client was active during the test.
51SwitchCiscoCisco Nexus 9000 C9516Used for Ethernet data connections between clients and storage systems. Large switch in use due to testing having been done in a large shared-infrastructure lab. Only the ports used for this test are listed in this report. See the 'Transport Configuration - Physical' section for connectivity details.
612LinecardCiscoN9K-X97160YC-EXCisco Nexus 9500 48p 1/10/25G SFP+ plus 4p 100G QSFP cloud-scale line card. Clients connected to the 10/25 ports on these cards.
74LinecardCiscoN9K-X9732C-EXCisco Nexus 9500 32x100G Ethernet Module. The AFF A800 connected to these cards.
824Fibre Channel Interface CardEmulexQuad Port 32 Gb FC X1135ALocated in Slots 2 and 5 of each controller; there were 2 such cards in each controller; they were not used for this test. They were in place because this is a large shared-infrastructure lab environment; no I/O was directed through these cards during this test.
91SwitchCiscoCisco Nexus 3000 C3232CUsed for 100 GbE cluster interconnections
1072ClientSuperMicroCustom build from Superserver 5018R-W baseClients are a custom build ordered from Supermicro. The base build was a Superserver 5018R-W. The custom build includes a X10SRW-F motherboard, a CSE-815TQ-600WB chassis, a RSC-R1UW-2E16-O-P Dual PCI-Express 3.0 x16 Riser, a RSC-R1UW-E8R-O-P Single PCI-Express 3.0 x8 Riser Card, Intel Xeon E5-1630 v4 Quad-core Haswell Processors, and 4 DDR 2400 16GB ECC/REG DIMMs. 1 used as Prime Client; 71 used to generate the workload.
1110944Software Enablement/LicenseNetAppOS-ONTAP1-CAP1-PREM-2PCapacity-based License, Per 0.1TB

Configuration Diagrams

  1. NetApp AFF A800 12-Node Cluster with FlexGroup

Component Software

Item NoComponentTypeName and VersionDescription
1LinuxOSCentos Linux 6.10 (Kernel 4.18.12)Operating System (OS) for the 72 clients
2ONTAPStorage OS9.5Storage Operating System
3Data SwitchOperating System7.0(3)I6(1)Cisco switch NX-OS (system software)
4Cluster SwitchOperating System7.0(3)I6(1)Cisco switch NX-OS (system software)

Hardware Configuration and Tuning - Physical

Parameter NameValueDescription
MTU9000Jumbo Frames configured for Cluster Interconnect ports
MTU9000Jumbo Frames configured for data ports

Hardware Configuration and Tuning Notes

NetApp AFF A800 storage controller 100 GbE ports used for cluster interconnects and HA connections (8 per HA pair) were set up with MTU of 9000. Data network was set up with MTU of 9000.

Software Configuration and Tuning - Physical

Parameter NameValueDescription
rsize,wsize65536NFS mount options for data block size
protocoltcpNFS mount option for protocol
nfsvers3NFS mount option for NFS version
somaxconn65536Max tcp backlog an application can request

Software Configuration and Tuning Notes

Tuned the necessary client parameters as shown above, for communication between clients and storage controllers over Ethernet, to optimize data transfer and minimize overhead.

Service SLA Notes


Storage and Filesystems

Item NoDescriptionData ProtectionStable StorageQty
13.84 TB NVMe SSDs used for data and storage operating system; used to build three RAID-DP RAID groups per storage controller node in the clusterRAID-DPYes288
2960GB NVMe M.2 device, 2 per HA pair; used as boot medianoneYes12
Number of Filesystems1
Total Capacity671.5 TiB
Filesystem TypeNetApp FlexGroup

Filesystem Creation Notes

The single FlexGroup consumed all data volumes from all of the aggregates across all of the nodes. In order to support the high number of files required by the benchmark, immediately after creation the default inode density of the FlexGroup was increased. At the same time the FlexGroup's snapshot reserve percentage was also set to zero. These actions were accomplished with a single 'volume modify' command with option '-files 12000000000' and option '-percent-snapshot-space 0'.

Storage and Filesystem Notes

The storage configuration consisted of 6 AFF A800 HA pairs (12 controller nodes total). The two controllers in each HA pair were connected in a SFO (storage failover) configuration. Together, all 12 controllers (configured as 6 HA Pairs) comprise the tested AFF A800 HA cluster. Stated in the reverse, the tested AFF A800 HA cluster consists of 6 HA Pairs, each of which consists of 2 controllers (also referred to as nodes).

Each storage controller was connected to its own and partner's NVMe drives in a multi-path HA configuration.

All NVMe SSDs were in active use during the test. In addition to the factory configured RAID Group housing its root aggregate, each storage controller was configured with two 21+2 RAID-DP RAID Groups. There were 2 data aggregates on each node, each of which consumed one of the node's two 21+2 RAID-DP RAID Groups. 8x volumes, holding benchmark data, were created within each aggregate. "Root aggregates" hold ONTAP operating system related files. Note that spare (unused) drive partitions are not included in the "storage and filesystems" table because they held no data during the benchmark execution.

A storage virtual machine or "SVM" was created on the cluster, spanning all storage controller nodes. Within the SVM, a single FlexGroup volume was created using the two data aggregates on each controller. A FlexGroup volume is a scale-out NAS single-namespace container that provides high performance along with automatic load distribution and scalability.

Transport Configuration - Physical

Item NoTransport TypeNumber of Ports UsedNotes
125 GbE and 100 GbE96For the client-to-storage network, the AFF A800 Cluster used a total of 24x 100 GbE connections from storage to the switch, communicating via NFSv3 over TCP/IP to 72 clients, via one 25 GbE connection to the switch for each client. MTU=9000 was used for data switch ports. The benchmark was conducted in a large shared-infrastructure lab; only the ports shown and documented were used on the Cisco Nexus C9516 switch for this benchmark test.
2100 GbE24The Cluster Interconnect network is connected via 100 GbE to a Cisco C3232 switch, with 4 connections to each HA pair.

Transport Configuration Notes

Each NetApp AFF A800 HA Pair used 4x 100 GbE ports for data transport connectivity to clients (through the Cisco C9516 switch), Item 1 above. Each of the clients driving workload used one 25GbE port for data transport. All ports on the Item 1 network utilized MTU=9000. The Cluster Interconnect network, Item 2 above, utilized MTU=9000. All interfaces associated with dataflow are visible to all other interfaces associated with dataflow.

Switches - Physical

Item NoSwitch NameSwitch TypeTotal Port CountUsed Port CountNotes
1Cisco Nexus C951625GbE and 100 GbE Switch969672 client-side 25 GbE data connections; 24 storage-side 100 GbE data connections. Only the ports on the Cisco Nexus C9516 used for the solution under test are included in the total port count.
2Cisco Nexus C3232100 GbE Switch3224For Cluster Interconnect

Processing Elements - Physical

Item NoQtyTypeLocationDescriptionProcessing Function
124CPUStorage Controller2.10 GHz Intel Xeon Platinum 8160NFS, TCP/IP, RAID and Storage Controller functions
272CPUClient3.70 GHz Intel Xeon E5-1630NFS Client, Linux OS

Processing Element Notes

Each of the 12 NetApp AFF A800 Storage Controllers contains 2 Intel Xeon 8160 processors with 24 cores each; 2.10 GHz, hyperthreading disabled. Each client contains 1 Intel Xeon E5-1630 processor with 4 cores at 3.70 GHz, hyperthreading enabled.

Memory - Physical

DescriptionSize in GiBNumber of InstancesNonvolatileTotal GiB
Main Memory for each NetApp AFF A800 HA Pair12806V7680
NVDIMM (NVRAM) Memory for each NetApp AFF A800 HA pair1286NV768
Memory for each client; 71 of these drove the workload6472V4608
Grand Total Memory Gibibytes13056

Memory Notes

Each storage controller has main memory that is used for the operating system and caching filesystem data. Each controller also has NVRAM; See "Stable Storage" for more information.

Stable Storage

The AFF A800 utilizes non-volatile battery-backed memory (NVRAM) for write caching. When a file-modifying operation is processed by the filesystem (WAFL) it is written to system memory and journaled into a non-volatile memory region backed by the NVRAM. This memory region is often referred to as the WAFL NVLog (non-volatile log). The NVLog is mirrored between nodes in an HA pair and protects the filesystem from any SPOF (single-point-of-failure) until the data is de-staged to disk via a WAFL consistency point (CP). In the event of an abrupt failure, data which was committed to the NVLog but has not yet reached its final destination (disk) is read back from the NVLog and subsequently written to disk via a CP.

Solution Under Test Configuration Notes

All clients accessed the FlexGroup from all the available network interfaces.

Unlike a general-purpose operating system, ONTAP does not provide mechanisms for non-administrative users to run third-party code. Due to this behavior, ONTAP is not affected by either the Spectre or Meltdown vulnerabilities. The same is true of all ONTAP variants including both ONTAP running on FAS/AFF hardware as well as virtualized ONTAP products such as ONTAP Select and ONTAP Cloud. In addition, FAS/AFF BIOS firmware does not provide a mechanism to run arbitrary code and thus is not susceptible to either the Spectre or Meltdown attacks. More information is available from https://security.netapp.com/advisory/ntap-20180104-0001/.

None of the components used to perform the test were patched with Spectre or Meltdown patches (CVE-2017-5754,CVE-2017-5753,CVE-2017-5715).

Other Solution Notes

ONTAP Storage Efficiency techniques including inline compression and inline deduplication were enabled by default, and were active during this test. Standard data protection features, including background RAID and media error scrubbing, software validated RAID checksum, and double disk failure protection via double parity RAID (RAID-DP) were enabled during the test. WARMUP_TIME has been set to a value of 600 seconds.


Please reference the configuration diagram. 72 clients were used to generate the workload; 1 client acted as Prime Client to control the 71 other clients. Each client used one 25 GbE connection, through a Cisco Nexus C9516 switch. Each storage HA pair had 4x 100 GbE connections to the data switch. The filesystem consisted of one ONTAP FlexGroup. The clients mounted the FlexGroup volume as an NFSv3 filesystem. The ONTAP cluster provided access to the FlexGroup volume on every 100 GbE port connected to the data switch (16 ports total). Each cluster node had 2 Logical Interfaces (LIFs) per 100GbE Port, for a total of 4 LIFs per node, for a total of 48 LIFs for the AFF A800 cluster. Each client created mount points across those 48 LIFs symmetrically.

Other Notes


Other Report Notes

NetApp is a registered trademark and "Data ONTAP", "FlexGroup", and "WAFL" are trademarks of NetApp, Inc. in the United States and other countries. All other trademarks belong to their respective owners and should be treated as such.

Generated on Wed Mar 13 16:19:32 2019 by SpecReport
Copyright © 2016-2019 Standard Performance Evaluation Corporation