SPEC SFS®2014_swbuild Result: IBM Corporation - IBM Spectrum Scale 4.2.1 with Cisco UCS and IBM FlashSystem 900

IBM Corporation	SPEC SFS2014_swbuild = 240 Builds
IBM Spectrum Scale 4.2.1 with Cisco UCS and IBM FlashSystem 900	Overall Response Time = 1.32 msec

IBM Spectrum Scale 4.2.1 with Cisco UCS and IBM FlashSystem 900
Tested by	IBM Corporation
Hardware Available	February 2016
Software Available	September 2016
Date Tested	November 2016
License Number	11
Licensee Locations	Raleigh, NC USA

IBM Spectrum Scale helps solve the challenge of explosive growth of unstructured data against a flat IT budget.

Spectrum Scale provides unified file and object software-defined storage for high performance, large scale workloads on-premises or in the cloud. Spectrum Scale includes the protocols, services and performance required by many industries, Technical Computing, Big Data, HDFS and business critical content repositories. IBM Spectrum Scale provides world-class storage management with extreme scalability, flash accelerated performance, and automatic policy-based storage tiering from flash through disk to tape, reducing storage costs up to 90% while improving security and management efficiency in cloud, big data & analytics environments.

Cisco UCS is the first truly unified data center platform that combines industry- standard, x86-architecture servers with networking and storage access into a single system. The system is intelligent infrastructure that is automatically configured through integrated, model-based management to simplify and accelerate deployment of all kinds of applications. The system's x86-architecture rack and blade servers are powered exclusively by Intel(R) Xeon(R) processors and enhanced with Cisco innovations. These innovations include the capability to abstract and automatically configure the server state, built-in virtual interface cards (VICs), and leading memory capacity. Cisco's enterprise-class servers deliver world-record performance to power mission-critical workloads. , Cisco UCS is integrated with a standards-based, high-bandwidth, low-latency, virtualization-aware 10-Gbps unified fabric, with a new generation of Cisco UCS fabric enabling an update to 40 Gbps.

Solution Under Test Bill of Materials

Item No	Qty	Type	Vendor	Model/Name	Description
1	1	Blade Server Chassis	Cisco	UCS 5108	The Cisco UCS 5108 Blade Server Chassis features flexible bay configurations for blade servers. It can support up to eight half-width blades, up to four full-width blades, or up to two full-width double-height blades in a compact 6-rack-unit (6RU) form factor.
2	4	Blade Server, Spectrum Scale Node	Cisco	UCS B200 M4	UCS B200 M4 Blade Servers, each with: 2X Intel Xeon processors E5-2680 v3 (24 core per node) 256 GB of memory
3	2	Fabric Interconnect	Cisco	UCS 6332-16UP	Cisco UCS 6300 Series Fabric Interconnects support line-rate, lossless 40 Gigabit Ethernet and FCoE connectivity.
4	2	Fabric Extender	Cisco	UCS 2304	Cisco UCS 2300 Series Fabric Extenders can support up to four 40-Gbps unified fabric uplinks per fabric extender connecting Fabric Interconnect.
5	4	Virtual Interface Card	Cisco	UCS VIC 1340	The Cisco UCS Virtual Interface Card (VIC) 1340 is a 2-port, 40 Gigabit Ethernet, Fibre Channel over Ethernet (FCoE)-capable modular LAN on motherboard (mLOM) mezzanine adapter.
6	2	FlashSystem	IBM	9840-AE2	Each FlashSystem was configured with 12 2.9TB IBM MicroLatency modules (feature code AF24).

Configuration Diagrams

Component Software

Item No	Component	Type	Name and Version	Description
1	Spectrum Scale Nodes	Spectrum Scale File System	4.2.1.1	The Spectrum Scale File System is a distributed file system that runs on the Cisco UCS B200 M4 servers to form a cluster. The cluster allows for the creation and management of single namespace file systems.
2	Spectrum Scale Nodes	Operating System	Red Hat Enterprise Linux 7.2 for x86_64	The operating system on the Spectrum Scale nodes was 64-bit Red Hat Enterprise Linux version 7.2.
3	FlashSystem 900	Storage System	1.4.4.2	The FlashSystem software covers all aspects of administering, configuring, and monitoring the FlashSystem 900.

Hardware Configuration and Tuning - Physical

Spectrum Scale Nodes
Parameter Name	Value	Description
numaMemoryInterleave	yes	Enables memory interleaving on NUMA based systems.
multipath device: path_selector	queue-length 0	Determines which algorithm to use when selecting paths. With this value the path with the least amount of outstanding I/O is selected.
multipath device: path_grouping_policy	multibus	Determines which grouping policy to use for a set of paths. The multibus value causes all paths to be placed in one priority group.

Hardware Configuration and Tuning Notes

The first configuration parameter was set using the "mmchconfig" command on one of the nodes in the cluster. The multipath device parameters were set in the multipath.conf file on each node. A template multipath.conf file for the FlashSystem can be found in the "Implementing IBM FlashSystem 900" Redbook, published by IBM.

Software Configuration and Tuning - Physical

Spectrum Scale Nodes
Parameter Name	Value	Description
ignorePrefetchLUNCount	yes	Specifies that only maxMBpS and not the number of LUNs should be used to dynamically allocate prefetch threads.
maxblocksize	1M	Specifies the maximum file system block size.
maxMBpS	10000	Specifies an estimate of how many megabytes of data can be transferred per second into or out of a single node.
maxStatCache	0	Specifies the number of inodes to keep in the stat cache.
pagepoolMaxPhysMemPct	90	Percentage of physical memory that can be assigned to the page pool
scatterBufferSize	256K	Specifies the size of the scatter buffers.
workerThreads	1024	Controls the maximum number of concurrent file operations at any one instant, as well as the degree of concurrency for flushing dirty data and metadata in the background and for prefetching data and metadata.
maxFilesToCache	11M	Specifies the number of inodes to cache for recently used files that have been closed.
pagepool	96G	Specifies the size of the cache on each node.
nsdBufSpace	70	Sets the percentage of the pagepool that is used for NSD (Network Shared Disk) buffers.
nsdMaxWorkerThreads	3072	Sets the maximum number of threads to use for block level I/O on the NSDs.
nsdMinWorkerThreads	3072	Sets the minimum number of threads to use for block level I/O on the NSDs.
nsdMultiQueue	64	Specifies the maximum number of queues to use for NSD I/O.
nsdThreadsPerDisk	3	Specifies the maximum number of threads to use per NSD.
nsdThreadsPerQueue	48	Specifies the maximum number of threads to use per NSD I/O queue.
nsdSmallThreadRatio	1	Specifies the ratio of small thread queues to small thread queues.

Software Configuration and Tuning Notes

The configuration parameters were set using the "mmchconfig" command on one of the nodes in the cluster. Both the nodes used mostly default tuning parameters. A discussion of Spectrum Scale tuning can be found in the official documentation for the mmchconfig command and on the IBM developerWorks wiki.

Service SLA Notes

There were no opaque services in use.

Storage and Filesystems

Item No	Description	Data Protection	Stable Storage	Qty
1	FlashSystem 900 volumes, 700 GiB each, used as Network Shared Drives for Spectrum Scale.	RAID-5	Yes	64
2	FlashSystem 900 volumes, 100 GiB each, used to store the operating system of each Spectrum Scale node.	RAID-5	Yes	4

Number of Filesystems	1
Total Capacity	44800 GiB
Filesystem Type	Spectrum Scale File System

Filesystem Creation Notes

A single Spectrum Scale file system was created with a 128 MiB block size for data and metadata, 4 KiB inode size, and a 64 MiB log size. The file system was spread across all of the Network Shared Disks (NSDs). Each client node mounted the file system. The file system parameters reflect values that might be used in a typical small file, metadata intensive environment.

The nodes each had an ext4 file system that hosted the operating system.

Storage and Filesystem Notes

Each of the FlashSystem 900 systems had 12 2.9 TiB flash modules. On each system one module was used as a spare and the remaining 11 modules were configured into a RAID-5 array. At that point volumes were created and mapped to the hosts, which were the Spectrum Scale nodes.

There were two sets of volumes used in the benchmark. On one of the FlashSystem 900 systems 4 100 GiB volumes were created, and each was mapped to a single host to be used for the node operating system. On each of the 2 FlashSystem 900 systems 32 700 GiB volumes were created, and each volume was mapped to all 4 hosts. These volumes were configured as NSDs by Spectrum Scale and used as storage for the Spectrum Scale file system.

The cluster used a single-tier architecture. The Spectrum Scale nodes performed both file and block level operations. Each node had access to all of the NSDs, so any file operation on a node was translated to a block operation and serviced on the same node.

Transport Configuration - Physical

Item No	Transport Type	Number of Ports Used	Notes
1	40 GbE cluster network	4	Each node connects to a 40 GbE administration network with MTU=1500
2	16 Gbps SAN	24	There were 16 total connections from storage and 8 total connections from servers.

Transport Configuration Notes

Each of the Cisco UCS B200 M4 blade servers comes with a Cisco UCS Virtual Interface Card 1340. The two port card supports 40 GbE and FCoE. To the operating system on the blade servers the card appears as a NIC for Ethernet and as an HBA for fibre channel connectivity. Physically the card connects to the UCS 2304 fabric extenders via internal chassis connections. The eight total ports from the fabric extenders connect to the UCS 6332-16UP fabric interconnects. The two fabric interconnects function as both 16 Gbps FC switches and as 40 Gbps Ethernet switches.

Switches - Physical

Processing Elements - Physical

Memory - Physical

Stable Storage

Item No	Switch Name	Switch Type	Total Port Count	Used Port Count	Notes
1	Cisco UCS 6332-16UP #1	40 GbE and 16 Gbps FC	40	12	The default configuration was used on the switch.
2	Cisco UCS 6332-16UP #2	40 GbE and 16 Gbps FC	40	12	The default configuration was used on the switch.

Item No	Qty	Type	Location	Description	Processing Function
1	8	CPU	Spectrum Scale client nodes	Intel Xeon CPU E5-2680 v3 @ 2.50GHz 12-core	Spectrum Scale nodes, load generator, device drivers

Description	Size in GiB	Number of Instances	Nonvolatile	Total GiB
Spectrum Scale node system memory	256	4	V	1024
Grand Total Memory Gibibytes	1024

All of the storage used by the benchmark was non-volatile flash storage. Modified writes were not acknowledged as complete until the data was written to the FlashSystem 900s. Each FlashSystem has two battery modules that in the case of a power failure allow the system to remain powered long enough for all of the data in the system's write cache to be committed to the flash modules.

Solution Under Test Configuration Notes

SPEC SFS®2014_swbuild Result

Performance

Product and Test Information