Standard Performance Evaluation Corporation
The increasing demand for energy-efficient IT equipment has resulted in the need for power and performance benchmarks. In response, the SPEC community established the SPECpower Committee, an initiative to augment existing or create new industry-standard benchmarks and tools with a power/energy measurement. Leading engineers and scientists in the fields of benchmark development and energy efficiency, representing a diverse and competitive set of companies, made a commitment to tackle this task. The development of the first industry-standard benchmark that measures the power and performance characteristics of server-class compute equipment began on January 26 2006.
The SPECpower committee includes representatives from AMD, Dell, HPE, IBM, Intel, Microsoft, and the University of Würzburg. Veteran SPEC benchmark developers Hansfried Block, Greg Darnell, and Karin Wulf are supporting contributors.
The SPEC PTDaemon
The power temperature daemon (also known as PTDaemon) is used to offload the work of controlling a power analyzer or temperature sensor during measurement intervals to a system other than the SUT. It hides the details of different power analyzer interface protocols and behaviors from the benchmark software, presenting a common TCP/IP-based interface that can be readily integrated into different benchmark harnesses. Benchmarks already using PTDaemon include SPECpower_ssj®2008, SPECweb®2009, and SPECvirt_sc®2013, the SERT™ suite, and the Chauffeur™ WDK.
In December of 2007, SPECpower_ssj2008 was released, which exercises the CPUs, caches, memory hierarchy, and the scalability of shared memory processors on multiple load-levels. The benchmark runs on a wide variety of operating systems and hardware architectures. In version 1.10, which was released on April 15 2009, SPEC augmented SPECpower_ssj2008 with multi-node support (e.g., blade-support). Several enhancements and code changes to all benchmark components, documentation updates, and run and reporting rules enhancements were included in version 1.11, released September 13, 2011 and version 1.12, released March 30, 2012.
The SERT Suite
The work on the SERT suite started around 2009 with the ground-breaking design of the Chauffeur framework, followed with the design and implementation of the workload, automated hardware and software discovery, and GUI. After a series of successful Alpha and Beta test phases, the SERT suite was released on February 26, 2013. The U.S. Environmental Protection Agency (EPA) adopted the SERT suite for their ENERGY STAR for Server Certification program 2 months later on March 15, 2013. The SPECpower Committee released four upgrades over the years, in order to further enhance the usability of the SERT suite, and to increase its platform support.
For version 2.0, one of the main focuses was the reduction of its run time and improvements on the automated hardware and software discovery, in order to save the user testing time and costs. The support of additional power analyzers and interfaces was adopted as well in order to provide a wider choices for users. Also some major improvements of the memory subsystem worklets, for more accurate characterization of memory, were implemented. Simultaneously to these efforts, the SPECpower Committee in collaboration with the more academic-focused SPEC Power Research Group, conducted large scale experiments in order to determine the server efficiency metric for SERT Suite 2.0 which enabled its release on March 28, 2017.
The Chauffeur Worklet Development Kit (WDK)
The Chauffeur framework from SERT is made available as the Chauffeur Worklet Development Kit (WDK). This kit can be used to develop new workloads (or "worklets" in Chauffeur terminology). Researchers can also use the WDK to configure worklets to run in different ways, in order to mimic the behavior of different types of applications. These features can be used in the development and assessment of new technologies such as power management capabilities. The first version of the Chauffeur WDK was released on May 18, 2014, and succeeded by the 1.1.0 release on May 26, 2015 and version 2.0 on April 18, 2017.
The Chauffeur WDK 2.0 further simplifies workload development for researchers analyzing computer performance and power consumption. Targeting both industry and academia, the WDK provides the infrastructure for SERT 2.0, providing run-time & platform configuration support to simplify worklet development. This major release includes significant enhancements to the hardware detection, customization options of generating HTML reports, and developer documentation. It has now reduced memory requirements for the Director when signing results files and reduced the size of the result output for large systems or clusters. Chauffeur WDK 2.0 added worklet-specific normalization of results and an updated list of supported operating systems and the latest PTDaemon integration for power analyzers and temperature sensors, along with data collection, validation and reporting.
The Power and Performance Benchmark Methodology
The Power and Performance Benchmark Methodology document is intended for performance benchmark designers and implementers who want to integrate a power component into their benchmark. The document may also serve as an introduction to those who need to understand the relationship between power and performance metrics in computer systems benchmarks. The assumption is that the business model and benchmark application are already selected and may already be implemented. Guidance is provided for including power metrics in existing benchmarks, as well as altering existing benchmarks and designing new ones to provide a more complete view of energy consumption. Benchmarks and tools already using this methodology document include SPECpower_ssj2008, SPECweb2009, SPECvirt_sc2013, the SERT suite, and the Chauffeur WDK.
SPEC and the names SERT, SPECpower, Chauffeur, and PTDaemon are trademarks of the Standard Performance Evaluation Corporation. Additional product and service names mentioned herein may be the trademarks of their respective owners. Copyright © 1988-2017 Standard Performance Evaluation Corporation (SPEC). All rights reserved.