SPEC Power and Performance Committee

The SPEC Power and Performance Committee is developing the first generation SPEC benchmark for evaluating the power and performance characteristics of volume server class computers.

The drive to create the power and performance benchmark comes from the recognition that the IT industry, computer manufacturers, and governments are increasingly concerned with the energy use of servers. Currently, many vendors report some energy efficiency figures, but these are often not directly comparable due to differences in workload, configuration, test environment, etc. SPEC is defining server power measurement standards in the same way we have done for performance. Development of this benchmark provides a means to measure power in conjunction with a performance metric. This should help IT managers to consider power characteristics along with other selection criteria to increase the efficiency of data centers.

Some of the challenges include defining the scope we expect to address, selecting and creating workload(s), defining test conditions, specifying a measurement protocol, and providing for extension of the metric to a wider scope.

Benchmark Goals

The committee develops a means to fairly and consistently report system energy use under various usage levels.

The initial benchmark addresses one subset of server workloads: the performance of server side Java. It exercises the CPUs, caches, memory hierarchy and the scalability of shared memory processors (SMPs) as well as the implementations of the JVM (Java Virtual Machine), JIT (Just-In-Time) compiler, garbage collection, threads and some aspects of the operating system.

The benchmark will run on a wide variety of operating systems and hardware architectures and should not require extensive client or storage infrastructure.

The power measurements are made at the AC input to the system under test.

A Complete Benchmark Package

In keeping with SPEC tradition, the package consists of a variety of tools, rules, and documentation:

OS Independent Workload
Server Energy Measurement Methodology
Extensible Software Framework
- Power and Temperature Daemon
- Control and Collection System
Run and Reporting Rules
Documentation
- User Guide
- Design Documents

Power Analyzer Requirements

The Committee has set requirements for: Calibration, Crest Factor, Measurements, Logging, Power Accuracy, Power Resolution

Minimum power meter requirements (Draft)

SPECpower will require that the power analyzer must have an interface that allows its measurements to be read by the benchmark harness. This increases ease of use since synchronization of power and performance data is automated. Power analyzers which meet the other requirements generally come with an interface option. Several power analyzers are currently supported and meet these requirements.

Environment Conditions

The system must be operating in its specified operating ranges, i.e. normal operating environment. SPECpower requires a minimum ambient temperature of at least 20ºC.

graph - example: temperataure over time in data center

Benchmark Harness

For the ease of use, the benchmark harness ensures the synchronization of the measured performance, power, and environmental data.

Current Release Status

The committee's goal is to release the benchmark before the end of 2007. We are currently testing the benchmark beta version.

Committee Membership

Current active SPEC member companies committed to developing a power and performance measurement standard are AMD, Dell, Fujitsu Siemens Computers, HP, Intel, IBM, and Sun Microsystems. SPEC Associates also participating in the committee are: University of California - Berkeley and Virginia Polytechnic Institute and State University, as well as Lawrence Berkeley National Laboratory.

SPEC welcomes organizations to join and participate in our work, please contact info@spec.org for additional information.

SPEC's Response to EPA's Energy Star Server Draft (30. August 2007)

The first SPECpower benchmark will be a leap forward in the state of the art in power and performance analysis of Enterprise Computer Servers. We would like to work closely with the EPA in defining additional requirements to enhance our benchmark suite to satisfy EPA needs, where feasible.

The first SPECpower benchmark is only focusing on one application space. In order to represent a wide spectrum of the application space, a means for introducing additional work-loads must be a part of the Tier II plan. We welcome EPA input in prioritizing future work.

SPEC and EPA's Energy Star Enterprise Server and Data Center Energy Efficiency Initiatives

EPA Technical Workshop on Energy Efficient Servers and Datacenters in the U.S. (February 16, 2007, Santa Clara, CA)
- EPA Study on Server and Datacenter Energy Efficiency
- SPEC at the EPA Workshop
Energy Efficiency Server Benchmark Technical Workshop (March 27, 2006, Santa Clara, CA)
- Server Efficiency Benchmark Development Effort
- SPEC at the EPA Workshop

Standard Performance Evaluation Corporation