The SPEC 30th Anniversary: Better Benchmarks Since 1988
Like a lot of institutions, the Standard Performance Evaluation Corporation (SPEC) sprang out of frustration, according to co-founder John Mashey.
"All the performance folks and systems architects were just so sick of being hassled with dumb benchmarks."
Thirty years later, SPEC has grown from a handful of members and a single benchmark to 125 members in 22 countries, presiding over nearly two dozen active benchmarks and tools for measuring performance and energy efficiency, conducting research into a wide range of performance issues, and sponsoring international conferences and symposiums.
Here's a timeline encapsulating the history of the world's leading computer benchmarking organization.
1988
SPEC incorporates on November 14.
1989

- SPECmark (subsequently known as SPEC CPU89) introduced with a VAX 11/780 as a reference machine and 10 computation-intensive benchmark programs (average size of 2.5 billion dynamic instructions per program).
1992
- SPEC92 (subsequently known as SPEC CPU92) debuts with 20 benchmark programs and up to 10.2 billion dynamic instruction count.
1993

- SPECsfs 93, a one-test NFS benchmark suite derived from the Nhfsstone workload, is introduced.
1994
- The SPEC High Performance Group (SPEC/HPG) is formed.
- The Graphics Performance Characterization (GPC) group releases the SPECviewperf benchmark for graphics performance.
1995

- SPEC CPU95 goes live, with a SPARCstation 10 model 40 as its reference machine and 18 benchmark programs with up to 520.4 billion dynamic instruction count.
1996

- SPEC/HPG releases SPEChpc96, a benchmark suite that measures the performance of HPC systems running industrial-style applications.
- The GPC Group (now SPEC/GWPG) joins SPEC and releases SPECviewperf 5.0.
- SPEC announces SPECweb96, the first standardized benchmark for measuring the performance of web servers.
1997
- SPECsfs 97 released with major upgrades, including support for both NFS protocol version 3 and version 2.
- The SPEC Application Performance Characterization (SPECapc) project group is formed under SPEC/GWPG.
1998


- SPEC releases SPECjvm98, the first industry-standard benchmark that measures performance for Java virtual machine (JVM) client platforms.
- SPECapc releases Solidworks 98 and Quake II application benchmarks.
- SPECviewperf 6 is released with a new viewset and support for multiple primitive types and textures.
1999
- SPECweb99 is released with a new framework and workload that reflect industry changes.

- SPECapc releases new application benchmarks for Pro/ENGINEER and SolidWorks CAD/CAM software.
- On December 30, SPEC rings in the new millennium with SPEC CPU2000, comprising 26 benchmarks and dynamic instruction counts that reach 546.7 billion.
2000
- SPEC/HPG announces SPEComp2001, a new benchmark development effort based on OMP applications.
- SPEC releases its first server-side Java benchmark, SPEC JBB2000.
- SPECapc releases a new benchmark for Unigraphics V15.
2001
- SPEC releases SPECmail 2001, a benchmark for evaluating the performance of mail servers.
- SPEC OMP2001, a benchmark suite based on OpenMP applications for evaluating the performance of SMP systems, goes live.
- SPECapc releases new benchmarks for systems running 3ds Max, Solid Edge and Unigraphics software.
2002

- SPEC releases SPECjAppServer2001, a client/server benchmark that measures the performance of Java Enterprise Application Servers.
- SPEC/HPG releases SPEC OMPL2001, a new benchmark suite that measures the performance of computing systems running large scalable applications.

- New SPECviewperf 7.0 enables general state changes to be made during frames, bringing the benchmark much closer to real-world behavior.
- The SPEChpc 2002 benchmark, derived from real HPC applications and practices, is released.
- SPECapc updates its benchmarks for new versions of Solidworks, 3ds Max and Pro/E applications.
2003

- SPECapc updates its benchmarks for new versions of Solid Edge, Solidworks and Maya applications.
- SPEC/HPG releases V3.0 of the OMP benchmark.
2004
- SPEC releases SPECjAppServer2004, a multi-tier benchmark for measuring the performance of Java 2 Enterprise Edition (J2EE) application servers.
- SPECviewperf 8 released with four new viewsets and application tracing that gets closer to application behavior in the real world.
- SPECapc updates its benchmarks for new versions of Solid Edge, Solidworks, Maya and 3ds Max applications.
2005

- SPEC releases its JBB2005 benchmark for evaluating server-side Java performance.
- SPECweb 2005 released with three real-world workloads and dynamic content in PHP and JSP.
- SPECapc updates its benchmarks for new versions of Maya and 3ds Max applications.
2006

- SPEC CPU2006 is released with 29 benchmarks exercising new areas of functionality for CPUs, memory systems, and compilers.
- The SPECpower committee is formed to develop standard methods and metrics for comparing energy use of server-class computers.
- New SPECviewperf 9 features two new viewsets and major code changes. A Linux/Unix version is also released.
- SPECapc updates its benchmarks for new versions of 3ds Max, NX and Pro/E applications.
2007

- SPECpower_ssj2008, the first industry-standard benchmark that measures power consumption in relation to performance for server-class computers, goes live.
- SPEC forms a new subcommittee to develop standard methods of comparing performance for servers using the Session Initiation Protocol (SIP).
- SPECviewperf 10 adds performance measurement for full-scene anti-aliasing and multithreading.
- SPEC/HPG releases MPI2007 to measure the performance of parallel computing systems and clusters running Message-Passing Interface (MPI) applications.
- SPECjms2007, a new benchmark designed to measure performance of enterprise messaging servers based on JMS (Java Message Service), makes its debut.
- SPECapc updates its benchmarks for 3ds Max, NX, Solid Edge and Solidworks.
2008
- SPECsfs2008, designed to evaluate the speed and request-handling capabilities of file servers utilizing the NFSv3 and CIFS protocols, is released.
- SPEC releases SPECjvm2008, a benchmark suite for measuring the performance of a Java Runtime Environment (JRE).
- SPECmail2008, designed to measure the performance of enterprise mail servers compliant with SMTP and IMAP4 protocols, makes its debut.
- SPECviewperf 10 for Linux/Unix is released.
2009

- The SPECpower committee embarks on a project to develop a Server Efficiency Rating Tool (SERT) for use by the EPA.
- SPECmail2009, a new version of the enterprise mail server benchmark, goes live.
- SPECweb2009 is released with new power/performance measurement.
- SPECjEnterprise2010 is released to measure performance for Java Enterprise Edition 5 or later application servers, databases and infrastructure.
- SPECapc releases new benchmark for Lightwave 3D and updates its Maya application benchmark.
2010

- SPECviewperf 11 released with a new GUI and fully updated viewsets based on traces from professional applications.
- SPECvirt_sc2010, SPEC's first benchmark for virtualized server consolidation, goes live.

- SPEC/HPG releases MPI2007 v2.0, which adds the MPIL2007 suite to allow testing up to 2048 ranks.
- SPEC and ACM launch the first International Conference on Performance Engineering (ICPE).
2011

- SPEC's Research Group (SPEC RG) is formed to develop standard scenarios and workloads for emerging technologies and applications.
- SPEC releases SPECsip_Infrastructure2011, designed to evaluate a system's ability to act as a SIP server.
- SPECapc releases updated benchmark for the 3ds Max application.
- SPEC RG establishes the annual Kaivalya Dixit Distinguished Dissertation Award.
2012

- SPEC forms a new Cloud subcommittee.
- SPEC releases SPEC OMP 2012, an updated benchmark for measuring performance and energy efficiency using applications based on the OpenMP 3.1 standard.
- SPEC/GWPG forms a workstation performance characterization (SPECwpc) group.
- The first two SPEC Kaivalya Dixit Distinguished Dissertation Awards are presented.
- SPEC RG establishes the Security Benchmarking and Cloud working groups.
- SPECapc releases updated benchmarks for Creo, NX and Maya applications.
2013
- The Server Efficiency Rating Tool (SERT), comprising synthetic worklets to test power consumption data at different load levels, makes its debut.

- SERT testing becomes mandatory part of the U.S. Environmental Protection Agency's (EPA) ENERGY STAR for Servers specification.
- SPECjbb2013, developed from the ground up to measure performance based on the latest Java application features, is released.
- SPECvirt_sc2013, the latest virtualized server benchmark, goes live.

- SPECwpc releases the first benchmark to measure all key aspects of workstation performance based on diverse professional applications.
- SPECviewperf 12 released with extensible architecture and new viewsets representing energy and medical applications.
- SPECapc updates benchmarks for NX and Solidworks applications.
2014

- SPEC ACCEL V1.0, a benchmark suite that measures performance with computationally intensive parallel applications, goes live.
- SPEC releases the Chauffeur WDK (Worklet Development Kit) Tool, designed to simplify the development of workloads for measuring both performance and energy efficiency.
- SPEC SFS2014, SPEC's updated benchmark suite to measure performance of storage configurations, is released.
- Fifth annual ICPE conference held in Dublin.
- SPECapc updates its benchmark for the 3ds Max application.
- SPEC RG forms working groups for Big Data and DevOps Performance.
2015

- SPECwpc 2.0 is released with improved storage workloads and better scalability measurement for multi-core systems.
- SPECjbb2015, an enhanced Java server benchmark that now supports virtualized hosts, is released.
- SPECapc updates benchmarks for Creo and Solidworks.
2016

- SPEC holds benchmarking symposiums and technical meetings in Beijing, China.
- SPEC Cloud_IaaS 2016, a new benchmark that measures the performance of infrastructure-as-a-service (IaaS) cloud implementations, is released.
- SPECapc updates its benchmark for the NX application.
2017

- SPEC releases SPEC CPU2017, an all-new version of its flagship performance evaluation suite, with 43 individual benchmarks organized into four categories.
- SERT 2.0, which adds a single-value metric and numerous other improvements, is released.
- SPEC establishes the Power working group.
- Version 2.0 of the Chauffeur Workload Development Kit (WDK) is released.
- SPECapc updates its benchmark for the Maya application.
2018

- SPEC releases SPECjEnterprise2018 Web Profile, a new benchmark measuring performance of Java EE 7 Web Profile application servers.
- SPEC offers HPG benchmarks (ACCEL, MPI2007, OMP2012) free of charge to qualified non-profit organizations worldwide.
- SPECviewperf 13 is released with new and updated viewsets, redesigned GUI, and improved scoring and reporting methods.
- SPECworkstation 3 (formerly SPECwpc) is released with new and updated workloads throughout the benchmark suite.
- SPEC Cloud IaaS 2018 is released with new workload versions and a wide range of methodology updates.
- EPA's ENERGY STAR Version 3.0 for Computer Servers defines active state efficiency thresholds based on SERT.
- SPECapc updates its benchmark for the Maya application.
- SPEC RG creates the Quality of Experience working group.