The SPEC 30th Anniversary: Better Benchmarks Since 1988
Like a lot of institutions, the Standard Performance Evaluation Corporation (SPEC) sprang out of frustration, according to co-founder John Mashey.
"All the performance folks and systems architects were just so sick of being hassled with dumb benchmarks."
Thirty years later, SPEC has grown from a handful of members and a single benchmark to 125 members in 22 countries, presiding over nearly two dozen active benchmarks and tools for measuring performance and energy efficiency, conducting research into a wide range of performance issues, and sponsoring international conferences and symposiums.
Here's a timeline encapsulating the history of the world's leading computer benchmarking organization.
1988
SPEC incorporates on November 14.
1989
![1989-VAX-11-780.jpg](1989-VAX-11-780.jpg)
- SPECmark (subsequently known as SPEC CPU89) introduced with a VAX 11/780 as a reference machine and 10 computation-intensive benchmark programs (average size of 2.5 billion dynamic instructions per program).
1992
- SPEC92 (subsequently known as SPEC CPU92) debuts with 20 benchmark programs and up to 10.2 billion dynamic instruction count.
1993
![SPEC_SFS_1.0_results-1993-1998.jpg](SPEC_SFS_1.0_results-1993-1998.jpg)
- SPECsfs 93, a one-test NFS benchmark suite derived from the Nhfsstone workload, is introduced.
1994
- The SPEC High Performance Group (SPEC/HPG) is formed.
- The Graphics Performance Characterization (GPC) group releases the SPECviewperf benchmark for graphics performance.
1995
![1995-Sun_SparcStation_10_with_CRT.jpg](1995-Sun_SparcStation_10_with_CRT.jpg)
- SPEC CPU95 goes live, with a SPARCstation 10 model 40 as its reference machine and 18 benchmark programs with up to 520.4 billion dynamic instruction count.
1996
![SPEChpc96.jpg](SPEChpc96.jpg)
- SPEC/HPG releases SPEChpc96, a benchmark suite that measures the performance of HPC systems running industrial-style applications.
- The GPC Group (now SPEC/GWPG) joins SPEC and releases SPECviewperf 5.0.
- SPEC announces SPECweb96, the first standardized benchmark for measuring the performance of web servers.
1997
- SPECsfs 97 released with major upgrades, including support for both NFS protocol version 3 and version 2.
- The SPEC Application Performance Characterization (SPECapc) project group is formed under SPEC/GWPG.
1998
![jvm98benchmarks.jpg](jvm98benchmarks.jpg)
![SPECviewperf_DesignReview_viewset-1996.jpg](SPECviewperf_DesignReview_viewset-1996.jpg)
- SPEC releases SPECjvm98, the first industry-standard benchmark that measures performance for Java virtual machine (JVM) client platforms.
- SPECapc releases Solidworks 98 and Quake II application benchmarks.
- SPECviewperf 6 is released with a new viewset and support for multiple primitive types and textures.
1999
- SPECweb99 is released with a new framework and workload that reflect industry changes.
![SPEC-CPU2000-Henning.jpg](SPEC-CPU2000-Henning.jpg)
- SPECapc releases new application benchmarks for Pro/ENGINEER and SolidWorks CAD/CAM software.
- On December 30, SPEC rings in the new millennium with SPEC CPU2000, comprising 26 benchmarks and dynamic instruction counts that reach 546.7 billion.
2000
- SPEC/HPG announces SPEComp2001, a new benchmark development effort based on OMP applications.
- SPEC releases its first server-side Java benchmark, SPEC JBB2000.
- SPECapc releases a new benchmark for Unigraphics V15.
2001
- SPEC releases SPECmail 2001, a benchmark for evaluating the performance of mail servers.
- SPEC OMP2001, a benchmark suite based on OpenMP applications for evaluating the performance of SMP systems, goes live.
- SPECapc releases new benchmarks for systems running 3ds Max, Solid Edge and Unigraphics software.
2002
![jAppServer2001-designdoc.jpg](jAppServer2001-designdoc.jpg)
- SPEC releases SPECjAppServer2001, a client/server benchmark that measures the performance of Java Enterprise Application Servers.
- SPEC/HPG releases SPEC OMPL2001, a new benchmark suite that measures the performance of computing systems running large scalable applications.
![ugs03viewset-SPECviewperf7-2002.jpg](ugs03viewset-SPECviewperf7-2002.jpg)
- New SPECviewperf 7.0 enables general state changes to be made during frames, bringing the benchmark much closer to real-world behavior.
- The SPEChpc 2002 benchmark, derived from real HPC applications and practices, is released.
- SPECapc updates its benchmarks for new versions of Solidworks, 3ds Max and Pro/E applications.
2003
![2003-SPECapc_for_Solidworks_2003.jpg](2003-SPECapc_for_Solidworks_2003.jpg)
- SPECapc updates its benchmarks for new versions of Solid Edge, Solidworks and Maya applications.
- SPEC/HPG releases V3.0 of the OMP benchmark.
2004
- SPEC releases SPECjAppServer2004, a multi-tier benchmark for measuring the performance of Java 2 Enterprise Edition (J2EE) application servers.
- SPECviewperf 8 released with four new viewsets and application tracing that gets closer to application behavior in the real world.
- SPECapc updates its benchmarks for new versions of Solid Edge, Solidworks, Maya and 3ds Max applications.
2005
![SPECjbb2005.jpg](SPECjbb2005.jpg)
- SPEC releases its JBB2005 benchmark for evaluating server-side Java performance.
- SPECweb 2005 released with three real-world workloads and dynamic content in PHP and JSP.
- SPECapc updates its benchmarks for new versions of Maya and 3ds Max applications.
2006
![CPU2006-integer_memory_use.jpg](CPU2006-integer_memory_use.jpg)
- SPEC CPU2006 is released with 29 benchmarks exercising new areas of functionality for CPUs, memory systems, and compilers.
- The SPECpower committee is formed to develop standard methods and metrics for comparing energy use of server-class computers.
- New SPECviewperf 9 features two new viewsets and major code changes. A Linux/Unix version is also released.
- SPECapc updates its benchmarks for new versions of 3ds Max, NX and Pro/E applications.
2007
![SPECpower_ssj2008-set-up.jpg](SPECpower_ssj2008-set-up.jpg)
- SPECpower_ssj2008, the first industry-standard benchmark that measures power consumption in relation to performance for server-class computers, goes live.
- SPEC forms a new subcommittee to develop standard methods of comparing performance for servers using the Session Initiation Protocol (SIP).
- SPECviewperf 10 adds performance measurement for full-scene anti-aliasing and multithreading.
- SPEC/HPG releases MPI2007 to measure the performance of parallel computing systems and clusters running Message-Passing Interface (MPI) applications.
- SPECjms2007, a new benchmark designed to measure performance of enterprise messaging servers based on JMS (Java Message Service), makes its debut.
- SPECapc updates its benchmarks for 3ds Max, NX, Solid Edge and Solidworks.
2008
- SPECsfs2008, designed to evaluate the speed and request-handling capabilities of file servers utilizing the NFSv3 and CIFS protocols, is released.
- SPEC releases SPECjvm2008, a benchmark suite for measuring the performance of a Java Runtime Environment (JRE).
- SPECmail2008, designed to measure the performance of enterprise mail servers compliant with SMTP and IMAP4 protocols, makes its debut.
- SPECviewperf 10 for Linux/Unix is released.
2009
![2009-2010-SERT_workload_efficiency-med.jpg](2009-2010-SERT_workload_efficiency-med.jpg)
- The SPECpower committee embarks on a project to develop a Server Efficiency Rating Tool (SERT) for use by the EPA.
- SPECmail2009, a new version of the enterprise mail server benchmark, goes live.
- SPECweb2009 is released with new power/performance measurement.
- SPECjEnterprise2010 is released to measure performance for Java Enterprise Edition 5 or later application servers, databases and infrastructure.
- SPECapc releases new benchmark for Lightwave 3D and updates its Maya application benchmark.
2010
![2011-SPECviewperf11-lightwave-viewset.jpg](2011-SPECviewperf11-lightwave-viewset.jpg)
- SPECviewperf 11 released with a new GUI and fully updated viewsets based on traces from professional applications.
- SPECvirt_sc2010, SPEC's first benchmark for virtualized server consolidation, goes live.
![2010-ICPE2010.jpg](2010-ICPE2010.jpg)
- SPEC/HPG releases MPI2007 v2.0, which adds the MPIL2007 suite to allow testing up to 2048 ranks.
- SPEC and ACM launch the first International Conference on Performance Engineering (ICPE).
2011
![2011-spec_research.jpg](2011-spec_research.jpg)
- SPEC's Research Group (SPEC RG) is formed to develop standard scenarios and workloads for emerging technologies and applications.
- SPEC releases SPECsip_Infrastructure2011, designed to evaluate a system's ability to act as a SIP server.
- SPECapc releases updated benchmark for the 3ds Max application.
- SPEC RG establishes the annual Kaivalya Dixit Distinguished Dissertation Award.
2012
![2012-cloud.png](2012-cloud.png)
- SPEC forms a new Cloud subcommittee.
- SPEC releases SPEC OMP 2012, an updated benchmark for measuring performance and energy efficiency using applications based on the OpenMP 3.1 standard.
- SPEC/GWPG forms a workstation performance characterization (SPECwpc) group.
- The first two SPEC Kaivalya Dixit Distinguished Dissertation Awards are presented.
- SPEC RG establishes the Security Benchmarking and Cloud working groups.
- SPECapc releases updated benchmarks for Creo, NX and Maya applications.
2013
- The Server Efficiency Rating Tool (SERT), comprising synthetic worklets to test power consumption data at different load levels, makes its debut.
![2013-Energy_Star_logo.png](2013-Energy_Star_logo.png)
- SERT testing becomes mandatory part of the U.S. Environmental Protection Agency's (EPA) ENERGY STAR for Servers specification.
- SPECjbb2013, developed from the ground up to measure performance based on the latest Java application features, is released.
- SPECvirt_sc2013, the latest virtualized server benchmark, goes live.
![2013-WPC_GUI.jpg](2013-WPC_GUI.jpg)
- SPECwpc releases the first benchmark to measure all key aspects of workstation performance based on diverse professional applications.
- SPECviewperf 12 released with extensible architecture and new viewsets representing energy and medical applications.
- SPECapc updates benchmarks for NX and Solidworks applications.
2014
![2014-ACCEL-Results.jpg](2014-ACCEL-Results.jpg)
- SPEC ACCEL V1.0, a benchmark suite that measures performance with computationally intensive parallel applications, goes live.
- SPEC releases the Chauffeur WDK (Worklet Development Kit) Tool, designed to simplify the development of workloads for measuring both performance and energy efficiency.
- SPEC SFS2014, SPEC's updated benchmark suite to measure performance of storage configurations, is released.
- Fifth annual ICPE conference held in Dublin.
- SPECapc updates its benchmark for the 3ds Max application.
- SPEC RG forms working groups for Big Data and DevOps Performance.
2015
![2015-SPECwpc2.0.jpg](2015-SPECwpc2.0.jpg)
- SPECwpc 2.0 is released with improved storage workloads and better scalability measurement for multi-core systems.
- SPECjbb2015, an enhanced Java server benchmark that now supports virtualized hosts, is released.
- SPECapc updates benchmarks for Creo and Solidworks.
2016
![2016-DavidatAsiaSummit.jpg](2016-DavidatAsiaSummit.jpg)
- SPEC holds benchmarking symposiums and technical meetings in Beijing, China.
- SPEC Cloud_IaaS 2016, a new benchmark that measures the performance of infrastructure-as-a-service (IaaS) cloud implementations, is released.
- SPECapc updates its benchmark for the NX application.
2017
![2017-CPU-suite-growth.png](2017-CPU-suite-growth.png)
- SPEC releases SPEC CPU2017, an all-new version of its flagship performance evaluation suite, with 43 individual benchmarks organized into four categories.
- SERT 2.0, which adds a single-value metric and numerous other improvements, is released.
- SPEC establishes the Power working group.
- Version 2.0 of the Chauffeur Workload Development Kit (WDK) is released.
- SPECapc updates its benchmark for the Maya application.
2018
![2018-jEnterprise2018-hardware-environment.jpg](2018-jEnterprise2018-hardware-environment.jpg)
- SPEC releases SPECjEnterprise2018 Web Profile, a new benchmark measuring performance of Java EE 7 Web Profile application servers.
- SPEC offers HPG benchmarks (ACCEL, MPI2007, OMP2012) free of charge to qualified non-profit organizations worldwide.
- SPECviewperf 13 is released with new and updated viewsets, redesigned GUI, and improved scoring and reporting methods.
- SPECworkstation 3 (formerly SPECwpc) is released with new and updated workloads throughout the benchmark suite.
- SPEC Cloud IaaS 2018 is released with new workload versions and a wide range of methodology updates.
- EPA's ENERGY STAR Version 3.0 for Computer Servers defines active state efficiency thresholds based on SERT.
- SPECapc updates its benchmark for the Maya application.
- SPEC RG creates the Quality of Experience working group.