Skip navigation

Standard Performance Evaluation Corporation

Facebook logo LinkedIn logo Twitter logo Google+ logo
 
 

The SPEC 30th anniversary

Better benchmarks since 1988


Like a lot of institutions, the Standard Performance Evaluation Corporation (SPEC) sprang out of frustration, according to co-founder John Mashey.

"All the performance folks and systems architects were just so sick of being hassled with dumb benchmarks."

Thirty years later, SPEC has grown from a handful of members and a single benchmark to 125 members in 22 countries, presiding over nearly two dozen active benchmarks and tools for measuring performance and energy efficiency, conducting research into a wide range of performance issues, and sponsoring international conferences and symposiums.

Here's a timeline encapsulating the history of the world's leading computer benchmarking organization.

1988

  • SPEC incorporates on November 14.
  • 1989

  • SPEC CPU89 introduced with a VAX 11/780 as a reference machine and 10 computation-intensive benchmark programs (average size of 2.5 billion dynamic instructions per program).
  • 1992

  • SPEC CPU92 debuts with 20 benchmark programs and up to 10.2 billion dynamic instruction count.
  • 1993

  • SPECsfs 93, a one-test NFS benchmark suite derived from the Nhfsstone workload, is introduced.
  • 1994

  • The SPEC High Performance Group (SPEC/HPG) is formed.
  • The Graphics Performance Characterization (GPC) group releases the SPECviewperf benchmark for graphics performance.
  • 1995

  • SPEC CPU95 goes live, with a SPARCstation 10 model 40 as its reference machine and 18 benchmark programs with up to 520.4 billion dynamic instruction count.
  • 1996

  • SPEC/HPG releases SPEChpc96, a benchmark suite that measures the performance of HPC systems running industrial-style applications.
  • The GPC Group (now SPEC/GWPG) joins SPEC and releases SPECviewperf 5.0.
  • SPEC announces SPECweb96, the first standardized benchmark for measuring the performance of web servers.
  • 1997

  • SPECsfs 97 released with major upgrades, including support for both NFS protocol version 3 and version 2.
  • The SPEC Application Performance Characterization (SPECapc) project group is formed under SPEC/GWPG.
  • 1998

  • SPEC releases SPECjvm98, the first industry-standard benchmark that measures performance for Java virtual machine (JVM) client platforms.
  • SPECapc releases Solidworks 98 and Quake II application benchmarks.
  • SPECviewperf 6 is released with a new viewset and support for multiple primitive types and textures.
  • 1999

  • SPECweb99 is released with a new framework and workload that reflect industry changes.
  • SPECapc releases new application benchmarks for Pro/ENGINEER and SolidWorks CAD/CAM software.
  • On December 30, SPEC rings in the new millennium with SPEC CPU2000, comprising 26 benchmarks and dynamic instruction counts that reach 546.7 billion.
  • 2000

  • SPEC/HPG announces SPEComp2001, a new benchmark development effort based on OMP applications.
  • SPEC releases its first server-side Java benchmark, SPEC JBB2000.
  • SPECapc releases a new benchmark for Unigraphics V15.
  • 2001

  • SPEC releases SPECmail 2001, a benchmark for evaluating the performance of mail servers.
  • SPEC OMP2001, a benchmark suite based on OpenMP applications for evaluating the performance of SMP systems, goes live.
  • SPECapc releases new benchmarks for systems running 3ds Max, Solid Edge and Unigraphics software.
  • 2002

  • SPEC releases SPECjAppServer2001, a client/server benchmark that measures the performance of Java Enterprise Application Servers.
  • SPEC/HPG releases SPEC OMPL2001, a new benchmark suite that measures the performance of computing systems running large scalable applications.
  • New SPECviewperf 7.0 enables general state changes to be made during frames, bringing the benchmark much closer to real-world behavior.
  • The SPEChpc 2002 benchmark, derived from real HPC applications and practices, is released.
  • SPECapc updates its benchmarks for new versions of Solidworks, 3ds Max and Pro/E applications.
  • 2003

  • SPECapc updates its benchmarks for new versions of Solid Edge, Solidworks and Maya applications.
  • SPEC/HPG releases V3.0 of the OMP benchmark.
  • 2004

  • SPEC releases SPECjAppServer2004, a multi-tier benchmark for measuring the performance of Java 2 Enterprise Edition (J2EE) application servers.
  • SPECviewperf 8 released with four new viewsets and application tracing that gets closer to application behavior in the real world.
  • SPECapc updates its benchmarks for new versions of Solid Edge, Solidworks, Maya and 3ds Max applications.
  • 2005

  • SPEC releases its JBB2005 benchmark for evaluating server-side Java performance.
  • SPECweb 2005 released with three real-world workloads and dynamic content in PHP and JSP.
  • SPECapc updates its benchmarks for new versions of Maya and 3ds Max applications.
  • 2006

  • SPEC CPU2006 is released with 29 benchmarks exercising new areas of functionality for CPUs, memory systems, and compilers.
  • The SPECpower committee is formed to develop standard methods and metrics for comparing energy use of server-class computers.
  • New SPECviewperf 9 features two new viewsets and major code changes. A Linux/Unix version is also released.
  • SPECapc updates its benchmarks for new versions of 3ds Max, NX and Pro/E applications.
  • 2007

  • SPECpower_ssj2008, the first industry-standard benchmark that measures power consumption in relation to performance for server-class computers, goes live.
  • SPEC forms a new subcommittee to develop standard methods of comparing performance for servers using the Session Initiation Protocol (SIP).
  • SPECviewperf 10 adds performance measurement for full-scene anti-aliasing and multithreading.
  • SPEC/HPG releases MPI2007 to measure the performance of parallel computing systems and clusters running Message-Passing Interface (MPI) applications.
  • SPECjms2007, a new benchmark designed to measure performance of enterprise messaging servers based on JMS (Java Message Service), makes its debut.
  • SPECapc updates its benchmarks for 3ds Max, NX, Solid Edge and Solidworks.
  • 2008

  • SPECsfs2008, designed to evaluate the speed and request-handling capabilities of file servers utilizing the NFSv3 and CIFS protocols, is released.
  • SPEC releases SPECjvm2008, a benchmark suite for measuring the performance of a Java Runtime Environment (JRE).
  • SPECmail2008, designed to measure the performance of enterprise mail servers compliant with SMTP and IMAP4 protocols, makes its debut.
  • SPECviewperf 10 for Linux/Unix is released.
  • 2009

  • The SPECpower committee embarks on a project to develop a Server Efficiency Rating Tool (SERT) for use by the EPA.
  • SPECmail2009, a new version of the enterprise mail server benchmark, goes live.
  • SPECweb2009 is released with new power/performance measurement.
  • SPECjEnterprise2010 is released to measure performance for Java Enterprise Edition 5 or later application servers, databases and infrastructure.
  • SPECapc releases new benchmark for Lightwave 3D and updates its Maya application benchmark.
  • 2010

  • SPECviewperf 11 released with a new GUI and fully updated viewsets based on traces from professional applications.
  • SPECvirt_sc2010, SPEC's first benchmark for virtualized server consolidation, goes live.
  • SPEC/HPG releases MPI2007 v2.0, which adds the MPIL2007 suite to allow testing up to 2048 ranks.
  • SPEC and ACM launch the first International Conference on Performance Engineering (ICPE).
  • 2011

  • SPEC's Research Group (SPEC RG) is formed to develop standard scenarios and workloads for emerging technologies and applications.
  • SPEC releases SPECsip_Infrastructure2011, designed to evaluate a system's ability to act as a SIP server.
  • SPECapc releases updated benchmark for the 3ds Max application.
  • SPEC RG establishes the annual Kaivalya Dixit Distinguished Dissertation Award.
  • 2012

  • SPEC forms a new Cloud subcommittee.
  • SPEC releases SPEC OMP 2012, an updated benchmark for measuring performance and energy efficiency using applications based on the OpenMP 3.1 standard.
  • SPEC/GWPG forms a workstation performance characterization (SPECwpc) group.
  • The first two SPEC Kaivalya Dixit Distinguished Dissertation Awards are presented to Luk Van Ertvelde of Ghent University and Kai Sachs of TU Darmstadt.
  • SPEC RG establishes the Security Benchmarking and Cloud working groups.
  • SPECapc releases updated benchmarks for Creo, NX and Maya applications.
  • 2013

  • The Server Efficiency Rating Tool (SERT), comprising synthetic worklets to test power consumption data at different load levels, makes its debut.
  • SERT testing becomes mandatory part of the U.S. Environmental Protection Agency's (EPA) ENERGY STAR for Servers specification.
  • SPECjbb2013, developed from the ground up to measure performance based on the latest Java application features, is released.
  • SPECvirt_sc2013, the latest virtualized server benchmark, goes live.
  • Kaivalya Dixit Distinguished Dissertation Award presented to Shicong Meng of Georgia Institute of Technology.
  • SPECwpc releases the first benchmark to measure all key aspects of workstation performance based on diverse professional applications.
  • SPECviewperf 12 released with extensible architecture and new viewsets representing energy and medical applications.
  • SPECapc updates benchmarks for NX and Solidworks applications.
  • 2014

  • SPEC ACCEL V1.0, a benchmark suite that measures performance with computationally intensive parallel applications, goes live.
  • SPEC releases the Chauffeur WDK (Worklet Development Kit) Tool, designed to simplify the development of workloads for measuring both performance and energy efficiency.
  • SPEC SFS2014, SPEC's updated benchmark suite to measure performance of storage configurations, is released.
  • Fifth annual ICPE conference held in Dublin.
  • Kaivalya Dixit Distinguished Dissertation Award presented to Anshul Gandhi of Carnegie Mellon University.
  • SPECapc updates its benchmark for the 3ds Max application.
  • SPEC RG forms working groups for Big Data and DevOps Performance.
  • 2015

  • SPECwpc 2.0 is released with improved storage workloads and better scalability measurement for multi-core systems.
  • SPECjbb2015, an enhanced Java server benchmark that now supports virtualized hosts, is released.
  • SPECapc updates benchmarks for Creo and Solidworks.
  • Kaivalya Dixit Distinguished Dissertation Award presented to Nadav Amit of Technion, Israel Institute of Technology.
  • 2016

  • SPEC holds benchmarking symposiums and technical meetings in Beijing, China.
  • SPEC Cloud_IaaS 2016, a new benchmark that measures the performance of infrastructure-as-a-service (IaaS) cloud implementations, is released.
  • SPECapc updates its benchmark for the NX application.
  • Kaivalya Dixit Distinguished Dissertation Award presented to Yufei Ren of Stony Brook University.
  • 2017

  • SPEC releases SPEC CPU2017, an all-new version of its flagship performance evaluation suite, with 43 individual benchmarks organized into four categories.
  • SERT 2.0, which adds a single-value metric and numerous other improvements, is released.
  • SPEC establishes the Power working group.
  • Version 2.0 of the Chauffeur Workload Development Kit (WDK) is released.
  • SPECapc updates its benchmark for the Maya application.
  • Kaivalya Dixit Distinguished Dissertation Award presented to Scott Beamer III of University of California, Berkeley.
  • 2018

  • SPEC releases SPECjEnterprise2018 Web Profile, a new benchmark measuring performance of Java EE 7 Web Profile application servers.
  • SPEC offers HPG benchmarks (ACCEL, MPI2007, OMP2012) free of charge to qualified non-profit organizations worldwide.
  • SPECviewperf 13 is released with new and updated viewsets, redesigned GUI, and improved scoring and reporting methods.
  • SPECworkstation 3 (formerly SPECwpc) is released with new and updated workloads throughout the benchmark suite.
  • SPEC Cloud IaaS 2018 is released with new workload versions and a wide range of methodology updates.
  • EPA's ENERGY STAR Version 3.0 for Computer Servers defines active state efficiency thresholds based on SERT.
  • SPECapc updates its benchmark for the Maya application.
  • SPEC RG creates the Quality of Experience working group.
  • Kaivalya Dixit Distinguished Dissertation Award presented to Aleksandar Milenkoski of Julius-Maximilians-University (JMU) Wuerzburg.