SPEC

The SPEC 30th Anniversary: Better Benchmarks Since 1988

Like a lot of institutions, the Standard Performance Evaluation Corporation (SPEC) sprang out of frustration, according to co-founder John Mashey.

"All the performance folks and systems architects were just so sick of being hassled with dumb benchmarks."

Thirty years later, SPEC has grown from a handful of members and a single benchmark to 125 members in 22 countries, presiding over nearly two dozen active benchmarks and tools for measuring performance and energy efficiency, conducting research into a wide range of performance issues, and sponsoring international conferences and symposiums.

Here's a timeline encapsulating the history of the world's leading computer benchmarking organization.

1988

SPEC incorporates on November 14.

1989

1989-VAX-11-780.jpg
  • SPECmark (subsequently known as SPEC CPU89) introduced with a VAX 11/780 as a reference machine and 10 computation-intensive benchmark programs (average size of 2.5 billion dynamic instructions per program).

1992

  • SPEC92 (subsequently known as SPEC CPU92) debuts with 20 benchmark programs and up to 10.2 billion dynamic instruction count.

1993

SPEC_SFS_1.0_results-1993-1998.jpg
  • SPECsfs 93, a one-test NFS benchmark suite derived from the Nhfsstone workload, is introduced.

1994

  • The SPEC High Performance Group (SPEC/HPG) is formed.
  • The Graphics Performance Characterization (GPC) group releases the SPECviewperf benchmark for graphics performance.

1995

1995-Sun_SparcStation_10_with_CRT.jpg
  • SPEC CPU95 goes live, with a SPARCstation 10 model 40 as its reference machine and 18 benchmark programs with up to 520.4 billion dynamic instruction count.

1996

SPEChpc96.jpg
  • SPEC/HPG releases SPEChpc96, a benchmark suite that measures the performance of HPC systems running industrial-style applications.
  • The GPC Group (now SPEC/GWPG) joins SPEC and releases SPECviewperf 5.0.
  • SPEC announces SPECweb96, the first standardized benchmark for measuring the performance of web servers.

1997

  • SPECsfs 97 released with major upgrades, including support for both NFS protocol version 3 and version 2.
  • The SPEC Application Performance Characterization (SPECapc) project group is formed under SPEC/GWPG.

1998

jvm98benchmarks.jpg SPECviewperf_DesignReview_viewset-1996.jpg
  • SPEC releases SPECjvm98, the first industry-standard benchmark that measures performance for Java virtual machine (JVM) client platforms.
  • SPECapc releases Solidworks 98 and Quake II application benchmarks.
  • SPECviewperf 6 is released with a new viewset and support for multiple primitive types and textures.

1999

  • SPECweb99 is released with a new framework and workload that reflect industry changes.
SPEC-CPU2000-Henning.jpg
  • SPECapc releases new application benchmarks for Pro/ENGINEER and SolidWorks CAD/CAM software.
  • On December 30, SPEC rings in the new millennium with SPEC CPU2000, comprising 26 benchmarks and dynamic instruction counts that reach 546.7 billion.

2000

  • SPEC/HPG announces SPEComp2001, a new benchmark development effort based on OMP applications.
  • SPEC releases its first server-side Java benchmark, SPEC JBB2000.
  • SPECapc releases a new benchmark for Unigraphics V15.

2001

  • SPEC releases SPECmail 2001, a benchmark for evaluating the performance of mail servers.
  • SPEC OMP2001, a benchmark suite based on OpenMP applications for evaluating the performance of SMP systems, goes live.
  • SPECapc releases new benchmarks for systems running 3ds Max, Solid Edge and Unigraphics software.

2002

jAppServer2001-designdoc.jpg
  • SPEC releases SPECjAppServer2001, a client/server benchmark that measures the performance of Java Enterprise Application Servers.
  • SPEC/HPG releases SPEC OMPL2001, a new benchmark suite that measures the performance of computing systems running large scalable applications.
ugs03viewset-SPECviewperf7-2002.jpg
  • New SPECviewperf 7.0 enables general state changes to be made during frames, bringing the benchmark much closer to real-world behavior.
  • The SPEChpc 2002 benchmark, derived from real HPC applications and practices, is released.
  • SPECapc updates its benchmarks for new versions of Solidworks, 3ds Max and Pro/E applications.

2003

2003-SPECapc_for_Solidworks_2003.jpg
  • SPECapc updates its benchmarks for new versions of Solid Edge, Solidworks and Maya applications.
  • SPEC/HPG releases V3.0 of the OMP benchmark.

2004

  • SPEC releases SPECjAppServer2004, a multi-tier benchmark for measuring the performance of Java 2 Enterprise Edition (J2EE) application servers.
  • SPECviewperf 8 released with four new viewsets and application tracing that gets closer to application behavior in the real world.
  • SPECapc updates its benchmarks for new versions of Solid Edge, Solidworks, Maya and 3ds Max applications.

2005

SPECjbb2005.jpg
  • SPEC releases its JBB2005 benchmark for evaluating server-side Java performance.
  • SPECweb 2005 released with three real-world workloads and dynamic content in PHP and JSP.
  • SPECapc updates its benchmarks for new versions of Maya and 3ds Max applications.

2006

CPU2006-integer_memory_use.jpg
  • SPEC CPU2006 is released with 29 benchmarks exercising new areas of functionality for CPUs, memory systems, and compilers.
  • The SPECpower committee is formed to develop standard methods and metrics for comparing energy use of server-class computers.
  • New SPECviewperf 9 features two new viewsets and major code changes. A Linux/Unix version is also released.
  • SPECapc updates its benchmarks for new versions of 3ds Max, NX and Pro/E applications.

2007

SPECpower_ssj2008-set-up.jpg
  • SPECpower_ssj2008, the first industry-standard benchmark that measures power consumption in relation to performance for server-class computers, goes live.
  • SPEC forms a new subcommittee to develop standard methods of comparing performance for servers using the Session Initiation Protocol (SIP).
  • SPECviewperf 10 adds performance measurement for full-scene anti-aliasing and multithreading.
  • SPEC/HPG releases MPI2007 to measure the performance of parallel computing systems and clusters running Message-Passing Interface (MPI) applications.
  • SPECjms2007, a new benchmark designed to measure performance of enterprise messaging servers based on JMS (Java Message Service), makes its debut.
  • SPECapc updates its benchmarks for 3ds Max, NX, Solid Edge and Solidworks.

2008

  • SPECsfs2008, designed to evaluate the speed and request-handling capabilities of file servers utilizing the NFSv3 and CIFS protocols, is released.
  • SPEC releases SPECjvm2008, a benchmark suite for measuring the performance of a Java Runtime Environment (JRE).
  • SPECmail2008, designed to measure the performance of enterprise mail servers compliant with SMTP and IMAP4 protocols, makes its debut.
  • SPECviewperf 10 for Linux/Unix is released.

2009

2009-2010-SERT_workload_efficiency-med.jpg
  • The SPECpower committee embarks on a project to develop a Server Efficiency Rating Tool (SERT) for use by the EPA.
  • SPECmail2009, a new version of the enterprise mail server benchmark, goes live.
  • SPECweb2009 is released with new power/performance measurement.
  • SPECjEnterprise2010 is released to measure performance for Java Enterprise Edition 5 or later application servers, databases and infrastructure.
  • SPECapc releases new benchmark for Lightwave 3D and updates its Maya application benchmark.

2010

2011-SPECviewperf11-lightwave-viewset.jpg
  • SPECviewperf 11 released with a new GUI and fully updated viewsets based on traces from professional applications.
  • SPECvirt_sc2010, SPEC's first benchmark for virtualized server consolidation, goes live.
2010-ICPE2010.jpg
  • SPEC/HPG releases MPI2007 v2.0, which adds the MPIL2007 suite to allow testing up to 2048 ranks.
  • SPEC and ACM launch the first International Conference on Performance Engineering (ICPE).

2011

2011-spec_research.jpg
  • SPEC's Research Group (SPEC RG) is formed to develop standard scenarios and workloads for emerging technologies and applications.
  • SPEC releases SPECsip_Infrastructure2011, designed to evaluate a system's ability to act as a SIP server.
  • SPECapc releases updated benchmark for the 3ds Max application.
  • SPEC RG establishes the annual Kaivalya Dixit Distinguished Dissertation Award.

2012

2012-cloud.png
  • SPEC forms a new Cloud subcommittee.
  • SPEC releases SPEC OMP 2012, an updated benchmark for measuring performance and energy efficiency using applications based on the OpenMP 3.1 standard.
  • SPEC/GWPG forms a workstation performance characterization (SPECwpc) group.
  • The first two SPEC Kaivalya Dixit Distinguished Dissertation Awards are presented.
  • SPEC RG establishes the Security Benchmarking and Cloud working groups.
  • SPECapc releases updated benchmarks for Creo, NX and Maya applications.

2013

  • The Server Efficiency Rating Tool (SERT), comprising synthetic worklets to test power consumption data at different load levels, makes its debut.
2013-Energy_Star_logo.png
  • SERT testing becomes mandatory part of the U.S. Environmental Protection Agency's (EPA) ENERGY STAR for Servers specification.
  • SPECjbb2013, developed from the ground up to measure performance based on the latest Java application features, is released.
  • SPECvirt_sc2013, the latest virtualized server benchmark, goes live.
2013-WPC_GUI.jpg
  • SPECwpc releases the first benchmark to measure all key aspects of workstation performance based on diverse professional applications.
  • SPECviewperf 12 released with extensible architecture and new viewsets representing energy and medical applications.
  • SPECapc updates benchmarks for NX and Solidworks applications.

2014

2014-ACCEL-Results.jpg
  • SPEC ACCEL V1.0, a benchmark suite that measures performance with computationally intensive parallel applications, goes live.
  • SPEC releases the Chauffeur WDK (Worklet Development Kit) Tool, designed to simplify the development of workloads for measuring both performance and energy efficiency.
  • SPEC SFS2014, SPEC's updated benchmark suite to measure performance of storage configurations, is released.
  • Fifth annual ICPE conference held in Dublin.
  • SPECapc updates its benchmark for the 3ds Max application.
  • SPEC RG forms working groups for Big Data and DevOps Performance.

2015

2015-SPECwpc2.0.jpg
  • SPECwpc 2.0 is released with improved storage workloads and better scalability measurement for multi-core systems.
  • SPECjbb2015, an enhanced Java server benchmark that now supports virtualized hosts, is released.
  • SPECapc updates benchmarks for Creo and Solidworks.

2016

2016-DavidatAsiaSummit.jpg
  • SPEC holds benchmarking symposiums and technical meetings in Beijing, China.
  • SPEC Cloud_IaaS 2016, a new benchmark that measures the performance of infrastructure-as-a-service (IaaS) cloud implementations, is released.
  • SPECapc updates its benchmark for the NX application.

2017

2017-CPU-suite-growth.png
  • SPEC releases SPEC CPU2017, an all-new version of its flagship performance evaluation suite, with 43 individual benchmarks organized into four categories.
  • SERT 2.0, which adds a single-value metric and numerous other improvements, is released.
  • SPEC establishes the Power working group.
  • Version 2.0 of the Chauffeur Workload Development Kit (WDK) is released.
  • SPECapc updates its benchmark for the Maya application.

2018

2018-jEnterprise2018-hardware-environment.jpg
  • SPEC releases SPECjEnterprise2018 Web Profile, a new benchmark measuring performance of Java EE 7 Web Profile application servers.
  • SPEC offers HPG benchmarks (ACCEL, MPI2007, OMP2012) free of charge to qualified non-profit organizations worldwide.
  • SPECviewperf 13 is released with new and updated viewsets, redesigned GUI, and improved scoring and reporting methods.
  • SPECworkstation 3 (formerly SPECwpc) is released with new and updated workloads throughout the benchmark suite.
  • SPEC Cloud IaaS 2018 is released with new workload versions and a wide range of methodology updates.
  • EPA's ENERGY STAR Version 3.0 for Computer Servers defines active state efficiency thresholds based on SERT.
  • SPECapc updates its benchmark for the Maya application.
  • SPEC RG creates the Quality of Experience working group.