SPEC Fair Use Rule

Updated 3 November 2014

Change history:


11 April 2011 - The SPEC Fair Use rule has been re-written to:

  1. Promote greater Fair Use consistency across SPEC benchmarks; and
  2. Where Fair Use rules differ among SPEC benchmarks, make it easier to find differences.

22 June 2011 - Editorial clarifications

  1. Emphasize that comparisons of non-compliant numbers must not be deceptive.
  2. Explain the term "major release" as used in the rule about comparisons.
  3. Clarify example for normalized historical comparisons.
  4. Clarify definition of Close Proximity.
  5. Prefer term "licensee" rather than synonyms.
  6. Minor editorial clarifications.

18 August 2011 - Add SPECsip_Infrastructure2010. Correct the metrics list for SPECweb2009.

7 February 2013 - Add SPECjbb2013, SPEC OMP2012.

25 February 2013 - Add SERT.

13 March 2014 - Add SPEC Accel, Chauffeur-WDK. Note retirement of SPECjAppServer2004, SPECjbb2005, SPECmail2001 and SPECmail2009, SPEC OMP2001, SPECvirt_sc2010, SPECweb2005 and SPECweb2009.

3 November 2014 - Add SPEC SFS2014.


Introduction

Consistency and fairness are guiding principles for SPEC. To help assure that these principles are met, the following requirements must be met by any organization or individual who makes public use of SPEC benchmark results.

Section I lists general requirements that apply to public use of all SPEC benchmarks. Section II lists additional specific requirements for individual benchmarks.

As of early 2011, SPEC is in the process of removing Fair Use rules from individual benchmark run rules documents, centralizing that information here for the convenience of users. It is intended that this document provide all of the information that is needed for compliance with Fair Use. In the event of contradiction between this rule and individual benchmark run rules, this SPEC-wide rule shall prevail.

Contents

I. General Requirements For Public Use
   of All SPEC Benchmark Results

A. Requirements List

1. Compliance

2. Data Sources

3. Clear and correct, as of a specific date

4. Trademarks

5. Required Metrics

6. Comparisons

B. Generic Example

C. Compliance Exceptions

1. Academic/Research Usage

2. Estimates

D. Derived Values

E. Non-SPEC Information

F. Retired Benchmarks

1. Disclosure

2. Benchmarks that require review

3. Normalized historical comparisons

II. Requirements For Public Use of Individual Benchmark Results

SPEC ACCEL       SPECapc for 3ds Max 9       SPECapc for LightWave 3D v9.6     SPECapc for Maya 2009       SPECapc for Pro/ENGINEER Wildfire 2.0     SPECapc for Solid Edge V19     SPECapc for SolidWorks 2007     SPECapc for UGS NX4     SPEC CPU2006     SPEC jAppserver2004     SPECjbb2005     SPECjbb2013     SPECjEnterprise2010     SPECjms2007     SPECjvm2008     SPECmail2001     SPECmail2009     SPEC MPI2007     SPEC OMP2001     SPEC OMP2012     SPECpower_ssj2008     SPEC SFS2014     SPECsfs2008     SPECsip_Infrastructure2011     SPECviewperf 11     SPECvirt_sc2010     SPECvirt_sc2013     SPECweb2005     SPECweb2009     SERT     Chauffeur WDK    

III. Definitions

Basis for Comparison     By Location     Close Proximity     Compliant Result     Derived Values     Disallowed Comparisons     Estimate     Major Release     Non-Compliant Number    Required Metric     SPEC Metric    

IV. Violations Determination, Penalties, and Remedies

I. General Requirements For Public Use of All SPEC Benchmark Results

I.A. Requirements List

  1. Compliance. Claimed results must be compliant with that benchmark's rules. See definition: compliant result. (Certain Exceptions may apply.)

  2. Data Sources

    1. Source(s) must be stated for quoted SPEC results.

    2. Such sources must be publicly available, from SPEC or elsewhere.

    3. The licensee (the entity responsible for the result) must be clearly identifiable from the source.

    4. The date that the data was retrieved must be stated.

    5. The SPEC web site (http://www.spec.org) or a suitable sub page must be noted as a resource for additional information about the benchmark.

  3. Clear and correct, as of a specific date

    1. Statements regarding SPEC, its benchmarks, and results published by SPEC, must be clear and correct.

    2. A claim must state a date as of which data was retrieved.

    3. A claim may compare newly announced compliant results vs. data retrieved earlier.

    4. There is no requirement to update a claim when later results are published.

    For example, an Acme web page dated 28 January 2011 announces performance results for the Model A and claims "the best SPECweb2009 performance when compared vs. results published at www.spec.org as of 26 January 2011". If SPEC publishes better results on 1 February, there is no requirement to update the page.

  4. Trademarks.

    1. Reference must be made to the SPEC trademark. Such reference may be included in a notes section with other trademark references (SPEC trademarks are listed at http://www.spec.org/spec/trademarks.html).

    2. SPEC's trademarks may not be used to mislabel something that is not a SPEC metric.

      For example, suppose that a Gaming Society compares performance using a composite of a weighted subset of SPEC CPU2006 plus a weighted subset of SPECviewperf 11, and calls its composite "GamePerfMark". The composite, weighting, and subsetting are done by the Society, not by SPEC. The composite may be useful and interesting, but it may not be represented as a SPEC metric. It would be a Fair Use violation to reference it as "SPECgame".

  5. Required Metrics. In the tables below, some benchmarks have Required Metrics. Public statements must include these.

  6. Comparisons. It is fair to compare compliant results to other compliant results. Enabling such comparisons is a core reason why the SPEC benchmarks exist. Each benchmark product has workloads, software tools, run rules, and review processes that are intended to improve the technical credibility and relevance of such comparisons.

    When comparisons are made,

    1. SPEC metrics may be compared only to SPEC metrics.

    2. The basis for comparison must be stated.

    3. Results of one benchmark are not allowed to be compared to a different benchmark
      (e.g. SPECjAppServer2004 to TPC-C; or SPECvirt_sc2010 to SPECweb2005).

    4. Results of a benchmark may not be compared to a different major release of the same benchmark
      (e.g. SPECweb2005 to SPECweb2009). Exception: normalized historical comparisons may be made as described under Retired Benchmarks.

    5. Comparisons of non-compliant numbers. The comparison of non-compliant numbers to compliant results is restricted to certain exceptional cases described later in this Fair Use Rule (Academic/Research usage; Estimates, for those benchmarks that allow estimates; Normalized Historical Comparisons). Where allowed, comparisons that include non-compliant numbers must not be misleading or deceptive as to compliance. It must be clear from the context of the comparison which numbers are compliant and which are not.

Back to Contents

I.B. Generic Example

This example for a generic SPEC benchmark illustrates the points above. See also the examples for specific benchmarks below, for additional requirements that may apply.

Example: New York, NY, January 28, 2011: Acme Corporation announces that the Model A achieves 100 for SPECgeneric2011, a new record among systems running Linux [1].
[1] Comparison based on best performing systems using the Linux operating system published at www.spec.org as of 26 January 2011. SPEC® and the benchmark name SPECgeneric® are registered trademarks of the Standard Performance Evaluation Corporation. For more information about SPECgeneric2011, see www.spec.org/generic2011/.

Back to Contents

I.C. Compliance Exceptions

Exceptions regarding the compliance requirement are described in this section.

  1. Academic/research usage. SPEC encourages use of its benchmarks in research and academic contexts, on the grounds that SPEC benchmarks represent important characteristics of real world applications and therefore research innovations measured with SPEC benchmarks may benefit real users. SPEC understands that academic use of the SPEC benchmarks may be seen as enhancing the credibility of both the researcher and SPEC.

    Research use of SPEC benchmarks may not be able to meet the compliance requirement.

    Examples: (1) Testing is done with a simulator rather than real hardware. (2) The software innovation is not generally available or is not of product quality. (3) The SPEC test harness is modified without approval of SPEC.

    SPEC has an interest in protecting the integrity of the SPEC metrics, including consistency of methods of measurement and the meaning of the units of measure that are defined by SPEC benchmarks. It would be unfair to those who do meet the compliance requirements if non-compliant numbers were misrepresented as compliant results. Therefore, SPEC recommends that researchers consider using the SPEC workload, but do not call the measurements by the SPEC metric name.

    The requirements for Fair Use in academic/research contexts are:

    1. It is a Fair Use violation to imply, to the reasonable reader, that a non-compliant number is a compliant result.
    2. Non-compliance must be clearly disclosed. If the SPEC metric name is used, it is recommended that (nc), for non-compliant, be added after each mention of the metric name. It is understood that there may be other ways to accomplish this in context, for example adding words such as "experimental" or "simulated" or "estimated" or "non-compliant".
    3. Diagrams, Tables, and Abstracts (which, often, are excerpted and used separately) must have sufficient context on their own so that they are not misleading as to compliance.
    4. If non-compliant numbers are compared to compliant results it must be clear from the context which is which.
    Example: The Acme Corporation Model A achieves SPECint2006  100 in testing published at www.spec.org. Our Research Compiler improves the same hardware to SPECint2006  125(nc). The notation (nc), for non-compliant, is used because our compiler does not meet SPEC's requirements for general availability.

    Other Fair Use Requirements Still Apply. This section discusses an exception to only the compliance requirement from the Requirements List. Fair Use in academic/research context must still meet the other requirements, including but not limited to making correct use of SPEC results with dated citations of sources.

  2. Estimates. Some SPEC benchmarks allow estimates, as shown in the tables below. Only for those benchmarks, it is acceptable to compare estimates to compliant results provided that:

    1. Estimates must be clearly identified as such.

    2. Each use of a SPEC metric as an estimate must be clearly marked as an estimate.

    3. If estimates are used in graphs, the word "estimated" or "est." must be plainly visible within the graph, for example in the title, the scale, the legend, or next to each individual number that is estimated.

    4. Licensees are encouraged to give a rationale or methodology for any estimates, together with other information that may help the reader assess the accuracy of the estimate.

      Example 1: The Acme Corporation Model A achieves SPECint2006  100 in testing published at www.spec.org. The Bugle Corporation Model B will nearly double that performance to SPECint2006  198(est). The notation (est), for estimated, is used because SPECint2006 was run on pre-production hardware. Customer systems, planned for Q4, are expected to be similar.

      Example 2: Performance estimates are modeled using the cycle simulator GrokSim Mark IV. It is likely that actual hardware, if built, would include significant differences.

Back to Contents

I.D. Derived Values

It is sometimes useful to define a numeric unit that includes a SPEC metric plus other information, and then use the new number to compare systems. This is called a Derived Value.

Examples: SPECint_rate2006 per chip
SPECvirt_sc2010 per gigabyte

Note: the examples above are not intended to imply that all derived values use ratios of the form above. The definition is intentionally broad, and includes additional examples

  1. Derived values are acceptable, provided that they follow this Fair Use rule, including but not limited to using compliant results, listing sources for SPEC result data, and including any required metrics.

  2. A derived value must not be represented as a SPEC metric. The context must not give the appearance that SPEC has created or endorsed the derived value. In particular, it is a Fair Use violation, and may be a Trademark violation, to form a new word that looks like a SPEC metric name when there is no such metric.

    Not Acceptable: SPECint_chiprate2006
    SPECvirt_sc2010gigs
  3. If a derived value is used as the basis of an estimate, the estimate must be correctly labeled. A derived value may introduce seeming opportunities to extrapolate beyond measured data. For example, if 4 different systems all have the same ratio of SPECwhatever per chip, it can be tempting to estimate that another, unmeasured, system will have the same ratio. This may be a very good estimate; but it is still an estimate, and must be correctly labeled. If used in public, it must be for a benchmark that allows estimates.

Back to Contents

I.E. Non-SPEC Information

  1. A basis of comparison or a derived value may use information from both SPEC and non-SPEC sources.

  2. SPEC values truthfulness and clarity at all times:

  3. Disclaimer.    SPEC is not responsible for non-SPEC information. The SPEC Fair Use rule is limited to the information derived from SPEC sources. (Other rules may apply to the non-SPEC information, such as industry business standards, ethics, or Truth in Advertising law.)

  4. SPEC may point out non-SPEC content.    SPEC reserves the right to publicly comment to distinguish SPEC information from non-SPEC information.

  5. Integrity of results and trademarks.    The non-SPEC information must not be presented in a manner that may reasonably lead the reader to untrue conclusions about SPEC, its results, or its trademarks.

Examples

Example 1 (basis): ACME Corporation claims the best SPECjEnterprise2010 performance for systems available as (example 1a) rack mount, or (1b) with more than 8 disk device slots, or (1c) with Art Deco paint. Bugle Corporation asserts that the basis of comparison is irrelevant or confusing or silly. Bugle may be correct. Nevertheless, such irrelevance, confusion, or silliness would not alone be enough to constitute a SPEC Fair Use violation.

Example 2 (derived value): ACME claims that its model A has better SPECint_rate2006 per unit of cooling requirement than does the Bugle Model B. SPEC is not responsible for judging thermal characteristics.

Example 3: ACME claims the "best SPECmpiM_2007 performance among industry-leading servers". This claim violates the requirement that the basis must be clear.

Example 4: ACME computes SPECint_rate2006 per unit of cooling, but inexplicably selects SPECint_rate_base2006 for some systems and SPECint_rate2006 for others. The computation violates the requirement that the SPEC information must be accurate, and may also violate the requirement that a claim should not lead the reasonable reader to untrue conclusions about SPEC's results.

Back to Contents

I.F. Retired Benchmarks

  1. Disclosure. If public claims are made using a retired benchmark, with compliant results that have not been previously reviewed and accepted by SPEC, then the fact that the benchmark has been retired and new results are no longer being accepted for review and publication by SPEC must be plainly disclosed.

    Example: The Acme Corporation Model A achieves a score of 527 SPECjvm98. Note: SPECjvm98 has been retired and SPEC is no longer reviewing or publishing results with that benchmark. We are providing this result as a comparison to older hardware that may still be in use at some customer sites.
  2. Benchmarks that require review. Some benchmarks require that SPEC review and accept results prior to public use. For such benchmarks, the review process is not available after benchmark retirement, and therefore no new results may be published.

  3. Normalized historical comparisons. When SPEC releases a new major version of a benchmark, the SPEC metrics are generally not comparable to the previous version, and there is no formula for converting from one to the other. Nevertheless, SPEC recognizes that there is value in historical comparisons, which are typically done by normalizing performance across current and one or more generations of retired benchmarks, using systems that have been measured with both the older and newer benchmarks as the bridges for the normalization. Historical comparisons are inherently approximate because picking differing 'bridge' systems may yield differing ratios and because an older workload exercises different system capabilities than a more modern workload.

    Normalized historical comparisons are acceptable only if their inherently approximate nature is not misrepresented. At minimum:

    1. It must not be claimed that SPEC metrics for one benchmark generation are precisely comparable to metrics from another generation.
    2. The approximate nature must be apparent from the context.
      For example, a graph shown briefly in a presentation is labelled "Normalized Historic Trends for SPEC<benchmark>". As another example, in a white paper (where the expectation is for greater detail than presentations), the author explicitly calls out that workloads have differed over time, and explains how numbers are calculated.

Back to Contents

II. Requirements for Public Use of Individual Benchmark Results

For further detail about the meaning of SPEC metrics, the individual benchmark run rules may be consulted. The benchmark names at the top of each table are links to that benchmark's run rules.

Back to Contents

SPEC ACCEL

SPEC.org Submission
Requirements
None. Submission to SPEC is encouraged, but is not required.
SPEC Metrics
  • The bottom line metrics: SPECaccel_ocl_base, SPECaccel_acc_base, SPECaccel_ocl_peak, SPECaccel_acc_peak, SPECaccel_ocl_energy_base, SPECaccel_acc_energy_base, SPECaccel_ocl_energy_peak, SPECaccel_acc_energy_peak
  • Median individual benchmark SPECratios
  • Median run times of the individual benchmarks
Required Metrics None
Conditionally
Required Metrics
For an individual benchmark, if a result other than the median is mentioned, then the median from the same set must also be mentioned.
Use of Estimates Estimates are allowed if clearly identified. Power measurement metrics are not allowed to be estimated
Disallowed
Comparisons
No additional requirements beyond the requirements that results not be compared to other benchmarks.

Back to Contents

SPECapc for 3ds Max 9

SPEC.org Submission
Requirements
None. Submission to SPEC is encouraged, but is not required.
SPEC Metrics Rendering Composite, Graphics Composite, Shaders Composite
Required Metrics Rendering Composite, Graphics Composite
Conditionally
Required Metrics
Shaders composite if using DirectX mode
Use of Estimates Estimates are allowed if clearly identified.
Disallowed
Comparisons
No additional requirements beyond the requirements that results not be compared to other benchmarks.

Back to Contents

SPECapc for LightWave 3D v9.6

SPEC.org Submission
Requirements
None. Submission to SPEC is encouraged, but is not required.
SPEC Metrics Interactive Composite, Render Composite, Multitask Composite
Required Metrics None
Conditionally
Required Metrics
None
Use of Estimates Estimates are allowed if clearly identified.
Disallowed
Comparisons
No additional requirements beyond the requirements that results not be compared to other benchmarks.

Back to Contents

SPECapc for Maya 2009

SPEC.org Submission
Requirements
None. Submission to SPEC is encouraged, but is not required.
SPEC Metrics Graphics, CPU, I/O, Overall Composite
Required Metrics None
Conditionally
Required Metrics
None
Use of Estimates Estimates are allowed if clearly identified.
Disallowed
Comparisons
No additional requirements beyond the requirements that results not be compared to other benchmarks.

Back to Contents

SPECapc for Pro/ENGINEER Wildfire 2.0

SPEC.org Submission
Requirements
None. Submission to SPEC is encouraged, but is not required.
SPEC Metrics Graphics Wireframe, Graphics Shaded, CPU, I/O, File Time, Overall Composite
Required Metrics None
Conditionally
Required Metrics
None
Use of Estimates Estimates are allowed if clearly identified.
Disallowed
Comparisons
No additional requirements beyond the requirements that results not be compared to other benchmarks.

Back to Contents

SPECapc for Solid Edge V19

SPEC.org Submission
Requirements
None. Submission to SPEC is encouraged, but is not required.
SPEC Metrics CPU Composite, Graphics Composite, File I/O Composite, Overall Composite
Required Metrics
Conditionally
Required Metrics
Use of Estimates Estimates are allowed if clearly identified.
Disallowed
Comparisons
No additional requirements beyond the requirements that results not be compared to other benchmarks.

Back to Contents

SPECapc for SolidWorks 2007

SPEC.org Submission
Requirements
None. Submission to SPEC is encouraged, but is not required.
SPEC Metrics CPU Intensive, File I/O Intensive, Graphics Composite, Overall Composite
Required Metrics None
Conditionally
Required Metrics
None
Use of Estimates Estimates are allowed if clearly identified.
Disallowed
Comparisons
No additional requirements beyond the requirements that results not be compared to other benchmarks.

Back to Contents

SPECapc for UGS NX4

SPEC.org Submission
Requirements
None. Submission to SPEC is encouraged, but is not required.
SPEC Metrics CPU Composite, File I/O Composite, Graphics Composite, Overall Composite
Required Metrics None
Conditionally
Required Metrics
None
Use of Estimates Estimates are allowed if clearly identified.
Disallowed
Comparisons
No additional requirements beyond the requirements that results not be compared to other benchmarks.

Back to Contents

SPEC CPU2006

SPEC.org Submission
Requirements
None. Submission to SPEC is encouraged, but is not required. Compliant results may be published independently.
SPEC Metrics
  • The bottom line metrics: SPECint_base2006, SPECint2006, SPECfp_base2006, SPECfp2006, SPECint_rate_base2006, SPECint_rate2006, SPECfp_rate_base2006, SPECfp_rate2006
  • Individual benchmark SPECratios
  • Median run times of the individual benchmarks
Required Metrics None.
Conditionally
Required Metrics
If a run time other than the median is quoted, then the median must also be quoted.
Use of Estimates
  1. Estimates are allowed if clearly identified.
  2. It is permitted to estimate any of the SPEC Metrics listed above.
  3. It is permitted to estimate a peak metric without providing a corresponding base estimate
Disallowed
Comparisons
No additional requirements beyond the requirements that results not be compared to other benchmarks.

Back to Contents

(Retired) SPEC jAppserver2004

RETIRED SPECjAppServer2004 was retired on November 30, 2010.
  • All public use of results for this benchmarkmust plainly disclose that the benchmark has been retired, as described above.
  • No further submissions will be accepted for publication at www.spec.org
  • SPEC is no longer reviewing results for this benchmark.
  • Independent publication of new results is not allowed (because the benchmark results required review by SPEC before publication)
SPEC.org Submission
Requirements
Results must be reviewed and accepted by SPEC prior to public disclosure.
SPEC Metrics SPECjAppServer2004 JOPS@Category
Required Metrics SPECjAppServer2004 JOPS@Category
Conditionally
Required Metrics
Use of Estimates Not allowed
Disallowed
Comparisons

In addition to the requirements that results not be compared to other benchmarks:

Results between different categories (see the run rules section on Standard Vs. Distributed) within SPECjAppServer2004 may not be compared.

Back to Contents

(RETIRED) SPECjbb2005

RETIRED SPECjbb2005 was retired on October 1, 2013 in favor of its successor, SPECjbb2013.
  • All public use of results for this benchmark must plainly disclose that the benchmark has been retired, as described above.
  • No further submissions will be accepted for publication at www.spec.org.
  • SPEC is no longer reviewing results for this benchmark.
  • Rule-compliant results may be published independently, provided that the fact of retirement is plainly disclosed
SPEC.org Submission
Requirements
Vendors may publish compliant results independently, provided that the first use of input.expected_peak_warehouse property by the vendor be reviewed by the subcommittee to determine compliance with run rules section 2.3. Future publications by the vendor using input.expected_peak_warehouse do not require review unless the technical reason for setting the flag differs from what was previously accepted by the subcommittee.
SPEC Metrics SPECjbb2005 bops, SPECjbb2005 bops/JVM
Required Metrics
  1. The number of jvms used in the benchmark must be stated
  2. Both throughput metrics (SPECjbb2005 bops and SPECjbb2005 bops/JVM) must be stated.
Conditionally
Required Metrics
Use of Estimates Not allowed
Disallowed
Comparisons
No additional requirements beyond the requirements that results not be compared to other benchmarks.

Back to Contents

SPECjbb2013

SPEC.org Submission
Requirements
Vendors may publish compliant results independently, provided that run does not produce any warning or invalid messages and all run and reporting rules are followed. Any result which has warnings can only be used once accepted by the OSGjava subcommittee.
SPEC Metrics SPECjbb2013-<category> max-jOPS and SPECjbb2013-<category> critical-jOPS where <category>: [Composite / MultiJVM / Distributed]
Required Metrics Both metrics SPECjbb2013-<category> max-jOPS and SPECjbb2013-<category> critical-jOPS must be stated in proximity.
Conditionally
Required Metrics
Use of Estimates Not allowed
Disallowed
Comparisons

In addition to the requirements that results not be compared to other benchmarks:

  1. Any comparison across categories SPECjbb2013-Composite, SPECjbb2013-MultiJVM and SPECjbb2013-Distributed is prohibited

Back to Contents

SPECjEnterprise2010

SPEC.org Submission
Requirements
Results must be reviewed and accepted by SPEC prior to public disclosure.
SPEC Metrics SPECjEnterprise2010 EjOPS
Required Metrics SPECjEnterprise2010 EjOPS
Conditionally
Required Metrics
Use of Estimates Not allowed
Disallowed
Comparisons
No additional requirements beyond the requirements that results not be compared to other benchmarks.

Back to Contents

SPECjms2007

SPEC.org Submission
Requirements
Results must be reviewed and accepted by SPEC prior to public disclosure.
SPEC Metrics SPECjms2007@Category
Required Metrics
Conditionally
Required Metrics
Use of Estimates Not allowed
Disallowed
Comparisons
No additional requirements beyond the requirements that results not be compared to other benchmarks.

Back to Contents

SPECjvm2008

SPEC.org Submission
Requirements
None. Submission to SPEC is encouraged, but is not required. Compliant results may be published independently.
SPEC Metrics SPECjvm2008 Base ops/m and SPECjvm2008 Peak ops/m
Required Metrics
Conditionally
Required Metrics
Use of Estimates Not allowed
Disallowed
Comparisons
No additional requirements beyond the requirements that results not be compared to other benchmarks.

Back to Contents

(Retired) SPECmail2001

RETIRED SPECmail2001 was retired on October 31, 2011.
  • All public use of results for this benchmark must plainly disclose that the benchmark has been retired, as described above.
  • No further submissions will be accepted for publication at www.spec.org.
  • SPEC is no longer reviewing results for this benchmark.
  • Rule-compliant results may be published independently, provided that the fact of retirement is plainly disclosed
SPEC.org Submission
Requirements
None. Submission to SPEC is encouraged, but is not required. Compliant results may be published independently.
SPEC Metrics SPECmail2001 and SPECmail2001_users
Required Metrics
Conditionally
Required Metrics
Use of Estimates Not allowed
Disallowed
Comparisons
No additional requirements beyond the requirements that results not be compared to other benchmarks.

Back to Contents

(Retired) SPECmail2009

RETIRED SPECmail2009 was retired on October 31, 2011.
  • All public use of results for this benchmark must plainly disclose that the benchmark has been retired, as described above.
  • No further submissions will be accepted for publication at www.spec.org.
  • SPEC is no longer reviewing results for this benchmark.
  • Rule-compliant results generated in a test location which has previously produced an accepted compliant result may be published independently, provided that the fact of retirement is plainly disclosed. Independent publication of new results generated in test locations which have not met this requirement is not allowed
SPEC.org Submission
Requirements
By location, as defined below.
SPEC Metrics SPECmail_Ent2009, SPECmail_Ent2009Secure
Required Metrics
Conditionally
Required Metrics
Use of Estimates Not allowed
Disallowed
Comparisons
No additional requirements beyond the requirements that results not be compared to other benchmarks.

Back to Contents

SPEC MPI2007

SPEC.org Submission
Requirements
None. Submission to SPEC is encouraged, but is not required. Compliant results may be published independently.
SPEC Metrics
  • The bottom line metrics:
      SPECmpiM_base2007, SPECmpiM_peak2007, SPECmpiM_2007,
      SPECmpiL_base2007, SPECmpiL_peak2007, SPECmpiL_2007
  • Median individual benchmark SPECratios
  • Median run times of the individual benchmarks
Required Metrics None.
Conditionally
Required Metrics
For an individual benchmark, if a result other than the median is mentioned, then the median from the same set must also be mentioned.
Other Required Information CPU description (number of chips and cores), and degree of parallelism (MPI ranks).
Use of Estimates
  1. Estimates are allowed if clearly identified.
  2. It is permitted to estimate any of the SPEC Metrics listed above.
  3. It is permitted to estimate a peak metric without providing a corresponding base estimate
Disallowed
Comparisons
No additional requirements beyond the requirements that results not be compared to other benchmarks.

Back to Contents

(Retired) SPEC OMP2001

RETIRED SPEComp2001 was retired on January 16, 2013.
  • All public use of results for this benchmark must plainly disclose that the benchmark has been retired, as described above.
  • No further submissions will be accepted for publication at www.spec.org.
  • SPEC is no longer reviewing results for this benchmark.
  • Rule-compliant results may be published independently, provided that the fact of retirement is plainly disclosed
SPEC.org Submission
Requirements
None. Submission to SPEC is encouraged, but is not required. Compliant results may be published independently.
SPEC Metrics
  • The bottom line metrics: SPECompMbase2001, SPECompMpeak2001, SPECompM2001,
    SPECompLbase2001, SPECompLpeak2001, SPECompL2001
  • Median individual benchmark SPECratios
  • Median run times of the individual benchmarks
Required Metrics None.
Conditionally
Required Metrics
For an individual benchmark, if a result other than the median is mentioned, then the median from the same set must also be mentioned.
Other Required Information CPU description (number of chips and cores), and degree of parallelism (OpenMP threads).
Use of Estimates Estimates are allowed if clearly identified.
Disallowed
Comparisons
No additional requirements beyond the requirements that results not be compared to other benchmarks.

Back to Contents

SPEC OMP2012

SPEC.org Submission
Requirements
None. Submission to SPEC is encouraged, but is not required. Compliant results may be published independently.
SPEC Metrics
  • The bottom line metrics: SPECompG_base2012, SPECompG_peak2012, SPECompG_2012,
    SPECompG_energy_base2012, SPECompG_energy_peak2012
  • Median individual benchmark SPECratios
  • Median run times of the individual benchmarks
Required Metrics None.
Conditionally
Required Metrics
For an individual benchmark, if a result other than the median is mentioned, then the median from the same set must also be mentioned.
Other Required Information CPU description (number of chips and cores), and degree of parallelism (OpenMP threads).
Use of Estimates Estimates are allowed if clearly identified.
Disallowed
Comparisons
No additional requirements beyond the requirements that results not be compared to other benchmarks.

Back to Contents

SPECpower_ssj2008

SPEC.org Submission
Requirements
By location, as defined below.
SPEC Metrics
  • SPECpower_ssj2008 overall ssj_ops/watt
  • for a specific target load level, its ssj_ops and Average Active Power(W).
Required Metrics

SPECpower_ssj2008 overall ssj_ops/watt

The required metric must be listed in close proximity to any other measured data from the disclosure or any derived value.

Conditionally
Required Metrics
ConditionRequirement
1. Comparison of performance or power data from the same target load level Both the performance and the power results for that target load must be disclosed in close proximity.
2. Comparison of SUTs with different numbers of nodes The number of nodes for each SUT must be disclosed in close proximity.
3. Comparison of one target load level on one system and a different target load level on another system The performance and power at the 100% target load level must be disclosed in close proximity.
4. Comparison of Active Idle points The performance and power at the 100% target load level must be disclosed in close proximity.
5. Derived value based on a subset of performance and/or power information from a multi-node result The number of nodes and the calculation method must be disclosed in close proximity.
Examples
  1. When fully loaded, Server X provides more performance and consumes less power than Server Y. Server X scores: (95,853 ssj_ops and 276W) @ 100% target load vs. Server Y: (40,852 ssj_ops and 336W) @ 100%. The SPECpower_ssj2008 overall ssj_ops/watt are Server X: 203 and Server Y: 87.4 [1].
  2. Server X provides greater efficiency than Server Y. The SPECpower_ssj2008 overall ssj_ops/watt for 4-node Server X is 203 and for 2-node Server Y is 87.4 [1].
  3. Server X does not pass 250W until near full load, whereas Server Y reaches it much earlier. Server X scores (79,346 ssj_ops and 252W) @ 90% target load while Server Y scores (8,237 ssj_ops and 254W) @ 30%. When fully loaded, Server X scores (95,853 ssj_ops and 276W) @ 100% and Server Y scores (40,852 ssj_ops and 336W) @ 100%. The SPECpower_ssj2008 overall ssj_ops/watt are Server X: 203 and Server Y: 87.4 [1]
  4. Server X uses only 50W at the Active Idle point, compared to 255W at Active Idle for Server Y. Server X scores (185,000 ssj_ops and 200W) @ 100% target load and Server Y scores (240,000 ssj_ops and 200W) @ 100%. The SPECpower_ssj2008 overall ssj_ops/watt are Server X: 512 and Server Y: 450 [1]
  5. Server X provides better performance and uses less power than the individual nodes of Server Y. The single node server X scores (766 ssj_ops and 1,050W) @ 100% target load level. Server Y is a 10-node server which scores (2,550 ssj_ops and 10KW) @ 100% -- which means that on average each of the nodes uses (255 ssj_ops and 1000W) @ 100%. The SPECpower_ssj2008 overall ssj_ops/watt results are Server X: 415 and Server Y: 325 [1]
[1] Comparison based on results for the named systems as published at www.spec.org as of 26 January 2011. SPEC® and the benchmark name SPECpower_ssj® are registered trademarks of the Standard Performance Evaluation Corporation. For more information about SPECpower, see www.spec.org/power_ssj2008/.
Use of Estimates Not allowed
Disallowed
Comparisons

In addition to the requirements that results not be compared to other benchmarks:

  • Comparisons may not be drawn between a Server and a Personal System. See the "System Designation" definitions in the benchmark run rules
  • The calibration throughputs must not be used (use the 100% target load level instead).

Back to Contents

SPEC SFS2014

SPEC.org Submission
Requirements
None. Submission to SPEC is encouraged, but is not required. Compliant results may be published independently.
SPEC Metrics
  • SPECsfs2014_database and corresponding Overall Response Time
  • SPECsfs2014_vdi and corresponding Overall Response Time
  • SPECsfs2014_vda and corresponding Overall Response Time
  • SPECsfs2014_swbuild and corresponding Overall Response Time
Required Metrics

Peak SPECsfs2014_database Ops/sec or SPECsfs2014_vdi Ops/sec or SPECsfs2014_vda Ops/sec or SPECsfs2014_swbuild Ops/sec and the ORT (overall response time)

Conditionally
Required Metrics
Use of Estimates Not allowed
Disallowed
Comparisons
No additional requirements beyond the requirements that results not be compared to other benchmarks. SPEC SFS2014 results for different workloads may not be compared.

Back to Contents

SPECsfs2008

SPEC.org Submission
Requirements
None. Submission to SPEC is encouraged, but is not required. Compliant results may be published independently.
SPEC Metrics
  • SPECsfs2008_nfs.v3 and corresponding Overall Response Time
  • SPECsfs2008_cifs and corresponding Overall Response Time
Required Metrics Peak SPEC SFS2008_nfs or SFS2008_cifs Ops/sec and the ORT (overall response time)
Conditionally
Required Metrics
Use of Estimates Not allowed
Disallowed
Comparisons
No additional requirements beyond the requirements that results not be compared to other benchmarks.

Back to Contents

SPECsip_Infrastructure2011

SPEC.org Submission
Requirements
By location, as defined below.
SPEC Metrics
  • SPECsip_Infrastructure2011 Supported Subscribers
Required Metrics SPECsip_Infrastructure2011 Supported Subscribers
Conditionally
Required Metrics
None
Use of Estimates Not allowed
Disallowed
Comparisons

No additional requirements beyond the requirements that results not be compared to other benchmarks.

Back to Contents

SPECviewperf 11

SPEC.org Submission
Requirements
None. Submission to SPEC is encouraged, but is not required. Compliant results may be published independently.
SPEC Metrics
  • catia-03 geometric mean, ensight-04 geometric mean, lightwave-01 geometric mean, maya-03 geometric mean, proe-05 geometric mean, sw-03 geometric mean, tcvis-02 geometric mean, snx-01 geometric mean
  • Any subtest of the SPECViewperf 11 viewsets
Required Metrics None
Conditionally
Required Metrics
None
Use of Estimates Estimates are allowed if clearly identified.
Disallowed
Comparisons
No additional requirements beyond the requirements that results not be compared to other benchmarks.

Back to Contents

(Retired) SPECvirt_sc2010

RETIRED SPECvirt_sc2010 was retired on February 26, 2014 in favor of its successor, SPECvirt_sc2013.
  • All public use of results for this benchmarkmust plainly disclose that the benchmark has been retired, as described above.
  • No further submissions will be accepted for publication at www.spec.org
  • SPEC is no longer reviewing results for this benchmark.
  • Independent publication of new results is not allowed (because the benchmark results required review by SPEC before publication)
SPEC.org Submission
Requirements
Results must be reviewed and accepted by SPEC prior to public disclosure.
SPEC Metrics
  • SPECvirt_sc2010, or
  • SPECvirt_sc2010_PPW, or
  • SPECvirt_sc2010_ServerPPW
Required Metrics
  • SPECvirt_sc2010, or
  • SPECvirt_sc2010_PPW, or
  • SPECvirt_sc2010_ServerPPW

The required metric must be listed in close proximity to any other measured data from the disclosure or any derived value.

Conditionally
Required Metrics
Use of Estimates Not allowed
Disallowed
Comparisons

In addition to the requirements that results not be compared to other benchmarks:

  1. SPECvirt_sc2010 supports three categories of results listed below. Cross category comparisons are disallowed, however a licensee has the option of submitting results from the same test run to multiple categories.

    • Performance-only, which produces the SPECvirt_sc2010 metric,
    • Performance/Power of the Total System Under Test, which produces the SPECvirt_sc2010_PPW metric, and
    • Performance/Power of the Server only, which produces the SPECvirt_sc2010_ServerPPW metric
  2. SPECvirt_sc2010 uses modified versions of SPECweb2005, SPECjAppServer2004, and SPECmail2008 for its virtualized workloads, as these are established industry-standard workloads. These workloads have been modified to focus on stressing particular aspects of the SUT's resources (CPU, memory, network, disk) typical of server consolidation environments. As such, the modifications are significant enough that comparisons between the original benchmarks and the versions used in SPECvirt_sc2010 are not allowed.

Back to Contents

SPECvirt_sc2013

SPEC.org Submission
Requirements
Results must be reviewed and accepted by SPEC prior to public disclosure.
SPEC Metrics
  • SPECvirt_sc2013, or
  • SPECvirt_sc2013_PPW, or
  • SPECvirt_sc2013_ServerPPW
Required Metrics
  • SPECvirt_sc2013, or
  • SPECvirt_sc2013_PPW, or
  • SPECvirt_sc2013_ServerPPW

The required metric must be listed in close proximity to any other measured data from the disclosure or any derived value.

Conditionally
Required Metrics
Use of Estimates Not allowed
Disallowed
Comparisons

In addition to the requirements that results not be compared to other benchmarks:

  1. SPECvirt_sc2013 supports three categories of results listed below. Cross category comparisons are disallowed, however a licensee has the option of submitting results from the same test run to multiple categories.

    • Performance-only, which produces the SPECvirt_sc2013 metric,
    • Performance/Power of the Total System Under Test, which produces the SPECvirt_sc2013_PPW metric, and
    • Performance/Power of the Server only, which produces the SPECvirt_sc2013_ServerPPW metric
  2. SPECvirt_sc2013 uses modified versions of SPECweb2005, SPECjAppServer2004, SPECmail2008, and SPEC CINT2006 for its virtualized workloads, as these are established industry-standard workloads. These workloads have been modified to focus on stressing particular aspects of the SUT's resources (CPU, memory, network, disk) typical of server consolidation environments. As such, the modifications are significant enough that comparisons between the original benchmarks and the versions used in SPECvirt_sc2013 are not allowed.

Back to Contents

(Retired) SPECweb2005

RETIRED SPECweb2005 was retired on January 12, 2012.
  • All public use of results for this benchmark must plainly disclose that the benchmark has been retired, as described above.
  • No further submissions will be accepted for publication at www.spec.org.
  • SPEC is no longer reviewing results for this benchmark.
  • Rule-compliant results may be published independently, provided that the fact of retirement is plainly disclosed
SPEC.org Submission
Requirements
None. Submission to SPEC is encouraged, but is not required. Compliant results may be published independently.
SPEC Metrics
  • Overall metric: SPECweb2005
  • Individual workload metrics: SPECweb2005_Banking, SPECweb2005_Ecommerce, SPECweb2005_Support
Required Metrics None
Conditionally
Required Metrics
None
Use of Estimates Not allowed
Disallowed
Comparisons
No additional requirements beyond the requirements that results not be compared to other benchmarks.

Back to Contents

(Retired) SPECweb2009

RETIRED SPECweb209 was retired on January 12, 2012.
  • All public use of results for this benchmark must plainly disclose that the benchmark has been retired, as described above.
  • No further submissions will be accepted for publication at www.spec.org.
  • SPEC is no longer reviewing results for this benchmark.
  • Rule-compliant results generated in a test location which has previously produced an accepted compliant result may be published independently, provided that the fact of retirement is plainly disclosed. Independent publication of new results generated in test locations which have not met this requirement is not allowed
SPEC.org Submission
Requirements
By location, as defined below.
SPEC Metrics
  • Overall Metrics: SPECweb2009_JSP_Peak; SPECweb2009_PHP_Peak; SPECweb2009_ASPX_Peak; SPECweb2009_JSP_Power; SPECweb2009_PHP_Power; SPECweb2009_ASPX_Power
  • Individual workload metrics: SPECweb2009_JSP_Banking, SPECweb2009_PHP_Banking, SPECweb2009_ASPX_Banking, SPECweb2009_JSP_Ecommerce, SPECweb2009_PHP_Ecommerce, SPECweb2009_ASPX_Ecommerce, SPECweb2009_JSP_Support, SPECweb2009_PHP_Support, SPECweb2009_ASPX_Support
Required Metrics If any data from a full disclosure is used, then one of the following must also be included:
  • SPECweb2009_JSP_peak and SPECweb2009_JSP_Power; or
  • SPECweb2009_PHP_peak and SPECweb2009_PHP_Power; or
  • SPECweb2009_ASPX_peak and SPECweb2009_ASPX_Power
Conditionally
Required Metrics
Use of Estimates Not allowed
Disallowed
Comparisons

In addition to the requirements that results not be compared to other benchmarks:

Different scripting languages tend to have different workload characteristics and are based on technologies that are not directly comparable. Therefore, only comparisons between results using the same scripting language are allowed. For instance, comparison between results using JSP and the ones that use PHP or ASPX will not be allowed.

Back to Contents

SERT

SPEC.org Submission
Requirements
These requirements are defined by the energy efficiency agencies that specify the use of the SERT for their programs (e.g. the United States Environmental Protection Agency’s ENERGY STAR program). SPEC does not dictate these requirements.
SPEC Metrics

There is no single metric for this version of the SERT.

Required Metrics

SPEC does not require a metric to be published for the SERT information.

Agencies using the SERT for their energy efficiency programs may require specific information to be published.

Conditionally
Required Metrics
Use of Estimates Not allowed
Disallowed
Comparisons

In addition to the requirements that results not be compared to results from other benchmarks or tools:

  • Competitive comparisons that promote the use of one product over another and use numeric data generated by the SERT are expressly disallowed.
  • The only information provided by the SERT that can be used for marketing collateral is the qualification of a server configuration or server family for an energy efficiency program such as EPA ENERGY STAR.
  • The only information provided by the SERT that can be used for public comparison when removed from the context of the full Power and Performance Datasheet is the ENERGY STAR qualification of a server configuration or server family, or a similar qualification defined by another Agency. All other publicly available information is provided in the datasheet and references must be made to this document in its entirety.
  • If the tool is used for research to generate information outside of the ENERGY STAR program or similar programs, the information may not be compared to the results that are associated with an official energy efficiency program, such as ENERGY STAR, and competitive comparisons may not be made using the data generated.

Back to Contents

Chauffeur WDK

SPEC.org Submission
Requirements
Not applicable
SPEC Metrics

Information generated with worklets running under this harness may not be treated as a SPEC metric and are not endorsed by SPEC.

Required Metrics

Not applicable

Conditionally
Required Metrics
Not applicable
Use of Estimates Not applicable
Disallowed
Comparisons

In addition to the requirements that results not be compared to results from other benchmarks or tools:

  • Competitive comparisons that promote the use of one product over another and use numeric data generated by the Chauffeur WDK are expressly disallowed.

Back to Contents

III. Definitions

Basis for Comparison Information from a compliant result may be used to define a basis for comparing a subset of systems, including but not limited to memory size, number of CPU chips, operating system version, other software versions, or optimizations used. Other information, not derived from SPEC, may also be used to define a basis, for example, cost, size, cooling requirements, or other system characteristics. The basis must be clearly disclosed.
By Location

For benchmarks designated as having a submission requirement "By location", these requirements apply:

Each licensee test location (city, state/province and country) must measure and submit a single compliant result for review, and have that result accepted by the technically relevant subcommittee, before publicly disclosing or representing as compliant any result for the benchmark.

After acceptance of a compliant result from a test location, the licensee may publicly disclose future compliant results produced at that location without prior acceptance by the subcommittee.

The intent of this requirement is that the licensee test location demonstrates the ability to produce a compliant result.

Note that acceptance of a result for one SPEC benchmark does not relieve a licensee of the requirement to complete the procedure for any other SPEC benchmark(s) that also require initial acceptance by location.

Close Proximity In the same paragraph or an adjacent paragraph for written materials; or visible simultaneously for visual materials. The font must be legible to the intended audience.
Compliant Result

(i) The set of measurements, logs, full disclosure report pages, and other artifacts that are the output of a process that follows the run and reporting rules of a SPEC benchmark. Depending on the benchmark and its rules, the process may have many steps and many ingredients, such as specific software, hardware, tuning, documentation, availability of support, and timeliness of shipment. To find the rules for a specific benchmark, click its name in the tables above.

(ii) A number within such set that is labelled as a SPEC metric.

Note that benchmark reporting pages include other types of information, such as the amount of memory on the system. It is not allowed to represent such other information as a SPEC metric, although it may be used to define a Basis for Comparison.

SPEC reviews results prior to publication on its web site, but the accuracy and compliance of the submission remains the responsibility of the benchmark licensee. See the disclaimer.

Derived Value

A unit that is a numerical function of one or more SPEC Metrics, rather than the original metric. The function may be a constant divisor, to normalize performance to a comparison system of interest. The function may bring in quantities that are some other characteristic(s) of the system. Such other characteristics may include information from both SPEC result pages and from non-SPEC sources.

Examples: "SPECint_rate2006 per chip" (metric is divided by number of chips reported on SPEC disclosure)
"Cubic feet per SPECint_rate2006" (a non-SPEC quantity is divided by the metric)
"Normalized SPECsfs2008_cifs" (metric is divided by result for a comparison system)
"GamePerfMark", from the trademark section above.

This definition is intentionally broad, encompassing any function that includes a SPEC metric as one of the inputs.

Disallowed Comparisons As mentioned above, results of one benchmark may not be compared to a different benchmark, nor to a different major release of the same benchmark. Individual benchmarks may forbid other comparisons, typically where such comparisons are considered inherently misleading.
Estimate

An estimate is an alleged value for a SPEC metric that was not produced by a run rule compliant test measurement.

For purposes of this definition, it does not matter whether the alleged value for the metric was produced by extrapolating from known systems, or by cycle accurate simulation, or by whiteboard or dartboard, or by normal testing with the exception of a single missing mandatory requirement (e.g. the 3 month availability window). If the alleged value is not from a rule-compliant run, then it is an estimate.

The usage of estimates is limited.

Major Release

For purposes of this fair use rule, the term "major release" references a change in the year component of a benchmark product name, for example SPECjvm98 vs. SPECjvm2008.

Non-Compliant Number

A value for a SPEC metric that fails to meet all the conditions for a compliant result.

Usage Note: By the definition of Estimate, above, a non-compliant number is also an estimate; and, of course, an estimate does not comply with the run rules. Therefore, the terms are sometimes interchangeable. In practical usage, an estimate may bear no relationship to any measurement activity; whereas a non-compliant number is typically the product of running the SPEC-supplied tools in a manner that does not comply with the run rules. In such cases, the tools may print out numbers that are labelled with SPEC metric units, but the values that are printed are not compliant results. Such values are sometimes informally called "non-compliant results", but for the sake of clarity, this document prefers the term "non-compliant number".

Required Metric A SPEC metric whose value must be supplied. Individual benchmark sections above list whether they have required metrics. If so, then when any data is used from a full disclosure report, then the values for this/these metric(s) must also be used.
SPEC Metric

(i) A unit of measurement defined by a benchmark, such as response time or throughput for a defined set of operations. The available units for each benchmark are named in the tables above, and are defined within the benchmark run rules (which can be found by clicking the benchmark name in the tables above).

Example: SPECjvm2008 Peak ops/m.

(ii) A specific value measured by such a unit.

Example: 320.52 SPECjvm2008 Peak ops/m.

Usage Note: Both senses are used in this document, and it is expected that the sense is clear from context. For example, the prohibition against calling a derived value by a SPEC metric name is sense (i): do not define your own unit of measurement and then apply SPEC's trademarks to that unit. As another example, the rules for SPECpower_ssj2008 require disclosure of SPECpower_ssj2008 overall ssj_ops/watt, which is sense (ii): one is required to supply the value measured for a particular system.

A printed SPEC metric value is not necessarily a Compliant Result: SPEC provides tools that display values for SPEC metrics, such as the above example of "320.52 SPECjvm2008 Peak ops/m". Although SPEC's tools help to enforce benchmark run rules, they do not and cannot automatically enforce all rules. Prior to public use, the licensee remains responsible to ensure that all requirements for a compliant result are met. If the requirements are not met, then any printed values for the metrics are non-compliant numbers.




Back to Contents




IV. Violations Determination, Penalties, and Remedies

SPEC has a process for determining fair use violations and appropriate penalties and remedies that may be assessed.




Back to Contents