Q1: What will SPEC be doing with regard to reported computer security vulnerabilities and their mitigation?
Q2: What actions has SPEC taken?
Q3: Why only the listed SPEC benchmarks?
Q4: What are these security vulnerabilities?
Q5: Why is SPEC only looking at these vulnerabilities?
Q6: I plan on running a SPEC benchmark; what should I do about the security issues?
Q7: I plan on publishing results measured with a SPEC benchmark, what should I do?
Q8: What will SPEC do with previously published results on the SPEC website?
Q9: How should I compare results in light of these patches?
Q10: If I want more information, what should I do?

Security Mitigations and SPEC benchmarks

SPEC has received questions asking how reported computer security vulnerabilities and their mitigation impact SPEC benchmarks and results. Below, in a question and answer format, is a summary of SPEC's current approach and recommendations.

Q1: What is SPEC doing with regard to reported computer security vulnerabilities and their mitigation?

As noted on www.spec.org, SPEC is a non-profit corporation formed to establish, maintain and endorse standardized benchmarks and tools to evaluate performance and energy efficiency. SPEC also reviews results as to their compliance with run rules, and, after review, publishes (on www.spec.org) rule compliant results from SPEC benchmark licensees. Note that SPEC does not perform any testing of its own nor does SPEC provide editorial comment/judgement on rule compliant results; thus in the context of security vulnerabilities and their mitigation, SPEC expects:

to continue to review results that are submitted and ensure that, when published, they are compliant with the rules for a given SPEC benchmark,
each organization within SPEC to evaluate security vulnerability reports and determine what course of action, if any, is required for their benchmarks.

SPEC is not expecting to provide a summary of performance impacts, nor evaluations of the functionality nor performance impacts of security mitigation provided by various vendors.

Q2: What actions has SPEC taken?

For certain benchmarks SPEC has required results published on its web site to provide information about mitigation status. The list of benchmark and the general format of this information is described in Q7 below.

Q3: Why only the listed SPEC benchmarks?

SPEC has chosen not to take action for any of the SPEC benchmarks that have been retired and for which SPEC is no longer accepting submission. For some active SPEC benchmarks, there are no requirements at this point in time.

Q4: What are these security vulnerabilities?

More information on the security vulnerabilities can be found at the Common Vulnerabilities and Exposures (CVE) website (https://cve.mitre.org) as well as websites like Wikipedia. SPEC is identifying them on the benchmark results disclosure by the CVE identifier (e.g. - CVE-2017-5754) and the colloquial name (e.g. - Meltdown).

Q5: Why is SPEC only looking at these vulnerabilities?

These are the security vulnerabilities that, in SPEC's view, were/are of most interest to the consumers of SPEC benchmark results.

Q6: I plan on running a SPEC benchmark; what should I do about the security issues?

If you are planning on running a SPEC benchmark, be aware of the rules for that particular SPEC benchmark. In particular, be aware that for rule compliant runs that the configuration being tested must be recommended for running a class of programs that includes programs other than the SPEC benchmarks. The configuration must also be available, documented and supported. Be aware of what your hardware or software vendors has communicated with regard to patches for these issues; some vendors may require patches to maintain support.

Q7: I plan on publishing results measured with a SPEC benchmark, what should I do?

As mentioned above, the published results need to be compliant with the SPEC rules at time of publication (and general availability requires availability for a certain period of time after publication). If publishing by submitting to SPEC, you will need to ensure that your run and your method of reporting conform to the benchmark license and rules. In light of the security issues mentioned, SPEC would emphasize the following things:

Ensure that the configuration is documented such that it can be reproduced (i.e. - include information about BIOS versions, patch version, etc.)
Ensure that the configuration tested is recommended for running a class of programs more than the SPEC benchmarks as well as being available, documented and supported
- This would include applying to your system all vendor required patches relevant to the particular SPEC benchmark.

Additionally, to enable readers of the SPEC website additional information to evaluate the results on the SPEC website, SPEC will require notes of the following general form in the "General Notes" section of the benchmark submission for the following benchmarks: ACCEL, Cloud IaaS 2016, CPU2017, JBB2015, jEnterprise2010, MPI2007, OMP2012, SFS2014, SPEC VIRT_SC 2013

Yes/No/NA: The test sponsor attests, as of date of publication, that CVE-2017-5754 (Meltdown) is mitigated in the system as tested and documented.
Yes/No/NA: The test sponsor attests, as of date of publication, that CVE-2017-5753 (Spectre variant 1) is mitigated in the system as tested and documented.
Yes/No/NA: The test sponsor attests, as of date of publication, that CVE-2017-5715 (Spectre variant 2) is mitigated in the system as tested and documented.

The specific format can be confirmed with SPEC prior/during the submission process. The above documentation requirements may change from time to time. SPEC may add to the above list, or may remove mitigations from the above list if they become ubiquitous. If you apply a patch, but disable it fully or partially, the answer to the corresponding question should be No, not Yes. Answering "No" to any of these questions does not make the result automatically unpublishable but the submitter should expect that SPEC will be asking for more information to verify that the system as tested is supported. Answering "NA" is an assertion that the system as tested is not subject to this security vulnerability. Note that SPEC may request further information in such cases; for example, pointers to public statements from the vendor that assert that the particular CVE is not relevant for that system. To facilitate publication, SPEC may require submitters to actively participate in the SPEC results review process. Some benchmarks allow publication outside the SPEC web site. For such results, the rules still apply and SPEC may ask licensees to provide clarification or additional information.

Q8: What will SPEC do with previously published results on the SPEC website?

SPEC is not planning to automatically add additional information to existing results. Nor will SPEC automatically indicate that historical results are non-compliant. SPEC believes there is a value in a historical database; SPEC does not automatically take any actions with regard to results as systems are retired by vendors or software is revised. If there is a belief that a result on the SPEC website is non-compliant, SPEC should be contacted with details about the results and the reasoning for the concern.

Q9: How should I compare results in light of these patches?

Any public comparisons that are made with SPEC benchmarks are expected to follow the SPEC Fair Use rules (https://www.spec.org/fairuse.html). These rules include a requirement about how comparisons should be made (https://www.spec.org/fairuse.html#Comparisons) which includes having a basis for the comparison made. SPEC is expecting that by requiring the additional information mentioned to be in public results people will be able to make informed comparisons.

Q10: If I want more information, what should I do?

If you have questions about the performance of a specific system in light of the security issues or patches, you should contact the hardware and/or software vendors. If there are questions about the SPEC submission and results process, please contact SPEC at: info@spec.org.

Standard Performance Evaluation Corporation