SPEC CPU® v8 Benchmark Search Program

SPEC® holds to the principle that better benchmarks can be developed from actual applications. With this in mind, SPEC is once again seeking to encourage those outside of SPEC to assist us in locating applications that could be used in the next CPU-intensive benchmark suite, currently under development within SPEC and currently designated as the SPEC CPU® v8 benchmark.

The SPEC CPU v8 Benchmark Search Program is sponsored by the Standard Performance Evaluation Corporation (SPEC, a non-profit consortium; identified as "Sponsor" in this document). The program is designed to encourage and recognize achievements of members of the academic and industrial community who provide and develop application code and datasets that can be used as components of the next revision of the SPEC CPU benchmark suite.

For each step of the process that a submission passes, SPEC will compensate the Program Submitter (in recognition of the Submitter's effort and skill). A submission that passes all of the steps and is included in the next SPEC CPU benchmark suite will receive up to $9000 overall and a license for the new benchmark suite when released.

What is SPEC looking for?

Overall, the SPEC CPU benchmarks have provided performance metrics for comparing systems on a known compute-intensive workload, with emphasis on the system's processor(s), memory hierarchy, and compiler(s). SPEC is expecting the next SPEC CPU benchmark suite to continue this trend.

A good benchmark candidate is:

Used by real users.
Compute bound, or can have its compute bound portion excerpted.
Portable or can be ported to multiple hardware architectures and operating systems with reasonable effort.
State of the art for the given field.
Derived from a representative application.
Capable of solving problems of varying sizes. The SPEC CPU 2017 suite used up to 4 workloads, in various capacities, for its benchmarks.
Reasonably predictable as to its code path. For example, minor differences in floating point accuracy across platforms should not cause the application to do wildly different work.

SPEC's approach for the SPEC CPU 2017 suite may be found at https://www.spec.org/cpu2017/Docs/overview.html#AboutBenchmarks

Sample application areas

SPEC has held to the principle that better benchmarks are based on real applications, and these applications can come from any area of work. For example, the current SPEC CPU 2017 suite includes applications from the following areas:

Interpreters
Compilers
Route planning
XML to HTML conversion
Video compression
AI game theory
General data compression
Explosion modeling
Physics: relativity
Molecular dynamics
Ray tracing

Fluid dynamics
Weather forecasting
Discrete Event simulation - computer network
Biomedical imaging optical tomography with finite elements
3D rendering and animation
Atmosphere modeling
Wide-scale ocean modeling (climate level)
Image manipulation
Computational Electromagnetics
Regional ocean modeling

SPEC is not limiting consideration to just these areas. A proposed application may come from any area or field of work. It is also acceptable for parallel/threaded codes or applications to be submitted.

Languages

Traditionally, the SPEC CPU suites have used C, C++, and Fortran.
Submissions using these languages should be compatible with compilers for C18 (aka "C17" or "ISO/IEC 9899:2018"), C++17, and Fortran 2008.
Submissions using other languages may be considered at SPEC's discretion if the language is commonly available across multiple platforms.
Note that Java applications traditionally are covered by the SPEC Java benchmarks, and would therefore be unlikely to be accepted into a SPEC CPU suite.

The SPEC CPU v8 Benchmark Search Program Official Rules

No purchase is necessary to participate.
In all cases, the burden of proof is on the Submitters.
SPEC will evaluate the submitted benchmarks for inclusion in the suite.
All evaluation and awards are within the sole discretion of SPEC.

Who May Submit Benchmarks?

*VOID WHERE PROHIBITED BY LAW.*
Not open to any employees or contractors of any of the SPEC member companies and their immediate families.
Entries may be submitted on an individual entrant basis (defined as a natural person), or on a group or team basis, in which case the registration and submission will identify a leader of such group or team.
Corporate entities are not eligible.
The award program is sponsored by Standard Performance Evaluation Corporation, 7001 Heritage Village Plaza, Suite 225, Gainesville, VA 20155, U.S.A.

Participation Constitutes Agreement to Official Rules

Participation in the Search Program constitutes Submitter's full and unconditional agreement to and acceptance of these Official Rules.

Timing

Submissions for the first step in the search program will be accepted by SPEC beginning 25-Feb-2020 and ending 13-Mar-2023 at 11:59 pm Pacific Time (America/Los_Angeles).
Status updates through the rest of the process will be provided to the Submitters at least every three months until the release of the next SPEC CPU suite.

How To Enter

Step 1: Complete the legal paperwork

Submit a Search Program Proposal to SPEC through the entry form found at: https://www.spec.org/cpu/cpuv8/entry_form.html including:

Submitter's name and contact information (email address and shipping address),
Name of the application,
Programming(s) language used and approximate lines of code,
A description of what the application does and how it is used, and
A statement declaring that the Submitter has the authority to give SPEC the right to distribute the application as part of a SPEC benchmark suite.

Upon receipt of the proposal, SPEC will acknowledge its receipt within seven (7) working days. Within four (4) weeks, SPEC will notify the Submitter as to whether or not SPEC wishes to further evaluate the entry.

If SPEC wishes to further evaluate the entry, the Submitter must provide a signed "Permission to Use" form for the submitted code, data sets, and any other material provided to SPEC. Additionally, any documents provided by the SPEC office for tax purposes need to be completed and returned to SPEC.

Once the Permission To Use form has been received, and signed by both the Submitter and SPEC, technical work may begin.

Step 2: Initial porting and workload ($1500 upon successful completion)

If accepted for entry into Step 2, SPEC will provide the Submitter with:

A copy of the SPEC CPU benchmark framework
Information on how to prepare the application
A SPEC Primary Contact.

With the assistance of the Primary Contact, the Submitter must cause the application to:

2a. Work within SPEC's environment

Build and run with the SPEC CPU toolset. SPEC is expecting to use the tool set developed for the SPEC CPU 2017 suite which is described at https://www.spec.org/cpu2017/Docs/runcpu.html
- Note that SPEC's approach to porting benchmarks does not allow the use of an automatic configuration process such as GNU autoconf or Metaconfig for each target platform.
- In some cases, this may require a significant effort to re-work the benchmark.
- The use of -D compilation switches for portability is allowed, where needed.
Exclude or disable platform dependent code (such as gcc-specific extensions and GNU-specific header files) unless needed for portability.
For C/C++ applications:
- Return an exit code of 0 if it runs successfully, or non-zero otherwise.
- For the main routine, take one of these two forms:
```
int main(void); 
            or 
int main(int argc, char * argv[]); 
```

2b. Compute for at least one minute

Include an initial workload (input data set) that causes the application to run for at least one minute when:
- compiled with basic optimization (for example: gcc -O) and
- tested on a system with a single-copy SPECrate®2017_int_base rating of about 5 or a single-copy SPECrate®2017_fp_base rating of about 5.
Demonstrate at least 95% compute-bound behavior in the above test (for example, using the Unix time command, observe that user+sys > .95 * real)
Notes:
- The SPEC Primary Contact will arrange testing on such a system.
- If it is not practical to perform this test on SPEC's system (perhaps due to early porting limitations), SPEC may at its option accept evidence of initial workload testing elsewhere, and may scale results as needed (for example, instead of >1 minute on a system rated at ~5, SPEC may accept >2 minutes on a system rated ~2.5).
- Step 3 requires several workload sizes, and has more specific requirements than the Step 2 workload.

2c. Define valid answers

Include a set of expected outputs that demonstrate correct operation with the supplied input data set.
Define how to validate that application outputs match the expected outputs. SPEC's tools can be used to:
- check for exact answers, or
- check for answers within a defined tolerance, or
- check for images that are similar.
For information on validation capabilities in the current SPEC CPU 2017 benchmark, see https://www.spec.org/cpu2017/Docs/utility.html#specdiff and https://www.spec.org/cpu2017/Docs/utility.html#imagevalidate

2d. Generate valid answers on both Windows® and Unix® operating systems

When compiled with basic optimization (for example: gcc -O), generate outputs that meet the above validation criteria using:

At least one Unix platform (such as AIX, BSD, HP-UX, Linux, macOS, or Solaris)
At least one Windows platform
Note: For Step 2, it is not required that all tested platforms produce valid answers. It is expected that some porting work will continue during later steps.

2e. Include documentation

Provide a completed SPEC Benchmark Description file -- similar to those available for the current SPEC CPU 2017 suite at https://www.spec.org/cpu2017/Docs/#benchmarks -- covering topics such as:

Author(s) contact information, including an email address that the author is willing to publish
Inputs - including: How can workloads be created?
Outputs

2f. Verify and fully describe the licensing

Audit all material supplied to SPEC to ensure that license requirements are met for all included components.
In the documentation:
- Reference specific versions of all licenses.
- Provide full detail, similar to the information for the SPEC CPU 2017 suite https://www.spec.org/cpu2017/Docs/licenses.html#benchmarks
- For open source applications, include the original (unmodified) sources prior to any changes that were done in order to adapt the application for SPEC.

2g. Step 2 Award

If, in SPEC's sole judgment, the submission does not meet these criteria, the Submitter will be notified as to what expectations were not met.

If the submission meets these criteria, the Submitter is awarded $1500.

2h. Movement to the next step

After the submitter has completed the requirements for this step, SPEC will then decide whether to move the benchmark into the next step. Note that SPEC, in its sole judgment, may decide at any time to discontinue work on a submission. It is possible that a submission may receive a monetary award for completing a step and nevertheless not proceed to the next step.

Step 3: Provide workloads and demonstrate profile ($1500 or $2500 upon successful completion)

During Step 3, the Submitter must provide 3 workloads, or, optionally, 4 workloads:

a test workload
a train workload
a SPECrate reference workload, or a SPECspeed reference workload, or both.

Each workload must include:

Input data
Output data that can be used to validate correct operation.
Validation criteria. When validating output, SPEC's tools can be used to check for exact answers, or for answers within a defined tolerance, or for images that are similar. The capabilities for the SPEC CPU 2017 suite are described at https://www.spec.org/cpu2017/Docs/utility.html#specdiff and https://www.spec.org/cpu2017/Docs/utility.html#imagevalidate

When compiled with moderately advanced optimization (such as gcc -O2 -fnoinline -fno-builtin), the benchmark must produce valid answers for each workload on at least one Unix system and at least one Windows system.

3a. Test workload

A short test (perhaps 10 or 20 seconds) that confirms that the compiled application is functional.

3b. Train workload

A proposed medium training input.
This input should run for about 10-25% of the runtime of a reference workload and reflect a typical usage (in the Submitter's opinion) of the submitted application.
This workload will be considered as input for use with compilers that do feedback directed optimization.

3c. SPECrate® reference workload (Integer or Floating Point)

A proposed SPECrate reference workload must:

Require no more than 1.8 GiB as measured by SPEC using ps -o vsz
Require about 3 to 5 minutes run time on a system with a single-copy SPECrate®2017_int_base rating of approximately 6 (for integer candidates) or a single-copy SPECrate®2017_fp_base rating of approximately 10 (for floating point candidates).

Testing of workload timing: Workload timing will be evaluated by SPEC. For Step 3, SPEC will use an x86_64 Linux system; one chip; one hardware thread per core; and advanced compiler optimization -- for example, but not limited to, gcc -m64 -Ofast.
SPECrate workloads will be evaluated using runcpu --copies=1.
SPECspeed workloads will be evaluated using runcpu --threads=40.

3d. SPECspeed® Floating Point reference workload

A proposed SPECspeed Floating Point reference workload must:

Use multiple hardware threads for parallel processing via one of these methods:
- OpenMP. It is recommended that benchmark candidates use traditional features such as those in OpenMP v3.1. If newer features are present in a candidate, they will be evaluated on a case-by-case basis.
- Native language parallelism (such as C++17 Parallel Standard Template Library (PSTL) or Fortran 2008 DO CONCURRENT)
Require no more than 60 GiB of main memory as measured by SPEC using ps -o vsz
Require about 5 to 10 minutes run time on a system with a SPECspeed®2017_fp_base rating of approximately 120 (1 chip / 40 cores / 40 hardware threads)

3e. SPECspeed® Integer reference workload

A proposed SPECspeed Integer reference workload must:

Require no more than 60 GiB of main memory as measured by SPEC using ps -o vsz
Require about 5 to 10 minutes run time on a system with a SPECspeed®2017_int_base rating of approximately 9 (1 chip / 40 cores / 40 hardware threads).

Note: it is encouraged, but not required, to use multiple hardware threads for parallel processing with a SPECspeed Integer reference workload via one of the same methods as described just above for SPECspeed Floating Point reference workloads.

3f. Profile

SPEC will profile the benchmark with the supplied reference workload(s), and expects that:

At least 95% of the time is compute bound on both Windows and Unix. This requirement will not be waived.
At least 95% of the execution time is spent in the supplied code.
- SPEC may consider granting waivers for this requirement where it can be demonstrated that library code - such as exp, or malloc - is typical of the application area.
- Such waivers will be solely at SPEC's discretion.
The executed instruction mix is either:
- Less than 1% floating point instructions, in which case it will be considered an integer benchmark, or
- 10% or more floating point instructions, in which case it will be considered a floating point benchmark
Notes:
- If a benchmark falls in between 1% and 10%, the submitter will be asked whether modifications can be made so that the benchmark would be more clearly integer or floating point.
- It is common for profiles to vary depending on optimization level.
- SPEC's profiling will be conducted using moderately advanced optimization, with inlining disabled so that individual functions may be observed (for example: gcc -O2 -fno-inline -fno-builtin)

3g. Step 3 awards

On completion of this step:

$1500 will be awarded if a single reference workload is provided (a SPECrate workload or a SPECspeed workload).
For submissions that include both a SPECrate reference workload and SPECspeed reference workload:
- $2500 will be awarded if, in the sole judgment of SPEC, the two workloads differ from each other in a substantive, non-trivial fashion. The criteria for SPEC's judgment will be whether the SPECspeed workload:
  - Uses parallelism to achieve greater speed, and/or
  - Effectively uses larger memory, and/or
  - Applies different portions of the benchmark source code.
- $1500 will be awarded if, in the sole judgment of SPEC, the two workloads are not substantially different.

3h. Movement to the next step

Step 4: Complete Code Testing ($1500 or $2000 upon successful completion)

During Step 4, the submitter must complete code testing, adjusting the benchmark as needed.

4a. Respond to a wide variety of testing

Demonstrate to SPEC's satisfaction that the submission runs on a variety of additional systems. These may include:
- Unix, Linux, macOS, and Windows systems
- Systems with varying memory models (e.g. "LP64", "P64", and so forth)
- Various hardware, potentially including both little endian and big endian systems
- Varying thread counts if a SPECspeed workload is provided
- Varying copy counts if a SPECrate workload is provided
Respond to failures that arise when compilers employ more advanced optimization techniques.
- It is typical to discover that greater optimization exposes problems that are not seen with lesser optimization.
- The submitter will need to assist with analysis to determine whether failures are due to the benchmark, or due to the specific platform (compiler, operating system, or hardware).
- If SPEC judges that the benchmark is at fault, the submitter is responsible to develop fixes to repair it.

4b. Improve standards compliance

Resolve standards problems to SPEC's satisfaction, when SPEC compiles the application using the language versions mentioned above (for example, gcc -std=c18 -pedantic)
SPEC always prefers to use code that conforms to the relevant language standards.
It is understood that in some cases this may not be practical (e.g. for entrenched user practices that are, in fact, commonly accepted by compilers.)

4c. Complete remaining documentation

Complete any remaining documentation required for the SPEC infrastructure, such as the benchmark flags file. (For information on how flags files were done for the SPEC CPU 2017 suite, see https://www.spec.org/cpu2017/Docs/flag-description.html)
Ensure that all remaining portability problems are documented and/or fixed.

4d. Adjust if required

During testing, SPEC may determine that adjustments are required (for example, if advanced optimization causes a benchmark to perform substantially different work than intended).
SPEC reserves the right to require adjustments to both source code and workloads during Step 4.

4e. Step 4 awards

On completion of the above requirements:

$1500 will be awarded if a single reference workload is provided (a SPECrate reference workload or a SPECspeed reference workload)
If both SPECrate and SPECspeed workloads are provided, then:
- $2000 will be awarded if the two workloads differ from each other in a substantive, non-trivial fashion.
- $1500 will be awarded if the two workloads are not substantially different.
- The judgment of whether the workloads differ will be made solely by SPEC, using the criteria listed earlier in this document.
- Note: SPEC may make a different determination at the conclusion of Step 4 than it made at the conclusion of Step 3 (for example, if parallelism is newly added or newly deleted).
  - If it is newly determined that the workloads differ, the submitter shall receive an adjustment for their Step 3 award, that is, an additional $1000 for Step 3, along with the Step 4 award of $2000.
  - If it is newly determined that the workloads do not differ, the submitter is not required to repay any portion of their Step 3 award, and will receive $1500 for Step 4.

4f. Movement to the next step

Step 5: Recommendation to the Open System Group (no monetary award)

If the application is accepted by the SPEC CPU Subcommittee and meets all criteria, including those stated in Steps 1 to 4, the CPU Subcommittee will consider whether to recommend the benchmark to the SPEC Open Systems Group for inclusion in the new suite.

If the CPU subcommittee recommends the application for inclusion in the new suite, the submission moves to Step 6. (No monetary award is associated with this step.)

Otherwise, the SPEC CPU Subcommittee will notify the Submitter that the submission was removed from consideration.

Step 6: Acceptance into the SPEC CPU Suite ($3000 if accepted)

If the application is recommended to and is accepted by the Open Systems Group, in its sole discretion, then the application is included in the suite and the Submitter will receive $3000 and a license for the suite when it is released.

Number of Entries and Return of Entries

Submitters may enter as many times as they wish. However, each entry must be accompanied by a separate registration. Sponsor will not return any entries to the Submitters.

Personal Data

Sponsor will use its best effort to use personal data about Submitter only for the administration of the award program, in accordance with these Official Rules and for internal marketing purposes. Submitter agrees to such use of the personal data.

Note that the benchmark documentation described above does expect that the submitter will provide an email address that can be published.

Important Terms

Sponsor is not responsible for lost, late, incomplete, illegible or misdirected mail or e-mail, for failed, partial or garbled computer transmissions, or for technical failures of any kind. Sponsor reserves the right to cancel or modify the awards if fraud, technical failures or other events or circumstances impair the integrity of the award program as determined by Sponsor in its sole discretion.

Submitter agrees that Sponsor's only obligations are to submit entries from eligible submissions to the SPEC CPU Subcommittee, according to the procedures and criteria set forth in these Official Rules, and to award the compensation set forth herein, to Submitters as criteria in SPEC's sole judgment are met, subject to the terms and conditions herein.

Submitter will be held harmless from any liability incurred from the intended or unintended use of the application as a benchmark or application.

Submitter agrees that Sponsor's obligations are solely to fairly and adequately consider any entry submitted, and that Submitter is not entitled to and shall not seek any further compensation.

Submitter hereby grants Sponsor a perpetual, non-exclusive license to copy, modify, display, distribute, and sub-license the entry without geographical limitations or further compensation to Submitter of any kind, provided that Sponsor's sole use of the entry will be for the purposes of inclusion of the entry in Sponsor's CPU benchmark suite. Submitter agrees that e-mail shall satisfy any writing requirement, which may apply to intellectual property licenses. Upon request of Sponsor, Submitter agrees to obtain written consent from the owner of the copyright in the application, if that person is not Submitter, and to execute any documents required to effectuate the terms of these Official Rules.

Submitter warrants that it has the right to submit this application and information to SPEC and to grant the license rights granted herein, and that to Submitter's knowledge use of the application will not infringe or violate any intellectual property rights (including but not limited to patent, trademark, trade secret, and copyright) or license rights of any third party. Submitter agrees to notify SPEC of any claim of such infringement or violation within fifteen (15) days of its receipt of the claim. Submitter shall defend, indemnify and hold harmless SPEC, its successors, assigns and licensees for all costs (including attorneys' fees) arising out of any breach of these warranties and representations.

All issues and questions concerning the construction, validity, interpretation and enforceability of these Official Rules, or the rights and obligations of Submitter and Sponsor in connection with the Award Program, shall be governed by, and construed in accordance with, the laws of the Commonwealth of Virginia, U.S.A., without giving effect to any choice of law or conflict of law rules or provisions (whether of the Commonwealth of Virginia, U.S.A., or any other jurisdiction), which would cause the application of the laws of any jurisdiction other than the Commonwealth of Virginia, U.S.A.

Taxes, customs duties, fees, freight charges and other related charges, on prizes are the sole responsibility of the winners. Winners may waive the right to receive compensation. Sponsor will reimburse winners up to U.S. $1,000 for payments made for customs duties, fees, freight charges (but not taxes, including taxes on income, goods and services, and value added taxes), upon presentation by submittal of original and official receipts for such payments.

Submittal Notification

Submitters will be notified by electronic means of each successful completion of each step and may be required to complete an affidavit of eligibility and publicity release (except where prohibited by law), which must be returned within 20 days of receipt by Submitter. Failure to sign and return the affidavit or release or to comply with any term or condition in these Official Rules may result in disqualification and forfeiture of compensation.

This Program is subject to all applicable laws. Void where prohibited.

All questions about the search program can be directed to SPEC at info@spec.org.

Updates to this document

Sep-2022: Extend the due date for program entries.

Sep-2021: This document was updated to reference the C18 (aka C17) standard instead of C11, on the grounds that C18 provides only minor technical corrections to C11.

Feb-2022: Extend the due date for program entries. Update the sections on workload timing (SPECrate, SPECspeed floating point, SPECspeed integer), including a better explanation of SPEC's test methods. Clarify the recommendations for use of OpenMP features.

Jun-2022: Fix example in step 4b to use one of the required language versions.