SPEC subject: SPEC Run Rules for CFP92 date: January 23, 1992 from: SPEC Steering Committee ABSTRACT This document provides guidelines for how to run the benchmarks, how to report the results (which is explained more fully in the document SPEC Reporting Rules) and how to report results for modified versions of benchmarks. It contains several examples of necessary types of changes. Memorandum to Performance Analysts SPEC subject: SPEC Run Rules for CFP92 date: January 23, 1992 from: SPEC Steering Committee 1. General The general philosophy behind this set of rules for running the SPEC benchmarks is to ensure repeatability and documentation of all factors pertinent to duplicating reported performance results. Suggestions for improving this run methodology should be made to SPEC for consideration in future releases. In general: - All times reported must be for runs that produce correct results. - Single user mode is allowed. This must be documented in the report. - These benchmarks have been run across a number of platforms with several compilers, but are not guaranteed to be free of language extensions that are commonly found, but not part of industry standard language specifications. - Use of software features (in pre-processors, compilers, etc.) which invoke, generate or use software designed specifically for any of the SPEC Benchmark Releases is not allowed. - Source code may not be modified unless it is required for portability. See Section 4 (Source Code Changes) for specifics. 2. Makefiles,_Compilers_And_Libraries Generic Makefiles are provided for convenience and insight into running the benchmarks. Typically a vendor supplied wrapper is used in conjunction with the generic Makefile. These are provided for convenience and can be replaced to facilitate running benchmarks on proprietary operating systems or to take advantage of optimizations not easily activated with the current method. It is anticipated that the build process which is capable of building the highest performance executable will be employed, even if a different - 2 - set of optimizations are activated for each benchmark. For example, one SPEC benchmark can be built with one set of options and another can be built with an entirely different set of options. Special libraries may be used as long as they 1. are products announced for availability to customers within 6 months of the date of test, 2. do not require changing benchmark source code, 3. implicitly replace routines in the benchmark source code, 4. are not "benchmark specific" (i.e.targeted specifically for a SPEC benchmark). Libraries can be specified in the Makefile. Source code changes to eliminate default high level language subroutines so that optimized library versions can be used is not allowed. Accuracy verification specifications (i.e. to diff or spiff) must not be overridden in Makefiles (or any other manner). Only SPEC approved alternative input files are allowed. Use of these alternate input files must be documented in the footnotes section. It should be recognized that each vendor is continuously improving its compilation systems and that new optimizations may be required to achieve published performance ratings. If you are unable to reproduce a vendor's published rating, please contact the vendor and ensure that you have the latest Makefiles and equivalent system configuration. 3. Pre-Processors - Use of a pre-processor on unmodified source code that is part of the identified released software is allowed. - Use of benchmark specific pre-processor features is not allowed. - Documented pre-processor options may be used, and should be specified either in the testing report or in the vendor Makefile (M.ident) provided on the SPEC tape. - 3 - 4. Source_Code_Changes SPEC policy allows only portability changes to the benchmark source code. When source code changes are made, a diff listing of the changes must be included in the testing report. SPEC encourages performance comparisons to be made across systems running the unmodified SPEC code, or if necessary, with the minimum required portability changes. - Portability o Source code changes required for standards compliance should be described in the testing report. Appropriate standards documents should be cited. All such changes should be reported to SPEC. SPEC may consider incorporating such changes in future releases. Whenever possible, SPEC will strive to develop and enhance benchmarks to be POSIX compliant. SPEC recognizes that some source code changes may be necessary in order to meet specific language implementations. Such changes must be documented, appropriate references must be cited and results marked as portability changes (denoted by * on any publicly reported results). Once these changes are incorporated into the tape including alternate source files, then the asterisk may be dropped. o The double precision (real*8) floating point in SPEC benchmarks is intended to be 64 bits. If this is not the case on the machine being tested, modify the appropriate declarations and forward to SPEC the source code changes required to accommodate the system. Over time, SPEC desires to use compiler options to eliminate the need for such changes. o If any portability change is backed off, one of three things must happen: i. benchmark code will not compile, or ii. benchmark code does not execute, or iii. benchmark code produces the wrong result. - 4 - 5. Alternate_Source_Files Some portability changes made to the source files may be generally useful to other SPEC users. These are classes of codes that meet the portability requirement (i.e., code won't compile, or won't execute, or produces wrong result), but are required by one or more target system. In such instances, the files are placed in a directory named "src.alt" in the benchmark directory. All files placed in this directory have been reviewed and approved by the SPEC Steering Committee; alternate source files are otherwise not permitted. When alternate source files have been approved for inclusion, however, the M.vendor makefile must be modified to retrieve the appropriate version. In general, original copies of codes which are to be replaced by alternate copies are preserved in the directory "src.std" for safe-keeping. When an alternate source file is used, it must be annotated in the footnotes section of the report. The asterisk (*) is not used to denote portability changes when alternate source files are employed. 6. Alternate_Results_Files On occasion, one platform will cause a variance in the results of the benchmark which may not necessarily be incorrect. In such cases, an alternate results directory, called "result.alt", contains additional valid results. Only results that have been reviewed and approved by the SPEC Steering Committee are placed in this directory. Otherwise, all results must compare with the files in the "result.ref" directory. In such cases, the M.vendor makefile must be modified to accommodate alternate results. If the use of alternate results are used, it must be specified in the footnotes section of the report.