The OpenGL Performance Characterization Project Rules
Version 1.0
Last Updated 4/8/99

 I. Overview

  1. General Philosophy

    1. The OpenGL Performance Characterization Project of SPEC/GPC (henceforth abbreviated as OPC) believes the user community will benefit from an objective series of tests, which can serve as common reference and be considered as part of an evaluation process.

    2. The OPC seeks to develop benchmarks for generating accurate OpenGL performance measures in an open, accessible and well-publicized manner.

    3. The OPC wishes to contribute to the coherence of the field of OpenGL performance measurement and evaluation so that vendors will be better able to present well-defined performance measures; and customers will be better able to compare and evaluate vendors' products and environments.

    4. The OPC will provide formal beta software to members and final software releases to the public in a timely fashion.

    5. Hardware and software used to run the OPC benchmarks must provide a suitable environment for running typical OpenGL programs.

    6. OPC reserves the right to adapt its benchmarks as it deems necessary to preserve its goal of fair and useful benchmarking (e.g. remove benchmark, modify benchmark code or data, etc). If a change is made to the suite, OPC will notify the appropriate parties (i.e. OPC members and users of the benchmark) and OPC will re-designate the metrics (e.g. changing the metric from DRV-04 composite to DRV-05 composite). In the case that a benchmark is removed in whole or in part, OPC reserves the right to republish in summary form "adapted" results for previously published systems, converted to the new metric. In the case of other changes, such a republication may necessitate re-testing and may require support from the original test sponsor.

  2. Overview of Optimizations

    1. OPC is aware of the importance of optimizations in producing the best system performance. OPC is also aware that it is sometimes hard to draw an exact line between legitimate optimizations that happen to benefit OPC benchmarks and optimizations that specifically target OPC benchmarks. However, with the list below, OPC wants to increase awareness of implementers and end-users to issues of unwanted benchmark-specific optimizations that would be incompatible with OPC's goal of fair benchmarking.

    2. To ensure that results are relevant to end-users, OPC expects that the hardware and software implementations used for running OPC benchmarks adhere to a set of general rules for optimizations.

  3. General Rules for Optimization

    1. Optimizations must generate correct images for a class of programs, where the class of programs must be larger than a single OPC benchmark or OPC benchmark suite. Correct images are those deemed by the majority of the OPC electorate to be sufficiently adherent to the OpenGL specification for the targeted end-user community (e.g. users of OpenGL on PDAs would have lower quality expectations than those using high-end workstations).

    2. Optimizations must improve performance for a class of programs where the class of programs must be larger than a single OPC benchmark or OPC benchmark suite and applicable to at least one end user application.

    3. For any given optimization a system must generate correct images with and without said optimization. An optimization must not reduce system stability.

    4. The vendor encourages the implementation for general use (not just for running a single OPC benchmark or OPC benchmark suite).

    5. The implementation is generally available, documented and supported by the providing vendor.

    6. It is expected that vendors would endorse the general use of these optimizations by customers who seek to achieve good application performance.

    7. No pre-computed (e.g. driver cached) images, geometric data, or OpenGL state may be substituted within an OPC benchmark on the basis of detecting that said benchmark is running (e.g. pattern matching of command stream or recognition of benchmark's name).

    8. Every OpenGL implementation in both immediate and display list mode must fully process every GL element presented to it that will impact the frame buffer and GL state.

    9. Differences to the frame buffer between immediate and display list modes must not exceed 0.01% of the number of pixels in the window.

    10. In the case where it appears the guidelines in this document have not been followed, OPC may investigate such a claim and request that the optimization in question (e.g. one using OPC benchmark-specific pattern matching) be removed and the results resubmitted. Or, OPC may request that the vendor correct the deficiency (e.g. make the optimization more general purpose or correct problems with image generation) before submitting results based on the optimization.

[APC Project] [GPC Home] [MBC Project] [OPC Project] [PLB Project] [XPC Project]