Skip navigation

Standard Performance Evaluation Corporation

Facebook logo LinkedIn logo Twitter logo Google+ logo
 
 

SPEC High Performance Group
Policies & Procedures

Updated: August 3, 2011, Version 3.1.1
Approved: August 3, 2011

1.0 General Policies and Principles

The intent of this section is to provide an overview of the general policies and principles of the SPEC organization, as context for the policies of the High Performance Group (HPG). The controlling documents for governance of SPEC are the Articles of Incorporation and Bylaws, filed with the State of California, and available from the SPEC Administrator.

1.1 SPEC's Purpose

The primary purpose of the Standard Performance Evaluation Corporation (SPEC) is to develop suites of benchmark programs that are effective and fair in comparing the performance of high performance computing systems, and to assure that they are readily available to manufacturers and users of such systems.

1.2 SPEC's Formation

SPEC was formed from the instigation and sponsorship of Electronic Engineering Times (E.E. Times), and by the cooperative development work of Hewlett-Packard Co., Sun Microsystems Inc., Apollo Computer Inc., and MIPS Computer Systems Inc.

1.3 The Corporation

Upon approval by the representatives of the founding companies, their effort to develop a benchmark standardizing activity was converted into a non-profit Corporation of the state of California, on November 14, 1988.

The Corporation has a Board of Directors, President, and a staff to carry out the business of SPEC. The board has established three technical committees:

  1. The Graphics and Workstation Performance Group (GWPG)
  2. The High Performance Group (HPG) and
  3. The Open Systems Group (OSG)

Each group has a steering committee to manage and supervise the development of applicable benchmarks. Each steering committee may sponsor a number of subcommittees or working groups to facilitate benchmark development.

The Board may establish additional technical committees.

1.4 Meetings

The annual meeting of SPEC members will be held in January.

Election of the Board of Directors and members of the steering committee(s) and a financial report by the Board will be a necessary part of that meeting. The Board of Directors will meet at least quarterly.

1.5 The Board

To facilitate a fair and balanced representation on the Board, SPEC's policy is that the Board of Directors has staggered two-year terms, with half of the Board elected each year. Decisions of the board require agreement of the majority. Elections to the Board will be held at the annual meeting to be held at the beginning of the calendar year. Board positions are held by individuals, not by companies. Currently there are eight elected directors on the board.

Each member of each SPEC technical steering committee (OSG, GPC, and HPG) may cast one vote per open board seat. E.g., if a company is a member of two steering committees, it may cast two votes per open board seat.

1.5.1 Officers

The officers of the corporation shall be a President, zero or more Vice Presidents as may be determined by the Board, a Secretary, a Treasurer and such other Assistant Secretaries and/or Assistant Treasurers as may be deemed necessary. Officers shall be nominated and elected by the Board of Directors at the Board's annual meeting and shall hold office for one (1) year or until their successors shall have been duly elected and qualified.

1.5.2 Administrative Duties

SPEC has an administrative organization that carries out the publicity, accounting, organizational and other necessary work of the Corporation. The administrative staff reports to the Board, through the Vice President. The budget of the staff is developed by the Vice President and the Treasurer and approved by the Board.

1.6 Membership

Any company or organization that wishes to join SPEC may join provided they pay the initiation fee and annual dues as determined by the Board of Directors.

All members are eligible to stand for election to the steering committee seats, to nominate candidates for offices and the Board of Directors, and to champion proposed benchmarks. All members have the opportunity to review and comment on all benchmarks developed by HPG. This can be done through active participation in the subcommittee developing the suite or during the benchmark's final call for comments.

The fees and dues for membership must not discriminate among prospective members.

1.7 Copyrights and Trademarks

The standards and documents authorized by the Corporation may be copyrighted or trademarked, as appropriate.

The purpose of the copyrights and trademarks is to assure the computer industry that programs and documents developed by SPEC are genuine. The intent of protecting by trademarks and copyrights is not to restrict the use or availability or make any significant revenue from them, but rather to assure their general availability, promote confidence in them, and prevent fraud.

1.8 Openness

Information about the activities and developments of SPEC are intended to be readily available to all parties with legitimate interests in the performance of advanced computer systems. The board is responsible to carry out this policy.

SPEC meetings must be open to representatives of each member company. The Board may invite interested parties who are not SPEC members, but who have a legitimate interest in its work, to the meetings.

Meetings of SPEC's technical steering committees and their subcommittees are generally closed to the press. A meeting may be opened to the press by agreement of the steering committee or by agreement of the Board and the meeting's chair.

The annual meeting of the SPEC membership must be open to any interested party with a legitimate interest in the work of the Corporation, including the press.

 

2.0 High Performance Group

The High Performance Group (HPG) focuses on the development of benchmarks for high performance computers, including but not limited to, symmetric multiprocessor systems, clusters, distributed memory parallel systems, and vector parallel supercomputers.

2.1 High Performance Group's Formation

SPEC's High Performance Group was formed in 1994, with the cooperation of Convex Computer Corp., Cray Research Inc., Dartmouth College, Digital Equipment Corp., Fujitsu America, Hewlett-Packard, IBM, Intel Scalable Systems Division, Kuck & Associates, NEC/HNSX Supercomputers, Silicon Graphics Inc., the University of Illinois, the University of Michigan and the University of Minnesota.

2.2 High Performance Group and SPEC

The High Performance Group (HPG) is chartered to manage and supervise the technical development of the SPEC/HPG benchmarks suites. In case of any conflict between provisions within this Charter and Resolutions enacted by HPG, the Resolutions shall prevail. In case of any conflict between HPG's Resolutions and SPEC's Articles of Incorporation and Bylaws, the Articles of Incorporation and Bylaws shall prevail.

2.3 High Performance Group Steering Committee

The voting membership of the High Performance Group constitutes the High Performance Steering Committee (HPSC). HPSC is formally charged with managing and supervising the development of SPEC/HPG benchmarks.

2.4 Participation Levels

Terminology note about SPEC "Membership": SPEC is a California nonprofit mutual benefit corporation. Governing law has certain formal terminology regarding membership (see, for example, California Corporations Code sections 5056 and 7331). SPEC's Bylaws authorize only one class of membership in the corporation. It is understood that one sometimes sees the term "member" used in a looser sense, and indeed the law appears to recognize that the term "member" may be used in a less formal sense (7333). However, this section 1.6 follows the formal usage of the Bylaws, which clearly say that SPEC has only one class of "Membership".

Any company or organization interested in the technical development of HPG benchmarks is welcome to participate and must be willing to abide by SPEC's Bylaws, the Fair Use Guidelines, the Guidelines for Handling SPEC-confidential Material, HPG's Policies and Procedures and pay the initiation fee and annual dues determined by the Board of Directors.

HPG provides for three types of participation: Sustaining Membership, Not-for-profit Associates and Supporting Contributors. Universities and non-profit institutions are eligible to become sustaining members, but at their option may prefer to join as "Non-profit Associates", as described in Section 2.4.2.1.

2.4.1 Sustaining Memberships

Sustaining members are dues-paying members and they accrue all the rights, privileges and responsibilities that full membership entails. Sustaining memberships are available to computing-related commercial and non-commercial entities as specified in the Bylaws . HPG's fees and dues for membership can not discriminate among prospective members.

Sustaining members may become active participants in any working group or subcommittee of HPG.

All sustaining members are eligible to nominate candidates for offices and the Board of Directors, and to champion proposed benchmarks. All members have the opportunity to review and comment on all benchmarks developed by HPG, either through active participation in the subcommittee developing the suite or during the benchmark's final call for comments.

Sustaining members may become active participants in any working group or subcommittee of HPG.

Sustaining members are eligible to stand for election to the steering committee seats, to nominate candidates for offices and the Board of Directors, and to champion proposed benchmarks.

Sustaining members have the opportunity to review and comment on all benchmarks developed by HPG. This can be done through active participation in the subcommittee developing the suite or during the benchmark's final call for comments.

2.4.2 Non-Membership Roles

SPEC does make an effort to accommodate the participation of academic institutions and other non-profit organizations that might be unable to meet the full requirements for Sustaining Membership. These accommodations are made by special arrangement of HPG and can include reduced (or waived) dues and other arrangements. These accommodations generally fall into two categories:

  • SPEC Associates, those which pay a reduced fee and received full benefits, and
  • SPEC Supporting Contributor, whose contributions and access is restricted to the one specified benchmark development process.

2.4.2.1 Non-profit Associates

To encourage participation from universities and other non-profit institutions interested in its work, SPEC maintains a SPEC Associate role that enables non-profit organizations to share in the SPEC process, but not to participate in general member voting.

Associates must be willing to abide by SPEC's Bylaws, the Fair Use Guidelines, the Guidelines for Handling SPEC-confidential Material, HPG's Policies and Procedures and pay (potentially reduced) initiation fees and annual dues as determined by the Board of Directors.

Non-profit Associates have the opportunity to review and comment on all benchmarks developed by the HPG. This can be done through active participation in the subcommittee developing the suite or during the benchmark's final call for comments.

2.4.2.2 Supporting Contributor

The role of a Supporting Contributor is available to a commercial organization, to an academic institution, or to an individual by invitation only. A subcommittee may issue an invitation to an interested party to apply for this role. The application must be approved by the High Performance Group and the Board.

Supporting contributors may participate in only one subcommittee and have no voting rights and pay no dues. Supporting contributors may help develop new benchmark suites, provide expertise to HPG or help provide other support within HPG.

2.4.4 Table of Membership Rights and Privileges

  SUSTAINING MEMBER ASSOCIATE SUPPORTING CONTRIBUTOR
DUES FULL DISCOUNT NO
GENERAL MEMBERSHIP VOTING YES NO NO
PARTICIPATE IN FACE-TO-FACE MEETING, CONFERENCE CALLS, BENCHMARK DEVELOPMENT YES YES YES
GET WORK IN PROGRESS YES YES YES - if providing support for code;
otherwise NO.
ELIGIBLE TO ELECT HPG OFFICERS YES YES NO
ELIGIBLE TO VOTE FOR BOARD MEMBERS YES NO NO
ELIGIBLE TO PARTICIPATE IN ANY SUBCOMMITTEE ANY/ALL ANY/ALL ANY/ALL IF SUPPORT IS PROVIDED
COPIES OF HPG SUITES ALL ALL ONLY WHERE SUPPORT PROVIDED
SPEC/HPG EMAIL ALIAS ANY/ALL ANY/ALL 1
ACCESS TO SPEC/HPG SERVERS YES YES LIMITED - Controlled by Chair to selected pages only.
RESULT PUBLICATION ON WWW.SPEC.ORG UNLIMITED UNLIMITED Dependent on new suite support; otherwise, non-member price set by HPG.
TERM LIMIT Annual Renewal Annual Renewal ONE YR., RENEWABLE.
HPG may terminate at any time.
RESULTS REVIEW YES YES May participate in the results review session for which their results were submitted, but they have no vote.
ELIGIBLE FOR HPG OFFICE YES YES YES

2.5 HPG Officers, Elections, Voting, & Meetings

The HPG officers include a Chairperson, Vice Chairperson, Secretary and Release Manager(s) elected from the membership. Elections for officers will be held at the beginning of each calendar year to be set by the Chair. The HPG officers serve for one year and are eligible for re-election. If any elected office becomes vacant for any reason, HPG shall duly elect a replacement to serve the remaining term.

Each HPG Sustaining Member shall have one vote in all matters. Groups of Five Associates may combine to cast one vote; the single vote cast by each group of Associates is equal to a single Sustaining Member vote. When there are fewer than five Associates available to form a combinatorial group, those Associates present will be deemed a sufficient combinatorial group to cast one vote. Supporting Contributors are not entitled to voting rights.

To conduct business there must be a quorum. The group's quorum is a simple majority of the voting membership. All HPG votes shall be open and public votes, except the election of HPG officers, which is by secret ballot. Exceptions to the secret ballot may be made if there is only one candidate per office. A voting member may vote 'yes', 'no', 'abstain', or 'pass' when the vote is first called. If the member votes 'pass', the chair will return to that member to finalize their vote as either 'yes', no', or 'abstain' after the remaining members have voted. The motion will pass if the number of 'yes' votes exceeds the number of 'no' votes. Votes of 'abstain', count only toward establishing quorum. It is the responsibility of the chair to make sure that member's reasons for voting 'no' or 'abstain' are recorded in the minutes. Absent Members will be marked as absent, unless proxies have been received.

A voting member not attending two consecutive duly noticed meetings, including face to face meetings and conference calls, will have their voting rights suspended. The suspension of voting rights will take place at the beginning of the next scheduled meeting. Voting rights are reinstated at the beginning of the second of two consecutive meetings that the member attends.

HPG's Face-to-Face meetings shall be held no less than twice a year. One meeting shall be held in conjunction with SPEC's annual meeting. One of the other meetings will accommodate HPG's membership. Teleconferences are usually held every week, for no more than one hour. The schedule for these meetings is posted on the private page.

2.6 HPG Guidelines for Result Submission and Review

Each benchmark suite produced by HPG includes a set of specific run and reporting rules that the licensee must follow to produce a publishable result. These basic tenets of these rules require:

  • Proper use of the SPEC benchmark tools as provided.
  • Availability of an appropriate full disclosure report.
  • Support for all of the appropriate protocols and standards.

Furthermore, SPEC expects that any public use of results from this benchmark suite shall be for systems and configurations that are appropriate for public consumption and comparison. Thus, it is also expected that:

  • Hardware and software used to run this benchmark must provide a suitable environment for the application area(s) targeted by the benchmark.
  • Optimizations utilized must improve performance for a larger class of workloads than just the ones defined by this benchmark suite.
  • The tested system and configuration is generally available, documented, supported, and encouraged by the providing vendor(s).

SPEC reserves the right to adapt the benchmark codes, workloads, and rules of its benchmarks as deemed necessary to preserve the goal of fair benchmarking. SPEC will notify members and licensees whenever it makes changes to a benchmark or its run and reporting rules.

2.6.1 Results Review

HPG's results review process is detailed on the HPG private page (reviewprocess.html).The standard review cycle is two weeks. In cases where the submitter has made source code changes the review cycle is a minimum of four weeks. At the beginning of the review cycle, a primary reviewer is assigned by the Chair to review the results. The primary reviewer is encouraged to offer his preliminary comments on the results within one week. At the end of the review cycle, the results are accepted or rejected by a majority vote of the committee. Once a result is accepted, the full disclosure report is posted on the SPEC web site, and once posted on the SPEC web site (http://www.spec.org/), the information contained in the result disclosure is considered public information.

If HPG votes that the submission is not compliant and should not be published on SPEC's web site, the submitter may acquiesce to the majority position and withdraw the submission. Alternately, the submitter may make an appeal to the SPEC Board of Directors. The submitter may withdraw, correct, or replace the questioned submission at any time during this process.

If minor corrections are made to a submission, e.g. documentation of tuning options or product names, which do not affect performance, then the corrected result remains on the same publication schedule as the original submission. If major corrections are made, e.g. a new run, different performance results, or different system configuration, then the corrected result starts review over as a new submission.

To assure fairness to the submitter, HPG has established the following guidelines regarding the proper use of review materials:

  • New source code changes cannot be used by another vendor for a period of six weeks commencing from the day the results are published on SPEC website .
  • Trying to surpass a result currently under review is considered common practice and is not prohibited.
  • Published information found in press releases etc. can be used at any time.

Members participating in the review of results submissions are expected to be familiar with the Guidelines for Handling SPEC Information, particularly Section A.7.0 dealing with Results Review (see Appendix A.)

2.6.2 Handling HPG Run Rule Violations in Published Results

HPG may also [vote to] undertake a re-review of a previously published result, if significant questions as to its adherence to the applicable run and reporting rules arise. HPG reserves the ability to denote a publish result as non compliant if it has been found in violation of the run and reporting rules.

The following process has been established to handle instances where run and reporting rule violations are discovered in results that have been published on SPEC's web site.

Once HPG identifies a result, published on the public web site is in violation of the benchmark's run and reporting rules, the result may re-enter a review process unless the submitter agrees with the finding of non-compliance.

If the subcommittee's review returns a finding that the result is not in compliance with the benchmark's run and reporting rules, the following should occur.

The results are modified:

  1. To remove the metric values (summary & individual) and replace with "NC" (non compliant).
  2. Add a header to the disclosure stating that the result did not comply to the rules and the reason(s) why.
  3. The submitter, with the committee's approval of the text, may add additional wording into a remedy section. The remedy section is expected to explain how the vendor has addressed the problem. For example, mentioning that a new result has been submitted on the same platform with the issue resolved is the expected typical usage of this section.

2.6.3 Handling Continued Availability in Published Results

The various SPEC benchmarks' Run and Reporting Rules have clauses that at a certain time after the first publication (on SPEC's web site or otherwise), all components of the System Under Test, (SUT), must be available for general customer shipment. The purpose of this requirement is:

  1. To ensure that the results refer to a real, available system;
  2. To enable others to verify reproducibility of the results.

For the case that this availability status cannot be maintained over a minimum time length, the following rules apply:

Either at the request of the test sponsor or as a result of a resolution, HPG can undertake a re-review of a previously published result if a performance relevant component ceases to be available for 30 days or more within 90 days after initial overall availability (i.e. initial availability of all components of the SUT). If the committee's review returns a finding that this non availability condition holds for a performance relevant component, the result is marked "Not Available" (NA) in the following way:

  1. The metric values (summary and individual) are removed and replaced with "NA" (not available).
  2. A header is added to the disclosure stating that the result is for a system that is, at a time specified in the header, not available.
  3. The test sponsor can, with the committee's approval of the text, add additional wording into a remedy section. The remedy section can for example, identify the part that is not available; contain the reason for non availability, and/or an estimate when all components of the SUT are expected to be available again. If and when the SUT becomes available, the submitter may notify HPG that the product is available. With HPG's agreement, the NA marking will be removed from the result page, however the page will retain the notice that the page was marked NA along with the history of the NA marking. This restarts the clock for the continuous availability requirement covered in this section.

For results not published by SPEC, HPG can ask the publisher to withdraw the publication.

"Performance relevant" is defined in the various benchmarks' Run and Reporting Rules' reproducibility clauses.

Replacement by a new similar component with equivalent or better performance is considered acceptable and not a reason for an NA marking.

In cases where the Run Rules allow use of a beta product, this beta product or a later equivalent or better performing regular product (see clause above) must remain available.

Note: Effective for results published on or after July 1, 2002.

2.6.4 Results for Electronically Equivalent Systems

HPG encourages result submitters to run the benchmark on all systems. Submitters are responsible for the accuracy of their results. There may be cases where a submitter may want to submit the same results file for multiple submissions. This is acceptable if that performance is representative of those systems (e.g., just the power supply or chassis is different between the systems). A definition of what is acceptable as "electronically equivalent" systems is provided in the next section.

As part of HPG review process, the submitter should expect to be asked to justify why the systems should have the same performance. It would be within the purview of the HPG committee to ask for a rerun on the exact system, if the technical criteria are not satisfied. In cases where HPG accepts the submitter's claim of electronic equivalence, the submitter would have to add a line in the notes section of each of the result stating that the systems are electronically equivalent. For example, if a submitter submits the same result file for Model A and Model, the following note needs to be added to both the results:

Example: "Vendor's Model A and Vendor's Model B are electronically equivalent."

2.6.5 Definition of Electronic Equivalence

There are no noticeable differences in the behavior of the systems down to the level of electronic signals.

Examples of when a system is considered electronically equivalent include:

  • Installation - for example a system that is sold as both a desk side system and rack mount system (where the only difference is the casing) would be considered electronically equivalent. Another example is systems that are sold in a large case (to allow installation of disks internally) and a small case (which requires an external case for disks) but which are otherwise identical.
  • Packaging - for example a system where the vendor has changed the name and/or model number and face plate without changing the internal hardware is considered electronically equivalent.

Examples of when a system is not considered electronically equivalent include:

  • Different number of types of slots or buses - even if unused, hardware differences such as these may change the behavior of the system at peak performance. These systems are usually referred to as 'functionally equivalent'
  • Vendor fails to convince the committee on technical merits that systems are electronically equivalent.

2.7 HPG Technical Support Guidelines

The members of HPG are responsible for providing adequate resources to handle technical support questions regarding current benchmark suites. It is the goal of technical support to provide timely response to questions regarding the nature and usage of these benchmark suites to help facilitate testing by other SPEC benchmark licensees. The purpose of SPEC's technical support is NOT to provide OS, compiler or other vendor tool support or assistance. At the technical support person's discretion, the user may be referred to a third party, who may charge a fee for such support.

For each released benchmark, the SPEC administrator will maintain an email alias (i.e. omp2001support@spec.org). HPG may establish a quarterly rotation schedule among its active members (based on license number) to serve as technical support person. In special cases, a member may be assigned to provide support for specific subset of technical support questions for an extended period or to handle support for a specific geography. It is the Chair's responsibility is to see that questions are answered promptly and provide backup when the vendor on rotation becomes unavailable. The current support representative or chairs may assign a technical support call to another member, if they believe vendor specific knowledge may be required to resolve the call.

The SPEC administrator remains the initial point of contact for technical support questions. The administrator will make the determination as to which email alias the problem is reported to.

2.8 HPG General Member Guidelines

HPG is open to any organizations that have an interest in the use and development of standardized performance benchmarks for high performance computer systems. By joining the HPG, members may:

  • Receive copies of all currently released HPG benchmark suites.
  • Freely distribute copies of HPG benchmark suites within their company, including wholly owned subsidiaries. Country export and any other restrictions must be adhered to as per the license agreement.
  • Submit an unlimited number of benchmark results to HPG for review and publication on SPEC's website.
  • Become active members by contributing engineering resources to the development of new benchmark suites.
  • Join any or all of HPG email aliases and receive access to the members only website.
  • Attend any or all meetings of HPG and SPEC general membership, and open SPEC Board of Directors meetings.
  • Stand for election to HPG according to the guidelines above.
  • Receive access to preliminary (beta) versions of benchmarks currently in development for internal review and to provide feedback to the appropriate subcommittee.
  • Vote on any proposed benchmark suite submitted for General Membership approval.

Members are expected to uphold the policies and principles endorsed by SPEC and the High Performance Group. This includes policies described in this document and amended by resolutions of HPG; SPEC's Articles of Incorporation and Bylaws; policies adopted by the SPEC Board of Directors; and any applicable set of benchmark run and reporting rules. Members are also expected to remain members in good standing by the prompt remittance of annual dues. Membership privileges will be suspended if dues have not been received by the SPEC administrator by March 31st. Membership privileges will be restored once dues have been paid.

2.9 Fair Use of SPEC HPG Benchmark Results

Consistency and fairness are guiding principles for SPEC. To help assure that these principles are met, any organization or individual who makes public use of SPEC benchmark results must do so in accordance with the SPEC Fair Use Rule, as posted at http://www.spec.org/fairuse.html.

2.10 Other Resources

HPG can, after notification in writing to the Board of Directors, enlist the aid of advisors and consultants from the industry and academia. Certain organizational activities require board participation and approval. For example, the board must approve press releases, set prices for benchmark licenses, and allocate resources of SPEC administrative staff and Public Relations to assist with the non technical aspects of releasing a new benchmark suite.

This work may be facilitated by a joint committee of the board and the HPG (e.g. Planning Committee) or proposals may be brought directly to the board by the HPG Chair.

2.11 Benchmark Release Approval

The process to approve the release of new benchmark suites consists of several steps. HPG must finalize the benchmark and prepare a beta version that can be recommended for general membership review. General membership refers to all the members of HPG and not just the members who actively worked on developing and testing the benchmark suite.

When this beta is ready, the committee must send out a final call for comments to the General Membership. A member has four weeks from that time to review and comment on the benchmark. The member should send their comments back to HPG.

At the conclusion of the final call for comments period, the voting membership of HPG will take part in a final vote to approve the benchmark for release.

 

Appendix A. Guidelines for Handling SPEC Information

A.1.0 Introduction

SPEC is an industry consortium whose membership includes many competing for-profit corporations. Membership in SPEC is open to all persons who are directly and materially affected by SPEC's activities, and participation in SPEC activities is generally open to all members. However, in order to protect the interests of SPEC and its members, and to encourage free and open participation and discussion to further SPEC's objectives, public release of SPEC internal information may be prohibited without explicit approval from the SPEC organization concerned. Violators of this policy will be subject to the full force of the SPEC penalties process.

A.2.0 Anti-Trust Compliance

Since all SPEC meetings are to be conducted in compliance with all applicable laws, including anti-trust laws, the following policies shall be followed in the course of SPEC meetings, telephone meetings, and electronic exchanges: Any discussions that relate to the validity or cost of patent use should be avoided. Any discussions or any material relating to an ongoing litigation shall be avoided, except for the Board in executive session with advice of counsel to discuss matters affecting SPEC. Any discussions of pricing or issues that would violate US antitrust laws shall be avoided.

A.3.0 How Information is Classified and Unclassified

SPEC officers and the SPEC Administrator may explicitly designate certain information classified and mark it appropriately. Any member that wishes to be sure that its information is treated as SPEC Confidential should clearly label each page (electronic or hard copy) as such. It is the member's responsibility to assure that such information is appropriately labeled. Information not explicitly marked but which meets the descriptions in this policy must be taken as though it were so marked.

Classified information may become public through the authorized public release of the information by SPEC or by a member. If classified information is disclosed without authorization (leaked), that does not automatically remove its classification and relieve participants of the obligation to protect that information from further disclosure. However, if an unauthorized disclosure is widespread, then a SPEC officer may redesignate that information as public, stating the reason for so doing.

A.4.0 Classifications

SPEC identifies two levels of internal information which must be protected by SPEC members and associates, employees, and contractors, in accordance with these policies. "SPEC Confidential" is the ordinary classification of information which should not be disclosed outside SPEC. "SPEC Confidential Need-to-Know" refers to the most sensitive information described below.

A.4.1 SPEC Confidential Need-to-Know

This is information that must be restricted to only those individuals who must have it in order to carry out their SPEC responsibilities. In meetings, this information is discussed only when the board adjourns to executive session. Only the author of such a document, or the SPEC President, can change the classification or the approved reader list. Examples of Confidential Need-to-Know information, includes:

  • Information concerning litigation
  • Certain matters that relate to the formation of contracts with third parties
  • Employee salaries, performance appraisals, and other personnel matters
  • Administrative (root) passwords of SPEC business computers

A.4.1.1 Document Handling Guidelines

Documents of this sensitivity class will always be clearly labeled "SPEC Confidential Need-to-Know" on each page. The documents may be numbered for control purposes.

SPEC Private Documents must have a cover page which serves to:

  1. Hide the first page of the document,
  2. Document the approved reader list and,
  3. Document the date the information is no longer SPEC Confidential, if any.

Such documents should always be carefully transported. Documents should be kept inside an envelope, clearly labeled with the approved reader's name, when not in active use. Only persons on the approved reader list are privy to the document contents.

Email should be used sparingly for communicating such information. Messages should be sent directly to the intended recipient(s) rather than to any mail alias. Messages should be retained no longer than absolutely necessary, and should be kept separate from other messages. Encryption should be employed where possible.

A.4.2 SPEC Confidential

The following information is "SPEC Confidential":

  • Unpublished benchmark results, including configuration details, under SPEC review,
  • Benchmarks under development, including run rules, code, tools, workloads, benchmark and
  • Workload analyses, and benchmark design,
  • The content of SPEC meetings,
  • Messages received on SPEC members email aliases,
  • Passwords to the SPEC members web site, and to accounts on SPEC computers,
  • The content of the SPEC members web site,
  • SPEC financial data and
  • SPEC email and phone lists.

Note that non-members of SPEC should not be on mailing aliases. Please note that SPEC offers various levels of membership to fill different needs for information access.

A.5.0 Who May Have Access to SPEC Confidential Information

Confidential information is provided to SPEC members through their designated primary SPEC representatives, and those representatives of SPEC subcommittees designated by the member's primary SPEC representative. It is the responsibility of those representatives to determine who within a member company should have access to the information in order to carry out the member's SPEC activities; and it is their responsibility to ensure that anyone who does have access to SPEC Confidential information understands and agrees to abide by these guidelines. A member's SPEC representatives should use whatever internal controls are appropriate in their company to ensure that the member complies with SPEC's policies on confidentiality.

A.6.0 Meetings

Information about the activities and developments of SPEC are intended to be readily available to all parties with legitimate interests in the performance of advanced computer systems. The Board is responsible to carry out this policy.

The annual meeting of the SPEC membership must be open to any interested party with a legitimate interest in the work of the Corporation, including the press. Other meetings of each SPEC group and committee must be open to representatives of every member of that group. It is expected that meetings of any SPEC group will also be open to members of any other SPEC group, to encourage wider participation and exchange of ideas. However, a SPEC group may set a more restrictive policy for some or all meetings, subject to Board review. Interested parties who are not SPEC members, but who have a legitimate interest in its work, may be invited to its meetings by the SPEC group holding the meeting.

A.6.1 Press Participation

Press representatives who are not SPEC members are generally excluded from SPEC meetings other than the annual meeting, unless the chair of the SPEC group or subcommittee, invites the press to attend. If the press is in attendance, the chair should announce press attendance. The press is encouraged to use such attendance opportunities to gain background information in technical areas of performance benchmarking. SPEC is primarily an engineering organization which depends on frank and open exchange and debate of views. Views expressed in SPEC meetings are not expected necessarily to reflect official positions of SPEC member companies and organizations. SPEC asks that the press show restraint in any coverage based on SPEC meetings, so as not to jeopardize our open exchange of views.

A.6.2 Audio/Visual Recording

SPEC committees may apply restrictions on the use of audio recording, video recording, or photography equipment where they may impede free discussion, or where they are disruptive. Such restrictions should be clearly identified, in advance, to attendees.

A.7.0 Results Review

Benchmark results submitted for SPEC review, as for publication on the SPEC web site, are considered SPEC Confidential. This includes the benchmark result, the configuration, availability dates, and tuning notes. Within a member company this information should be shared with only those people necessary to provide a thorough review of the results, per paragraph 5.0.

Information which the submitting company makes public is not considered confidential. E.g., a company may issue a press release or briefing listing performance of new systems and, around the same time, submit those benchmark results for SPEC publication. In this case, details included in the full SPEC disclosure under review which were not included in the company's public announcement, are not considered SPEC Confidential because the run rules of each benchmark require the full SPEC disclosure to be publicly available.

If results are later published, then the result is public. If results are withdrawn or not accepted during SPEC review and are not published, then they remain SPEC Confidential. E.g., if a company submits results that are not accepted for SPEC publication, you may not make public use of that fact.

New and unique optimization techniques that appear in a result disclosure should not be utilized by another member until the original submission has completed its review cycle. The subcommittee may decide to suspend a result in review should this guideline not be followed. Trying to surpass a result currently under review is considered common practice and is not prohibited. Published information found in press releases, public web sites, manuals, etc. can be used at any time.

A.7.1 New Benchmark Announcements

More restrictive confidentiality rules may be applied by the SPEC groups on the occasion of results review for SPEC's announcement of a new benchmark suite, in addition to all of the preceding rules.

A.8.0 SPEC Press Relations

Press releases from SPEC must be approved by the SPEC group concerned, and by the Board.

SPEC members may occasionally be interviewed by the press, e.g. on the subject of newly released benchmarks. Normally SPEC's PR consultant will participate in such interviews in order to see that SPEC is treated fairly, and to assure all SPEC members that information is being represented in an unbiased manner. The Board Press Committee may on a case by case basis, waive this requirement, based on such factors as: degree of controversy of the subject, potential for appearance of conflict of interest by the spokesman, legal advice, advice from SPEC's PR consultant, recommendation of steering committee and technical committees and their chairs, experience of spokesman, experiences with the reporter, etc.

 

Appendix B. Guidelines for General Availability

One of the principal tenets of SPEC benchmarks is that the implementation of the System under Test (SUT) must be generally available, documented and supported by the providing vendor. The run and reporting rules for each benchmark includes this statement. The available, documented, and supported trio is frequently referred to with the single term: general availability.

The purpose of including general availability requirements is to ensure that the systems and their hardware and software components actually represent real products that solve real business and computational problems. It is also intended to ensure that the benchmarked system does not contain so-called benchmark specials which improve benchmark performance but fail one or more of the tests of general availability. These tests include:

1. Has the system/component shipped or has the vendor committed to shipping the finished product within a time specified by the specific benchmark's run rules, usually 3 to 6 months from the date the result was made public?

A finished product will have the vendor's endorsement that the product is suitable for production use in an environment comparable to the one represented by the benchmark. Typically this version will be designated as the final released version and not designated as a prototype or a preliminary release (i.e. alpha, beta, field test, etc.).

Shipping the product refers to the vendor's standard process for getting said finished product into the hands of the end-user base. This can include putting it in a box and physically transporting it to end-user sites or simply advising the customer base that the product can be obtained on demand, for example by having an announced URL where the product can be downloaded or purchase information can be found.

The commitment to ship a product within a specific time frame provided in a results disclosure must be taken as a public statement of intent. Any potential end-user should be able to confirm the vendor's intent to ship the finished product within the stated time frame and that information should not be considered secret once the result has been publicly disclosed.

Shipping a product generally requires that one or more instances of the final product be in end-user hands or enroute to end-users with delivery imminent and based on shipping method used.

2. Is the product documented such that there are written descriptions on the product, how to install it, how to configure it and how and when to use it?

The product documentation should be consistent with the deployment of the product in the benchmark such that there are no undocumented features employed solely for the benchmark. Availability of documentation should be concurrent with product availability.

3. Is the product supported such that the vendor has established clear expectations and procedures with its end-users with regards to problem resolution? Also, is the level of support reasonable for the type of product?

Levels of support can clearly have a wide range depending on the type of product. An end-user acquiring a supported product will have access to some method of problem resolution consistent with the product's importance within the system. An inadequately supported product would be one that most consumers of that type of product could not afford either due to excessive costs imposed by the vendor for support or due to the risk involved in using a product with insufficient recourse to problem resolution.

The requirement for support does not preclude the use of open source software products or other products where support is typically acquired separately.

The requirement for support is present to prevent vendors from deploying benchmark specials that may seemingly meet availability and documentation tests but which are subtly discouraged from practical and common use by imposing unusually high costs and/or lengthy or inadequate methods for problem resolution.

4. Does the product meet the vendor's own definition of general availability, in addition to all the other tests specified by SPEC?

Many vendors have their own internal processes for product development that set specific requirements for products to be considered generally available. If so, those requirements must be met for the product to be considered generally available by SPEC, in addition to the tests listed here. If the vendor has no such definition, then all other tests specified by SPEC must still be met.

A vendor is not required to discuss or disclose its internal processes to SPEC. However the SPEC representative is responsible to identify the internal definition most closely matching SPEC's intent of general availability, and applying that definition to any statements to SPEC regarding general availability. Note that the vendor may use different terms, e.g. full customer availability, open shipping status, phase 4 exit, etc.

5. Does the vendor represent the product to customers as being generally available?

This is not a clear cut test, as products may have several different distribution channels, and a product may not become generally available in all channels simultaneously. However, it is an indication that a product may not be generally available if the vendor specifically says in some venue that it is not. E.g.:

  • Web page saying product is not available.
  • Web page listing other configurations of the product as available which fails to list the published configuration. (E.g., web page offers 1.5 GHz and 2.0 GHz models for sale but doesn't mention the published 2.5 GHz model.)
  • Sales person will not take orders for the product.
  • Statement by official company spokesman saying product is not available.

It is an indication that a product may be generally available if the vendor specifically says so in some venue. E.g.,

  • Web page saying product is available.
  • Advertisement for the product
  • Sales person or reseller, possibly in a different locale, will take orders for the product.
  • Statement by official company spokesman saying product is available.

In case of conflicting information, SPEC will weigh the preponderance of the evidence with the standard that it show the product to be generally available in some channel.

The rules for each benchmark may have slightly different requirements related to general availability and may specify other types of benchmark specials to avoid. The overall goal of ensuring fair benchmarking for the vendors submitting results and the end-users reviewing those results remains the same.

Note: The term "benchmark special" refers to any optimization which only improves benchmark performance.