CPU98 Search Program Schedule and Notes

Schedule

Definitions

Available Benchmarks:

An available benchmark is one that has been submitted to SPEC and meets the following criteria:

• Source code must be available for world wide distribution.
• Must be POSIX and ANSI compliant.
• Must be portable across at least two operating systems (UNIX and WIN/NT are suggested).
• Must be portable across a little endian and a big endian hardware platform.
• Must be submitted to SPEC in the SPEC tool framework.
• Must have sufficient and checkable output verification.
• Must spend at least 95% of run time in it's own code.
• Must be 99% CPU bound in 128MB of RAM.
• Must include a description file.

Only available benchmarks are assigned benchmark numbers.

CPU Subcommittee Meetings:

The SPEC CPU Subcommittee will meet at all normal SPEC meetings (currently every two months) and by teleconference every two weeks (currently for CPU95 results review).

Subcommittee Vote:

Any interested members and associate members of the SPEC OSG may be a member of the CPU Subcommittee. One vote is allowed per member/associate member. Of those voting, 2/3rds of those voting yes or no must vote yes for subcommittee vote to pass.

Benchmark project leaders:

Any member of the SPEC CPU Subcommittee can be a project leader for a benchmark. The project leader is responsible for shepherding the benchmark through the SPEC CPU benchmark process:

• Making it an "available benchmark"
• Helping with further porting.
• Developing the workload.

Benchmark cycle:

Given limited resources and the fact that too much work in parallel causes more overhead, the CPU subcommittee will try to develop benchmarks in cycles, where a smaller number of benchmarks are looked at over a smaller period of time. Each cycle will be approximately 3 months and will work at follows:

• Prior to each cycle:
• "Available benchmarks" will be placed in a benchmark queue.
• At the beginning of a cycle:
• There will be a summary discussion of the "available benchmarks", primarily focusing on their application area and any known characteristics.
• There will then be a "prioritization vote" on the "available benchmarks". The Subcommittee will then move down this ordered list looking for project leaders. It is expected that each active member company would accept leadership of at least one benchmark.
• During the cycle:
• Project leaders would work on the benchmarks (i.e. - aiding in portability issues, crafting workloads, etc.)
• Each member company would also work on testing the benchmarks as the project leaders develop them.
• Updates to the source tree would be planned for every two weeks. (May change as we see how the process works).
• Evaluation and discussion of technical aspects of the benchmarks is encouraged at this time.
• Project leaders can drop their benchmarks (i.e. - impossible to port), put it at the bottom of the prioritized benchmark queue and get the next one out of the queue. - Note: It is hoped members are still working on new ideas for "available benchmarks" to be added to the queue for consideration in the next cycle.
• At the end of the cycle:
• Each benchmark being worked on is discussed and placed in one of three categories (by subcommittee vote in the following order):
• Ready for release: placed aside for integration into the final suite.
• Still needs to be worked on; placed back into the prioritized benchmark queue (at the top)
• Failing these two votes a benchmark is dropped due to lack of consensus. Reasons for no votes must be elaborated.
• The subcommittee will evaluate if additional cycles need to be pursued. With current resources four cycles are currently planned.
• At the end of the last cycle as determined by the subcommittee:
• The subcommittee will propose to the parent committee that they are ready for approval for a general member vote. The parent committee will either do this or ask that the subcommittee reconsider and enter another cycle (this would be an OSG decision for the CPU subcommittee).

Schedule for CPU98 candidates:

TBD

Go To: [Home] [SPEC] [OSG] | [CPU98] [Search] | [CPU95]