Standard Performance Evaluation Corporation

SPEC Eliminates Libcurses.a from 072.sc

By Bodo Parady
Sun Microsystems, Inc.
Menlo Park, Calif.

Published December, 1994; see disclaimer.

Introduction

In the interests of increasing the usefulness of CINT92 results, on Sept. 13, 1994, the Open Systems Steering Committee (OSSC) of SPEC adopted a change to the Run Rules for the 072.sc benchmark. This change may affect the way in which you run the benchmark. It may also improve the results you and others publish for that benchmark and for the metrics which depend on it: SPECint92, SPECbase_int92, SPECrate_int92 and SPECrate_base_int92.

This change in the Run Rules allows the use of an OSSC-supplied "common curses" library, to be compiled and used as part of the 072.sc benchmark code. This may be used in place of the libcurses.a supplied by the operating system. If you would like a revised copy of the 072.sc benchmark code, including the "common curses" library, send electronic mail to spec-ncga@cup.portal.com. Please direct questions to Dianne Rice, SPEC's Administrator.

Details of the "common curses" Run Rules Change for 072.sc:

On Sept. 13, 1994, the Open Systems Steering Committee voted to allow the use of so-called "common curses" for the 072.sc benchmark. The purpose is to eliminate differences in reported performance due to vendor-specific implementations of the "curses" library.

The 072.sc benchmark is based on the public domain spreadsheet, sc. It measures the performance of spreadsheet entries, calculations and updates, while performing procedure calls to routines that would typically display the results on a standard "curses"-driven display.

The problem identified with 072.sc is in the use of libcurses.a -- a commonly found, vendor-supplied, library which provides a standard interface to user visual display screens.

In the SPEC benchmark version of sc, the output stream which normally goes to the screen is redirected to a file, and not sent to an actual display. The OSSC recently discovered that under this circumstance, some implementations of libcurses.a do not perform all the work that would be required were the stream of output characters actually going to a display device.

This screen output forms a significant portion of the total workload of the benchmark. Thus these differences in implementation have led to published results for 072.sc that, from one vendor's system to another, do not reflect the same amount of work.

The solution agreed to by the OSSC to eliminate the great disparity in workload for this benchmark is to eliminate the workload of the curses library entirely, and to supply a "common" curses library for use with 072.sc.

The decision to eliminate the workload of the curses library from the benchmark is consistent with the intent of the OSSC that the CINT92 (and CFP92) benchmarks should highlight the performance of code compiled from source, and not code supplied by the operating system or one of its libraries. Given that the purpose of the CPU benchmarks is to reflect most strongly the performance of the CPU, compiler, and memory hierarchy, the performance of utility libraries distracts from the desired measurement.

SPEC members report that the effect of a "common" curses on their 072.sc result times, varies from a reduction of 4 percent to over 40 percent. For those vendors at the high end of variation, this can result in an increase of almost 10 percent in the reported SPECint92 or SPECrate_int92 results. The importance to users of the CINT92 metrics of the OSSC's decision is that these results will be more comparable, with variation produced by performance in libcurses.a eliminated from reported values.

The major issue with this is that some old results will not benefit directly from this leveling. This downside, in the view of the OSSC, is outweighed by having all present and future results reflect a more comparable benchmark.

Dr. Bodo Parady is with Sun Microsystems in Menlo Park, Calif. He serves as the Technical and OSSC representative for Sun Microsystems.