Non-baseline Results Eliminated from SPEC SFS Benchmark

By Brian Pawlowski
Network Appliance Corporation
Mountain View, Calif.

Published December, 1994; see disclaimer.

The SPEC steering committee and SFS subcommittee reviewed SFS results submission process at the Munich meeting held July 1994.

Currently two categories of results are defined for SFS. Baseline results that asserted that the NFS server being reported adhered to the stable storage requirement for NFS Version 2 (as described in the SFS documentation), and non- baseline results that did not make this assertion. Non-baseline results could only be submitted with baseline results for the same configuration.

In retrospect, the SFS committee felt the existence of the non-baseline result type was confusing. Further, since the introduction of the SFS suite, no valid non-baseline results have been published. Not even by those vendors that support server operation with stable storage guarantees disabled.

Looking toward the future, the SFS subcommittee felt that any future submission of a non-baseline result would detract value from the SFS suite, since the result would not be comparable with existing baseline submissions. Vendors have submitted many results since the introduction of the SFS benchmark, providing a base of results to be used for comparing NFS server performance. The SFS subcommittee felt that, for the SPECSFS_A93 workload, maintaining a single set of requirements for server configurations in support of comparable results was of critical importance to the usefulness of SFS benchmark results. Any non-baseline submission would confuse customers of the SFS results.

The SFS subcommittee moved to eliminate the non-baseline results reporting from SFS to maintain the value of the benchmark and the usefulness of the results in comparing NFS servers, and the OSSC ratified the following proposal:

R94.07.04

(SFS discontinuance of non-baseline): The SFS Run Rules are changed to allow only baseline results.

Further, the OSSC reaffirmed:

R94.07.03

(SFS uniform access): SPEC OSSC reaffirms that the "Uniform Access Rule" for SFS 1.0 as described in the September 1993 SPEC Newsletter applies to all architectures.

Though originally introduced during a review of SFS Cluster results, the original proposal was always intended to apply to all SFS submissions.

SFS 1.1 documentation was modified to reflect these resolutions.

The SPEC SFS subcommittee at Munich continued investigation of new workloads to augment the current single workload in the SFS suite. Benchmark results representing different application workloads will provide customers of SFS results with more accurate predictors of performance for their particular environment. In addition, the committee is evaluating addition distributed file system protocols for inclusion in the SFS suite, including NFS Version 3.

Brian Pawlowski is the project leader for performance at Network Appliance Corp. He has been involved with the LADDIS benchmark development for several years.

Copyright (c) 1994 Standard Performance Evaluation Corporation