SPEC logo


SPECsfs97_R1 Benchmark Press Release

1: What is SPEC SFS 3.0 and how does this benchmark compare to other network file system (NFS) benchmarks?

SPEC SFS 3.0 is the latest version of the Standard Performance Evaluation Corp.'s benchmark that measures NFS file server throughput and response time. It differs from other NFS benchmarks in that it provides a standardized method for comparing performance across different vendor platforms. The benchmark was written to be client-independent and vendor-neutral. Results are validated through peer review before publication on SPEC's public Web site <http:// www.spec.org/osg/sfs97r1/>

2: Does this benchmark replace the SPEC SFS 2.0 suite?

Yes. Now that SPEC SFS 3.0 is available, SFS 2.0 licenses are no longer being sold. Results from SFS 2.0 will no longer be accepted by SPEC for publication.

3: Can SPEC SFS 3.0 results be compared to SFS 2.0 results?

No. Although the benchmarks are similar, they cannot be compared, since SFS 3.0 uses a different file selection algorithm, its results can only be compared with other SFS 3.0 results.

4: What improvements have been made to SPEC SFS 3.0?

In addition to general code improvements, SPEC SFS3.0 includes three major enhancements: 1. Fixes in the file set selection and workload mechanisms. 2. Support for Linux clients. 3. The benchmark distribution CD contains updated pre-compiled and tested binaries.

5: How was the SPEC SFS 3.0 workload determined?

The SPEC SFS 3.0 workload is based primarily on a survey of more than 1,000 servers in different application environments. The survey found that 60 percent of these users have similar mixes of NFS operations. The workload in SFS 3.0 more accurately presents the intended workload that was used in SFS 2.0.

6: What is the metric for SPEC SFS 3.0?

SPEC SFS 3.0 has two performance measurement metrics: SPECsfs97_R1.v2 for NFS protocol version 2 and SPECsfs97_R1.v3 for NFS protocol version 3. Both metrics include a throughput measure (in operations per second) and an overall response time measure (the average response time per operation).

7: Are the metrics for SPEC SFS 3.0 different than the metric for SFS 2.0?

No. SFS 3.0 maintains the same metrics that were used in SFS 2.0. The overall response time and peak throughput. The larger the peak throughput the better. The lower the overall response time the better. The overall response time is an indicator of how quickly the system under test responds to NFS operations over the entire range of the tested load. In real-world situations, servers are not run continuously at peak throughput, so peak response time provides only minimal information. The overall response time is a measure of how the system will respond under an average load. Mathematically, the value is derived by calculating the area under the curve divided by the peak throughput.

8: How widespread is NFS version 3?

NFS version 3 has been shipping on systems for more than six years and is available for most systems that support NFS version 2.

9: What is the correlation between the TPC (Transaction Processing Council) benchmarks and SPEC SFS 3.0?

There is no correlation; the benchmarks measure totally different aspects of system performance.

10: Is SPEC SFS 3.0 a CPU- or I/O-intensive benchmark?

SPEC SFS 3.0 is a system-level benchmark that heavily exercises CPU, mass storage and network components. The greatest emphasis is on I/O, especially as it relates to operating and file system software. To obtain the best performance for a system running SFS 3.0, the vendor will typically add additional hardware -- such as memory, disk controllers, disks, network controllers and buffer cache -- to help alleviate I/O bottlenecks and to ensure that server CPUs are used fully.

11: For what computing environment is SPEC SFS 3.0 designed?

The benchmark was developed for load-generating clients running in the UNIX environment. But since the load-generating clients execute the benchmark code, SPEC SFS 3.0 can be used to test the performance of any NFS server, regardless of the underlying environment. Porting is required, however, for non-UNIX environments.

12: Can users measure NFS performance for workloads other than the one provided within SPEC SFS 3.0?

Yes, users can measure their own workloads by making changes to the SPECsfs97_R1 benchmark mix parameters to reflect the new measurements. The SPEC SFS 3.0 User's Guide details how this can be done. Workloads created by users cannot, however, be compared with SFS 3.0 results, nor can they be published in any form, as specified within the SFS 3.0 license.

13: To what extent is the server's measured performance within SPEC SFS 3.0 affected by the client's performance?

SPEC has written SFS 3.0 to minimize the effect of client performance on SPECsfs97_R1 results.

14: Why have no companies reported SPECsfs97_R1 results in conjunction with this announcement?

SPEC SFS 3.0 is a system-level benchmark that requires scheduling substantial resources for testing. SPEC expects member companies to report results in the near future.

15: How does SPEC validate numbers that it publishes?

Results published on the SPEC Web site have been reviewed by SPEC members for compliance with the SFS 3.0 run and disclosure rules, but there is no monitoring beyond that compliance check. The vendors that performed the tests and submitted the performance numbers have sole responsibility for the results. SPEC is not responsible for any measurement or publication errors.

16: Are the reported SFS 3.0 configurations typical of systems sold by vendors?

Yes and no. They are similar to large server configurations, but the workload is heavier than that found on smaller server configurations. SPEC has learned from experience that today's heavy workload is tomorrow's light workload. For some vendors, the configurations are typical of what they see in real customer environments, particularly those incorporating high-end servers. For other vendors, SFS 3.0 configurations might not be typical. Question 17: Do the SFS 3.0 run and disclosure rules allow results for a clustered server? Answer : Yes, cluster configurations are allowed as long as they conform strictly to the even distribution of all resources as defined by the SFS 3.0 run and disclosure rules.

18: Why do so few published results approach SPEC's response-time threshold cutoff of 40 milliseconds?

It is important to understand first that SPECsfs97_R1 run rules do not require that the throughput curve be carried out to 40 ms; they only state that the results cannot be reported for a response time higher than 40 ms. There are several reasons why results do not approach the threshold cutoff. Optimally configured servers often will achieve their maximum throughput at response times lower than the cutoff. Additionally, some vendors emphasize maximum throughput while others concentrate on fast response time. It does not indicate a problem with the results if the curve is not carried out to 40 ms, and those reviewing results should not try to predict what the throughput curve might be past the reported point.

19: Why was the response-time threshold reduced from 50 ms for SFS 1.1 to 40 ms for SFS 2.0 and SFS 3.0 ?

The lower response-time threshold reflects advances in server technologies since the release of SFS 1.1 in January 1995.

20: What resources are needed to run the SPEC SFS 3.0 benchmark?

In addition to a server, a test bed includes several clients and an appropriate number of networks. The server must have enough memory, disks and network hardware to saturate the CPU. The test bed requires at least one network and each network must have sufficient client capacity to saturate the network(s). A minimum of 64 MB of memory is required for each client, although in most cases 128 MB is needed. SFS 3.0 contains enhancements that reduce the amount of memory that the benchmark consumes on each client. Requirements are detailed in the SFS 3.0 User's Guide. To facilitate accuracy of reported vendor results, SFS 3.0 includes an entire NFS implementation. Examples of typical load-generating configurations can be found on the SPEC Web site: <http://www.spec.org/osg/sfs97r1/>.

21: What is the estimated time needed to set up and run SPEC SFS 3.0?

Hardware setup and software installation time depend on the size of the server and the complexity of the test beds. Many servers require large and complex test beds. The SFS 3.0 software installs relatively quickly. A SPECsfs97_R1 submission from a vendor includes at least 10 data points, with each data point taking about 20 to 30 minutes to complete.

22: What shared resources does SPEC SFS 3.0 use that might limit performance?

Shared resources that might limit performance include disk controllers, disks, network controllers, network concentrators, network switches and clients.

23: SPEC's CPU95 benchmark defines compiler optimization flags that can be used in testing. Does SPEC SFS 3.0 set tuning parameters?

When submitting results for SPEC review, vendors are required to supply a description of all server tuning parameters within the disclosure section of the reporting page.

24: Can a RAM disk be used within a SPEC SFS 3.0 configuration?

SPEC enforces strict storage rules for stability. Generally, RAM disks do not meet these rules, since they often cannot survive cascading failure-recovery requirements unless an uninterruptible power supply (UPS) with long survival lines is used.

25: How will the choice of networks affect SFS 3.0 results?

Different link types and even different implementations of the same link type might affect the measured performance -- for better or worse -- of a particular server. Consequently, the results measured by clients in these situations might vary as well.

26: Is SPEC SFS 3.0 scalable with respect to CPU, cache, memory, disks, controllers and faster transport media?

Yes, like SFS 2.0, the new benchmark is scalable as users migrate to faster technologies.

27: What is the price of a SPEC SFS 3.0 license and when will it be available?

SPEC SFS 3.0 is available now on CD-ROM for $900. Contact the SPEC office: Standard Performance Evaluation Corporation (SPEC) 6585 Merchant Place, Suite 100 Warrenton, VA 20187, USA Phone: 540-349-7878 Fax: 540-349-5992 E-Mail: info@spec.org

28: How much is an upgrade from SFS 2.0 to SFS 3.0?

The upgrade is free for those who have purchased SFS 2.0 licenses within the last three months and $300 for other SFS 2.0 licensees. Upgrades are available through the SPEC office.

29: Can users get help in running SPEC SFS 3.0?

The majority of questions should be answered in the SPEC SFS 3.0 User's Guide. There is also useful information on the SPEC Web site: <http://www.spec.org/osg/sfs97r1/>.

Running the benchmark

30: Do I need to measure NFSv2 _and_ NFSv3? TCP and UDP?

No. NFSv2 and NFSv3 are considered separate workloads and you only need to measure and disclose the ones you want.

31: How do I get started running the SPECsfs97_R1 benchmark?

Please read the User's Guide in its entirety.

32: I am running into problems setting up and running the benchmark. What can I do?

Most of the problems relating to the SPECsfs97_R1 benchmark can be resolved by referring to appropriate sections of the User's Guide, especially the Troubleshooting section.

33: I have read the User's Guide. But I am still running into problems. What can I do next?

Looking at the sfslog.* and sfscxxx.* files can give one an idea as to what may have gone wrong. As a last resort, one can contact SPEC. It is assumed that such calls are from people who have read the User's Guide completely, and have met all the prerequisites for setting up and running the benchmark.

34: How does one abort a run?

One needs to kill all SFS related processes on all clients and on the prime client and re-run the benchmark. The processes are sfs, sfs3, sfs_syncd and sfs_prime.

35: For a valid run, which parameters are required to be unchanged?

Information is provided in the sfs_rc file, and this is enforced by the benchmark. If invalid parameter values are selected, the benchmark reports an invalid run.

36: Is there a quick way to debug a testbed?

Read the User's Guide, ping server from client, try mount the server file systems from the client using the client's real NFS implementation, rsh from prime client to the other clients and reverse, run benchmark with one client and one file system.

37: When I specify 1000 NFSops/sec in the sfs_rc, the results report only 996 NFSops/sec requested, why is it less?

Unlike SFS 1.1, the sfs_rc file specifies the total number of NFSops/sec across all of the clients used. Because the benchmark only allow specifying an even number of NFSops/sec, the actual requested ops/ sec may be less due to rounding down. For example, 1000 NFSops/sec requested over 6 clients will result in each client generating 166 NFSops/sec for an aggregate of 996 NFSops/sec.

38: The number of operations/second that I achieve is often slightly higher or slightly lower than the requested load. Is this a problem?

No, the benchmark generates operations using random selection and dynamic feedback to pace correctly. This will result in small difference from the actual requested load.

Tuning the Server

39: What are a reasonable set of parameters for running the benchmark?

Study existing results' pages with configuration information similar to your system configuration.

40: When I request loads of 1000, 1300, 1600 NFSops, I get 938, 1278, and 1298 NFSops, respectively. Why do I not get the requested load?

This may happen when one has reached the server limit for a particular configuration. One needs to determine the bottleneck, and possibly tune and/or enhance the server configuration.

41: How do I increase the performance of our server?

One may need to add, as necessary, one or more of the following: disks, memory, controllers, processors, etc.

Submission of Results

42: We have a valid set of results. How do we submit these results to SPEC?

See the section SFS tools. The new submission tool documentation is in that section.