SPEC SFS 3.0 is the latest version of the Standard Performance Evaluation Corp.'s benchmark that measures NFS file server throughput and response time. It differs from other NFS benchmarks in that it provides a standardized method for comparing performance across different vendor platforms. The benchmark was written to be client-independent and vendor-neutral. Results are validated through peer review before publication on SPEC's public Web site <http:// www.spec.org/osg/sfs97r1/>
Yes. Now that SPEC SFS 3.0 is available, SFS 2.0 licenses are no longer being sold. Results from SFS 2.0 will no longer be accepted by SPEC for publication.
No. Although the benchmarks are similar, they cannot be compared, since SFS 3.0 uses a different file selection algorithm, its results can only be compared with other SFS 3.0 results.
In addition to general code improvements, SPEC SFS3.0 includes three major enhancements: 1. Fixes in the file set selection and workload mechanisms. 2. Support for Linux clients. 3. The benchmark distribution CD contains updated pre-compiled and tested binaries.
The SPEC SFS 3.0 workload is based primarily on a survey of more than 1,000 servers in different application environments. The survey found that 60 percent of these users have similar mixes of NFS operations. The workload in SFS 3.0 more accurately presents the intended workload that was used in SFS 2.0.
SPEC SFS 3.0 has two performance measurement metrics: SPECsfs97_R1.v2 for NFS protocol version 2 and SPECsfs97_R1.v3 for NFS protocol version 3. Both metrics include a throughput measure (in operations per second) and an overall response time measure (the average response time per operation).
No. SFS 3.0 maintains the same metrics that were used in SFS 2.0. The overall response time and peak throughput. The larger the peak throughput the better. The lower the overall response time the better. The overall response time is an indicator of how quickly the system under test responds to NFS operations over the entire range of the tested load. In real-world situations, servers are not run continuously at peak throughput, so peak response time provides only minimal information. The overall response time is a measure of how the system will respond under an average load. Mathematically, the value is derived by calculating the area under the curve divided by the peak throughput.
NFS version 3 has been shipping on systems for more than six years and is available for most systems that support NFS version 2.
There is no correlation; the benchmarks measure totally different aspects of system performance.
SPEC SFS 3.0 is a system-level benchmark that heavily exercises CPU, mass storage and network components. The greatest emphasis is on I/O, especially as it relates to operating and file system software. To obtain the best performance for a system running SFS 3.0, the vendor will typically add additional hardware -- such as memory, disk controllers, disks, network controllers and buffer cache -- to help alleviate I/O bottlenecks and to ensure that server CPUs are used fully.
The benchmark was developed for load-generating clients running in the UNIX environment. But since the load-generating clients execute the benchmark code, SPEC SFS 3.0 can be used to test the performance of any NFS server, regardless of the underlying environment. Porting is required, however, for non-UNIX environments.
Yes, users can measure their own workloads by making changes to the SPECsfs97_R1 benchmark mix parameters to reflect the new measurements. The SPEC SFS 3.0 User's Guide details how this can be done. Workloads created by users cannot, however, be compared with SFS 3.0 results, nor can they be published in any form, as specified within the SFS 3.0 license.
SPEC has written SFS 3.0 to minimize the effect of client performance on SPECsfs97_R1 results.
SPEC SFS 3.0 is a system-level benchmark that requires scheduling substantial resources for testing. SPEC expects member companies to report results in the near future.
Results published on the SPEC Web site have been reviewed by SPEC members for compliance with the SFS 3.0 run and disclosure rules, but there is no monitoring beyond that compliance check. The vendors that performed the tests and submitted the performance numbers have sole responsibility for the results. SPEC is not responsible for any measurement or publication errors.
Yes and no. They are similar to large server configurations, but the workload is heavier than that found on smaller server configurations. SPEC has learned from experience that today's heavy workload is tomorrow's light workload. For some vendors, the configurations are typical of what they see in real customer environments, particularly those incorporating high-end servers. For other vendors, SFS 3.0 configurations might not be typical. Question 17: Do the SFS 3.0 run and disclosure rules allow results for a clustered server? Answer : Yes, cluster configurations are allowed as long as they conform strictly to the even distribution of all resources as defined by the SFS 3.0 run and disclosure rules.
It is important to understand first that SPECsfs97_R1 run rules do not require that the throughput curve be carried out to 40 ms; they only state that the results cannot be reported for a response time higher than 40 ms. There are several reasons why results do not approach the threshold cutoff. Optimally configured servers often will achieve their maximum throughput at response times lower than the cutoff. Additionally, some vendors emphasize maximum throughput while others concentrate on fast response time. It does not indicate a problem with the results if the curve is not carried out to 40 ms, and those reviewing results should not try to predict what the throughput curve might be past the reported point.
The lower response-time threshold reflects advances in server technologies since the release of SFS 1.1 in January 1995.
In addition to a server, a test bed includes several clients and an appropriate number of networks. The server must have enough memory, disks and network hardware to saturate the CPU. The test bed requires at least one network and each network must have sufficient client capacity to saturate the network(s). A minimum of 64 MB of memory is required for each client, although in most cases 128 MB is needed. SFS 3.0 contains enhancements that reduce the amount of memory that the benchmark consumes on each client. Requirements are detailed in the SFS 3.0 User's Guide. To facilitate accuracy of reported vendor results, SFS 3.0 includes an entire NFS implementation. Examples of typical load-generating configurations can be found on the SPEC Web site: <http://www.spec.org/osg/sfs97r1/>.
Hardware setup and software installation time depend on the size of the server and the complexity of the test beds. Many servers require large and complex test beds. The SFS 3.0 software installs relatively quickly. A SPECsfs97_R1 submission from a vendor includes at least 10 data points, with each data point taking about 20 to 30 minutes to complete.
Shared resources that might limit performance include disk controllers, disks, network controllers, network concentrators, network switches and clients.
When submitting results for SPEC review, vendors are required to supply a description of all server tuning parameters within the disclosure section of the reporting page.
SPEC enforces strict storage rules for stability. Generally, RAM disks do not meet these rules, since they often cannot survive cascading failure-recovery requirements unless an uninterruptible power supply (UPS) with long survival lines is used.
Different link types and even different implementations of the same link type might affect the measured performance -- for better or worse -- of a particular server. Consequently, the results measured by clients in these situations might vary as well.
Yes, like SFS 2.0, the new benchmark is scalable as users migrate to faster technologies.
SPEC SFS 3.0 is available now on CD-ROM for $900. Contact the SPEC office: Standard Performance Evaluation Corporation (SPEC) 6585 Merchant Place, Suite 100 Warrenton, VA 20187, USA Phone: 540-349-7878 Fax: 540-349-5992 E-Mail: info@spec.org
The upgrade is free for those who have purchased SFS 2.0 licenses within the last three months and $300 for other SFS 2.0 licensees. Upgrades are available through the SPEC office.
The majority of questions should be answered in the SPEC SFS 3.0 User's Guide. There is also useful information on the SPEC Web site: <http://www.spec.org/osg/sfs97r1/>.
No. NFSv2 and NFSv3 are considered separate workloads and you only need to measure and disclose the ones you want.
Please read the User's Guide in its entirety.
Most of the problems relating to the SPECsfs97_R1 benchmark can be resolved by referring to appropriate sections of the User's Guide, especially the Troubleshooting section.
Looking at the sfslog.* and sfscxxx.* files can give one an idea as to what may have gone wrong. As a last resort, one can contact SPEC. It is assumed that such calls are from people who have read the User's Guide completely, and have met all the prerequisites for setting up and running the benchmark.
One needs to kill all SFS related processes on all clients and on the prime client and re-run the benchmark. The processes are sfs, sfs3, sfs_syncd and sfs_prime.
Information is provided in the sfs_rc file, and this is enforced by the benchmark. If invalid parameter values are selected, the benchmark reports an invalid run.
Read the User's Guide, ping server from client, try mount the server file systems from the client using the client's real NFS implementation, rsh from prime client to the other clients and reverse, run benchmark with one client and one file system.
Unlike SFS 1.1, the sfs_rc file specifies the total number of NFSops/sec across all of the clients used. Because the benchmark only allow specifying an even number of NFSops/sec, the actual requested ops/ sec may be less due to rounding down. For example, 1000 NFSops/sec requested over 6 clients will result in each client generating 166 NFSops/sec for an aggregate of 996 NFSops/sec.
No, the benchmark generates operations using random selection and dynamic feedback to pace correctly. This will result in small difference from the actual requested load.
Study existing results' pages with configuration information similar to your system configuration.
This may happen when one has reached the server limit for a particular configuration. One needs to determine the bottleneck, and possibly tune and/or enhance the server configuration.
One may need to add, as necessary, one or more of the following: disks, memory, controllers, processors, etc.
See the section SFS tools. The new submission tool documentation is in that section.