SPEC logo

SPECweb99 Release 1.02 User's Guide

1 Introduction
2 Installing SPECweb99

2.1 Pre-Installation Checklist
2.2 Client Setup

2.2.1 Installing and running the UNIX client
2.2.2 Compiling UNIX client software (if necessary)
2.2.3 Windows NT Client Setup
2.2.4 Compiling NT client software (if necessary)

2.3 Server Setup

3 Running SPECweb99

3.1 Customizing the rc file

3.1.1 Changeable benchmark parameters
3.1.2 Configuration description parameters
3.1.3 Benchmark Constants

3.2 Running the benchmark

4 Understanding the benchmark screen display

4.1 Test setup summary
4.2 Operational validation
4.3 Test Warnings
4.4 Results from each iteration
4.5 Screen display of ASCII output file

5 SPECweb99 Result Pages and Raw File

5.1 Result pages
5.2 Raw file

6 Troubleshooting

6.1 Server system
6.2 Client system

7 Performance Tuning

8 Submitting results


1 Introduction

SPECweb99 is a client/server benchmark for measuring the maximum number of simultaneous connections that a web server is able to support. The benchmark load is presented by client software on client machines networked to server machines running an HTTP server software.

This document is a practical guide for setting up and running a SPECweb99 test. This user's guide covers some, but not all of the rules and restrictions pertaining to SPECweb99. Before running a test, you should read the complete "Run and Reporting Rules" contained in the kit. For an overview of the benchmark architecture, see the SPECweb99 Whitepaper also contained in the kit.


2 Installing SPECweb99

2.1 Pre-Installation Checklist

Here is a checklist of steps to complete before installing the SPECweb99 benchmark software.

  1. Decide on the hardware configuration to use for the benchmark. You need to decide on both server and client systems. There are no requirements on your choice of clients. For optimal performance, you will need enough clients and networks to saturate the server. (The client and server software can run on the same system, however this will cause poor performance.)
  2. Choose HTTP server software. The SPECweb99 benchmark CD does not contain any HTTP daemon. The user is responsible for obtaining a legal copy of the HTTP daemon. Please note that NCSA HTTPD or Apache (or any other public domain software) are valid HTTP daemons for the purpose of the benchmark.
  3. Make sure you have enough disk space on all the machines.
    1. Client machines. The kit takes about 50MB to install. In addition, you'll need some space for various output files.
    2. Server machine(s). You will need space for the SPECweb99 file set, space for the POST data log, and space for the HTTP server log.
    3. Peak file size formulas for SPECweb99 server (in Mbytes)

      Files Size formula (approximate)
      file_set (25 + ( simultaneous_connections * .66)) * 4.88 Mbytes
      post log simultaneous_connections * 0.06 Mbytes
      HTTP server log consult HTTP server documentation

      The equations given are based on a speed of 400K bits/second and an average request size of 122K bits (based on the file sizes and the Zipf distribution used to select files). Using these values, a single connection can process a maximum of about 3.3 HTTP operations per second.

      The post log formula assumes the default run time of 20 minutes warmup, 20 minutes run-time and 5 minutes rampdown. The post log gets zeroed out between iterations. When calculating space requirements for the HTTP server logs, remember that the server logs do not get truncated between the 3 iterations.

      Note: Logs must be written to stable storage for a valid SPECweb99 benchmark run. Stable storage refers to non-volatile storage media. In the case of solid state disks, they should have battery back-up


2.2 Client Setup

This section describes the steps necessary to setup SPECweb99 clients. Detailed instructions are provided for the two major operating system types: UNIX and Windows NT. The setup instructions for most of the hardware platforms are very similar to the generic setup instructions for the two operating system versions. The SPECweb99 page of the SPEC website may contain additional instructions for particular platforms.

Note: You may use a mix of NT and UNIX based clients in your setup. Furthermore, a UNIX server can be tested with either NT or UNIX clients or a mix. An NT server can also be tested with a mix of either type of clients.

Designate one of your client machines as the prime client machine. The prime client will control the entire test and the other clients, so needs to be able to establish network connections to all the other clients and to the server. The prime client can be used just as the test controller or can serve as both controller and client. The following instructions assume it does both functions.

2.2.1 Installing and running the UNIX client

On all clients, do the following:

  1. Create a new account for the user 'spec'. Consult your operating system's documentation for instructions.
  2. Insert the SPECweb99 CD into the CDROM drive and mount the drive. Make sure the CD is readable by the new 'spec' user. Consult your operating system's documentation for instructions.
  3. Login as user 'spec'.
  4. Install the software You need to have a version of the Java Virtual Machine running on your client system. To install, type:
  5. java setup

    After accepting the license agreement, setup will ask you to "Select your architecture" followed by a list containing those operating systems and architectures for which the CD contains pre-compiled versions the SPECweb99 software. The list has 2 additional choices, none for installing client sources and toolsource for selecting SPEC tools sources.

    If your architecture is not on the list, install none for regular clients or toolsource on the prime client. Follow the instructions for compiling UNIX client software before continuing. Once you have installed the pre-built software or built it from sources you will have the following compiled executables on your client:

    Client executable

    Server software that is built with the client executables

    Prime-client tools

  6. Alternative installation methods: The CD contains a shell script, install.sh, that will install sources on your machine then compile the tools binaries. Follow the directions below for compiling the client executables. The CD also has several tar-and-gzipped precompiled kits that you can unpack to get the client software and tools. The precompiled kits have the suffix .tar.gz. The source files have the suffix .taz (in order to make them acceptable to Windows NT).
  7. Set Maximum Segment Size (MSS) of your network: Make sure that the TCP MSS is set to 1460 bytes (or less) on your system and the Maximum Transmission Unit (MTU) is 1500 bytes. This needs to be accomplished by platform-specific means outside the benchmark code itself. On many platforms, setting the MTU to 1500 automatically sets the MSS to 1460, which is (MTU - 40) bytes.
  8. Note: For a result to be valid the connections between a SPECweb99 load generating machine and the System Under Test (SUT) must not use a TCP MSS greater than 1460 bytes. An MTU of 1500 bytes is the standard packet-size for IP over Ethernet.

  9. Start the client daemon: You will need to start the client daemon on all the clients:
  10. Usage: ./client [options]
           -h    #   this help screen
           -D    #   debugging level
           -p    #   control port to listen on
           -i        used to indicate client is being run out of inetd
           -t    #   daemon idle timeout (set larger than anticipated
                     run)
           -c        start a control session on stdin
           -d        start in background as a daemon
           -m    #   size of shared memory segment (SYSV_IPC version only)
           -w   dir  work directory
           -s    #   Deck size
           -S    #   Deck cache size

2.2.2 Compiling UNIX client software (if necessary)

The SPECweb99 kit includes sources for all the tools and software needed to run SPECweb99. It also includes prebuilt executables for many platforms. Occasionally, however, you may need to build or rebuild the client tools or SPECweb99 client executable. You will need your own C compiler to do build the SPECweb99 tools and executables.

To build the tools, such as specperl install the sources option from the SPECweb99 CD, then the follow the directions in the README file in the <installation-directory>/tools/src directory.

To build the client executables, you must first run configure to build the makefile, then make to build the executables.

SPECweb99 has been written to run on multiple platforms. You need to use the configure script to create a makefile for your specific platform. For the most part, the configure script is smart enough to make the right decisions. But there are cases when the script cannot make the right decisions or the user may want to disable the use of certain features used in the code and instead choose an alternative method.

The following features are available in SPECweb99 and can be used as arguments to the configure script to enable or disable the feature. Enabling or disabling these features affects the way the client program is built.

Type 'configure --help' to see the generic options supported by configure.
Note: It should not be assumed that all these options are available or not available on a particular platform. Please check your product documentation before using these features.

--enable-rlimit
Enable the functions getrlimit() and setrlimit() to view/limit/control the consumption of a variety of system resources by a process and its children. In the case of SPECweb99, if this feature is enabled, it will allow the client program to have more open files and or sockets in a single process. The default behavior is to not use these functions
--enable-posix-threads
If a platform supports POSIX threads, the client program will be built using POSIX threads semantics. This will result in having a multi-threaded version of client instead of a multi-process version.
--enable-pthread_scope_system
When using POSIX threads, enabling or disabling this feature affects the contention-scope of the thread. Enabling this flag will enable a property of a thread (when the thread gets created) by which it will contend among system-wide resources. The default behavior is to not use this feature. Only use with --enable_posix_thread.
--enable-safe-gethostbyname
Enabling this feature means that gethostbyname() is thread-safe and can safely be used if the client is multi-threaded. Enabling this feature will not require enabling the next feature. The default behavior is to not assume the thread safe version of gethostbyname() is available.
--enable-gethostbyname_r
Enabling this feature means the reentrant/thread-safe version of gethostbyname() function, gethostbyname_r() is available on a particular platform and can be used instead of using gethostbyname() . The default behavior is to assume the existence of gethostbyname_r() .
Note: It makes sense to choose only one of safe-gethostbyname or gethostbyname_r depending on your platform.
--enable-safe-usleep
This feature needs to be considered if a platform does not support nanosleep() . In such a situation if usleep() is available on a platform and is thread-safe, it can be used in combination with sleep() . Otherwise the client program will use select() to emulate the sleep-feature. If nanosleep() is not present on a particular platform and if this argument is not passed to the configure script then the source code will use usleep() / sleep() regardless of whether it is thread-safe or not.
--disable-nanosleep
This feature is needed to disable the use of nanosleep() explicitly in order to override the default behavior.

 

Compile the benchmark software by typing the following command.

make all

This will build the following executables:

./client
./Cadgen99/cadgen99
./Upfgen99/upfgen99
./Wafgen99/wafgen99

2.2.3 Windows NT Client Setup

On all clients, do the following steps:

  1. Insert the SPECweb99 CD into the CDROM drive.
  2. Run SPECweb99.exe. This is an InstallShield binary
  3. Alternative installation method: contains 2 .taz files with sources. web99-v100.taz contains the client code and web99-v100-toolsource.taz contains the SPEC tools. The files are tarred and gzipped.
  4. Set Maximum Segment Size (MSS) of your network: Make sure that the TCP MSS is set to 1460 bytes (or less) on your system and the Maximum Transmission Unit (MTU) is 1500 bytes. To do this in Windows NT 4.0:
    1. Run the registry editor (REGEDT32.EXE)
    2. In the HKEY_LOCAL_MACHINE subtree, go to the key SYSTEM\CurrentControlSet\Services\adapterame\Parameters\Tcpip where adaptername is the network adapter that talks to the server
    3. Add a value "MTU" of type REG_DWORD with data of 1500.
    4. Do this for every adapter you will use to connect to the server.
    5. Reboot the machine for the added values to take effect.
  5. Start the client daemon, if you didn't install it as a service. If you didn't install client as a service type client -Z to run the client in the background. Alternatively you may type client -z install and client -z start to run as an NT service

2.2.4 Compiling NT client software (if necessary)

The SPECweb99 kit includes sources for all the tools and software needed to run SPECweb99. It also includes prebuilt executables for many platforms. Occasionally, however, you may need to build or rebuild the client tools or SPECweb99 client executable. You will need Microsoft Visual C++ to build the SPECweb99 executables and tools.

Note: in the following directions, %SPEC% refers to the installation directory.

To build the tools, such as specperl, get the tools sources from the SPECweb99 CD (see alternative installation methods, above) then follow the directions in the README file in the %SPEC%\tools\bin\src directory.

To rebuild the client software, you must install 'client' or 'prime client' from the CD. Both contain all the sources for building. The installation directory will be hereafter referred to %SPEC%. This is one of the environment variables you get after running shrc.bat.

The kit contains 4 separate executables that can be rebuilt. The 4 workspace files are in:

%SPEC%\Win32\client\client.dsw
%SPEC%\Win32\Cadgen99\Cadgen99.dsw
%SPEC%\Win32\Upfgen99\Upfgen99.dsw
%SPEC%\Win32\Wafgen99\Wafgen99.dsw

Note: The last 3 files, Cadgen99, Upfgen99, and Wafgen99 are utilities that get put on the server machine. It is highly unlikely that you will ever need to rebuild these.

To rebuild, open the workspace file, then from the "Build" menu, select either build or rebuild all. Developer Studio will place the resulting executable in a subdirectory called "Release". Make sure you move the new executable to the correct place before preceding. The kit is shipped with the executables in the following places:

%SPEC%\client.exe
%SPEC%\Cadgen99\cadgen99.exe
%SPEC%\Upfgen99\upfgen99.exe
%SPEC%\Wafgen99\wafgen99.exe

2.3 Server Setup (UNIX and Windows NT)

On the server the setup involves installing the HTTP daemon, creating the workload file set that will be accessed by the benchmark, and setting up the binaries for dynamic content.

The following instructions tell you how to run using the provided sample Perl implementation for the dynamic content. You may use your own API implementation in place of the sample Perl implementation. Please refer to the vendor's instructions for installation of the API binaries. The API implementation needs to conform to the specification in the "SPECweb99 Run and Reporting Rules". The API implementation must be submitted with your results and reviewed by the SPECweb99 committee.

  1. Install an HTTP daemon. Take note of the document root directory location. You will need this information for proper installation of other files on the server.
  2. Copy the "server" utilities to the HTTP server's document root directory. The utilities come with the SPECweb99 client kit and were either installed or built by you when you set up the clients. The following paths are with respect to the SPECweb99 installation directory:
  3. Wafgen/wafgen99[.exe]
    Cadgen99/cadgen99[.exe]
    Upfgen99/upfgen99[.exe]
    

    The wafgen99 builds the file_set. The other two utilities are called by the dynamic code during a Reset.

  4. Create the file set. Use the wafgen99 utility provided with the SPECweb99 kit to create the workload file_set on the server.
  5. Usage: wafgen99 [options] [ [startload] targetload ]
               -C dir       Change to 'dir' before creating files
               -h           this usage message
               -n conns     Set targetload for # simultaneous_connections
               -v           Increase verbosity (currently 2 levels)
               -s           If directory exists skip it
               -t           Test Mode, no random characters in files
               -r           If directory exists remove it
               -V           validate files, -V again to validate all
               -f load      Start directory load (# simultaneous_connections)
               -F num       Start directory number

    This will create a file_set directory tree. The SPECweb96 wafgen utility and the file set it creates are not usable for the SPECweb99 benchmark.

  6. Copy the sample Perl implementation to HTTP daemon root directory The sample Perl script specweb99-cgi.pl implements the dynamic (GET and POST) portion of the workload.
  7. You will need Perl 5.004 or above in order to run this script. You may install your own Perl, or use specperl that comes with the SPECweb99 kit. See the client setup notes for your operating system for information on installing the spec tools.

    UNIX only: Edit the top line of the file to point to perl or specperl on your system.

  8. Start the HTTP daemon Start the HTTP server daemon. Consult your HTTP server documentation for instructions.
  9. Configure your HTTP daemon to handle CGI perl requests Make sure that your HTTP daemon is configured properly to run perl CGI code. This may consist of enabling CGI operations, associating perl or specperl with certain files, or placing the specweb99-cgi.pl in a particular place. Consult the documentation for you web server for instructions.
  10. Test using telnet to make sure that your setup works. You should try some telnets to make sure that your HTTP server is working, that your clients can access your server and that your Perl or API implementations run properly. The following examples show how to do this. They assume that the SPECweb99 file_set created by wafgen99 resides directly under the HTTP server document root, that you installed the sample perl implementation, and specweb99-cgi.pl is in the cgi-bin directory underneath the document root.
  11. Connect to server and try a static get.

            telnet <servername> 80
            GET /file_set/dir00000/class0_0
    

    Connect to server and try a dynamic GET

            telnet <servername> 80
            GET /cgi_bin/specweb99-cgi.pl?/file_set/dir00000/class0_0
    

    Note: This is not the complete set of operations in the test. The manager will execute one of each kind of transaction prior to launching a test. The manager will halt if it finds any errors.


3 Running SPECweb99

SPECweb99 is controlled by the manager script on the prime client. It reads a resource configuration (rc) input file. To run SPECweb99, customize the rc file to reflect your benchmark setup, then run the test from the prime client with the manager script.

3.1 Customizing the rc file

Modify the rc file on the prime client. The rc file in the SPECweb99 kit is a template, containing every variable to describe a test. You may name the rc file anything you wish (and create as many as you like). The rc file has 3 main sections:

3.1.1 Changeable Benchmark parameters

SIMULTANEOUS_CONNECTIONS
Sets the number of load generators for the tests to be run. You may specify one value or a list of values. manager will run a separate test for each value in the list. During each test, N measurements will be taken, where N is the value assigned to ITERATIONS. If you use a list, each value results in a separate test with it's own output files, unless ITERATIONS=1. When ITERATIONS=1, all the values in SIMULTANEOUS_CONNECTIONS get reported in a single set of output files. This is for testing and tuning purposes only. For reporting purposes ITERATIONS must be set to 3. The form <first>-<last>x<increment> can be used to run a series of tests starting at <first>, ending at <last> and incrementing by <increment>.
CLIENTS
SERVER
Clients: client-name(port)[speed]{servername}
Server: server-name as seen by the prime-client
You may use multiple network adapters in a client by specifying that client multiple times with different {servername} settings and using the @@SERVER@@ form of URL_ROOT. For example:
       CLIENTS=box1{sut-fddi1} box1{sut-fddi2}
The speed setting gives relative speed of each client and the port setting is the port through which the prime-client communicates with that client. For example:
       CLIENTS=client1[2] client2(2223)
client1 is twice as fast as client2, client2 will open port 2223 to receive its messages from the prime client.
URL_ROOT
URL for the directory that contains "file_set". For example, if your file_set directory is in the web-server root then use "URL_ROOT=http://server1" (where server1 is the name of your web server). If you built the file_set in a directory named "specweb99" then use "URL_ROOT=http://server1/specweb99". To use multi-adapter clients use: URL_ROOT=http://@@SERVER@@/specweb99
DYNAMIC_ROOT
Or
DYN_GET_SCRIPT
DYN_CAD_SCRIPT
DYN_POST_SCRIPT
DYN_CMD_SCRIPT
The URL of the dynamic program(s) (e.g. CGI executable, ISAPI module, NSAPI module, etc). If your file_set directory is not in the web-server root then add a question mark and the URL to the directory that contains file_set.

For example, if file_set is in a directory named "specweb99" then you would use
" DYNAMIC_ROOT=http://server1/cgi-bin/specweb99-cgi.pl?/specweb99".

You may also use separate URL's for each class of DYNAMIC operation supported: POST, standard dynamic GET, custom ad rotation (CAD) GET and Commands (fetch, reset). You may set any or all of the 4 separate variables. The unset variables will default to DYNAMIC_ROOT.

DYN_CGI_SCRIPT
You must specify the URL for DYN_CGI_SCRIPT that is a CGI-based implementation of the standard dynamic GET. This will not default to DYNAMIC_ROOT.
HTTP_PROTOCOL
Used to specify HTTP version 1.0 or 1.1. If you leave this commented out then manager makes a connection to your web server to determine what version of the protocol to use. You can set the HTTP_PROTOCOL variable in order to force SPECweb99 to use a particular version of HTTP as the primary protocol. The primary protocol is used for 70% of requests (either HTTP 1.0 Keep-Alive or 1.1 Persistent connection request. The remaining 30% are HTTP 1.0 non-Keep-Alive rquests. The valid settings are "HTTP/1.0" or "HTTP/1.1".
WARMUP_TIME
Amount of time in seconds to run to get to steady state prior to the 1st measurement in the test. This time is intended to warm up any caches before taking a measurement. The minimum legal value is 1200 seconds.
WAIT_TO_BEGIN
Time to sleep in seconds before starting each iteration. This delay is taken after the prime client tells the clients about the workload and before telling them to start working. If test startup isn't synchronizing properly, this might need to be increased.
OUTPUT_NAME
All outputs get put in ./results/<OUTPUT_NAME>.nnn.xxx where nnn is an ascending test number and xxx is the output format type (i.e. raw, ps, pdf, asc, html).
RSH,CLEANUP_WORK,CLIENT_USER,CLIENT_DIR
Optional parameters used to control removal of SPECweb99 client work files from client directories before the benchmark starts. The use of this function is OPTIONAL and will not affect the validity of the run or the results. The SPECweb99 client leaves work files in a work directory after each run, and these files can accumulate and interfere with normal operation. By configuring this option, you can reduce or eliminate any such problems. There are usage examples in the sample RC file

3.1.2 Configuration description parameters

The Configuration description section should contain a full description of the testbed. It should have enough information to repeat the test results, including all necessary hardware, software and tuning parameters. All parameters with non-default values must be reported in the Notes section of the description.

The configuration description has 8 categories.

Each category contains variables for describing it. For example, Hardware contains variables for CPUs, caches, controllers, etc. If a variable doesn't apply, leave it blank. If no variable exists for a part you need to describe, add some text to the notes section. The notes sections can contain multiple lines by adding a 3-digit suffix to the variable. For example, the notes_http variable can become notes_http001, notes_http002, etc.

The rc file included in the kit contains examples for filling out these fields.

3.1.3 Benchmark Constants

Any changes to the benchmark constants will invalidate the run. However, changing these constants can be useful for tuning the system. Note: the -C switch in manager will reset these constants for a test run, so you needn't re-edit the rc file to get a legal run.

Test timing parameters

RAMPUP_TIME
"warmup" time before 2nd through nth iteration of the test (default = 300)
RUN_TIME
measurement time for each iteration (default = 1200)
RAMPDOWN_TIME
time after each iteration of the test (default = 300)
ITERATIONS
number of times to repeat the measurement (default = 3)

Mix parameters - control the percentage of requests of each type. Set to values between 0 and 1.

DYNAMIC_CONTENT
Percent of overall dynamic requests (default = 0.3 for 30% dynamic requests)
DYNAMIC_POST
Percent of dynamic requests that are posts (default = 0.16).
DYNAMIC_CAD_GET
Percent of dynamic requests that are Custom Ad Rotation GETS (default = .42)
DYNAMIC_CGI_GET
Percent of dynamic requests that are GETs calling the CGI code (default = .005)
Default transaction mix

Request type Percentage of mix
Static GET 70.00%
Standard Dynamic GET 12.45%
Dynamic GET with Custom Ad Rotation 12.60%
Dynamic POST 4.80%
Dynamic GET calling CGI code 0.15%

Other parameters

MAX_FILE
Number of files in each class. (default = 8)
USER_LINE_SPEED
Target line speed to emulate in bits/second (default = 400000)
USER_SPEED_LIMIT
Speed in bits/second below which a connection is deemed too slow. For a valid test, 95% of the connections must operate faster than the USER_SPEED_LIMIT (default = 320000)
REQUESTS_PER_KEEP_ALIVE
Average number of HTTP requests to do per "keep-alive" HTTP connection. For example, if a connection randomly selects 5 requests to keep alive it will open a connection to the web server, request a page, read the server response, do this request+read 4 more times, and then close the socket connection. Each of the 5 GETs on that socket will be for different URLs. (default = 10)
PERCENT_KEEP_ALIVE_REQUESTS
Percentage of GETs that to retrieve via "keep-alive" connections. Note that this is NOT the percentage of connections that will have the keep-alive header. (default = 0.7)
ABORTIVE_CLOSE
Set to 0 or 1 to indicate whether or not to use abortive closes. (default = 0)

3.2 Running the benchmark

Before running the benchmark check that the following are true:

Before running the benchmark, you must set up the SPEC environment by running the shrc file on the prime client. You only need to do this once per login session.

On NT:

On UNIX:

From the session you set up with shrc start the benchmark by executing the manager script with specperl:

        specperl manager [options]

manager tests all the operations on the server, tells all the clients what to run, tells them to run it, waits until the clients finish, then collects the results and creates the results summary in various output formats.

Usage: specperl manager [options] [option=val...] [file...] 
    -h           this help message
    -v   #       verbosity level
    -o   types   Produce output in the specified types
    -I           Do not abort on certain classes of errors
    -R           files are reformatted and not used as config files
    -l           Turn off client logging in log-client.nnn file
    -C           set all parameters need for a compliant run
    -D           turn on details in the ASCII output

The default input rc file is rc. You may specify your own rc file or files on the command line. If multiple rc files are used, they are run one after another, producing separate output files for each.

The output types available are: asc, html, ps, pdf, screen and raw. Results submitted to SPEC will appear on the SPEC website in the asc, html, ps and pdf formats. The manager script produces all output formats by default. It puts them into the results directory. In addition, the test details that appear on the screen get written to a res.<nnn> in the results directory. The 'detailed ASCII' output generated by the -D switch contains these same details. The 'detailed ASCII' format is for review, only, not publication.


4 Understanding the benchmark screen display

The manager script outputs data to the screen as it runs. This output also appears in the results/res.nnn file. The manager script writes the same data summarized on a per-client basis into the results/log-client.nnn file. The following is an annotated res file.

4.1 Test setup summary

The test preamble section gives information about the whole run, including:

The following example shows a problem with pdf and ps outputs.

Opened logfile results/res.248
WARNING: Unable to find PSPDF.pm -- no PDF or PS output will be made

Opened client logfile results/log-client.248

CLIENT LIST: 
  Client 1: name=localhost, port=, speed=1, host=
  Client 2: name=csg161, port=, speed=1, host=

4.2 Operational validation

Next, manager validates all the types of operations. When all the tests pass, you know that:

If any of these tests fail, manager will abort and let you correct the errors. For debugging purposes, you can set a failing operation type to 0% in the rc file to skip both the test and issuing of this operation. For example, if you haven't yet implemented the dynamic POST, set 'DYNAMIC_POST=0' in the rc file to run without it. In the following example, the CGI GET call fails.

Validating all paths for rc.short

  Checking Reset command with 'http://bbb116/specweb99/isapi/newz-xmit.so?comman
d/Reset&maxload=100&pttime=100&maxthread=100&exp=1,100&urlroot=http://bbb116/spe
cweb99'

  Checking static GET with 'http://bbb116/specweb99/file_set/dir00089/class3_6'

  Checking dynamic GET with 'http://bbb116/specweb99/isapi/newz-xmit.so?/specweb
99/file_set/dir00089/class1_6'

  Checking dynamic CGI GET with 'http://bbb116/specweb99/wrongcgi/specweb98-newc
gi?/specweb99/file_set/dir00089/class1_6'
  **ERROR**: Can't fetch file http://bbb116/specweb99/wrongcgi/specweb98-newcgi?
/specweb99/file_set/dir00089/class1_6 with a dynamic CGI GET request
Is DYNAMIC_ROOT (or DYN_xxx_SCRIPT) configured  correctly?
Error: 404 Not found

  Checking dynamic custom ad (CAD) GET with 'http://bbb116/specweb99/isapi/newz-
xmit.so?/specweb99/file_set/dir00089/class1_6'

  Checking dynamic POST with 'http://bbb116/specweb99/isapi/newz-xmit.so?/specwe
b99/file_set/dir00089/class1_6'


****** FATAL ERRORS FOUND *******

4.3 Test Warnings

The manager script warns you if you have reset any of the benchmark constants. For example:

WARNING: RUN WILL BE INVALID Invalid test: RUN_TIME is 30, must be 1200 Invalid test: RAMPUP_TIME is 30, must be 300 Invalid test: RAMPDOWN_TIME is 10, must be 300 Invalid test: WARMUP_TIME is 30, must be greater than 1200

4.4 Results from each iteration

The results from each iteration contains the following sections:

4.5 Screen display of ASCII output file

Lastly, you will see ASCII output format on the screen and in the res file. This is one of the formats available on the SPECweb99 result pages website. See the section on Result Pages for a description of this page.
Note:  Creation of the pdf output causes warnings to be displayed on some platforms. The following warnings are harmless:

Bad free() ignored at /usr/users/spec/spec/web99/bin/lib/site_perl/5.005/pdflib.pm line 282.
PDFlib warning: Symbol font Symbol doesn't use builtin encoding


5 SPECweb99 Result Pages and Raw File

The manager script can generate the output in four formats: ASCII, HTML, PDF and PostScript. They get written to the results directory with the names output.<nn>.[asc | html | pdf | ps]. In addition it creates an output.<nn>.raw file. Only the .raw file is needed to submit a result to SPEC, so errors in the creation of other output formats can generally be ignored.

5.1 Result pages

The result pages contain the following elements.

Hardware and Software information
The hardware vendor and model name and the HTTP Software vendor and software name/version.
SPECweb99 Metric
The median value from the three iterations. The metric for the benchmark is SPECweb99. It represents the number of simultaneous connections that a server can sustain at a given bitrate.
Performance of the 3 iterations (HTML and ASCII formats only)
The result from the 3 iterations of the run.
Availability and Configuration information
All the information from the Configuration description entries in the rc file.
Notes and Tuning Information
The Notes entries from the configuration description. This section contains detailed information about every component that used to generate the results. It includes information of both the software and hardware components. All system configuration information required to duplicate the performance results must be reported. This section also includes availability dates for the different components used.
Errors (if any)
Any errors encountered in the test inputs or results. To be valid, the test must have no errors reported in this section. Any errors will cause the phrase 'Invalid Test' to appear either next to the SPECweb99 result number (ASCII and HTML formats) or as a "watermark" across the page (PS and PDF formats).
Performance Details if the 3 iterations (ASCII detailed output only)
manager will produce a more detailed summary of the 3 iterations if you use the -D switch. This level of detail is for review purposes only and will not appear on the output pages displayed on the SPEC web site.

Sample output pages are included with the kit ( ASCII, ASCII-detailed, HTML, PDF, PS).

5.2 Raw file

The raw file contains all the inputs from the rc file and results from the test. You can make any of the other formats from the raw file with the manager -R command. The raw file is submitted to SPEC.


6 Troubleshooting

If problems are encountered, the following checklists may help identify the cause:

6.1 Server System

6.2 Client system


7 Performance Tuning

The first steps to tuning any benchmark, is analyzing the workload and look for possible bottlenecks in the configuration. There are a number of factors or bottlenecks that could affect the performance of the benchmark on your system.

These tuning tips are for configuring the system to generate the best possible performance numbers for SPECweb99. Please note that these should not necessarily be used to configure and size real world systems. Configuring and sizing real world HTTP server systems is an extremely difficult exercise. SPECweb99 quantifies certain important aspects of web server performance. However, other workload components may be involved in a given real world web server, e.g. directory name service, ftp, mail, news, etc. Due to the lack of accurate qualitative sizing guidelines, experience or even guesswork often becomes the determining factor in arriving at a configuration. Even experience of a similar environment can be misleading due to the huge variation in user and workload profiles across different end user sites.

Memory
Memory is an important consideration for this benchmark. Since the file set is read only (the dynamic code does not alter any of the files), there is potential for gain in performance from caching more of the file set in memory.
Network
SPECweb99 is a network intensive benchmark and to ensure accurate evaluation of your server, you may want to optimize the behavior of the network. The type of network subsystem needed depends on the peak bandwidth.
TCP/IP
HTTP is run over TCP/IP. Many TCP/IP implementations have tuning parameters for setting various buffer sizes and protocol features.
Disk I/O
The only writes that occur on the system are the ones to the HTTP server log and the ones to the POST log. Since all accesses to the server need to be logged as defined in the run rules, there is a potential for the log disk to become the bottleneck. Consider striping the log disk (in hardware or software) if the log disk utilization appears high.
Dynamic Content Implementation
The implementation used to handle the dynamic content portion of the benchmark can have a major impact on the performance. The CGI request portion of the benchmark must use CGI to form the reply, however the other dynamic requests don't have to. Many HTTP daemons support programming APIs that have much lower overhead for calling and for argument passing.

Note: The SPECweb99 kit supplies a sample implementation that is operational and correct, but is not a good choice for high performance.

HTTP Server Software
This is a critical component in determining the performance of the system. This will vary with the different HTTP daemons available. Please refer to the tuning guide for your particular HTTP daemon for guidance on tuning for the SPECweb99 workload.

Many HTTP daemons have companion products to accelerate static pages. These products typically use caching to return pages from memory, rather than disk

Client tuning
You will need to monitor the client CPU utilization to make sure that you have enough client power to saturate the server system. A rule of thumb to begin with is to have client systems operate at the 50% CPU utilization level. This will ensure that you are not bottlenecked on the client side.


8 Submitting results

Once you have a successful run, you can submit the result to the SPEC web committee for review by mailing the output.<nn>.raw file and the source code for the dynamic content implementation to subweb99 @ spec.org. When mailing the raw file, include it in the body of the mail message, don't send it as an attachment, and only mail one result per email message.

Save the HTTP daemon log file from the benchmark run as you may be asked to provide part of the log for review by the committee.

Note: The raw file uses the configuration and parameter information in the rc file. Please edit your rc file with the correct information prior to running the benchmark for submission.

Every submission goes through a two-week review process. During the review, members of the committee may ask for further information/clarifications on the submission. Once the result has been reviewed and approved by the committee, it is displayed on the SPEC web site at www.spec.org .