|
|
What is This Thing Called "SPECviewperf®"? |
|
SPECviewperf® is a portable OpenGL performance benchmark program written in C. It was developed by IBM. Later updates and significant contributions were made by SGI, Digital (Compaq, HP), 3Dlabs (Creative Labs) and other SPECopcSM project group members. SPECviewperf provides a vast amount of flexibility in benchmarking OpenGL performance. Currently, the program runs on most implementations of UNIX, Windows XP, Windows 2000, and Linux.
The OpenGL Performance Characterization (SPECopc) project group has endorsed SPECviewperf as its first OpenGL benchmark. Performance numbers based on SPECviewperf were first published in the Q4 1994 issue of The GPC Quarterly, a printed publication.
SPECopc group member companies have ported the SPECviewperf code to their operating systems and window environments. The SPECopc project group maintains a single source code version of the SPECviewperf code that is available to the public.
Benchmarking with SPECviewperf
SPECviewperf parses command lines and data files, sets the rendering state, and converts data sets to a format that can be traversed using OpenGL rendering calls. It renders the data set for a pre-specified amount of time or number of frames with animation between frames. Finally, it outputs the results.
SPECviewperf reports performance in frames per second. Other information about the system under test -- all the rendering states, the time to build display lists (if applicable), and the data set used -- are also output in a standardized report.
A "benchmark" using SPECviewperf is really a single invocation of SPECviewperf with command-line options telling the SPECviewperf program which data set to read in, which texture file to use, what OpenGL primitive to use to render the data set, which attributes to apply and how frequently, whether or not to use display lists, and so on. One quickly realizes that there are an infinite number of SPECviewperf "benchmarks" (an infinite number of data sets multiplied by an almost infinite number of command-line states).
Real-World Benchmarking
SPECopc project group members recognize the importance of real-world benchmarks. From the beginning, the group has sought benchmarks representative of the OpenGL rendering portion of independent software vendor (ISV) applications. Along these lines, the project group has come up with what it calls a viewset. A viewset is a group of individual runs of SPECviewperf that attempt to characterize the graphics rendering portion of an ISV's application.
Viewsets are generally not developed by the SPECopc project group; they come from the ISVs themselves. Members of the SPECopc project group often "sponsor" the ISV. Sponsorship entails helping the ISV in several areas, including how to obtain the SPECviewperf code, how to convert data sets to a SPECviewperf format, how to use SPECviewperf, how to create SPECviewperf tests to characterize the application, how to determine weights for each of the individual SPECviewperf tests based on application usage, and finally to help offer the viewset to the SPECopc project group for consideration as a standard SPECopc viewset. Any ISV wishing to develop a viewset should contact an SPECopc representative listed on the GPC Organization page.
Currently, there are six standard SPECopc application viewsets:
- 3dsmax-02, based on SPECapc for 3ds max 3.1 configured with the Open GL driver, includes three models containing an average of 1.5 million vertices each, and tests performance of scenes with different levels of lighting.
- drv-09, based on Intergraph's DesignReview model review package, has five different tests.
- dx-08, based on IBM's Data Explorer application, has 10 different tests.
- light-06, based on Discreet's Lightscape radiosity application, has four tests.
- proe-02, based on SPECapc for Pro/ENGINEER 2001, measures two models in three modes - shaded, wireframe and hidden-line removal (HLR).
- ugs-03, based on SPECapc for Unigraphics V17, tests performance based on an engine model containing 2.1 million vertices.
These type of applications typically render large data sets. All six viewsets represent relatively high-end applications.
The input to SPECviewperf is the command line. Here are the current command line options for SPECviewperf 7.1:
Program options:
Viewperf v7.1
Program options:
-polygon -pg <file> : Viewpoint object to be used in the tests
-triangle -tr <file> : Viewpoint object to be used in the tests
-quad -qd <file> : Viewpoint object to be used in the tests
-mesh -mh <file> : Mesh object to be used in the tests
-frameattr -fa <file> : Mesh3 attributes for each frame
-rendermode -rm <mode> : POINT, VECTOR, LINE, POLYGON, TMESH, TFAN,
TRIANGLE, or QUAD - default LINE
-vcriteria -vcrit : AUX Visual selection criteria - EXACT, MIN
- default MIN
-vid <id> : Ask AUX for visual with ID = <id>
-vaccum -vac : Ask AUX for an accumulation buffer visual
-valpha -val : Ask AUX for an alpha buffer visual
-vdepthbuffer -vz : Ask AUX for a depth buffer visual
-vstencil -vst : Ask AUX for a stencil buffer visual
-indirectrender -ir : Render indirect - default direct
-nodither -ndi : Disable dithering
-ortho -or : Parallel/Orthographic projection - default Perspective
-displaylist -dl : Render with display list mode
-vertexarray -va : Render with vertex array mode (OpenGL1.1)
-interleaved -il : Packing vertex data interleaved in memory
-colorper -cp <mode> : FRAME = Color per Frame,
PRIMITIVE = Color per Primitive,
VERTEX = Color per Vertex - default FRAME
-texture -tx <file> : Image for texturing
-texgen -txg <file> <mode> : <file> is image for enviroment mapping
<mode> is SPHERE_MAP, OBJECT_LINEAR, EYE_LINEAR
- default EYE_LINEAR
-texobj -txo <file> : Texture object mode <file> is list of images
-magfilter -magf <flt> : NEAREST, LINEAR - default NEAREST
-minfilter -minf <flt> : NEAREST, LINEAR, NEAREST_MIPMAP_NEAREST,
LINEAR_MIPMAP_NEAREST, NEAREST_MIPMAP_LINEAR,
LINEAR_MIPMAP_LINEAR - default NEAREST
-texenv -te <env> : Texture enviroment, MODULATE, DECAL, BLEND
- default DECAL
-texcomp -tc <num> : Texture components where <num> is 1,2,3, or 4
-default 3
-blend -bl : Enable Blending
-srcblendfunc -sbf : ZERO, ONE, DST_COLOR, ONE_MINUS_DST_COLOR, SRC_ALPHA,
ONE_MINUS_SRC_ALPHA, DST_ALPHA, ONE_MINUS_DST_ALPHA,
SRC_ALPHA_SATURATE - default SRC_ALPHA
-dstblendfunc -dbf : ZERO, ONE, SRC_COLOR, ONE_MINUS_SRC_COLOR, SRC_ALPHA,
ONE_MINUS_SRC_ALPHA, DST_ALPHA, ONE_MINUS_DST_ALPHA,
- default ONE_MINUS_SRC_ALPHA
-linewidth -lw <width> : Linewidth for wire/vector tests - default 1.0
-xwinsize -xws <side> : Size of test windows X dimension - default 700
-ywinsize -yws <side> : Size of test windows Y dimension - default 700
-numframes -nf <num> : Number of frames to be rendered during measurement
Takes priority over -mp
-numilights -nil <num> : Turns on <num> infinite lights - default 0
-numllights -nll <num> : Turns on <num> local lights - default 0
-colormaterial -cm <side> <mode> : <side> is FRONT, BACK, FRONT_AND_BACK - default FRONT
<mode> is AMBIENT, DIFFUSE, EMISSION, SPECULAR,
AMBIENT_AND_DIFFUSE
- default AMBIENT_AND_DIFFUSE
-backface -bf : Cull Backfacing primitives - default off
-frontface -ff : Cull Frontfacing primitives - default off
-singlebuffer -sb : Single buffer mode
-fog -fg : Enable fog
-linesmooth -ls : Enable line antialiasing
-polysmooth -ps : Enable polygon antialiasing
-facetnormal -fn : Use facet normals when lighting
-linestipple -lp : Enable line stipple
-polystipple -pp : Enable polygon stipple
-batch -bt <num> : Batch <num> primitives together per glBegin/glEnd
Valid with POINT, VECTOR, TRIANGLE, and QUADS
-polymodefront -pmf : POINT, LINE, or FILL - default FILL
-polymodeback -pmb : POINT, LINE, or FILL - default FILL
-flat -f : Set shademodel to FLAT - default GOURAUD
-zbuffer -zb : Enable zbuffer for tests - default off
-clip -c : Align object on 3D clip boundary
-lighttwoside -l2s : Light both sides of model
-localview -lv : Define local viewer for lit tests
-normalize -nm : Enable normal normalization
-minperiod -mp <num> : Set minimum testing period in seconds
-mblur <num> : Use motion blur with num being amount of decay
-aa_multi <x> <r> : Full scene antialiasing rendered x times at an
offset of r. r should be tuned to the viewset
-walkthru -wt : Walkthru mode
-geomrepeat -gr : Geometry repeat
-grab <script> : Screen capture
-grabonly -go : Don't run benchmark, just do screen capture
-swap <true/false> : Force byte swapping
-notitle -nt : Disable title screen
-startframe -sf <num> : Set starting frame of a walkthru
-framepause -fp : Pause after each frame waiting for enter key
-newview -nv : Enable new view frustum calculation
-fov <angle> : Set field of view in degrees
-spin x y z dx dy dz : Set rotation initial values and speeds (degrees)
-clipplane : Enables a user clip plane
-nomaterial : Disables material changes if present
-nolight : Disables viewperf lighting
-forcecalibrate : Forces viewperf calibration
-tracelights -tl : Use lights generated in scene
-norejection : Turns off trivial rejection for walk through.
-noframecounter : Turn off display of frame counter
-staticFrameCount -sfc <num> : Count of frames between data edits (default = 10)
-staticBlockCount -sbc <num> : Number of memory blocks changed per edit (default = 3)
-staticEditSize -ses <num> : Amount of frame data changed per edit (default = 0.001)
-staticEditsVisible -sev : Attempt to make edits visible in the frame during execution.
-staticEditsDisabled : Allow all data to be static, no frame edits for entire test.
The -threads option is applicable only if SPECviewperf has been compiled with the C Preprocessor variable MP defined. SPECviewperf should only be compiled this way if the vendor supplied makefile has defined the MP flag (-DMP).
Here is an example of the command line from test #1 of the proe-02 viewset:
viewperf -mh WorldCar6 -fa gradient -tracelights -va -zb -mp 12
-forcecalibrate -xws 960 -yws 960 -grab grab.scr
This command line is providing the following directions:
- invoke the SPECviewperf program
- -mh specifies the format of the data set is mesh format (binary)
- the data set file name is WorldCar6.mh3
- -fa specifies some attributes to be processed per-frame (binary)
- the attributes are stored in gradient.mha
- -tracelights specifies that the lights contained in the command stream should be used
- -va directs SPECviewperf to render using vertex arrays
- -zb enables z-buffering
- -mp 12 will set the minimum period to 12 seconds; SPECviewperf will run this test for at least 12 seconds
- -forcecalibrate forces SPECviewperf to calibrate before running the test. This allows the frame rate to be determined so that sufficient frames are processed during the test for the defined test period.
- -xws 960 -yws 960 this specifies the X and Y dimensions of the SPECviewperf window to be 960x960 pixels.
- -grab grab.scr directs SPECviewperf to capture a section of the screen after completing the performance test.
Upon entering the command line, the system under test will run the benchmark for at least 12 seconds. This is to insure a valid test result. During the test, the SPECviewperf code renders the data set, WorldCar6.mh3, which is the WorldCar model used in the SPECapc for Pro/ENGINEER 2001™, rotating it about the center of the window with the above specified characteristics.
What follows is the output of SPECviewperf 7.1 running the first test from proe-02:
--------------------
Viewset: ProEngineer
Test Number 01: WorldCar6 : Weight 13.50
--------------------
Viewperf Version 7.1
Viewperf Arguments -mh WorldCar6 -fa gradient -tracelights -va -zb -mp 12 -forcecalibrate -xws 960 -yws 960 -grab grab.scr
Month 3
Day 4
Year 2003
Host system HOSTNAME
Operating System Microsoft Windows NT
Operating System Version Version 5.1 (Build 2600: Service Pack 1)
Host Vendor unknown
Host Model AT/AT COMPATIBLE
Host CPU x86 Family 15 Model 2 Stepping 7
Host CPU Count unknown
Host Memory Size (MB) 1023
Host Primary Cache Size (KB) unknown
Host Secondary Cache Size (KB) unknown
Window System Win32
Driver Version Version of GFX driver
OpenGL Vendor Name of company making GFX card
OpenGL Version Version of OpenGL supported by GFX card
OpenGL Extensions List of all supported OpenGL Client extensions
OpenGL Renderer Name of GFX card
OpenGL Client Vendor GFX Card OpenGL client vendor string
OpenGL Client Version Version of OpenGL client supported by GFX card
OpenGL Client Extensions List of all supported OpenGLClient extensions
GLU Version 1.2.2.0 Microsoft Corporation
GLU Extensions List of all supported GLU extensions
Direct Rendering False
Double Buffer True
Stereo False
RGBA True
Color Index Size -1
Red Size 8
Green Size 8
Blue Size 8
Alpha Size 0
Accum Red Size 16
Accum Green Size 16
Accum Blue Size 16
Accum Alpha Size 16
Depth Size 24
Stencil Size 0
Auxiliary Buffer Count 0
Frame BufferLevel 0
Window Width (pixels) 960
Window Height (pixels) 960
Screen Width (pixels) 1280
Screen Height (pixels) 1024
Pixel Format ID 4
Visual Selection Criteria MINIMUM
Number of Execution Threads 1
Geometry File WorldCar6
Input Mode -mh
Minimum Test Period 12.000000
Number of Frames 0
Number of Primitives 319413
Number of Vertices per Frame 3917495
Number of Vertices per Primitive 12.264670
Estimated bytes per frame 0.000000
Static frame count 0.000000
Model edited every 10 frames
Amount model changed 0.001000
Toggle Mode NONE
Batching Count 0
Render Mode LINE
Color per COLOR_PER_FRAME
Orthographic Projection FALSE
Display List FALSE
Vertex Array (OpenGL1.1) TRUE
Interleaved Data FALSE
Clip Geometry FALSE
Walkthrough Mode FALSE
Geom Repeat Mode FALSE
Back Face Cull FALSE
Front Face Cull FALSE
Front Polygon Mode FILL
Back Polygon Mode FILL
Polygon Stipple Enable FALSE
Polygon Antialiasing Enable FALSE
Line Width 1.000000
Line Stipple Enable FALSE
Line Antialiasing Enable FALSE
Number of Infinite Lights 0
Number of Local Lights 0
Color Material Enable FALSE
Color Material Face FRONT
Color Material Mode AMBIENT_AND_DIFFUSE
Facet Normals FALSE
Two Sided Lighting Enable FALSE
Local Viewer Enable FALSE
Flat Shading FALSE
Fog Enable FALSE
Normal Normalization FALSE
User Clip Plane FALSE
Material Changes Disabled FALSE
Attribue Changes Disabled FALSE
Trace Lights Enabled TRUE
Texture Enable FALSE
TexObj Enable FALSE
Texture Generation Mode NO_TEXTURE_GENERATION
Texture File NONE
Texture Minification Filter NEAREST
Texture Magnification Filter NEAREST
Texture Environment Mode DECAL
Texture Components 3
Depth Test Enable TRUE
Blend Enable FALSE
Source Blend Function SRC_ALPHA
Destination Blend Function ONE_MINUS_SRC_ALPHA
Dithering Enable TRUE
Motion Blur Amount 0.000000
Full Scene Antialiasing Redraws 0
Full Scene Antialiasing Jitter Amount 0.000000
Number of frames run: 280, Test period: 12.187000 (sec)
23.0 frames/sec -- -mh WorldCar6 -rm LINE -mp 12 -cp FRAME -ir -va -zb -tracelights -vz
During the calibration period SPECviewperf tests to see how many frames need to be rendered to meet the minimum time period and then runs the test.
The last line is the number of frames per second, which is the fundamental metric for each test. This is the result that is printed in the summary and individual report pages within the GPC News SPECopc section. Also printed in the summary and individual report pages is a single composite number for each viewset. This composite number is derived using a weighted geometric mean methodology (essentially the product of the results of each test raised to the power of the test weightings).
SPECviewperf 7.1
SPECviewperf 7.1 introduces new data changes to discourage optimizations that benefit the benchmark, but might not improve real-world application performance. SPECviewperf 7.1 inserts a small amount of variation at regular intervals within its viewsets. This ensures that the test system examines and processes each frame individually and is more aligned with the behavior of real-world applications. A second frame counter that moves around within the frame has also been added to the benchmark. This provides visual verification of whether frame changes are being rendered. For more information, please read the documentation provided with the SPECviewperf source code.
Rob Putney / Simon Green / Chris Watts / Bill Licea-Kane / Allen Jensen
/ Ian Williams