SPEChpc™ 2021 Utilities

$Id$ Latest: www.spec.org/hpc2021/Docs/

Contents

Introduction

configpp

convert_to_development

dumpsrcalt

extract_config

extract_flags

extract_raw

flag_dump

go (Unix only)

index (UNSUPPORTED)

makesrcalt

ogo (Unix only)

port_progress

printkids.pl (UNSUPPORTED)

printpath.pl

rawformat

Editing rawfiles

Continued fields

test_date

specdiff

specinvoke

specmake

specperl

specpp

specrxp

spec<...>sum

spectar

specxz

sysinfo (UNSUPPORTED)

toolsver

Introduction

This document describes various useful utilities that ship with SPEChpc 2021.

configpp

configpp preprocesses a config file.
The preprocessor is automatically run every time that you use runhpc.
Running it separately is a good way to check whether your config file macros do what you hope they do.

Overview
configpp --config=file [--define symbol[=value]]... [--output=file]
configpp -c file       [-S symbol=[value]]...       [-o file]
Caution Try not to confuse with specpp
Don't say
You meant
configpp file  <— wrong
configpp -c file
configpp commonly-used options
-c file Same as --config
--config=file Config file to preprocess. If none specified, will look for default.cfg.
--define symbol[=value] Set the symbol to value. If you do not specify a value, behaves as an empty string in text contexts, 1 in numeric contexts, or true in logical contexts (because in Perl, 1 is true).
-o file same as --output
--output=file Where to send output. If not specified, will be sent to standard output. The suffix .cfg will be added if you have not already done so.
-S symbol=value same as --define
See also --help for additional options

Tips and Traps

q. Why did I get this message? Input file "-S" not readable

a. You wanted to preprocess a config file, but instead of typing configpp you said specpp.

q. Why did I get this message? Neither config file 'default.cfg' nor 'default.cfg.cfg' exist

a. You forgot to put --config= or -c in front of your config file name. Therefore,
$SPEC/config/default.cfg was tried, but it does not exist.

q. Why does it seem like the output of configpp is completely different than my input? Nothing matches at all!

a. You fell into the previous trap, only worse:
$SPEC/config/default.cfg actually exists on your system.

For much more information on the preprocessor, see its chapter chapter in SPEChpc 2021 Config Files.

convert_to_development

convert_to_development removes the SPEC safety checks which normally would verify the integrity of benchmark code, data, and tools.
Before using this utility, you should carefully consider the Tips and Traps below.

Usage
$ convert_to_development

Development tree conversion: /tmp/hpc2021

After this conversion, the benchmark tree named above will no longer be
usable for reportable runs.  In return, all of the safety features that
ensure that sources, data, and tools are not changed are removed.

This conversion is not reversable.  In order to regain the ability to do
reportable runs, it will be necessary to do a complete new reinstallation in
another location; re-running install.sh or install.bat will not be sufficient.

Do you still want to continue with this conversion?
yes
Successfully performed development tree conversion.
$ 

An academic or research user might consider using convert_to_development while investigating performance effects of alternate source code and data sets.

Tips and Traps

Not reportable Warning: It is not possible to use --reportable in a development tree.
All reports are marked Invalid.
Fair Use Measurements from a development tree are not official SPEC metrics. Any public use is subject to disclosure requirements, which are described in SPEC's Fair Use Rule.
One-way street Warning: After a tree is converted to development, it cannot be converted back. To do reportable runs, you will need to re-install, in a new location, using your original SPEC distribution media (ISO image).
Binaries not transferable Benchmark binaries built in one type of tree (reportable or development) cannot be used in the other type unless you set verify_binaries=no.
Alternatives Before using convert_to_development, you may want to check out use of a build sandbox, or the document Avoiding runhpc.

dumpsrcalt

dumpsrcalt displays the contents of a src.alt (a compressed tar file created by makesrcalt), without necessarily having to apply it.

Overview
dumpsrcalt [--summarize] src.alt 
See also dumpsrcalt --help for additional options

To find out what is in a src.alt, it may be easiest to just apply it.
Or, you can use dumpsrcalt, which displays the differences in a form that will be familiar to users of patch utilities. To find out more about the utility, please see its --help.

extract_config

extract_config reads a rawfile, decodes the config file that produced it, and writes the config file to a file or to standard output.

Overview
extract_config rawfile
See extract_config --help for options

The extracted config file will have these differences vs. the original:

An extracted config file will exactly match a config file as printed by rawformat --output_format=config with one exception: if you have edited the raw file, it will not include your edits.

Example: first, a config file is extracted:

$ specperl -I$SPEC/bin/harness $SPEC/bin/tools/extract_config $SPEC/result/hpc2021_sml.009.small.ref.rsf
Wrote "hpc2021_sml.009.small.ref.cfg"
$ mv hpc2021_sml.009.small.ref.cfg from_extract_config
   

The extracted config file was moved out of the way, so it won't be over-written when we run rawformat:

$ specperl $SPEC/bin/formatter/rawformat --output_format=config,nocheck $SPEC/result/hpc2021_sml.009.small.ref.rsf | grep format:
    format: cfg -> hpc2021_sml.009.small.ref.cfg, hpc2021_sml.009.small.ref.orig.cfg
   

If you edit a rawfile, the .orig.cfg shows what the config file looked like prior to editing. It matches the extracted config file (with the exception of having one extra line).

$ diff hpc2021_sml.009.small.ref.cfg from_extract_config 
454a455
> 
$  

extract_flags

extract_flags reads a rawfile, decodes any flags files that are stored within it, and writes them to one of several destinations.

Overview
extract_flags rawfile
See extract_flags --help for options

A flags file has a concept of a suggested filename. When writing an output file, that is the first preference.
A second choice would be to match the name of the input file.
A final choice would be to specify an arbitrary new name.
Each of these is shown in the examples below (use --help for a description of the switches).

$ extract_flags hpc2021_sml.009.small.ref.rsf 
Reading "hpc2021_sml.009.small.ref.rsf"
Wrote "gcc.xml"
$ specperl -I$SPEC/bin/harness $SPEC/bin/tools/extract_flags $SPEC/result/hpc2021_sml.009.small.ref.rsf
Reading "result/hpc2021_sml.009.small.ref.rsf"
Wrote "result/nv2021_flags.xml"
$ cat $SPEC/result/hpc2021_sml.009.small.ref.rsf | specperl -I$SPEC/bin/harness $SPEC/bin/tools/extract_flags --nofilename --basename=rutabaga
Reading from stdin
Wrote "rutabaga.xml" 
$

extract_raw

extract_raw reads an HTML, PDF, or PostScript report and extracts the rawfile that generated it.

Usage
extract_raw <report>
Caution You might or might not be able to extract from an HTML file if it was saved from a browser.
Try "Save as Web Page, HTML only", not "Save as Web Page, complete".

The raw file is the most important of the outputs.
If you have it, you can re-generate any of the others using rawformat.

q. I lost my rawfile. Do I have any hope?
a. Maybe. Did you save an HTML, PDF, or PostScript report?
$ ls
$ specperl -I$SPEC/bin/harness $SPEC/bin/tools/extract_raw $SPEC/result/hpc2021_sml.009.small.ref.pdf
result/hpc2021_sml.009.small.ref.pdf: Wrote "hpc2021_sml.009.small.ref.rsf"
$ specperl $SPEC/bin/formatter/rawformat --output_format=ps,html,txt,csv,config,nocheck hpc2021_sml.009.small.ref.rsf > /dev/null
$ ls result/hpc2021_sml.009.small*
result/hpc2021_sml.009.small.ref.cfg         result/hpc2021_sml.009.small.ref.orig.cfg  result/hpc2021_sml.009.small.ref.txt
result/hpc2021_sml.009.small.ref.flags.html  result/hpc2021_sml.009.small.ref.pdf       result/hpc2021_sml.009.small.test.rsf
result/hpc2021_sml.009.small.ref.html        result/hpc2021_sml.009.small.ref.rsf       result/hpc2021_sml.009.small.test.txt
$  

flag_dump

flag_dump reads an XML flag description and creates a human-readable HTML version from it.

Usage
flag_dump <XML flags file>...
         
Caution The utility assumes that you are fomatting a user flags file.
If you try to use it for one of the SPEC-provided flags files, it will probably refuse, because SPEC's files include mandatory flags, which user flags files are not allowed to use.

If you are writing a flags file, you should both:

  1. Verify XML correctness with specrxp.
  2. Verify HTML correctness of its included HTML components:

Example:

$ ls result/*.xml
result/nv2021_flags.xml
$ specperl -I$SPEC/bin/harness $SPEC/bin/tools/flag_dump  $SPEC/result/nv2021_flags.xml
Processing result/nv2021_flags.xml...wrote nv2021_flags.html
$  

go (Unix only)

go provides a handy alias to navigate a SPEChpc 2021 directory tree.
If you just say go (without any argument) your directory is set to $SPEC.
Other destinations are described in the table below.

Summary
go destination  [subdir]...  
Pre-requisite Source shrc or cshrc to set paths
Available destinations
top              $SPEC
bin              $SPEC/bin
config           $SPEC/config
Docs|docs        $SPEC/Docs
result           $SPEC/result

nnn.benchmark    $SPEC/benchspec/CPU/nnn.benchmark
nnn              Same, if nnn matches a benchmark
substring        Same, if substring matches the start of a benchmark name;
{build|exe|run}  If you are already under some  nnn.benchmark, goes to
                 nnn.benchmark/{build|exe|run} 

Notes

Examples

$ go
/spec/hpc2021/
$ go result
/spec/hpc2021/result
$ go 605 exe
/spec/hpc2021/benchspec/HPC/605.lbm_s/exe
   

If you abbreviate a benchmark name, the first matching benchmark is used.
Below, it would be easier to just type go 635.

$ go weather exe
/spec/hpc2021/benchspec/HPC/635.weather_s/exe
$ go tealeaf_s exe
/spec/hpc2021/benchspec/HPC/618.tealeaf_s/exe
$ go build
/spec/hpc2021/benchspec/HPC/618.tealeaf_s/build
   

You can string subdirectories together.

$ go bin lib Math BigInt
/spec/hpc2021/bin/lib/Math/BigInt
$   

index (UNSUPPORTED)

index prepares an HTML index for a set of results.

Usage

Populate a directory with results, then say:

      $SPEC/bin/scripts.misc/index
Caution This utility is officially unsupported.

As you are preparing sets of results, you might find it handy to use scripts.misc/index. The utility is officially unsupported because its primary intent is for use at the SPEC website; features may therefore be added, subtracted, or adjusted for convenience of the SPEC webmasters, rather than for user convenience. It might work for you; or you might use it as a basis for writing your own utility. You are welcome to try it out, but please do not submit support requests against it.

In this example, a SPEC documentation writer has lightly disguised his laptop as the fictional TurboBlaster 9000, and proceeds as if preparing for publication. He picks the runs that would be prepared for publication review, and copies them to a separate directory:
$ cd $GO/result
$ mkdir prep_for_pub
$ cp $SPEC/result/hpc2021_sml.009.small.ref.rsf prep_for_pub/TurboBlaster9000.small.mpi.ref
$ cp $SPEC/result/hpc2021_sml.010.small.ref.rsf prep_for_pub/TurboBlaster9000.small.omp.ref
$ cp $SPEC/result/hpc2021_sml.011.small.ref.rsf prep_for_pub/TurboBlaster9000.small.acc.ref
$ cp $SPEC/result/hpc2021_sml.014.small.ref.rsf prep_for_pub/TurboBlaster9000.small.tgt.ref
$ cd prep_for_pub/
$ vi *rsf 

As noted in the section on rawformat, a result rawfile may be edited to update reader fields, such as the system description. After doing so, the results are formatted, and the index is generated.

$ rawformat --flags $SPEC/config/flags/gcc.xml \
  -output_format=all *rsf > rawformat.log             
$ $SPEC/bin/scripts.misc/index
$ ls index.html
index.html
$ 

makesrcalt

makescralt packages alternative source into a compressed tar file, known as a src.alt.

Overview
mkdir         nnn.benchmark/src/src.alt/name
cp sources... nnn.benchmark/src/src.alt/name
cp README     nnn.benchmark/src/src.alt/name
makesrcalt    nnn.benchmark name
Caution SPEC must approve your src.alt before you publish any results using it.
makesrcalt commonly-used options and operands
--usewith <min>:<max> Versions eligible to use the alternate source.
If not specified, defaults to the currently-installed version.
nnn.benchmark

The benchmark that uses this alternate source.

<srcaltname> A name for your alternate source.
For convenience, make it short.
Use only letters, digits, and underscores.
See also --help for additional options

SPEC will post approved SPEChpc 2021 src.alts at www.spec.org/hpc2021/src.alt.
Users apply them via the config file src.alt option.

src.alt Development

If you believe that changes are needed to SPEC source code, here is the process to follow.

  1. Develop your changes in a build sandbox.
  2. Determine where the original benchmark sources are located. The Tiny, Medium and Large SPEChpc benchmarks share content that is located under a corresponding Small benchmark (6nn.benchmark_s).

  3. Create a directory nnn.benchmark/src/src.alt/name with your modified sources.
  4. Create a draft README in that directory. See previous postings at www.spec.org/hpc2021/src.alt.
    In your draft, kindly do NOT claim that it is approved; the release manager will do that at the last minute.
  5. Use makesrcalt as shown above.
  6. Test it with all workload sizes.
  7. If the sources are used by more than one benchmark, test all sizes for other affected benchmarks.
  8. Contact SPEC to get the src.alt approved.

makesrcalt Example

In this example, a comment is added to a source file:

$ pwd
/hpc2021/benchspec/HPC/605.lbm_s/src/src.alt/addAcomment
$ ls
README         lbm.c 
$ diff -u ../../lbm.c lbm.c 
80a81,83
> // Test src.alt example
> // This is only a test
>
$ specperl -I$SPEC/bin/harness $SPEC/bin/tools/makesrcalt 605.lbm_s addAcomment
Making 'addAcomment' src.alt for 605.lbm_s

Making differences and computing hashes...
Writing control file...
Gathering files...
Wrote /hpc2021/605.lbm_s.addAcomment.hpc2021.kit0.2.tar.xz

$ (Output subsetted for brevity.) 

ogo (Unix only)

ogo provides a handy alias that navigates *both*

ogo automatically picks a tree and a directory depending on whether you want to move to a read-only directory (e.g. lbm src) or an output directory (e.g. lbm build).

Summary
ogo destination  [subdir]...  
Pre-requisites 1. Source shrc or cshrc to set paths
2. Define the variable GO to match your output_root.
Caution C shell users must set GO to an actual path, not a symbolic link.
Available destinations are the same as for go.
If you want shorter messages, set the variable OGO_NO_WARN.

Notes

Example: Automating your output root using %{ENV_GO} and ogo

Set GO once, at login.
Config files can pick it up from an ENV macro

$ grep GO ~/.bashrc
export GO=/tmp/outputs
$ cat autoroot.cfg
output_root          = %{ENV_GO}  # pick up automatically from environment
iterations           = 1
label                = xyzzy
runlist              = 605.lbm_s
size                 = test
default:
   CC                = mpicc
   CC_VERSION_OPTION = -v
$ cat autoroot.sh 
runhpc --config=autoroot | grep txt
$ ./autoroot.sh 
    format: Text -> /tmp/outputs/result/hpc2021_sml.001.small.test.txt

ogo handily moves around in the directories from the above command.
It confirms your destination underneath $GO.

$ ogo
Using value in GO for output_root: /tmp/outputs
/tmp/outputs
$ ogo mcf run
Using value in GO for output_root: /tmp/outputs
/tmp/outputs/benchspec/HPC/605.lbm_s/run
$ ogo exe
Using value in GO for output_root: /tmp/outputs
/tmp/outputs/benchspec/HPC/605.lbm_s/exe
$ ls
lbm_s_base.xyzzy

If you get tired of the two-line message, you can shrink it by setting OGO_NO_WARN.

$ OGO_NO_WARN="quiet please" 
$ ogo lbm run
/tmp/outputs/benchspec/HPC/605.lbm_s/run

ogo uses the tree rooted at $SPEC if the destination is read-only (lbm src).

$ ogo lbm src
/spec/hpc2021/benchspec/HPC/605.lbm_s/src
$  

port_progress

port_progress analyzes log files and summarizes errors. It was primarily used during the benchmark porting phase for SPEChpc 2021 (thus its name), but it can still come in handy, for example when trying the benchmarks with a new compiler.

Usage
port_progress [-b benchmarks] [-c config] [-q] [--rate] [--skipnt] [--speed] [--table] [-a|logs...] 
port_progress commonly-used options
-b benchmarks Only report matching benchmarks (you can use a regexp).
-c config Only report matching config files (you can use a regexp).config.
-q Slightly less annoying verbosity
--table Much less annoying verbosity
--skipnt Please shut up about things that were Not Tested.
-a|logs... Report on All or on selected logs
See also --help for additional options

port_progress quickly summarizes the completion status for a series of runs. The report is organized into columns laid out by config file, tuning (base or peak), and workload size (Te for test, Tr for train, Ref for ref). Each benchmark and workload is summarized using codes such as:

S  - Success
CE - Compile error
RE - Run time error (did not finish; crashed or core dumped)
VE - Verification Error (it finished, but got the wrong answer)

In the example below, the tester has been trying several compiler versions (X, Y, and Z) using several config files. A report is requested only for the troublesome benchmarks - all of which worked with the most recent compiler.

$ specperl $SPEC/bin/tools/port_progress -q -a --skipnt --table -b 'lbm|tealeaf|pot3d|weather' result/*.log
Looking for benchmarks that match 'lbm|tealeaf|pot3d|weather'
SPEC hpc2021 v0.2.7
  Wed Oct 21 13:24:55 2020
Tester: tony   System: turbo5

                ------  ------  ------  ------  
                acc     mpi     omp     tgt  
                ------  ------  ------  ------  
                Base    Base    Base    Base    
                ------  ------  ------  ------  
                Te Ref  Te Ref  Te Ref  Te Ref  
                -- ---  -- ---  -- ---  -- ---  
605.lbm_s       S  S    S  S    S  S    S  S    
618.tealeaf_s   S  S    S  S    S  S    S  S    
628.pot3d_s     S  S    S  S    S  S    S  S    
635.weather_s   S  S    S  S    S  S    S  S    
   

printkids.pl

printkids.pl prints a table showing the execution times for each of a benchmark's subcomponents. Before using it, cd to a run directory.

Usage
printkids.pl [--columns=N] [--truncate] [--wrap] 
Windows: %SPEC%\bin\specperl %SPEC%\bin\tools\printkids.pl &etc
printkids.pl commonly-used options
--columns=N Limit output to the requested width
--truncate Content wider than the requested number of columns is chopped off (default).
--wrap Content wider than the requested number of columns is wrapped.
See also --help for additional options

Some SPEC benchmark executables are invoked multiple times. If you'd like to know how much time was spent on each invocation, you can examine the log file, or you can go to a run directory and use printkids.pl. In the example below from SPEC CPU2017, the ref run for 525.x264_r has three parts, and it is apparent that the first part is the shortest.

$ ogo 525 run
Using value in GO for output_root: /Users/somebody/spec/cpu2017/out.rc5
/Users/somebody/spec/cpu2017/out.rc5/benchspec/CPU/525.x264_r/run
$ cd run_base_refrate_mar10a-m64.0000/
$ printkids.pl --wrap --columns=90
  Seconds  Command
    46.58 x264_r_base.mar10a-m64 --pass 1 --stats x264_stats.log --bitrate 1000 --frames
                                 1000 -o BuckBunny_New.264 BuckBunny.yuv 1280x720
   159.59 x264_r_base.mar10a-m64 --pass 2 --stats x264_stats.log --bitrate 1000 --dumpyuv
                                 200 --frames 1000 -o BuckBunny_New.264 BuckBunny.yuv
                                 1280x720
   162.37 x264_r_base.mar10a-m64 --seek 500 --dumpyuv 200 --frames 1250 -o
                                 BuckBunny_New.264 BuckBunny.yuv 1280x720
=========
   368.54 Total by adding children      
   368.54 Total according to speccmds.out
$ 

printpath.pl

printpath.pl is a script that can be used on Windows to print the path in a format that is easier to read.
You can send the output to a file, which can then be easily edited.

NOTE: printpath.pl is not supported with SPEChpc 2021

Usage
specperl %SPEC%\bin\windows\printpath.pl

In the example below, printpath.pl makes it easier to notice that cygwin (a Windows/Unix compatibility product) is in the path. SPEC recommends that your %PATH% should not include Windows/Unix compatibility products, because these have sometimes contributed to hard-to-diagnose errors (link goes to a CPU2006 FAQ).


C:\cpu2017\rc5>echo %PATH%
C:\cpu2017\rc5\bin\windows;C:\cpu2017\rc5\bin;C:\ProgramData\Oracle\Java\javapat
h;C:\windows\system32;C:\windows;C:\windows\System32\Wbem;C:\windows\System32\Wi
ndowsPowerShell\v1.0\;c:\cygwin\bin;c:\unison

C:\cpu2017\rc5>specperl %SPEC%\bin\windows\printpath.pl
PATH=^
C:\cpu2017\rc5\bin\windows;^
C:\cpu2017\rc5\bin;^
C:\ProgramData\Oracle\Java\javapath;^
C:\windows\system32;^
C:\windows;^
C:\windows\System32\Wbem;^
C:\windows\System32\WindowsPowerShell\v1.0\;^
c:\cygwin\bin;^
c:\unison
   

rawformat

rawformat is the report writer for SPEChpc 2021. It creates HTML, PDF, and many other forms of output.
rawformat is automatically invoked by runhpc at the end of a performance test.
It can be invoked directly, by typing rawformat, or equivalently by typing runhpc --rawformat.
(The name rawformat indicates that its input is a raw data file generated by an experiment, which it transforms into various formats.)

Usage
rawformat [options] rawfiles...
rawformat commonly-used options
--basepeak Copy base results to peak
--basepeak=bench[:bench...] Copy base results to peak for the benchmarks specified
--baseonly Discard peak results and report only base
--flagsurl=URL Use the file at URL as a flags description file. This option may be specified multiple times.
--http_proxy=HOST[:PORT] Use HOST as a proxy when fetching flags files or suite updates. If unspecified, PORT defaults to 80.
--output_format=FORMAT[:...] Set the output format FORMAT is one of: all, cfg, check, csv, flags, html, mail, pdf, ps, raw, screen, text. By default, includes the check format, unless you include nocheck among your list of formats.
Options for rawformat can be included on your runhpc command line; therefore, for convenience, they are described in the document Using SPEChpc 2021: The 'runhpc' Command.
See also rawformat --help

You might use rawformat many times just before completing a project, as you go through cycles of the form
rawformat-review-edit-rawformat-review-edit-rawformat...
until finally you get to ...publish!

Editing rawfiles (ONLY above the fence!)

A result rawfile may be edited to update reader fields, such as the system description.
The fields that you are allowed to edit are the ones located above the "fence":

# =============== do not edit below this point =================== 

Caution: do not change your original rawfile. Keep it safely as a backup. Instead, copy the original to a new file, and then edit the copy.

Syntax: Rawfiles have some difference from config files:

Example: Edit the system name.
Just before Announcement Day, and after you have run all your tests,
you notice a typo in the system name:

"Turblaster 9000"
this should be
"Turbo Blaster 9000"

You may edit the rawfile, provided that you edit only the reader fields, above the fence.
Start by making a new copy:

$ grep hw_model $SPEC/result/hpc2021_sml.015.small.ref.rsf 
spec.hpc2021.interconnect_fs_hw_model: BI 100 Series
spec.hpc2021.node_compute_hw_model: Turblaster 9000
spec.hpc2021.node_fileserver_hw_model: BG650
$ mkdir prep_for_publication
$ cp $SPEC/result/hpc2021_sml.015.small.ref.rsf prep_for_publication/tb9000.hpc2021.sml.rsf
$ cd prep_for_publication/

Make your edits:

$ vi tb9000.hpc2021.sml.rsf 
$  diff $SPEC/result/hpc2021_sml.015.small.ref.rsf tb9000.hpc2021.sml.rsf
   36c36
< spec.hpc2021.node_compute_hw_model: Turblaster 9000
---
> spec.hpc2021.node_compute_hw_model: Turbo Blaster 9000

And re-generate your reports:

$ specperl $SPEC/bin/formatter/rawformat --output_format txt,html,csv tb9000.hpc2021.sml.rsf

Continued fields

When editing fields that have more than one line, you cannot mix numbered and un-numbered versions of a single field. Instead, you must use a number on both of them:

WRONG:
spec.hpc2021.sw_compiler: C/C++: Version 17.0.0.098 of Intel C/C++
spec.hpc2021.sw_compiler1: Compiler for Linux
RIGHT:
spec.hpc2021.sw_compiler000: C/C++: Version 17.0.0.098 of Intel C/C++
spec.hpc2021.sw_compiler001: Compiler for Linux

Test_date: field allowed only in rawfiles

For SPEChpc 2021, one field is ignored in config files, but may be set in a raw file:

test_date

When the tests were run. This field is populated automatically based on the clock in the system under test. Setting this in the config file will generate a warning and the setting will be ignored. If your system clock is incorrect, then the value may be edited in the raw file. It's better to avoid the necessity to edit, by setting your system clock properly.

specdiff

specdiff determines whether benchmark outputs match the SPEC-supplied expected outputs.
Benchmark developers define the validation criteria by specifying acceptable tolerances.

Usage
specdiff [--abstol n] [--calctol] [--obiwan] [--reltol n] [--skiptol n] file1  file2
Caution --calctol is often harder to interpret than expected.
specdiff commonly-used options
--abstol n absolute tolerance
--calctol calculate required tolerances. See cautionary note below.
-l|--lines n # of differences to print (-1 for all)
-m|--mis Write file2.mis containing information about miscompares
--obiwan allow off-by-one errors
--reltol n relative tolerance
--skiptol n # of differences to ignore
See also specdiff --help for additional options

Tolerances are written by runhpc to compare.cmd, which specinvoke reads when it calls specdiff.
If you are debugging a problem, you can look at how the validation is done and you can run specdiff by hand.

Example: Go to a run directory and find out what specdiff command would be generated using specinvoke's dry-run switch, -n.

$ go weather run
$ cd run_base_test_omp.0000 
$ specinvoke -n compare.cmd | grep specdiff
$SPEC/bin/specperl $SPEC/bin/harness/specdiff -m -l 10 --abstol 4e-07 --reltol 0.006 $SPEC/benchspec/HPC/535.weather_t/data/ref/output/output6.ref.txt output6.ref.txt 0<&- > output6.ref.txt.cmp
$  $SPEC/bin/specperl $SPEC/bin/harness/specdiff -m -l 10 --abstol 4e-07 --reltol 0.006 --calctol $SPEC/benchspec/HPC/535.weather_t/data/ref/output/output6.ref.txt output6.ref.txt
Calculated tolerances:
Maximum abstol: 1.00002e-10
Minimum abstol: 9.99947e-11
# of abstol errors: 0
Maximum reltol: 1.20429e-08
Minimum reltol: 2.38284e-12
# of reltol errors: 0
# of obiwan errors: 0
# of skiptol errors: 0
specdiff run completed
$

Caution to benchmark developers: The --calctol switch answers two separate questions, one at time: (1) Given all the other tolerances listed, what relative tolerance R would be needed in order for these files to be considered a match? and (2) Given all the other tolerances listed, what absolute tolerance A would be needed? Notice that:
Any particular difference D that is just barely covered by R may be much smaller than A.
Any particular difference D that is just barely covered by A may be much smaller than R.
Therefore, a benchmark developer who is considering a tolerance change should never blindly apply the results of --calctol. Consider a much smaller change, and recalculate.

specinvoke

specinvoke runs and times commands, after runhpc sets them up.
It can display commands needed to set up the environment and to change directories.

Usage
specinvoke [options] [commandfile]

If the commandfile is not named, looks for speccmds.cmd.

specinvoke commonly-used options
-n Print a dry run of commands, but do not actually do them.
-nn Include commands that set environment variables and commands that cd to directories.
See also --help for additional options

Environment variables: It is important to understand that specinvoke does not set environment variables; but you can cause them to be set by capturing the dry run output and executing it. The output of specinvoke -nn includes commands to:

The specinvoke utility by default looks for speccmds.cmd, which is how benchmarks are run. However, that is not its only use. The runhpc program also uses specinvoke for setting up certain benchmarks, via inputgen.cmd; and to drive comparisons, as described in the section on specdiff. You might want to use it to run benchmarks "by hand", or to customize the runs during a performance analysis exercise.

specmake

specmake is GNU make under another name (to avoid any possible conflicts with other versions of make on your system). It builds benchmark executables, driving compilers/linkers/other build components as needed. Typically, you do not use specmake directly; instead, you specify make variables in your config file. However, if something goes wrong, a useful debug strategy is to cd to the build directory, and try specmake directly until you figure out what the problem is.

Usage
specmake [options] [target]...
specmake commonly-used options
clean In a build directory, say specmake clean to remove existing objects, executables, and intermediate files in that directory.
-j <n> Allow up to the specified number of simultaneous jobs.
--keep-going Continue the build even if there is a failure.
-n Print a dry run of commands, but do not actually do them.
preprocess In a build directory, say specmake preprocess to create both Fortran and C/C++ preprocessed files. Note that these differ greatly by language, as described below.
TARGET={lbm_s} For benchmarks that build more than one executable, you will find in the build directory Makefiles that are named after each target.
Select the one you want to build via TARGET=
See also --help for additional options

When something goes wrong with a build, you can use the build directory as a "sandbox", trying out different therapies. Examples:

How many modules were not built?

cd nnn.benchmark/build/build_where_something_broke.0000
specmake -n | wc -l

I want to see as many errors as possible, not just the first one.

cd nnn.benchmark/build/build_where_something_broke.0000
specmake --keep-going 

Maybe it fails because my config file line CXXOPTIMIZE=-O3 is too aggressive.

cd nnn.benchmark/build/build_where_something_broke.0000
specmake clean
specmake CXXOPTIMIZE=-O2 

Maybe it fails because I need to try This, That, or Other.

cd nnn.benchmark/build/build_where_something_broke.0000
edit Makefile.spec # try This
specmake           # .......... still fails? 
edit Makefile.spec # try That
specmake           #  .......... still fails?
edit Makefile.spec # try Other
   ...&etc

specmake preprocess creates both Fortran intermediate files (name.fppized.f90) and C/C++ intermediate files (name.i).
These have very differing existence requirements, usefulness, and relevance:

Fortran preprocessed files are always created. Fortran preprocessing for SPEChpc 2021 benchmarks is never done by the compilers; it is done by specpp. Your Fortran compiler is not called until after name.fppized.f90 has been created. A reason you might say specmake preprocess is because you want to create all of them at once. Because your Fortran compiler does not see the original files, the executable always corresponds to the preprocessed source, and any line numbers reported by debugger messages may differ greatly from original source line numbering.

C/C++ preprocessed files are rarely created. C/C++ preprocessing is done by compilers, which may vary in how they do it. Typically, C/C++ compilers do not bother to save preprocessed output to intermediate files (name.i) unless requested to do so. A reason you might want to create them is in order to verify whether macros are behaving as expected; however, in some cases, the compiler might not promise that it is even capable of creating a name.i file that corresponds exactly to the object. Line numbers reported by debuggers normally match the line numbering of the original source, not the numbering of name.i files.

specperl

specperl is simply Perl, built under a known set of circumstances at the same time as other tools in the SPEC toolset, and renamed to avoid conflicts with any other versions of the program on your path. Your system probably already has Perl -- but it might be older or newer, or might not include the same CPAN additions.
You can compare your system Perl to specperl by comparing the output of "perl -V" vs. "specperl -V".

Interactive use: You can use specperl interactively if you have executed one of the three path-setting scripts (see list in Quick Start).
You can get help using specperldoc.
Examples:

specperl -v                           # print basic version info
specperl -V                           # print version details
specperl -e "print hex '1C000800'"    # One liner to convert a number
specperldoc perlre                    # Find out about regular expressions 
                                      # (useful for flag description files)
specperldoc -f hex                    # Lookup a function

Warning: specperl might not be the best choice for your own custom scripts. If you have a system Perl, it may be better adapted to the particular characteristics (or idiosyncracies) of your particular operating system and hardware. Caveat programmer, use at own risk.

How do the tools use specperl? When using Perl, a key requirement is to set paths correctly. For tools such as

configpp
convert_to_development
dumpsrcalt
extract_config
extract_flags
extract_raw
flag_dump
makesrcalt
port_progress
rawformat
runhpc
specdiff
specpp

the paths are set by tiny, similarly-named scripts in either $SPEC/bin
These set the path, then exec or call the real scripts, using something similar to

specperl -IdirectoryThis IdirectoryThat realScript

specpp

specpp is the Fortran preprocessor.

Usage
(typical)
specpp [-Dmacro] [-Idir] [-m c-comment.pm] [-w] 
Additional options: see specpp -h and the manpage
Caution Try not to confuse with configpp

Preprocessing is not a part of standard Fortran. Therefore, SPEC supplies a copy of Darren Miller's freely available preprocessor utility filepp, with minor modifications, as specpp. Using the same preprocessor on all platforms ensures that the preprocessing is done consistently.

When building benchmarks, your Fortran compiler is not invoked until after specpp has completed its work. Therefore, config.html points out:

For example, if a config file sets

FPORTABILITY   = --files=bigendian
FPPPORTABILITY = -DSPEC_ACCEL_AWARE_MPI
   

and if a benchmark uses mumble.F90, then specpp will create mumble.fppized.f90 prior to the invocation of the Fortran compiler. The preprocessor will see the flag -DSPEC_ACCEL_AWARE_MPI. The compiler will see the flag --files=bigendian and the compiler not see the flag -DSPEC_ACCEL_AWARE_MPI.

Usually, you won't need to know any more about specpp, because it is automatically invoked. If you are a benchmark developer, or if you are simply curious, the manpage is available.

Don't confuse specpp with configpp.

specrxp

specrxp validates the XML in a flags file.
It is simply Richard Tobin's RXP under a unique name to ensure no possible conflicts with other versions of rxp on your system.

Overview:
(validate silently)
specrxp -V -s <flags file>
Caution If it seems to hang, see below.
See the manpage for additional options

If you write a flags file, you should validate that you have written well-formed XML, using specrxp, typically with -V to request validation and -s to request silence (if there are no errors). Many more options can be found in the manpage.

q. Why does specrxp seem to hang?

a. You are behind a firewall. The following line, which must be present at the top of your flags file, needs internet access:

   SYSTEM "http://www.spec.org/dtd/cpuflags2.dtd"
   

To resolve the problem, you can:

  1. Copy the flags file to a system that does have access.
  2. Or define a proxy (e.g. export http_proxy=http://www-proxy.mycompany.com:80/)
  3. Or temporarily change the line to point at a local file. Change it back afterwards!

Reminder: after validating the XML, dump and then validate your HTML. Both steps are needed.

specmd5sum, specsha256sum, specsha512sum

specmd5sum, specsha256sum, and specsha512sum calculate and verify checksums. They are based on
md5sum, sha256sum, and sha512sum from GNU coreutils.

spectar

spectar is simply GNU tar, built under a known set of circumstances at the same time as other tools in the SPEC toolset, and renamed to avoid conflicts with any other versions of the program on your path. GNU tar, unlike some other archiving programs, correctly handles very long path names.

specxz

specxz is xz from XZ Utils, built under a known set of circumstances at the same time as other tools in the SPEC toolset, and renamed to avoid conflicts with any other versions of the program on your path.

Commonly-used switches work analogously to those in other compression utilities, such as -d to decompress. For details, say specxz --help

Warning: this idiom, commonly used on Unix systems, has been observed to fail on some tested Windows systems:

specxz -dc mumble.tar.xz | spectar -xf -  <-- may fail on Windows
   

If you see failures, break it into two commands.

specxz -d mumble.tar.xz                   <-- an alternative
spectar -xf mumble.tar 

sysinfo (Not used by SPEChpc)

sysinfo inventories various aspects of a SUT (System Under Test).
sysinfo is not used by SPEChpc since it can only gather information about the system on which 'runhpc' is invoked, which is usually just the head node of a cluster.
If useful, SPEC/HPG may consider adding support to sysinfo in order to gather system information across multiple nodes, but it may not be practical for all clusters.
For other SPEC benchmarks, such as SPEC CPU2017, it is called automatically by runcpu, but you can also run it directly from the command line.

Usage
sysinfo [-f] [-p] [-q] 
sysinfo options
-f Do not write descriptive fields. Write config file comments instead.
-p Do not write Platform Notes. Write config file comments instead.
-q Do not send progress updates to stderr

Location, platforms, and updates: The sysinfo utility is installed with SPEC CPU2017 as $SPEC/bin/sysinfo (Unix) or %SPEC%\bin\sysinfo (Windows). It supports multiple operating systems, including AIX, Linux, Mac OS X, Microsoft Windows, and Solaris. If your operating system is not supported, and if you wish to publish results, you should contact SPEC for assistance, because published results must use sysinfo.

SPEC may update the sysinfo tool from time to time, which users can download from the SPEC web site using runhpc --update. SPEC may require that published results use a specific version of the tool.

Why is sysinfo required for SPEC CPU2017 but not SPEChpc 2021?

sysinfo is very useful for SPEC CPU2017 since:

  1. It creates a contemporaneous record of the system under test.
  2. The record is created automatically, and is not subject to human transcription error.
  3. Therefore, sysinfo can remove ambiguity such as "I remember that I upgraded the version of the operating system, but I can't quite recall whether that was before or after I did the re-run of the floating point speed tests."

However, SPEC CPU2017 is measuring a single SUT. SPEChpc 2021 is often run from a head-node and run on one or more compute nodes. Thus, sysinfo would need to be run via a scheduler on all nodes under test, gather the results, remove duplications, and consolidate the information into a single report, which could be lengthy especially if there are many different types of systems on the cluster. While technically feasible, SPEC/HPG determined, at least for the initial release, to not include this feature.

How is it called? Can I turn it off? runcpu calls sysinfo while parsing the config file. The utility is found via the setting for sysinfo_program. By default, it uses the copy in the $SPEC/bin/ directory. You could temporarily turn it off using:

sysinfo_program = 

but it must be used for results that you would like to be able to publish.

Output types: By default, information will be added to the platform notes and to some of the system description fields. To see what will be added, run it from the command line:

$ $SPEC/bin/sysinfo -q
notes_plat_sysinfo_000 = Sysinfo program /Users/somebody/spec/cpu2017/rc4/bin/sysinfo
notes_plat_sysinfo_005 = Rev: r5214 of 2017-01-15 ea4c2974a34b51078facb71b0461e013
notes_plat_sysinfo_010 = running on hostname.local Sun Jan 15 08:45:49 2017
notes_plat_sysinfo_015 =  
notes_plat_sysinfo_020 = SUT (System Under Test) info as seen by some common utilities.
notes_plat_sysinfo_025 = For more information on this section, see
notes_plat_sysinfo_030 =    https://www.spec.org/hpc2021/Docs/config.html#sysinfo
notes_plat_sysinfo_035 =  
notes_plat_sysinfo_040 = Hardware:
notes_plat_sysinfo_045 =  
notes_plat_sysinfo_050 =     Hardware Overview:
notes_plat_sysinfo_055 =  
notes_plat_sysinfo_060 =       Model Name: MacBook Pro
notes_plat_sysinfo_065 =       Model Identifier: MacBookPro11,4
.
.
.
hw_model           = MacBookPro11,4
hw_nchips          = 1
hw_ncores          = 4
hw_scache          = 256 KB per Core
hw_tcache          = 6 MB
prepared_by        = somebody  (is never output, only tags rawfile)
sw_os              = macOS 10.12.2 (16C67)
sw_other           = Xcode 8.2.1 (11766.1)
$ 
   

Editing the output: Descriptive fields (such as hw_model above) may be edited in the usual manner. The Platform Notes (notes_plat_sysinfo_NNN) cannot be edited, but you may add additional notes. Put them into a subsection with a notes tag that falls alphabetically afterwards, for example:

notes_plat_update_100 = Because of a BIOS limitation, dmidecode and sysinfo do not 
notes_plat_update_110 = report Ranks correctly for DDR4 DIMMs.  The DIMM part numbers 
notes_plat_update_120 = are reported correctly.  Therefore, please see the manufacturer 
notes_plat_update_130 = memory data sheets to determine the number of Ranks.

Comment destination: If you use -f or -p to send output to config file comments, those comments will be visible only via output_format config. They do not appear in reports, and they are not written to the config file in your $SPEC/config directory. They are written to the copy in your result directory. This can make them easy to overlook, which is why the default is to write actual notes and fields, instead of writing only comments.

Avoiding warnings about conflicting fields: You might see warnings such as:

 WARNING: Your config file sets some fields that are also set by sysinfo:
  hw_cpu_name, sw_os
   

Fields created by sysinfo (such as hw_cpu_name) are written to a section named default: at the end of your config file. The warning means that you have already set one of the listed fields in your default: section. Because the sysinfo output comes at the end of your config file, its setting will override the earlier config file settings, which might not be desirable.

To get rid of sysinfo duplicate field warnings, put your fields into any named section other than default:.

Example: Brian is using a small system for SPECspeed testing, and a large server for SPECrate testing. Brian does not like the default output from sysinfo, because it only includes the bare amount of information about memory, and does not break the memory down by DIMM types:

default:
hw_memory = 32768 MB  fixme: format is: 'n GB (n x n GB nRxn PCn-nnnnnR-n, ECC)'

Therefore, he writes the into his config file:

intspeed,fpspeed:
   hw_memory = 32 GB (2 x 16 GB 4Rx4 PC4-2133P-L)
intrate,fprate;
   hw_memory = 1 TB (32 x 32 GB 4Rx4 PC4-2133P-L)
   

After writing the above, the warnings go away, and Brian's preferred disclosure always 'wins'.

Writing your own sysinfo program: If you wish, you can write your own utility, by following the rules below. If you plan to publish your results, you should contact SPEC to ask for approval of your variant.

  1. The sysinfo program can be any executable program or script. It does not have to be a Perl script.
  2. If you do want to write a Perl script, you can count on specperl being on the path (as in the example above).
  3. The sysinfo program is run prior to most other config file processing. This means that most variables of the form $[var] and ${var} will not work.
  4. You can use $[top] to find the top of your SPEChpc 2021 tree.
  5. The program should output lines of the form:
    name = value
  6. The lines should be output to standard output.
  7. Only informational fields (hw_*, sw_*, notes*, test_sponsor, license_num, tester, prepared_by) may be set; these settings will be inserted into the 'default:' section, and will not be able to override any per-benchmark settings.
  8. Illegal lines will be silently discarded.
  9. Standard config file comments are also allowed.
  10. Because the line checking is fairly simple-minded, some config file features such as line continuation, block quotes, and line appending are not available.
  11. The output from the sysinfo program will be included in the final config as if it had been part of the original config file.
  12. Arguments can be given to the command if necessary.
  13. If the sysinfo_program fails to generate any output, the tools will write a warning to the log file. Errors and warning messages from your program might not be captured to the log.

toolsver

toolsver prints information about the tools that were used by a rawfile

Usage
toolsver rawfile

SPEChpc™2021 Utilities: Copyright © 2021 Standard Performance Evaluation Corporation (SPEC)