Using SPECaccel® 2022: The 'runaccel' Command

$Id$ Latest: www.spec.org/hpg/accel2023/Docs/

1. Basics

1.1 Defaults

1.2 Syntax

1.3 Benchmarks and suites

1.4 Run order

1.5 Disk Usage

1.5.1 Directory tree

1.5.2 Hey! Where did all my disk space go?

1.6 Multi-user support and limitations
expid (partial solution) output_root (recommended)
1.7 Actions: build buildsetup report run runsetup setup validate
cleaning: clean clobber onlyrun realclean scrub trash
(alternative: Clean by hand)

2. Commonly used options

--action --check_version --config --flagsurl --help --ignore_errors --iterations --loose --output_format --pmodel New --rawformat --rebuild --reportable --threads --tune

3. Less common options

--baseonly --basepeak --nobuild --comment --define --delay --deletework --expid --fake --fakereport --fakereportable --[no]graph_auto --graph_max --graph_min --http_proxy --http_timeout --info_wrap_column --keeptmp --labelNew --log_timestamp --make_no_clobber --notes_wrap_column --output_root --preenv --reportonly --review --[no]setprocgroup --size --[no]table --test --undef --update --use_submit_for_compare --use_submit_for_speed --username --verbose --version

4 Removed/unsupported options

4.1 Not used with SPECaccel: --rate --speed --[no]feedback --parallel_setup

--copies --[no]power --parallel_test --parallel_test_workloads --ranks

4.2 Feature removed: --machine --maxcompares

4.3 Unsupported: --make_bundle --unpack_bundle --use_bundle

5 Quick reference

1. Basics

What is runaccel?   runaccel is the primary tool for SPECaccel. You use it from a Linux shell command line to build and run benchmarks, with commands such as these:

runaccel --config=eniac.cfg    --action=build 404.lbm
runaccel --config=colossus.cfg --threads=16   450.md

The first command compiles the benchmark named 404.lbm. The second runs the benchmark 450.md using 16 host threads.

New with SPECaccel: The former runspec utility is renamed runaccel in SPECaccel 2023.

Before reading this document: If you have not already done so, please install and test your SPECaccel distribution. This document assumes that you have already:

If you have not done the above, please see the brief instructions in the Quick Start guide, or the more detailed section "Testing Your Installation" Linux.

1.1 Defaults

The SPECaccel default settings described in this document may be adjusted by config files.

The order of precedence for settings is:

Highest precedence: runaccel command
Middle: config file
Lowest: the tools as shipped by SPEC

Therefore, when this document tells you that something is the default, bear in mind that your config file may have changed that setting. With luck, the author of the config file will tell you so.

1.2 Syntax

The syntax for the runaccel command is:

runaccel [options] [list of benchmarks to run]

Options are described in the following sections. There, you will notice that many options have both long and short names. The long names are invoked using two dashes, and the short names use only a single dash. For long names that take a parameter, you can optionally use an equals sign. Long names can also be abbreviated, provided that you still enter enough letters for uniqueness. For example, the following commands all do the same thing: 

runaccel --config=dianne_july25a --debug=99 accel
runaccel --config dianne_july25a --debug 99 accel
runaccel --conf dianne_july25a   --deb 99   accel
runaccel -c dianne_july25a       -v 99      accel

1.3 Benchmarks and Suites

In the list of benchmarks to run, you can use one or more individual benchmarks, such as 404.lbm, or you can run the entire suite, using the short tag "accel".

Benchmark names: Individual benchmarks can be named, numbered, or both.
Separate them with a space.
Names can be abbreviated, as long as you enter enough characters for uniqueness.
Each of the following commands does the same thing:

runaccel -c jason_july09d --noreportable 404.lbm 450.md 470.bt
runaccel -c jason_july09d --noreportable 404 450 470

To exclude a benchmark: Use a hat (^, also known as carat, typically found as shift-6). Note that if hat has significance to your shell, you may need to protect it from interpretation by the shell, for example by putting it in single quotes. 

bash-n.n.n$ runaccel --noreportable -c cathy_sep14c accel ^450
pickyShell% runaccel --noreportable -c cathy_sep14c accel '^450'

Turning off reportable: If your config file sets reportable=yes then you cannot run a subset unless you turn that option off.

[/usr/cathy/specaccel]$ runaccel --config cathy_apr21b --noreportable accel ^lbm

1.4 Run order

A reportable run does these steps:

  1. Ref: Run the reference workload

  2. Report:

Summarizing reportable run order: The order can be summarized as:

  setup for ref
          ref1, ref2 [, ref3] (*)

 (*) One benchmark at a time.  Third run only if --iterations=3.

Reportable order when more than one tuning is present: If you run both base and peak tuning, base is always run first. 

 setup for ref
          base ref1, base ref2 [, base ref3] (*)
          peak ref1, peak ref2 [, peak ref3] (*)

 (*) One benchmark at a time.  Third run only if --iterations=3.

Reportable order when more than one suite is present: If you start a reportable using more than one suite, all the work is done for one suite before proceeding to the next. Note SPECaccel 2023 currently has only a single suite.

1.5 Disk Usage

1.5.1 Directory tree

The structure of the SPECaccel directory tree is:

$SPEC - the root directory
   benchspec    - Some suite-wide files
   ACCEL        - The benchmarks
   bin          - Tools to run and report on the suite
   config       - Config files
   Docs         - HTML documentation
   result       - Log files and reports
   tmp          - Temporary files
   tools        - Sources for the SPECaccel tools

Within each of the individual benchmarks, the structure is:

nnn.benchmark - root for this benchmark
   build      - Benchmark binaries are built here
   data        
      all     - Data used by all runs (if needed by the benchmark)
      ref     - The timed data set
      test    - Data for a simple test that an executable is functional
      train   - Data for feedback-directed optimization (not used in SPECaccel. Contains same workload as test)
   Docs       - Documentation for this benchmark
   exe        - Compiled versions of the benchmark
   run        - Benchmarks are run here
   Spec       - SPEC metadata about the benchmark
   src        - The sources for the benchmark

Look for the output of your runaccel command in the directory $SPEC/result. There, you will find log files and result files. More information about log files can be found in the Config Files document.

The format of the result files depends on what was selected in your config file, but will typically include at least .txt for ASCII text, and will always include .rsf, for raw (unformatted) run data. More information about result formats can be found below, under --output_format. Note that you can always re-generate the output, using the --rawformat option, also documented below.

1.5.2 Hey! Where did all my disk space go?

When you find yourself wondering "Where did all my disk space go?", the answer is usually "The run directories." Most activity takes place in automatically created subdirectories of $SPEC/benchspec/ACCEL/*/run/. Other consumers of disk space underneath individual nnn.benchmark directories include the build/ and exe/ directories.

At the top of the directory tree, space is used by your config/ and result/ directories, and for temporary directories

$SPEC/tmp
output_root/tmp

Usually, the largest amount of space is in the run directories.

If you use the config file label feature, then directories are named to try to make it easy for you to hunt them down, For example, suppose George has a config file that he is using to test some new memory optimizations. He has set

label=GeorgeMemoryOpt

in his config file. In that case, the tools would create directories such as these:

$ pwd
/Users/george/accel/benchspec/ACCEL/404.lbm:
$ ls -d */*George*
build/build_base_GeorgeMemoryOpt.0000
exe/mcf_r_base.GeorgeMemoryOpt
run/run_base_ref_GeorgeMemoryOpt.0000
run/run_base_ref_GeorgeMemoryOpt.0001
run/run_base_ref_GeorgeMemoryOpt.0002
run/run_base_ref_GeorgeMemoryOpt.0003
run/run_base_ref_GeorgeMemoryOpt.0004
run/run_base_ref_GeorgeMemoryOpt.0005
run/run_base_ref_GeorgeMemoryOpt.0006
run/run_base_ref_GeorgeMemoryOpt.0007
run/run_base_ref_GeorgeMemoryOpt.0008
run/run_base_ref_GeorgeMemoryOpt.0009
run/run_base_ref_GeorgeMemoryOpt.0010
run/run_base_ref_GeorgeMemoryOpt.0011
run/run_base_ref_GeorgeMemoryOpt.0012
run/run_base_test_GeorgeMemoryOpt.0000
$

To get your disk space back, see the documentation of the various cleaning options, below.

1.6 Multi-user support

SPECaccel supports multiple users sharing an installation; however you must choose carefully regarding file protections. This section describes the multi-user features and protection options.

Features that are always enabled:

Limitations: The default methods impose two key limitations, which will not be safe in some environments:

  1. The directory tree must be writable by each of the users, which means that they have to trust each other not to modify or delete each others' files.
  2. Directories such as result/ and nnn.benchmark/exe/ and nnn.benchmark/run/ are not segregated by user. Therefore you can have only one version of (for example) 404.lbm/exe/lbm_base.Ofast and different users will have their result logs intermixed with each others in the result/ directory.

Partial solution(?) expid+conventions:
You can deal with limitation #2 if users adopt certain habits. For example, Swen could name all his config files swen-something.cfg. He could use runaccel --expid=swen or the corresponding config file expid=swen to cause his results to be placed under $SPEC/result/swen and binaries under nnn.benchmark/exe/swen. Unfortunately, this alleged solution still requires that the tree be writeable by all users, and will not help Swen at all when Chris comes along and blithely does one of the alternate cleaning methods.

Solution(?) Give up:
You could just choose to spend the disk space to give each person their own tree. For SPECaccel, this may increase disk space requirement by about 2 GB per user.

Recommended Solution: output_root. The recommended method uses 4 steps:

StepExample
(1) Protect most of the SPEC tree read-only chmod -R ugo-w $SPEC
(2) Allow shared access to the config directory. chmod 1777 $SPEC/config
chmod u+w $SPEC/config/*cfg
(3) Keep your own config files cp config/assignment1.cfg config/sunita.cfg
(4) Use the --output_root switch or
add an output_root to your config file. 		runaccel --output_root=~/accel
output_root = /home/${username}/accel

More detail:

  1. Most of the SPECaccel tree is shared, and can be protected read-only. For example, on a Unix system, you might set protections with:

    chmod -R ugo-w $SPEC

  2. The one exception is the config directory, $SPEC/config/ which needs to be a read/write directory shared by all the users, and config files must be writeable. On most Linux system, chmod 1777 is very useful: it lets anyone create files, which they own, control, and protect. (1777 is commonly used for /tmp for this very reason.)

    chmod 1777 $SPEC/config
chmod u+w $SPEC/config/*cfg

    3. New in SPECaccel: The SPEC installation tree may now be fully read-only including the config directory. However in this case, you must to provode the full path to the config file to runaccel since it can no longer be in the default location.
    Example: runaccel --config /path/to/config.cfg.

  3. Config files usually would not be shared between users. For example, students might create their own copies of a config file:

    Kevin enters:

    cd /cs403/accel
. ./shrc
cd config
cp assignment1.cfg kevin1.cfg
chmod u+w kevin1.cfg
runaccel --config=kevin1 --action=build 404.lbm

    Bert enters:

    cd /cs403/accel
. ./shrc
cd config
cp assignment1.cfg bert1.cfg
chmod u+w bert1.cfg
runaccel --config=bert1 --action=build 404.lbm

  4. Set output_root in the config files to change the destinations of the outputs. For example, if config files include (near the top):

    output_root=/home/${username}/accel
label=feb27a

    then these directories will be used for the above runaccel command:

    Kevin's directories 
    build: /home/kevin/accel/benchspec/ACCEL/404.lbm/build/build_base_feb27a.0001
Logs:  /home/kevin/accel/result
    Bert's 
    build: /home/bert/accel/benchspec/ACCEL/404.lbm/build/build_base_feb27a.0001
Logs:  /home/bert/accel/result

Navigation: Unix users can easily navigate an output_root tree using ogo

1.7 Actions

Most runaccel commands perform an action on a set of benchmarks.

(Exceptions: runaccel --rawformat or update.)

The default action is validate.
The actions are described in two tables below: first, actions that relate to building and running; and then actions regarding cleanup.

--action build Compile the benchmarks, using the config file specmake options.
--action buildsetup

Set up build directories for the benchmarks.
Copy the source files to the directory, and create the needed Makefiles.
Do not attempt to actually do the build.

This option may be useful when debugging a build: you can set up a directory and play with it as a private sandbox.
--action onlyrun

Run the benchmarks but do not verify that they got the correct answers.
You cannot use this option to report performance.

This option may be useful while applying SPECaccel for some other purpose, such as tracing instructions for a hardware simulator, or generating a system load while debugging an operating system feature.
--action report Synonym for --fakereport; see also --fakereportable.
--action run Synonym for --action validate.
--action runsetup

Set up the run directory (or directories).
If executables do not exist, build them.
Copy executables and data to the directory(ies).
Create control file specccmds.cmd but do not actually run any benchmarks.

This option may be useful when debugging a run.
See the runsetup sandbox example in the Utilities documentation.
--action setup Synonym for --action runsetup
--action validate Build (if needed), set up directories, run, check for correct answers, generate reports.
This is the default action.

Cleaning actions are listed in order from least thorough to most:

--action clean

Empty run and build directories for the specified benchmark set for the current user. For example, if the current OS username is set to Jeffrey and this command is entered:

runaccel --action clean --config may12a accel

then the tools will remove build and run directories with username Jeffrey for Small benchmarks generated by config file may12a.cfg.

--action clobber Clean + remove the corresponding executables.
--action trash Remove run and build directories for all users and all labels for the specified benchmarks.
--action realclean A synonym for --action trash
--action scrub Trash + remove the corresponding executables.
Caution Fake mode is not implemented for the cleaning actions.
For example, if you say runaccel --fake --action=clean the cleaning really happens.

Clean by hand:
If you prefer, you can clean disk space by entering commands such as the following:

rm -Rf $SPEC/benchspec/ACCEL/*/run
rm -Rf $SPEC/benchspec/ACCEL/*/build
rm -Rf $SPEC/benchspec/ACCEL/*/exe

The above commands not only empty the contents of the run and exe directories; they also delete the directories themselves. That's fine; the tools will re-create the run and exe directories if they are needed again later on.

result directories can be cleaned or renamed. Don't worry about creating a new directory; runaccel will do so automatically. You should be careful to ensure no surprises for any currently-running users. If you move result directories, it is a good idea to also clean temporary directories at the same time.
Example:

cd $SPEC
mv result old-result
rm -Rf tmp/
cd output_root     # (If you use an output_root)
rm -Rf tmp/

I have so much disk space, I'll never use all of it:

Run directories are automatically re-used for subsequent runs. If you prefer, you can ask the tools to never touch a used run directory. Do this by setting the environment variable:

     SPEC_RUNDIR_DEL

In this case, you should be prepared to do frequent cleaning, perhaps after reviewing the results of each run.

2 Commonly used options

Most users of runaccel will want to become familiar with the following options.

This section is organized alphabetically without regard to upper/lower case and without regard to presence or absence of no at the start of the switch.

--action action

--check_version

--config name

--copies number

--flagsurl URL[,URL...]

--help

--ignore_errors

--iterations number

--loose

--output_format format

Name|synonyms... Meaning
all implies all of the following except screen, check, and mail 
config
   cfg|cfgfile
   configfile
   conffile

config file used for this run, written as a numbered file in the result directory, for example, $SPEC/result/accel2023.005.ref.cfg

  1. The config file is saved on every run, as a compressed portion of the rawfile. Therefore, you can regenerate it later, if desired, using rawformat
  2. Results published by SPEC include your config file. Anyone can download it and try to reproduce your result.

  3. The config file printed by --output_format=config is not identical to the original:

    • The file name matches the other files for this result, not the name you had in your config/ directory.
    • It does not include protected comments
    • It includes a copy of the runaccel line that invoked it.
    • It tells you whether output_root was defined.
    • It includes any result edits you make after the run (see utility.html).
    • It does not include the HASH section.
 
check
   subcheck
   reportcheck
   reportable
   reportablecheck
   chk|sub|subtest|test
 Reportable syntax check (automatically enabled for reportable runs).
  • Causes the format of many fields to be checked, e.g. "Nov-2023", not "11/23" for hw_avail.
  • Consistent formats help readers, especially when searching.
  • check is included by default for reportable runs and when using --rawformat.
  • It can be disabled by adding nocheck to your list of formats.
 
csv
   spreadsheet

Comma-separated variable. If you populate spreadsheets from your runs, you probably should not cut/paste data from text files; you'll get more accurate data by using --output_format csv. The csv report includes all runs, more decimal places, system information, and even the compiler flags.

 
default
 implies HTML and text 
flag|flags
 Flag report. Will also be produced when formats that use it are requested (PDF, HTML). 
html
   xhtml|www|web
 web page 
mail
   mailto|email
 All generated reports will be sent to an address specified in the config file. 
pdf
   adobe
 Portable Document Format. This format is the design center for SPECaccel reporting. Other formats contain less information: text lacks graphs, postscript lacks hyperlinks, and HTML is less structured. (PDF does not appear as part of "default" only because some systems may lack the ability to read it.) 
postscript
   ps|printer|print
 PostScript 
raw
   rsf
 The unformatted raw results, written to a numbered file in the result directory that ends with .rsf (e.g. /spec/accel/result/accel2023.005.ref.rsf). Your raw result files are your most important, because the other formats are generated from them. 
screen|scr|disp
   display|terminal|term
 ASCII text output to stdout. 
text
   txt|ASCII|asc
 Plain ASCII text file

--pmodel model (New in SPECaccel)

--ranks N

--rawformat rawfiles

--rebuild

--reportable

--threads N

--tune tuning

3. Less common options

This section is organized alphabetically without regard to upper/lower case and without regard to presence or absence of no at the start of the switch.

--baseonly

--basepeak [bench,bench,...]

--nobuild

--comment "comment text"

--define SYMBOL[=VALUE]
--define SYMBOL:VALUE

--delay secs

--deletework

--expid subdir

--fake

--fakereport

--fakereportable

--[no]graph_auto

--graph_max N

--graph_min N

--http_proxy proxy[:port]

--http_timeout N

--info_wrap_columns N

--[no]keeptmp

--label name

--[no]log_timestamp

--make_no_clobber

--notes_wrap_columns N

--output_root directory

--preenv, --nopreenv

--review, --noreview

--setprocgroup, --nosetprocgroup

--size size[,size...]

--table, --notable

--test

--undef SYMBOL

--update

--use_submit_for_compare (Not tested with SPECaccel)

--use_submit_for_speed

--username name

--verbose n

--version

4 Removed/unsupported options

4.1 Options that are no longer needed

Rate and Speed

The CPU2006 feature --rate[link goes to CPU2006]
and the CPU2006 feature --speed[link goes to CPU2006]
are not needed in SPECaccel.

Parallel setup

The SPEC CPU2006 feature --parallel_setup[link goes to CPU2006]
and the CPU2006 feature --parallel_setup_prefork[link goes to CPU2006]
and the CPU2006 feature --parallel_setup_type[link goes to CPU2006]
are not used in SPECaccel.

--[no]feedback (Not used with SPECaccel)

--power --nopower (Not available with SPECaccel)

--train_with WORKLOAD (Not available with SPECaccel)

--parallel_test processes (Not available with SPECaccel)

--parallel_test_workloads workload,... (Not available with SPECaccel)

4.2 Features removed

The SPEC CPU2006 feature --machine[link goes to CPU2006]
was removed because it was rarely used; and the additional complexity and confusion that it caused was deemed not worthwhile.

The CPU2006 feature --maxcompares[link goes to CPU2006]
was removed due to complexity considerations when implementing the new parallel setup methods.

4.3 Unsupported

The SPEC CPU2006 feature --make_bundle[link goes to CPU2006]
and the CPU2006 feature --unpack_bundle[link goes to CPU2006]
and the CPU2006 feature --use_bundle[link goes to CPU2006]
have not been tested in the SPECaccel environment.
It is not known whether anyone uses the features, and they were deemed not a priority.
It is possible that you might be able to get them to work by following the CPU2006 instructions linked above, but no promises are made.

5 Quick reference

(This table is organized alphabetically, without regard to upper/lower case, and without regard to the presence of a leading "no").

-a Same as --action
--action action Do: build|buildsetup|clean|clobber| onlyrun|realclean|report|run|runsetup|scrub| setup|trash|validate
--basepeak Copy base results to peak (use with --rawformat)
--nobuild Do not attempt to build binaries
-c Same as --config
-C Same as --copies
--check_version Check whether an updated version of SPECaccel is available
--comment "text"Add a comment to the log and the stored configfile.
--config file Set config file for runaccel to use
--copies Set the number of copies for a SPECrate run
-D Same as --rebuild
-d Same as --deletework
--debug Same as --verbose
--define SYMBOL[=VALUE] Define a config preprocessor macro
--delay secs Add delay before and after benchmark invocation
--deletework Force work directories to be rebuilt
--dryrun Same as --fake
--dry-run Same as --fake
--expid=dir Experiment id, a subdirectory to use for results/runs/exe
-F Same as --flagsurl
--fake Show what commands would be executed.
--fakereport Generate a report without compiling codes or doing a run.
--fakereportable Generate a fake report as if "--reportable" were set.
--[no]feedback Control whether builds use feedback directed optimization
--flagsurl url Location (url or filespec) where to find your flags file
--graph_auto Let the tools pick minimum and maximum for the graph
--graph_min N Set the minimum for the graph
--graph_max N Set the maximum for the graph
-h Same as --help
--help Print usage message
--http_proxy Specify the proxy for internet access
--http_timeout Timeout when attempting http access
-I Same as --ignore_errors
-i Same as --size
--ignore_errors Continue with benchmark runs even if some fail
--ignoreerror Same as --ignore_errors
--info_wrap_column NSet wrap width for non-notes informational items
--infowrap Same as --info_wrap_column
--input Same as --size
--iterations N Run each benchmark N times
--keeptmp Keep temporary files
-L Same as --label
-l Same as --loose
--label label Set the label for executables, build directories, and run directories
--loose Do not produce a reportable result
--noloose Same as --reportable
-M Same as --make_no_clobber
--make_no_clobber Do not delete existing object files before building.
--mockup Same as --fakereportable
-n Same as --iterations
-N Same as --nobuild
--notes_wrap_column NSet wrap width for notes lines
-noteswrap Same as --notes_wrap_column
-o Same as --output_format
--output_format format[,format...] Generate: all|cfg|check|csv|flags|html|mail|pdf|ps|raw|screen|text
--output_root=dir Write all files here instead of under $SPEC
--parallel_test Number of test/train workloads to run in parallel
--pmodel Enable a node level parallel model
--[no]power Control power measurement during run
--preenv Allow environment settings in config file to be applied
-R Same as --rawformat
--ranks Set the number of MPI ranks to use.
--rawformat Format raw file
--rebuild Force a rebuild of binaries
--reportable Produce a reportable result
--noreportable Same as --loose
--reportonly Same as --fakereport
--[no]review Format results for review
-s Same as --reportable
-S SYMBOL[=VALUE] Same as --define
-S SYMBOL:VALUE Same as --define
--[no]setprocgroup [Don't] try to create all processes in one group.
--size size[,size...] Select data set(s): test|train|ref
--strict Same as --reportable
--nostrict Same as --loose
-T Same as --tune
--[no]table Do [not] include a detailed table of results
--threads=N Set number of host threads per MPI rank.
--test Run various perl validation tests on specperl
--train_with Change the training workload
--tune Set the tuning levels to one of: base|peak|all
--tuning Same as --tune
--undef SYMBOL Remove any definition of this config preprocessor macro
-U Same as --username
--update Check www.spec.org for updates to benchmark and example flag files, and config files
--username Name of user to tag as owner for run directories
--use_submit_for_compare If submit was used for the run, use it for comparisons too.
--use_submit_for_speed Use submit commands for SPECspeed (default is only for SPECrate).
-v Same as --verbose
--verbose Set verbosity level for messages to N
-V Same as --version
--version Output lots of version information
-? Same as --help

Using SPECaccel®2023: The 'runaccel' Command: Copyright © 2021 Standard Performance Evaluation Corporation (SPEC)