SPEChpc™ 2021 Config Files

A. What is a config file?

B. Benchmark selection

C. Five consumers

D. Syntax

1. Sections and scope

2. Comments

3. Whitespace

4. Quoting

5. Line continuation

6. Included files

7. Section markers

E. Section merging

F. Variables

1. Defining variables

2. $[square] substitution

3. Useful $[variables]

4. ${curly} interpolation

5. Useful ${variables}

6. Unsetting "%undef%"

7. Debug tips

A. Commonly used Make variables

CC, CXX, FC
PORTABILITY (+ several similar options)
OPTIMIZE (+etc.)
EXTRA_CFLAGS (+etc.)
LIBS
PASS1_OPTIMIZE (+etc.)

B. Parallel Model Selection

C. Creating your own Make variables

D. += is available but use with caution

E. Using buildsetup to create a sandbox

F. About Automatic Rebuilds

{C|CXX|F}C_VERSION_OPTION New and required with SPEChpc 2021

A. \$SHELLVAR variable substitution

B. Options

bench_post_setup build_pre_bench build_post_bench mointor_X post_setup submit

C. Using submit

1. Basic usage

2. Using scripts with submit for thread binding

3. Reporting

A. Descriptive fields

Test description

Benchmark description

Node description(s)

Accelerator description(s)

Adaptor description(s)

Interconnect description(s)

B. Fields can vary by scope

C. Additional notes for the reader

1. Notes sections

notes_comp_NNN notes_port_NNN notes_base_NNN notes_peak_NNN notes_submit_NNN notes_os_NNN notes_plat_NNN notes_part_NNN notes_NNN

2. Note numbering

3. Additional tags

4. Links in notes sections

A. Introduction

1. Example: Picking Model

2. Syntax Basics

B. Defining macros

1. In a config file

2. On the command line

3. Predefined macros: endian, hostname...
%{ENV_variable_name} New with SPEChpc 2021

4. Example: Adjusting the Environment

5. Redefinition

C. Un-doing macro definition

D. Using macros

E. Conditionals

1. %ifdef .. %endif

2. %ifndef .. %endif

3. %if .. %endif

4. %else

5. %elif

F. Informational directives

1. %warning

2. %error

3. %info New with SPEChpc 2021

4. %dumpmacros New with SPEChpc 2021

A. Help, I've got too many config files!

B. The log file

1. Useful Search Strings

2. About Temporary Debug Logs

3. Verbosity levels

D. Help, I've got too many log files!

E. Finding the build directory

F. Files in the build directory

G. For more information

I.A. What is a config file?

A SPEChpc config file is a file that defines how to build, run, and report on the SPEChpc benchmarks in a particular environment. It defines how SPEChpc 2021 interacts with your test environment, and lets you describe that environment in human-readable form.

A config file provides detailed control of testing, including:

• Tuning and portability, which you may customize for your compilers.
• The runtime environment, which you may customize for your operating system.
• A description of the SUT (System Under Test), which you should customize to create an accurate report.

Using your customized options, the SPEChpc tools automatically create Makefiles, build the benchmarks, run them, generate reports, and write log files.

Because they collect your options into one place, config files are key to result reproducibility.

For example, if a vendor publishes SPEChpc 2021 results for the SuperHero Model 42 at the SPEC web site www.spec.org/hpc2021, it is expected that a customer can demonstrate similar performance using only 3 ingredients:

1. The SPEChpc 2021 media, ordered from SPEC.
2. The software and hardware as specified in the full disclosure from the published result.
3. The config file, which is posted along with the results.

The compiler is described in the main config file on lines 10-12.
The hardware is described in SUT.inc which is included via line 6.

$ cat -n include.cfg
 1  iterations    = 1
 2  output_format = text
 3  output_root   = /tmp/example
 4  runlist       = 605.lbm_s
 5  size          = test
 6  test_sponsor    = Turboblaster
 7  tester          = Turboblaster
 8  include: SUT.inc
 9  default:
10    CC                = mpicc
11    CC_VERSION_OPTION = -V
12    sw_compiler000   = C/C++/Fortran: Version 9.2.0 of
13    sw_compiler001   = GNU Compiler Collection
14    sw_avail         = Aug-2019
15    sw_mpi_library   = OpenMPI Version 3.1.5
$

Note the software date (sw_avail) on line 14 above.
Today's SUT is the Turboblaster SuperHero IV.
Copy its information to the expected place.

$ cp Turboblaster.inc SUT.inc
$ cat Turboblaster.inc 
default:
node_compute_label = TurboBlaster 5000
node_compute_order = 1
node_compute_count = 2
node_compute_purpose = compute
node_compute_hw_vendor = Mega Technology
node_compute_hw_model = Turblaster 5000
node_compute_hw_cpu_name = Turbo CPU
node_compute_hw_ncpuorder = 1 chips
hw_avail  = Feb-2018
$

Note the hardware date (hw_avail) in the include file.
Both dates are referenced in reports:

$ cat include.sh 
runhpc --config=include | grep txt
grep avail /tmp/example/result/*txt
$ ./include.sh 
format: Text -> /tmp/example/result/hpc2021_sml.001.small.test.txt
Test sponsor: Turboblaster      Hardware availability: Feb-2018
Tested by:    Turboblaster      Software availability: Aug-2019
$ 

I.D.7. Section markers and Named sections

A named section begins with a section marker and continues until the next section marker or the HASH section is reached.

Named sections can be entered in any order.
Section markers can be repeated. Material from repeated sections is automatically consolidated.

A section marker is a one- to three-part string of the form:

   benchmark[,...]=tuning[,...]=label[,...]:

The three parts of a section marker are called the section specifiers, with allowed values:

Trailing default section specifiers may be omitted from a section marker. In the pairs below, in each case, the second line is equivalent to the first:

small=default=default:     605.lbm_s=base=default:      default=default=default:
small:                     605.lbm_s=base:              default:

I.E. Section merging and precedence rules

By constructing section markers, you determine how you would like your options applied. Benchmarks are built according to instructions in the sections that they match, subject to rules for combining sections and resolving conflicts among them. Sections are combined using these rules.
Click to go to an example.

• For the benchmark specifier, the precedence is:

  highest   named benchmark(s)
            suite name 
  lowest    default 

  (See also Appendix A)

• For the tuning specifier, base or peak has higher precedence than default.
• For the label specifier, any named label is at a higher precedence level than default.
• Combining sections:
  1. Combine sections that apply to a benchmark, if there is no conflict among them.

  2. If sections conflict with each other, the order of precedence is:

     highest     benchmark
                 tuning
     lowest      label 

• Section order:

  • For sections at differing precedence levels, order does not matter.

  • If a section occurs more than once, the settings are combined. If there are conflicts, the last instance wins.

  highest   named benchmark(s)
            suite name
  lowest    default

$ cat -n precedence_example1.cfg
     1  flagsurl             = $[top]/config/flags/gcc.xml
     2  iterations           = 1
     3  output_format        = text
     4  output_root          = /tmp/ptest
     5  runlist              = 605.lbm_s,705.lbm_l,805.lbm_l
     6  size                 = test
     7  default:
     8     CC_VERSION_OPTION = -v
     9     CC                = mpicc
    10  701.lbm_s:
    11     OPTIMIZE          = -O3
    12  small:
    13     OPTIMIZE          = -O2
    14  medium:
    15     OPTIMIZE          = -O1
    16  default:
    17     OPTIMIZE          = -O0

$ cat -n precedence_example2.cfg 
     1  flagsurl             = $[top]/config/flags/gcc.xml
     2  iterations           = 1
     3  output_format        = text
     4  output_root          = /tmp/ptest2
     5  runlist              = 605.lbm_s
     6  size                 = test
     7  default=default:
     8     CC_VERSION_OPTION = -v
     9     CC                = mpicc
    10     OPTIMIZE          = -O0
    11  default=base:
    12     OPTIMIZE          = -O1
    13  default=peak:
    14     OPTIMIZE          = -O3
$ cat precedence_example2.sh 
runhpc --config=precedence_example2 --tune=base | grep txt
runhpc --config=precedence_example2 --tune=peak | grep txt
cd /tmp/ptest2/result
grep 'O[0-9]' *txt
$ ./precedence_example2.sh 
    format: Text -> /tmp/ptest2/result/hpc2021.001.small.test.txt
    format: Text -> /tmp/ptest2/result/hpc2021.002.small.test.txt
hpc2021.001.small.test.txt: 605.lbm_s: -O1
hpc2021.002.small.test.txt: 605.lbm_s: -O3
$

$ cat -n precedence_example3.cfg 
 1  runlist              = 605.lbm_s
 2  size                 = test
 3  small=base=default:
 4     CC_VERSION_OPTION = -v
 5     CC                = mpicc
 6     OPTIMIZE          = -O0
 7  small=base=OhTwo:
 8     OPTIMIZE          = -O2
$ cat precedence_example3.sh 
runhpc --config=precedence_example3 --fake --label=OhTwo | grep lbm.c
$ ./precedence_example3.sh 
mpicc -c -o lbm.o -DSPEC  -DNDEBUG -O2 lbm.c
$  

$ cat -n precedence_example4.cfg 
  1  label                = wall
  2  runlist              = 605.lbm_s
  3  tune                 = peak
  4  default:
  5     CC_VERSION_OPTION = -v
  6     CC                = mpicc
  7  small:
  8     OPTIMIZE          = -O1 
  9  default=peak:
 10     COPTIMIZE         = -ftree-vectorize
 11  default=default=wall:
 12     EXTRA_COPTIMIZE   = -Wall
$ cat precedence_example4.sh  
runhpc --config=precedence_example4 --fake | grep lbm.c
$ ./precedence_example4.sh 
mpicc -c -o lbm.o -DSPEC -DNDEBUG -O1 -ftree-vectorize           -Wall     lbm.c
$  

highest     benchmark
            tuning
lowest      label 

$ cat -n precedence_example5.cfg 
  1  flagsurl             = $[top]/config/flags/gcc.xml
  2  iterations           = 1
  3  output_format        = text
  4  output_root          = /tmp/ptest
  5  size                 = test
  6  default:
  7     CC_VERSION_OPTION = -v
  8     CC                = mpicc
  9  small:
 10     OPTIMIZE          = -O3 
 11  default=peak:
 12     OPTIMIZE          = -O2
 13  default=default=wall:
 14     OPTIMIZE          = -O1
 15  default:
 16     OPTIMIZE          = -O0
$ cat precedence_example5.sh 
runhpc --fake --config=precedence_example5 --label=wall 705.lbm_l         | grep lbm.c
runhpc --fake --config=precedence_example5 --label=wall -T peak  705.lbm_l | grep lbm.c
runhpc --fake --config=precedence_example5 --label=wall -T peak  605.lbm_m | grep lbm.c
$ ./precedence_example5.sh 
mpicc -c -o lbm.o -DSPEC -DNDEBUG -O1    lbm.c
mpicc -c -o lbm.o -DSPEC -DNDEBUG -O2    lbm.c
mpicc -c -o lbm.o -DSPEC -DNDEBUG -O3    lbm.c
$  

$ diff --side-by-side precedence_example6a.cfg precedence_example6b.cfg
iterations           = 1                            iterations           = 1
output_format        = text                         output_format        = text
output_root          = /tmp/ptest                   output_root          = /tmp/ptest
runlist              = 605.lbm_s                    runlist              = 605.lbm_s
size                 = test                         size                 = test
default:                                            default:
   CC_VERSION_OPTION = -v                              CC_VERSION_OPTION = -v
   CC                = mpicc                             CC                = mpicc
605.lbm_s:                                        <
   OPTIMIZE          = -O3                        <
small:                                            <
   OPTIMIZE          = -O1                        <
default:                                            default:
   OPTIMIZE          = -O0                             OPTIMIZE          = -O0
                                                  > small:
                                                  >    OPTIMIZE          = -O1
                                                  > 605.lbm_s:
                                                  >    OPTIMIZE          = -O3
$ cat precedence_example6.sh
runhpc --fake --config=precedence_example6a | grep lbm.c
runhpc --fake --config=precedence_example6b | grep lbm.c
$ chmod +x precedence_example6.sh
$ ./precedence_example6.sh 
mpicc -c -o lbm.o -DSPEC -DNDEBUG -O3  lbm.c
mpicc -c -o lbm.o -DSPEC -DNDEBUG -O3  lbm.c
$  

 7  small:
12  small=default:
17  small=default=default:

$ cat -n precedence_example7.cfg
  1  label                = wall
  2  runlist              = 605.lbm_s
  3  tune                 = peak
  4  default:
  5     CC_VERSION_OPTION = -v
  6     CC                = mpicc
  7  small:
  8     OPTIMIZE          = -O1 
  9     EXTRA_CFLAGS      = -finline-functions
 10  medium:
 11     OPTIMIZE          = -O0 
 12  small=default:
 13     OPTIMIZE          = -O2 
 14     COPTIMIZE         = -ftree-vectorize
 15  medium=peak:
 16     OPTIMIZE          = -O0 
 17  small=default=default:
 18     OPTIMIZE          = -O3 
 19     EXTRA_COPTIMIZE   = -Wall
$ cat precedence_example7.sh 
runhpc --config=precedence_example7 --fake | grep lbm.c 
$ ./precedence_example7.sh 
mpicc -c -o lbm.o -DSPEC -DNDEBUG  
    -O3 -ftree-vectorize  -finline-functions  -Wall   lbm.c
$  

I.F. Variables and Variable Substitution

I.F.1. Defining variables

You can create your own runhpc variables using a line of the form

name = value

The name may contain only letters, digits, and underscores (a hyphen is NOT allowed).
Start with a letter.
You may indent your definitions if you wish (see: whitespace)

Exception: preprocessor macros are different on all of the above. Hyphens are allowed, use

%define name value

and the % must appear in column 1. You can indent after the % if you wish.

Conventions: Although not required, certain conventions are usually followed:

1. lower case is typically used for macros. When combining words, CamelCase may be useful.
2. lower case is also typically used for runhpc variables, combining words with underscores
3. CAPITALS are typically used when creating make variables
4. CAPITALS are also typically used for shell or environment variables.

Examples of the above (in the same order)

%ifndef processorNumaControl
%   define processorNumaControl firstTouch # macro
%endif
default:
    my_submit_cmd       = numactl -C $BIND # runhpc variable
    MYTUNE              = -O2 --math=SIMD  # make variable
    ENV_LD_LIBRARY_PATH = /opt/lib         # environment variable

The remainder of this section I.F is about runhpc variables -- the $[square] and ${curly} rows from the table at the top.

I.F.2. $[square] substitution at startup

Immediately after preprocessing, variables that are delimited by $[square brackets] are substituted.
Any value set in the config file can be substituted, provided that is visible in the current scope.
You can access the value of additional variables that you may have created.
See the list of useful variables below.
Perhaps the most useful is $[top] for the top of the SPEChpc 2021 tree, often found in contexts such as:

    flagsurl01          = $[top]/config/compiler.xml
    flagsurl02          = $[top]/config/platform.xml
    EXTRA_LIBS          = $[top]/mylibs
    preENV_LIBRARY_PATH = $[top]/lib64:$[top]/lib

Traps for the unwary: In some cases it may be obvious to the human which value to use, but the tools aren't as smart as you.

1. Timing: You cannot substitute variables that do not exist, or whose values aren't known when the config file is first read. For example, the label is not sorted out until later. Values that pertain to individual benchmarks (such as the current iteration number or the name of the benchmark being run) are not set until much later, when benchmarks are actually being run.
2. Command line: If a value is mentioned on both the runhpc command line and in the config file, the substitution might not do what you expect.

3. output_root:
   You cannot set an output_root that depends on a runhpc variable.
   You can set one that uses a macro:

   output_root=$[top]/my/path      # wrong
   output_root=${top}/my/path      # wrong
   output_root=%{ENV_SPEC}/my/path # right

Square substitution is done early. That comes in handy if you need a variable right away, for example, in order to use it with preENV.

$ cat EarlySub.cfg 
SW_DIR                 = /opt/path/to/compilers
preENV                 = 1
preENV_LD_LIBRARY_PATH = $[SW_DIR]/lib

$ cat EarlySub.sh
runhpc --config=EarlySub --fake 605.lbm_s | grep '^LD' | uniq
$ ./EarlySub.sh 
LD_LIBRARY_PATH=/opt/path/to/compilers/lib
$  

The example below uses variables defined in several named sections. The sections are delimited by section markers default: (line 8), default=base: (line 11), and default=peak: (line 15).

$ cat -n square.cfg
     1  expand_notes         = 1
     2  iterations           = 1
     3  output_format        = text
     4  output_root          = /tmp/square
     5  runlist              = 605.lbm_s
     6  size                 = test
     7  tune                 = base,peak
     8  default:
     9     CC                = mpicc
    10     CC_VERSION_OPTION = -v
    11  default=base:
    12     the_system        = STAR
    13     OPTIMIZE          = -O1
    14     notes_base_100    = base tuning uses '$[CC]' '$[OPTIMIZE]' on system '$[the_system]'
    15  default=peak:       
    16     OPTIMIZE          = -O2
    17     notes_peak_100    = peak tuning uses '$[CC]' '$[OPTIMIZE]' on system '$[the_system]'
    18  
$ cat square.sh 
runhpc --config=square | grep txt
grep tuning /tmp/square/result/hpc2021.001.small.test.txt
$ ./square.sh 
    format: Text -> /tmp/square/result/hpc2021.001.small.test.txt
     base tuning uses 'mpicc' '-O1' on system 'STAR'
     peak tuning uses 'mpicc' '-O2' on system ''
$  

Note that line 14 finds all three variables that it is looking for, but line 17 does not. If it is not clear why this happens, please see the descriptions of named sections and precedence above.

I.F.3. Useful $[square] variables

Useful $[variables] include:

You can access the initial value of most options that you can enter into a config file, including:

I.F.4. ${curly} substitution (interpolation) during a run
+ $unbracketed substitution

runhpc uses perl interpolation.
Only scalars (not: perl arrays and hashes) can be interpolated.

Example: on the notes100 line, you could optionally add say either ${lognum} or $lognum, but don't try taking the curly brackets away from ${size}.

$ cat just.cfg 
expand_notes      = 1
notes100          = Just ${size}ing, in run $lognum 
output_root       = /tmp/just
runlist           = 605.lbm_s
size              = test
CC                = mpicc
CC_VERSION_OPTION = -v
$ 
$ cat just.sh
runhpc -c just | grep txt
grep Just /tmp/just/result/hpc2021.001.small.test.txt
$   
$ ./just.sh 
    format: Text -> /tmp/just/result/hpc2021.001.small.test.txt
     Just testing, in run 001
$ 

Traps for the unwary

1. Try not to confuse ${curly} interpolation vs. %{curly} (macros)

2. Timing: Some variables are only defined at certain times, and a line that uses it might be interpolated at a different time. Therefore interpolation won't always do what you might wish. In particular, notes are not expanded in the context of a particular benchmark run, and therefore variables such as ${tune} are not useful within them.

3. output_root:
   You cannot set an output_root that depends on a runhpc variable.
   You can set one that uses a macro:

   output_root=$[top]/my/path      # wrong
   output_root=${top}/my/path      # wrong
   output_root=%{ENV_SPEC}/my/path # right

I.F.5. Useful ${curly} variables

These variables may be of interest:

For a complete list of the available variables relative to the current config file, set

expand_notes = 1
verbose      = 35 # or higher

Then, do a run that causes a command substitution to happen.
In the log, you will find many lines of the form:

 - Variables available for interpolation that have changed since the last list:
    (From config) $runmode = "speed"
    (From config) $size = "test"
 - Variables available for interpolation that have changed since the last list:
    (From config) $size = "ref"

I.F.6. Unsetting a variable with "%undef%"

It is sometimes useful to undo the setting of a variable that would otherwise be included from another section. This can be accomplished using the special value %undef%. In the example, line 14 undefines COPTIMIZE when compiling peak:

 $ cat -n dave.cfg 
 1  flagsurl             = $[top]/config/flags/gcc.xml
 2  iterations           = 1
 3  output_format        = text
 4  output_root          = /tmp/undef
 5  runlist              = 605.lbm_s
 6  size                 = test
 7  tune                 = base,peak
 8  default:
 9     CC_VERSION_OPTION = -v
10     CC                = mpicc
11     OPTIMIZE          = -O2
12     COPTIMIZE         = -fno-tree-pre
13  605.lbm_s=peak:
14     COPTIMIZE         = %undef%
15  
$ runhpc --config=dave | grep txt
    format: Text -> /tmp/undef/result/hpc2021.001.small.test.txt
$ cd /tmp/undef/benchspec/HPC/605.lbm_r/build
$ grep OPTIMIZE build_base_none.0000/Makefile.spec 
COPTIMIZE        = -fno-tree-pre
OPTIMIZE         = -O2
$ grep OPTIMIZE build_peak_none.0000/Makefile.spec 
COPTIMIZE        = 
OPTIMIZE         = -O2
$ 

Notice that in the build directory, COPTIMIZE is present for base and absent for peak.

I.F.7. Debug tips for runhpc variables

When debugging a config file that uses runhpc variables, try:

iterations       = 1
minimize_rundirs = 0
reportable       = 0
runlist          = (one or two benchmarks)
size             = test 
verbose          = 40

Using --fake will probably be informative. Look inside the log for the (case-sensitive) word 'From'.

II. Config file options for runhpc

This section documents options that control the operation of runhpc itself.

II.A. Precedence: config file vs. runhpc command line

In the list that follows, some items are linked to the document Using SPEChpc 2021 - the 'runhpc' Command because they can be specified either in a config file, or on the runhpc command line.

New with SPEChpc 2021, If an option is specified in both places, the command line wins.

II.B. Options

In the table that follows, if an option is documented as accepting the values "no" and "yes", these may also be specified as "false" and "true", or as "0" and "1".

The "Use In" column indicates where the option can be used:

submit = send_job job="${command}" rank_id=$rank

         send_job job="exe >out" rank_id=n

         send_job job="exe" rank_id=n >out

 size        = test
 tune        = peak
 label       = srini
+env_vars    = 1
 #
 605.lbm_s=peak:
+   ENV_LD_LIBRARY_PATH = /tmp/compiler/lib64:%{ENV_LD_LIBRARY_PATH}
    CC                  = alt_mpicc
    CC_VERSION_OPTION   = -v
    OPTIMIZE            = -O1
 

605.lbm_s=default:
#> I am posting this config file for use by others in the
#> company, but am forcing it to fail here because
#> I want to force users to review this section.
#>
#> Once you find your way here, you should test whether
#> bug report 234567 has been fixed, by using the first
#> line below.  If it has not been fixed, then use the
#> second.  In either case, you'll need to remove the
#> fail_build.
#>
#>   - Pney Guvaxre
#>     Boomtime, the 66th day of Confusion in the YOLD 3172

OPTIMIZE = -Osuperduper
# OPTIMIZE = -Omiddling
fail_build = yes

     flagsurl1 = mycompiler.xml
     flagsurl2 = myplatform.xml 

http_proxy = http://webcache.tom.spokewrenchdad.com:8080 

label=jun12.old.CC 
label=jun12.new.CC 
label=jun14-flagday 
label=jun15-robert.wants.yet.another.test

runhpc --label=sandra [...]
ERROR: The label 'sandra' defines no settings in the config file!

ranks               = 4
output_format        = text
runlist              = 605.lbm_s
default:
   CC_VERSION_OPTION = -v
   CC                = mpicc
default=base=OhZero:
   OPTIMIZE          = -O0
default=base=OhThree:
   OPTIMIZE          = -O3 

$ find . -type d -name '*OhZ*'
./benchspec/HPC/605.lbm_s/build/build_base_OhZero.0000
./benchspec/HPC/605.lbm_s/run/run_base_test_OhZero.0000
./benchspec/HPC/605.lbm_s/run/run_base_test_OhZero.0001
./benchspec/HPC/605.lbm_s/run/run_base_test_OhZero.0002
$ cd benchspec/HPC/605.lbm_s/build
$ grep OPTIMIZE */Makefile.spec
build_base_OhThree.0000/Makefile.spec:OPTIMIZE   = -O3
build_base_OhZero.0000/Makefile.spec:OPTIMIZE   = -O0
$ 

   mailto=fast.guy@welovebenchmarks.org
   output_format=text,mail

   mailto=fast.guy@welovebenchmarks.org
   output_format=text,mail
   mail_reports=text

runhpc --output_root=/local/home/junjie --config=mine 605.lbm_s 

default:
tiny:
small:
medium:
large:

preENV_LD_LIBRARY_PATH = %{ENV_SPEC}/libraries:%{ENV_LD_LIBRARY_PATH}

$ cd $SPEC   
$ specxz -dc nnn.benchmark.FixMumble.tar.xz | spectar -xvf -
$ cat README.nnn.benchmark.src.alt.FixMumble.txt

$ cat testme.cfg
action               = buildsetup
runlist              = nnn.benchmark
tune                 = base
default:
   CC                = mpicc
   CC_VERSION_OPTION = -v
nnn.benchmark=base=without:
   OPTIMIZE          = -O2
nnn.benchmark=base=with:
   OPTIMIZE          = -O2
   srcalt            = FixMumble

runhpc --label=without --config=testme
runhpc --label=with    --config=testme

III. Config file options for specmake

For SPEChpc you do not write Makefiles. Instead, you set Make variables in the config file, which are sent to a SPEC-supplied copy of GNU Make, known as specmake. Variables with a dollar sign and parentheses, aka "round brackets", are substituted by specmake. For example:

COMPILER_DIR = /usr/local/bin/
CC           = $(COMPILER_DIR)cc
CXX          = $(COMPILER_DIR)c++
FC           = $(COMPILER_DIR)f90

See below for more information on syntax of variables that you create and reference.

III.A. Commonly used Make variables

• Basic specmake variables are described in this section.
• Many more are described in the Make Variables document.
• The specmake utility can be run directly from the command line, as described in Utilities.

The following Make variables are frequently useful. When selecting where to put a flag, please bear in mind that the run rules require that portability flags must use PORTABILITY variables.

III.B. Parallel Model Selection

By default, the SPEChpc benchmarks use only MPI parallelism. However, you may also enable either OpenACC or OpenMP directives to enable node-level parallelism. The benchmark will then be hybrid MPI+OpenACC or MPI+OpenMP.

For OpenMP, you may select either "traditional" OpenMP using CPU task/threads, or "Target" OpenMP which includes OpenMP 5.0 'target' constructs that may use to target a host CPU or offload to accelerator device.

Note if you choose to use a node-level parallel model, the same model must be used for all benchmarks in base. For peak, you may select the model on a per benchmark basis.

To enable a node-level parallel mode, use the appropriate 'pmodel' setting. The tools then set the appropriate preprocessor define flag which makes the directives visible in the source. Note that not all compilers support all models and the compiler flag that enables the model must be set in your OPTIMIZE variable in the config file. Please consult your compiler's documentation for the supported models and options.

Note that using the define flags directly in you OPTIMIZE or PORTABILITY flags will cause the tools to error. This is to help reduce the possibiltiy of setting the pmodel to one model but then accidently enabling a different model.

The config file fragment below demonstrates how to conditionally include a parallel model selected via runhpc --define option:

% cat -n config/pmodel.cfg
     1  # If model is not set use Only MPI
     2  %ifndef %{model}
     3  pmodel=MPI
     4  %endif
     5
     6  # Compiler flags applied to all models
     7  default:
     8    CC             = mpicc
     9    OPTIMIZE       = -O3
    10    CXXPORTABILITY = --c++14
    11    submit = mpirun --bind-to none -mp $ranks $command
    12
    13  # OpenACC flags
    14  %if %{model} eq 'acc'
    15    pmodel = ACC
    16    OPTIMIZE += -acc
    17  %endif
    18
    19  # OpenMP (CPU Threaded) flags
    20  %if %{model} eq 'omp'
    21     pmodel = OMP
    22     OPTIMIZE += -mp
    23  %endif
    24
    25  # OpenMP (Target) flags
    26  %if %{model} eq 'omptgt'
    27     pmodel=TGT
    28     OPTIMIZE += -mp=gpu
    29  %endif
    30
    31  605.lbm_s=peak:
    32    pmodel = ACC
    33    OPTIMIZE = -acc -fast
    34
    35  # This will error since the model define flags
    36  # should only be set by the tools
    37  705.lbm_m=peak:
    38    OPTIMIZE = -mp -fast -DSPEC_OPENMP
    39
% runhpc -c pmodel --fake -ranks=4 605.lbm_s | grep lbm.c
mpicc -c -o lbm.o -DSPEC -DNDEBUG   -O3                  lbm.c
% runhpc -c pmodel --fake -ranks=4 --define model=acc 605.lbm_s | grep lbm.c
mpicc -c -o lbm.o -DSPEC -DNDEBUG   -O3 -acc           -DSPEC_OPENACC        lbm.c
% runhpc -c pmodel --fake -ranks=4 --define model=omp 605.lbm_s | grep lbm.c
mpicc -c -o lbm.o -DSPEC -DNDEBUG   -O3 -mp           -DSPEC_OPENMP        lbm.c
% runhpc -c pmodel --fake -ranks=4 --define model=omptgt 605.lbm_s | grep lbm.c
mpicc -c -o lbm.o -DSPEC -DNDEBUG   -O3 -mp=gpu           -DSPEC_OPENMP_TARGET        lbm.c
% runhpc -c pmodel --fake -ranks=4 --tune peak 605.lbm_s | grep lbm.c
mpicc -c -o lbm.o -DSPEC -DNDEBUG   -acc -fast           -DSPEC_OPENACC        lbm.c

Note for peak only OpenACC is used from 605.lbm_s since it's not guarded by the define.

Using a pmodel's define flag in the config will cause the following error:

% runhpc -c pmodel --fake -ranks=4 --tune peak 705.lbm_m
SPEC HPC(r) 2021 Benchmark Suites
Copyright 1995-2021 Standard Performance Evaluation Corporation (SPEC)

runhpc v.unknown
Using 'linux-x86_64' tools
Reading file manifests... read 16870 entries from 2 files in 0.08s (200294 files/s)
... cut ...
Compiling Binaries

ERROR: pmodel define flag "-DSPEC_OPENMP" may only be set by the tools.
Please remove it from the following line of your config file.
OPTIMIZE         = -acc -fast -DSPEC_OPENMP
... cut ... 

III.C. Creating your own Make variables

Variables with a dollar sign and parentheses, aka "round brackets", are substituted by specmake.

Deprecated feature alert: Although it is also possible to pass information to specmake using curly brackets: ${COMPILER_DIR}, this is not recommended. Instead, you should consistently use curly brackets to address runhpc and round brackets to address specmake. It is possible that a future version of runhpc may insist on interpolating curly brackets itself, rather than allowing specmake to do so.

Example:

$ cat makevar.cfg 
action            = build
runlist           = 605.lbm_s
default:
   DEBUG_SYMBOLS  = --debug:symbols=expanded_info_level:42
   EXTRA_FFLAGS   = $(GEE)
small=base:
   GEE            = -g $(DEBUG_SYMBOLS)
small=peak:
   GEE            = -g 

$ cat makevar.sh
runhpc --config=makevar --fake --tune=base | grep COMP:
runhpc --config=makevar --fake --tune=peak | grep COMP:
$ ./makevar.sh 
COMP: "cc -c -o lbm.o -g --debug:symbols=expanded_info_level:42 <source>"
COMP: "cc -c -o lbm.o -g <source>"
$  

The config file above creates two variables options (DEBUG_SYMBOLS and GEE). Both are passed to specmake, which interprets them. The results are shown above using the runhpc --fake option.

For an extensive example of variable substitution handled by specmake, see the SPEC CPU®2000 example at www.spec.org/cpu2000/docs/example-advanced.cfg. Search that file for LIBS, and note the long comment which provides a walk-through of a complex substitution handled by specmake.

III.D. The operator "+=" is available (but should be used with caution)

The operator "+=" adds to specmake variables. It may be convenient; it also may cause hard-to-diagnose bugs. Example:

$ cat tmp.cfg  
action  = build
runlist = 605.lbm_s
tune    = peak
default:
   OPTIMIZE   = -O1
fprate=default:
   OPTIMIZE  += --unroll
default=peak:
   OPTIMIZE  += --inner_unroll
605.lbm_s=peak:
   OPTIMIZE  += --outer_unroll
default=default=breakfast:
   OPTIMIZE   = --jelly_roll

$ runhpc --fake --config=tmp | grep lbm.c
cc -c -o lbm.o -DSPEC -DSPEC_CPU -DNDEBUG -O1 --inner_unroll --unroll --outer_unroll lbm.c
$   

Note that the options accumulate.

Caution: although the += operator adds flexibility, it may introduce hard-to-predict behavior, depending on precedence of section specifiers, the order of your config file, and other features, such as include files. Instead of using '+=' try picking different make variables for different purposes. For an example of hard-to-predict behavior, what will happen if you add --label=breakfast to the above runhpc command? (Try it.)

Recommendations:
Avoid += to prevent surprises.
Keep it simple.
Pick different makevars for different purposes.
Create conventions for your config files and write them down, in config file comments.
If you must use += review its effects carefully (--fake is your friend).

III.E Using action=buildsetup to create a sandbox

When debugging a set of build options, it is often useful to create a "sandbox" - that is, a directory where you can play with the benchmark and its options. This example creates a build sandbox with action buildsetup.

$ cat sandbox.cfg 
action        = buildsetup
label         = fast
output_root   = /tmp/demo_buildsetup
runlist       = 605.lbm_s
tune          = peak
default=peak:
   OPTIMIZE   = -fast
$ cat sandbox.sh 
runhpc --config=sandbox | grep log
grep Makefile.spec /tmp/demo_buildsetup/result/hpc2021.001.log
$ ./sandbox.sh 
The log for this run is in /tmp/demo_buildsetup/result/hpc2021.001.log
Wrote to makefile '/tmp/demo_buildsetup/benchspec/HPC/605.lbm_s/build/build_peak_fast.0000/Makefile.spec':

The action causes a directory to be created
There, the Makefile can be examined, used in a dry run, or modified as part of a testing effort.

$ cd /tmp/demo_buildsetup/benchspec/HPC/605.lbm_s/build/build_peak_fast.0000/
$ grep OPTIMIZE Makefile.spec 
OPTIMIZE         = -fast
$ specmake --dry-run
cc -c -o lbm.o -DSPEC -DNDEBUG   -fast                  lbm.c
cc -c -o main.o -DSPEC -DNDEBUG   -fast                  main.c
cc -c -o specrand/specrand.o -DSPEC -DNDEBUG   -fast                  specrand/specrand.c
cc      -fast          lbm.o main.o specrand/specrand.o             -lm         -o lbm

See also the chapter on specmake in SPEChpc 2021 Utilities and the sandbox examples in Avoiding runhpc.

III.F. About Automatic Rebuilds

The SPEChpc tools try to keep config files and binaries synchronized with each other.(*)
Edits to a config file may cause binaries to be rebuilt, sometimes to the surprise(**) of testers.

Testing option sensitivity: The first thing that happens in a rebuild is to delete the old binary.
If that is a potential problem (perhaps it takes a long time to build), you can test whether a config file change will cause a rebuild:

1. Copy the config file to a test version (say, mycopy.cfg)
2. Edit the copy.
3. Remove any lines that change defaults for ignore_errors or verify_binaries
4. Use: runhpc --config=mycopy --nobuild --size=test --iterations=1 ...

The command:

• completes quickly (because --size=test and iterations=1)
• exits if binaries are not up to date, because --nobuild

No binaries are harmed.

IV. Config file options for the shell

Some options in your config file cause commands to be executed by your shell (/bin/sh).

IV.A. \$SHELLVAR variable substitution

Substitution by the shell uses backslash dollar sign.
The backslash protects the variables from interpretation by runhpc.

$ cat tmp.cfg
expand_notes         = 1
iterations           = 1
runlist              = 605.lbm_s
size                 = test
tune                 = base,peak
default:
   CC                = mpicc
   CC_VERSION_OPTION = -v
default=base:
   submit    = echo home=$HOME, spec=$SPEC   > /tmp/chan; ${command}
default=peak:
   submit    = echo home=\$HOME, spec=\$SPEC > /tmp/nui;  ${command}
$ runhpc --config=tmp | grep txt
    format: Text -> /Users/bo/spec/hpc2021/result/hpc2021.697.small.test.txt
$ cd /tmp
$ cat chan nui
home=, spec=
home=/Users/chris, spec=/Users/chris/spec/hpc2021
$
   
In base, $HOME and $SPEC are gobbled up by runhpc, which
   obediently retuns their values: nothing at all.  
In peak, backslashes prevent runhpc from interpreting
   the variables, and the shell provides their
   values.

IV.B. Shell Options

Warning: SPEChpc config files can execute arbitrary shell commands.
Read a config file before using it.
Don't be root. Don't run as Administrator. Turn privileges off.

These options cause commands to be executed:

IV.C. Using submit

The config file feature submit allows you specify the command needed to launch your MPI command as well as include and process to core bindings. This section provides examples to demonstrate how submit works with several other config file features. You might also want to search published results at www.spec.org/hpc2021 for systems that are similar to your system.

IV.C.1. Basic usage of submit

You can use submit to specify the MPI or batch scheduler command used to launch your MPI jobs.

The example below runs 4 ranks using OpenMPI's "-bind-to cores" option to bind ranks to cores.

$ cat submitmpi.cfg 
ranks   = 4
runlist  = 605.lbm_s
submit   = mpirun --bind-to cores -np ${ranks} ${command}
$ cat submitmpi.sh
runhpc --fake --config=submitmpi | grep '^mpirun' | cut -b 1-75
$ ./submitmpi.sh 
mpirun --bind-to cores -np 4 ../run_base_ref_none.0000/lbm_base.none 
$  

IV.C.2. Using scripts with submit for thread binding per rank

You can use scripts with submit to specify the thread binding for individual MPI ranks. While setting environment variables such as OMP_PLACES in your config's environment setting or using utilities such as numactl work fine when there's a single rank per node, using these with multiple ranks would cause the threads from each rank to be bound to the same core. Several MPI and batch schedulers will perform this binding correctly for you. However in cases where it doesn't, you can provide a script to perform the binding.

The example below runs 8 ranks with each running 16 OpenMP threads. The example uses a perl script to determine the local rank number via OpenMPI's OMPI_COMM_WORLD_LOCAL_RANK environment variable, and then bind the rank's threads via the numactl. Note: please check your MPI distribution's documentation for the equivalent environment variable.

The sample for a two socket system with 128 cores with 8 NUMA nodes. You would need to modify the script for different systems.

$ cat bind8.pl
use strict;
my %bind;
$bind{0} = "0-15";
$bind{1} = "16-31";
$bind{2} = "32-47";
$bind{3} = "48-63";
$bind{4} = "64-79";
$bind{5} = "80-95";
$bind{6} = "96-111";
$bind{7} = "112-127";

my %mem;
$mem{0} = "0";
$mem{1} = "1";
$mem{2} = "2";
$mem{3} = "3";
$mem{4} = "4";
$mem{5} = "5";
$mem{6} = "6";
$mem{7} = "7";


my $rank = $ENV{OMPI_COMM_WORLD_LOCAL_RANK};
my $cmd = "numactl -C $bind{$rank} -m $mem{$rank} ";
#my $cmd = "taskset -c $bind{$rank} ";
while (my $arg = shift) {
        $cmd .= "$arg ";
}
my $rc = system($cmd);
exit($rc);

Assuming the bind8.pl is copied to the top directory in your SPEChpc installation, you would then use the following submit command to execute it:

$ cat config/submitmpi2.cdf
ranks    = 8
runlist  = 605.lbm_s
submit   = mpirun --bind-to cores -np ${ranks} perl $[top]/bind8.pl ${command}

IV.C.3. Reporting of submit usage

If you use the submit feature, a notes section will automatically be created to indicate that you have done so.

                                 Submit Notes
                                 ------------
    The config file option 'submit' was used.

You can add notes to that section, or customize it as you wish, by creating lines with notes_submit_NNN. The phrase The config file option 'submit' was used must appear somewhere in your customized notes. You can vary the capitalization of the phrase, you can even break it across multiple lines; it just needs to be present. If it is not, it will automatically be added. If you used scripts in your submit command, the full text of the script should be listed in the submit notes.

V. Config file options for readers

Whether or not you send your result to SPEC, you should fully disclose how you achieved the result. If it requires the installation of the GoFastLinker, you should say so. By setting the appropriate fields in the config file, you can cause information about the GoFastLinker to appear in the reports that are intended for humans.

V.A. Descriptive fields

Here are the fields that you can set to describe your testbed to readers:

      test_sponsor = Genius Compilers
      tester       = Pawtuckaway State College
      system_vendor    = TurboBlaster

The second set of parameters can be continued to multiple lines by repeating the variable name multiple times with different suffixes:

V.A.I Component attributes

A system running MPI may be a standalone SMP or a cluster of nodes with dedicated file servers. The SPEChpc 2021 reporting conventions allow you to describe highly heterogeneous systems by specifying the attributes of multiple kinds of nodes, interconnects and file servers.

The output report will contain a separate section describing each of these components. Each one of these sections consists of a formatted table derived from the fixed-field entries in the config-file, followed by free-form notes derived from the notes entries of the config-file.

Each component is described with a prefix that defines its type, followed by the field to be assigned:

        node_IBM655_label = IBM 655 Server                      # This system has only one kind of node
        node_IBM655_count = 16                                  # and uses 16 of them.
        node_IBM655_count = compute,fileserver                                  # One of the nodes is also the file server.
        node_IBM655_notes = Node 0 is the file server for the cluster.

        node_HP2000_label = HP 2000 Server
        node_HP3000_label = HP 3000 Server                      # This system has two kinds of nodes
        node_HP2000_count = 12                                  # with different numbers of each.
        node_HP3000_count = 20
        node_HP2000_purpose = compute
        node_HP3000_purpose = compute

        interconnect_GBEthernet_data_rate = 1Gb/sec                      # Details of the GB Ethernet interconnect.
        interconnect_InfiniBand_data_rate = 1Gb/sec                      # Details of the InfiniBand interconnect.

        node_SUN1000_label = SUN 1000 Disk Array Server   # Details of the SUN 1000 file server.
        node_SUN1000_count = 1                            # Only one of them.
        node_SUN1000_purpose = fileserver

If you don't declare any components of a given type, no subsection will appear in the report. For example, on an SMT system there will be only one kind of node, and no file server, since the node uses its own local file system. The interconnect is neither replaceable nor configurable so there are no details to report, either.

V.A.II Node description(s)

An SMP system consists of one node of one type. A heterogeneous cluster may consist of many nodes of several types. Most nodes are used for computation but others may act as file servers or provide other utilities. The fields in this section describe the hardware and software configuration of a given node, and are assigned as follows:

        node_HP2000_vendor = Hewlett-Packard Corporation

The available fixed-format fields are as follows:

Continuable fields:

V.A.III Accelerator description(s)

Included as part of the node description, the following fields describe the accelerator used. The fields may be left blank or set to "Not Applicable" if no accelerator was used.

V.A.IV. Adaptor description(s)

The nodes in a compute-cluster use adaptors to attach to the available interconnects. The disclosure convention provides the following fields to declare multiple adaptors of different kinds for each given node type, using a second level of tags

            node_HPCX_hw_adaptor_InfiniBand_model = PCI-Express DDR InfiniBand HCA
            node_HPCX_hw_adaptor_InfiniBand_count = 1

that allow for there being different kinds of adaptors. The defined fields are

V.A.V Interconnect description(s)

The nodes of a compute-cluster are joined via one or more interconnects that will transmit various kinds of data communication. The report will contain one section for each interconnect. An SMT system with no external file server will not use an interconnect for communication and file transfers, so none of the below fields will be filles in and the report will not contain any interconnect sections. The attributes of each interconnect are prefixed as follows:

        interconnect_GBEthernet_label = Gigabit Ethernet             # If more than one interconnect.
        interconnect_mpiComm_label    = MPI traffic interconnect     # It may be named by what it's used for.

First, the fixed-format fields:

Continuable fields..

V.B. Fields can vary by scope.

Fields can appear and disappear based upon scope. For example, if your Small runs used two Fortran compilers (which is allowed for peak), you could construct a config file that adjusts the fields accordingly:

$ cat tmp.cfg
expand_notes      = 1
test_sponsor      = Turboblaster, Inc.
output_format     = text
output_root       = /tmp/fake
default:
   hw_avail       = Feb-2021
   sw_avail       = Jan-2021
   sw_compiler1   = C: V42.0 of TurboBlaster C/C++
   sw_compiler2   = Fortran: V42.2 of TurboBlaster Fortran 
small:
   sw_avail       = Apr-2021
   sw_compiler3   = Fortran: V9.2.0 of gfortran
   notes_comp_100 = In base, all benchmarks use Turboblaster Fortran
   notes_comp_110 = In peak, some benchmarks use Turboblaster Fortran and some
   notes_comp_120 =          benchmarks use gfortran, as noted in the report.
$ runhpc --fakereportable --config=tmp small medium | grep .txt
    format: Text -> /tmp/fake/result/hpc2021_sml.001.small.txt
    format: Text -> /tmp/fake/result/hpc2021_med.001.medium.txt
$ cd /tmp/fake/result
$ grep avail hpc2021_sml.001.small.txt 
 Test sponsor: Turboblaster, Inc.             Hardware availability: Feb-2021
 Tested by:    Turboblaster, Inc.             Software availability: Apr-2021
$ grep avail hpc2021_med.001.medium.txt 
 Test sponsor: Turboblaster, Inc.             Hardware availability: Feb-2021
 Tested by:    Turboblaster, Inc.             Software availability: Jan-2021
$ grep ortran hpc2021_sml.001.small.txt 
                      Fortran: V42.2 of TurboBlaster Fortran
                      Fortran: V9.2.0 of gfortran
     In base, all benchmarks use Turboblaster Fortran
     In peak, some benchmarks use Turboblaster Fortran and some
              benchmarks use gfortran, as noted in the report.
$ grep ortran hpc2021_med.001.medium.txt 
                      Fortran: V42.2 of TurboBlaster Fortran
$   

In the above example, notice that both the compiler information and the availability date changed in the report, depending on the metric.

V.C. Additional notes for the reader

In addition to the pre-defined fields, you can write as many notes as you wish. These notes are printed in the report, using a fixed-width font. For example, you can use notes to describe software or hardware information with more detail beyond the predefined fields:

   notes_os_001  = The operating system used service pack 2 plus patches 
   notes_os_002  = 31415, 92653, and 58979.  At installation time, the 
   notes_os_003  = optional "Numa Performance Package" was selected.

V.C.1. Notes sections

There are various notes sections. If there are no notes in a particular section, it is not output, so you don't need to worry about making sure you have something in each section.

The sections, in order of appearance, are as follows:

1. notes_comp_NNN -- Notes about compiler invocation.
2. notes_port_NNN -- Notes about portability options.
3. notes_base_NNN -- Notes about base optimization options.
4. notes_peak_NNN -- Notes about peak optimization options.
5. notes_submit_NNN -- Notes about use of the submit option.
6. notes_os_NNN -- Notes about operating system tuning and changes.
7. notes_plat_NNN -- Notes about platform tuning and changes.
8. notes_part_NNN -- Notes about component parts (for kit-built systems).
9. notes_NNN -- General notes.

Notes about the submit command are described above, with the description of the submit option.

V.C.2. Note numbering

Start your notes with the name of the notes section where you want the note to appear, and then add numbers to define the order of the lines. Within a section, notes are sorted by line number. The NNN above is not intended to indicate that you are restricted to 3 digits; you can use a smaller or larger number of digits as you wish, and you can skip around as you like: for example, ex-BASIC programmers might naturally use line numbers 100, 110, 120... But note that if you say notes_plat782348320742972403 you just might encounter the dreaded (and highly unusual) "out of memory" error, so don't do that.

You can optionally include an underscore just before the number, but beware: if you say both notes_plat_105 and notes_plat105, both are considered to be the same line. The last one mentioned will replace the first, and it will be the only one output.

V.C.3. Additional tags

For all sections you can add an optional additional tag of your choosing before the numbers. Notes will be organized within the tags.

The intent of the feature is that it may allow you to organize your system information in a manner that better suits your own categories for describing it.

For example:

$ cat notes_tags.cfg 
iterations      = 1
output_format   = text
output_root     = /tmp/notes_tags
runlist         = 605.lbm_s
size            = test
small=base:
   CC                       = mpicc
   CC_VERSION_OPTION        = -V
   notes_part_greeting_011  = ++ how
   notes_part_greeting_20   = ++ you?
   notes_part_greeting_012  = ++ are
   notes_part_aname_1       = ++ Veronica,
   notes_part_080           = ++ hi

$ cat notes_tags.sh  
runhpc --config=notes_tags --fakereport 605.lbm_s | grep txt
grep '++' /tmp/notes_tags/result/hpc2021_sml.001.small.test.txt
$ ./notes_tags.sh
    format: Text -> /tmp/notes_tags/result/hpc2021_sml.001.small.test.txt
     ++ hi
     ++ Veronica,
     ++ how
     ++ are
     ++ you?
$ 

V.C.4. Links in notes sections

You can mention URLs in your notes section, and html reports will correctly render them as hyperlinks. For example:

notes_plat_001 = Additional detail may be found at
notes_plat_002 = http://www.turboblaster.com/servers/big/green/

If you like, you can use descriptive text for the link by preceding it by the word LINK and adding the descriptive text in square brackets:

LINK url AS [descriptive text]

The brackets may be omitted if your descriptive text is a single word, without blanks.

For example:

notes_plat_001 = Additional detail may be found at 
notes_plat_002 = LINK http://www.turboblaster.com/servers/big/green/ AS [TurboBlaster Servers]

When the above appears in an html report, it is rendered as:

 Additional detail may be found at 
 TurboBlaster Servers
      

And in a text report, it appears as:

                                Platform Notes
                                --------------
     Additional detail may found at 
     TurboBlaster Servers (http://www.turboblaster.com/servers/big/green/)

Since the text report is not a context in which the reader can click on a link, it is spelled out instead. Note that because the text report spells the link out, the text line is wider than in HTML, PS, and PDF reports. When deciding where to break your notes lines, you'll have to pick whether to plan line widths for text (which may result in thin-looking lines elsewhere) or plan your line widths for HTML/PS/PDF (which may result in lines that fall of the right edge with text). The feature notes_wrap_columns won't help you here, since it is applied before the link is spelled out.

VI. The config file preprocessor

VI.A. Introduction

The SPEChpc tools include a configuration file macro preprocessor. This section introduces preprocessor examples, and then discusses a few basics of syntax.

VI.A.1. Preprocessor Example: Picking a Model

In this example, the user is testing several configurations, and wants to pick an appropriate pmodel.

$ cat -n preprocessor_example1.cfg
     1  505.lbm_t,605.lbm_s,705.lbm_m,805.lbm_l:  
     2  %if %{model} eq "openacc"  
     3      pmodel=ACC
     4  %elif %{model} eq "openmp"
     5  %   if %{usegpu} == 1
     6         pmodel=TGT  
     7  %   else
     8         pmodel=OMP 
     9  %   endif
    10  %endif

Lines that start with percent (%) are preprocessor directives. The percent (%) character must be in column 1.

The config file preprocessor is called a macro processor because it allows you to define macros, which are brief abbreviations for longer constructs. In the example above, two macros are used: (%{model}, and %{usegpu}). Macro values are tested, and hunks of config file are included or excluded, using the conditionals %if (lines 2 and 5), and %elif (line 4). One of the conditionals includes an %else clause (line 7).

The preprocessor is automatically run whenever you use runhpc, and macros may be set on the runhpc command line. Or, you can run it separately, as configpp:

% specperl bin/harness/configpp -c preprocessor_example1.cfg --define model=openacc | grep -e lbm -e pmodel
505.lbm_t,605.lbm_s,705.lbm_m,805.lbm_l:
    pmodel=ACC
% specperl bin/harness/configpp -c preprocessor_example1.cfg --define model=openmp | grep -e lbm -e pmodel
505.lbm_t,605.lbm_s,705.lbm_m,805.lbm_l:
       pmodel=OMP
% specperl bin/harness/configpp -c preprocessor_example1.cfg -S model=openmp -S usegpu=1 | grep -e lbm -e pmodel
505.lbm_t,605.lbm_s,705.lbm_m,805.lbm_l:
       pmodel=TGT

Notice above that preprocessor variables can be set on the command line, using either --define or '-S'.

VI.A.2. Preprocessor Syntax Basics

Emphasis: Column 1. Always punch column 1.

%define foo
%    define bar
%		undef hello!

 %define foo           Space in the first column
	%define foo    Tab in the first column
#define foo            <--wrong!

One line per directive

VI.B. Defining macros

Macro names may only be composed of alphanumeric characters, underscores, and hyphens, and they ARE case-sensitive.

Macros can be defined in a config file or on the command line. Both ways are equivalent.

Macros set on the command-line are defined first; therefore, it is not possible to use the command line to override a macro definition that occurs in the config file itself. If you want the command line to "win", see the notes below about redefinition.

A macro that has not been defined will not be substituted.

A macro that has been defined, but not given a value,

• behaves as if it were 1 in a numeric context
• behaves as if it were true in a logical context (because for Perl, 1 is true)
• behaves as an empty string in a string context

See the example in the next section.

Having no value is not the same as having an empty string as the value. The following are NOT equivalent:

See the table about Truth and definition, below.

VI.B.1. In a config file

To define a macro in a config file, use the %define preprocessor directive. Note that no quoting is necessary when specifying the names of macros or their values.

# Define a simple macro
%define foo bar
# Now the macro called 'foo' has the value 'bar'

# It's not necessary for a macro to have a value to be useful
%define baz
# Now the macro called 'baz' is defined, but it has no value.

Above, the macro baz has been defined, but has not been assigned a value.
Nevertheless, if you use it in an expression, it will behave as if it has the value 1.

$ cat macroDefinedButNoValue.cfg 
%define baz
%ifdef %{baz}
%   info baz is defined. In text contexts, it looks like this: "%{baz}".
%endif
%if %{baz} 
%   info In a logical context, it behaves as if it were 1 (i.e. true)
%endif
%if %{baz} + 3  == 4
%   info In a numeric context, it behaves as if it were 1.
%endif
$ cat macroDefinedButNoValue.sh  
specperl bin/harness/configpp --config=macroDefinedButNoValue | grep INFO
$ ./macroDefinedButNoValue.sh 
  INFO: baz is defined. In text contexts, it looks like this: "".
  INFO: In a logical context, it behaves as if it were 1 (i.e. true)
  INFO: In a numeric context, it behaves as if it were 1.
$ 

VI.B.2. On the command line

Macros can also be defined on the command line, using the --define switch. You may find it convenient to abbreviate that switch, if you prefer. The following are entirely equivalent:

$ runhpc --define mymacro=something
$ runhpc -S mymacro=something

VI.B.3. Predefined macros endian, hostname... %{ENV_variable_name}

Some useful macros are predefined, as shown in the table below. New with SPEChpc 2021: The config file preprocessor can do environment variable substitution. Environment variables to be substituted use a percent sign and curly brackets, and the name is prefixed with the string "ENV_", as %{ENV_variable_name}.

Automating output_root: You can automatically bring an output_root into your config file using %{ENV_GO} and navigate around it, as shown in the example for the ogo utility.

VI.B.4. Example: Adjusting the Environment

Notice that %{ENV_variable_name} brings an environment variable into your config file; what if you want to push it back out to affect other things? In that case, you will also need the feature preenv, which causes runhpc to re-invoke itself with your requested variable in the environment.

For example, you could set up paths for your compiler using something like this:

$ cat -n test.cfg
     1  action       = build
     2  output_root  = /tmp/me
     3  rebuild      = 1
     4  runlist      = 605.lbm_s
     5  verbose      = 99
     6
     7  %define tentop /SW/compilers/GCC/Linux/x86_64/gcc-10.1.0
     8  %define ninetop  /SW/compilers/GCC/Linux/x86_64/gcc-9.2.0
     9
    10  %ifdef %{wantGcc9}
    11     preENV_PATH            = %{ninetop}/bin:%{ENV_PATH}
    12     preENV_LD_LIBRARY_PATH = %{ninetop}/lib64:%{ENV_LD_LIBRARY_PATH}
    13  % else
    14     preENV_PATH            = %{tentop}/bin:%{ENV_PATH}
    15     preENV_LD_LIBRARY_PATH = %{tentop}/lib64:%{ENV_LD_LIBRARY_PATH}
    16  %endif
    17
    18  default:
    19     CC                = mpicc
    20     CC_VERSION_OPTION = -v
    21
$ 

The config file above builds a single benchmark, 605.lbm_s, using the local paths for GCC 9.2 or 10.1. Notice that in both cases, the desired path is added to the existing path: for example, on line 12, the right side accesses the existing setting via %{ENV_LD_LIBRARY_PATH} and the left side applies it to the build, via preENV_LD_LIBRARY_PATH. On line 5, the config file sets verbose=99, so that runhpc will print all possible detail and we will be able to confirm the adjusted variables.

VI.B.5. Redefinition

You will receive a warning if a previously defined macro is re-defined. In the example below, the first config file just sets a macro; the second one tests before acting.

$ cat preproc.symbol.just_define.cfg 
%define build_ncpus 20
makeflags   = -j%{build_ncpus}
$ 
$ cat preproc.symbol.test_first.cfg 
%ifndef %{build_ncpus}
%   define build_ncpus 20
%endif
makeflags   = -j%{build_ncpus}
$ 

A run script uses both config files with and without a definition on the runhpc command line:

$ cat preproc.symbol.sh 
JUST_DEFINE="configpp --config=preproc.symbol.just_define.cfg"
 TEST_FIRST="configpp --config=preproc.symbol.test_first.cfg"
    GREP_IT='grep -e WARNING -e makeflags'
echo original:
$JUST_DEFINE --define build_ncpus=8 | $GREP_IT
$JUST_DEFINE                        | $GREP_IT
echo
echo test1st: 
$TEST_FIRST  --define build_ncpus=8 | $GREP_IT
$TEST_FIRST                         | $GREP_IT
$

Result: testing first with ifndef avoids a warning and does a better job of respecting the user's wishes.

$ ./preproc.symbol.sh 
original:
WARNING: Redefinition of preprocessor macro 'build_ncpus' on line 1
makeflags   = -j20
makeflags   = -j20

test1st:
makeflags   = -j8
makeflags   = -j20
$  

VI.C. Un-doing macro definition

Sometimes you want to make the preprocessor forget about a macro that you taught it. This is easily accomplished.

Macros can be undefined in two ways:

1. On the command line, using the --undef switch, or
2. In the configuration file, using the '%undef' preprocessor directive.

   %define foo bar
   # Now the macro called 'foo' has the value 'bar'
   
   %undef foo
   # Now it doesn't
   

   Note that no quoting is necessary when specifying the names of macros.

Like macro definition, the undefinition requests can't affect macros set in the config file because those definitions effectively happen after the un-definition. For this reason, command-line undefinition is basically useless; it can only undo macros also set on the command line.
So why was such a useless ability added to the tools? The writer likes orthogonality.

VI.D. Using macros

By now you're probably over that initial euphoric rush that comes from wantonly defining and undefining macros, and you're looking for something more. This is it!

When you want to use the value of a macro, you refer to it by name. Unfortunately, the syntax for this is not as simple as you might hope. It's not too complicated, though; to have the preprocessor expand the macro 'foo', just write

  %{foo}

in the place where you'd like it to appear. Given the following config file snippet:

%define foo Hello_
%define bar baz
%define foobar Huh?
%define foobaz What?
%define Hello_baz Please don't do this

Here's a handy table to see the various ways you can reference these values:

Easy, right? The following is also possible:

Because macro values can only be one line long, it's not possible to use the preprocessor to macro-ize large chunks of your config file at once, as may be common practice for advanced users of CPP.

A macro that has not been defined will not be substituted. Note the mention of %{FOO} on the last line of this config file.

$ cat macroDef.cfg 
action          = build
output_format   = text
output_root     = /tmp/mDef
rebuild         = 1
runlist         = 605.lbm_s
size            = test
teeout          = 1
intspeed=base:
   CXX                 = mpicc
   CXX_VERSION_OPTION  = -v 
   OPTIMIZE            = %{FOO}
$

The failure for the first test below is expected: the GNU C compiler does not know what to do with ${FOO}.

$ cat macroDef1.sh 
runhpc --config=macroDef | grep -e lbm.o -e ^Error -e uccess
$ ./macroDef1.sh 
mpicc -c -o lbm.o -DSPEC -DNDEBUG -I.  %{FOO} lbm.c 
Error with make 'specmake --output-sync build': check file 
   "/tmp/mDef/benchspec/HPC/605.lbm_s/build/build_base_none.0000/make.out"
$ cat /tmp/mDef/benchspec/HPC/605.lbm_s/build/build*/make.out
mpicc -c -o lbm.o -DSPEC -DNDEBUG -I.  %{FOO}  lbm.c
gcc: error: %{FOO}: No such file or directory
$

If you want to substitute an empty string, then assign it one.

$ cat macroDef2.sh 
runhpc --config=macroDef --define FOO="" | grep -e lbm.o -e ^Error -e uccess
$ ./macroDef2.sh 
mpicc -c -o lbm.o -DSPEC -DNDEBUG -I.      lbm.c 
Build successes for small: 605.lbm_s(base)
$ 

VI.E. Conditionals

Defining, undefining, and expanding macros is quite an enjoyable activity in and of itself, and can even be useful on occasion. However, conditionals add an entirely new dimension to config file processing: the ability to include and exclude entire sections of text based on macros and their values.

VI.E.1. %ifdef .. %endif

The %ifdef conditional provides a way to determine whether or not a particular macro has been defined. If the named macro has been defined, the conditional is true, and the text to the matching %endif is included in the text of the config file as evaluated by runhpc. Note that the matching %endif may not necessarily be the next %endif; conditionals may be nested.

For example, given the following section of a config file:

%define foo
%ifdef %{foo}
This text will be included
%endif

%ifdef %{bar}
This text will not be included
%endif

The preprocessor would produce the following output:

This text will be included

Note especially the quoting used for the macro names in the conditional; the only time macro name quoting may be omitted is when defining or undefining it.

VI.E.2. %ifndef .. %endif

The %ifndef conditional is the converse of %ifdef; If the named macro has not been defined, the conditional is true, and the text to the matching %endif is included in the text of the config file as evaluated by runhpc. Note that the matching %endif may not necessarily be the next %endif; conditionals may be nested.

Given a slightly modified version of the example from earlier:

%define foo
%ifndef %{foo}
Now THIS text will not be included
%endif

%ifndef %{bar}
This text WILL be included
%endif

The preprocessor would produce the following output:

This text WILL be included

VI.E.3. %if .. %endif

Checking whether or not a macro is defined is quite useful, but it's just a subset of the more general conditional facility available. This general form is

%if expression
...
%endif

The expression is evaluated using a subset of the Perl interpreter, so the possibilities for testing values are fairly broad. For example,

%ifdef %{foo}
...
%endif

is exactly equivalent to

%if defined(%{foo})
...
%endif

Likewise,

%ifndef %{foo}
...
%endif

is exactly equivalent to

%if !defined(%{foo})
...
%endif

Using the general form, it's possible to string conditionals together:

%if defined(%{foo}) && !defined(%{bar}) || %{baz} == 0
...
%endif

To compare versus a string value, you must supply quotes:

%if %{foo} eq 'Hello, Dave.'
...
%endif

You may perform basic math on macro values:

%if %{foo} * 2004 > 3737
...
%endif

You can do basic regular expression matching, as shown in the example for %info.

More precisely, the Perl operations allowed are the :base_core and :base_math bundles, with the ability to dereference and modify variables disallowed. For more details, see the source code for config.pl (the eval_pp_conditional subroutine) and Perl's own Opcode documentation.

$ cat test_define.cfg
%ifdef %{a}
   OPTIMIZE = -a
%endif
%if defined(%{b})   
   LDOPTIMIZE = -b
%endif
%if %{c}
   COPTIMIZE = -c
%endif
$ specperl bin/harness/configpp --config=test_define -S a=1 -S b=1 -S c=1 | grep OPT  
   OPTIMIZE = -a
   LDOPTIMIZE = -b
   COPTIMIZE = -c
$ specperl bin/harness/configpp --config=test_define -S a=0 -S b=0 -S c=0 | grep OPT
   OPTIMIZE = -a
   LDOPTIMIZE = -b

VI.E.4. %else

It's possible to get by without the "else" part of the classic "if .. then .. else" trio, but it's not any fun. It works as you'd expect:

%define foo
%ifndef %{foo}
This text will not be included
%else
This text WILL be included (from the else clause)
%endif

The preprocessor would produce the following output:

This text WILL be included (from the else clause)

Only one %else per conditional is allowed.

VI.E.5. %elif

%elif is another convenience that's been added. For those not familiar with CPP, it's an "else if" construct. You may have as many of these as you'd like. Given:

%define foo Hello!

%if !defined(%{foo})
This text will not be included
%elif defined(%{bar})
This text won't be included either
%elif '%{foo}' eq 'Hello!'
This text WILL be included (from the second elif clause)
%else
Alas, the else here is left out as well.
%endif

The preprocessor would produce the following output:

This text WILL be included (from the second elif clause)

VI.F. Informational Directives

It's often helpful to be able to warn or exit on certain conditions. Perhaps there's a macro that must be set to a particular value, or maybe it's just very highly recommended.

VI.F.1. %warning

%warning does just what you'd expect; when the preprocessor encounters this directive, it prints the text following to stdout and the current log file, along with its location within the file being read, and continues on.

$ cat warning.cfg 
%if !defined(%{somewhat_important_macro})
%   warning You have not defined somewhat_important_macro!
%endif
$ 
$ cat warning.sh 
specperl bin/harness/configpp --config=warning | grep -C2 WARNING
$ 
$ ./warning.sh 

***
  WARNING: You have not defined somewhat_important_macro!
           (From line 2 of
           /hpc2021/config/warning.cfg.)
$  

VI.F.2. %error

Like %warning, %error logs an error to stderr and the log file. Unlike %warning, though, it then stops the run.

Consider a slightly modified version of the previous example:

$ cat error.cfg 
%if !defined(%{REALLY_important_macro})
%   error You have not defined REALLY_important_macro!
%endif
$ 
$ cat error.sh 
specperl bin/harness/configpp --config=error > /tmp/out
echo runhpc exit code: $?
grep -C2 ERROR /tmp/out
$ ./error.sh 
runhpc exit code: 1

************************************************************************
  ERROR: You have not defined REALLY_important_macro!
         (From line 2 of
         /hpc2021/config/error.cfg.)
$ 

Unlike a warning, the error will be close to the last thing output. As you can see from the output of echo $? above, runhpc exited with an error code 1.

VI.F.3. %info

The %info directive prints a message preceded by the word INFO:. For example,

$ cat -n info.cfg 
   1  %if !defined(%{chip}) || %{chip} !~ m/(sparc|x86)/
   2  %   error Please use --define chip=sparc or --define chip=x86
   3  %endif
   4  
   5  %if  %{chip} eq "sparc"
   6  %   define default_build_ncpus 64
   7  %elif %{chip} eq "x86"
   8  %   define default_build_ncpus 20
   9  %endif
  10  %ifndef   %{build_ncpus}
  11  %   define  build_ncpus   %{default_build_ncpus}
  12  %endif
  13  
  14  %info Preprocessor selections: 
  15  %info    .    build_ncpus      %{build_ncpus}
  16  %info    .    chip             %{chip}
  17  
  18  make = specmake --jobs=%{build_ncpus} --load-average=%{build_ncpus} 
$
$ specperl bin/harness/configpp -c info --define chip=sparc | grep -e make -e INFO
INFO: Preprocessor selections:
INFO: .    build_ncpus      64
INFO: .    chip             sparc
make = specmake --jobs=64 --load-average=64
$ 
 

Note in the example above:

• Line 1 demonstrates basic perl regular expression matching
• Lines 14-16 use %info
• After all preprocessing has been completed, line 18 tells the specmake utility that it can start up to 64 compile jobs, provided that the system load average is no higher than 64.

VI.F.4. %dumpmacros

The %dumpmacros prints the values of all macros currently defined at the point where it appears. Each macro value is preceded by the word 'DBG:'. For example,

$ cat -n dumpmacros.cfg 
     1	%if !defined(%{chip}) || %{chip} !~ m/(sparc|x86)/
     2	%   error Please use --define chip=sparc or --define chip=x86
     3	%endif
     4	
     5	%if  %{chip} eq "sparc"
     6	%   define default_build_ncpus 64
     7	%elif %{chip} eq "x86"
     8	%   define default_build_ncpus 20
     9	%endif
    10	%ifndef   %{build_ncpus}
    11	%   define  build_ncpus   %{default_build_ncpus}
    12	%endif
    13	
    14	%dumpmacros
$
$ specperl bin/harness/configpp -c dumpmacros --define chip=sparc | grep -e DBG
  DBG: build_ncpus: '%{default_build_ncpus}'
  DBG: chip: 'sparc'
  DBG: default_build_ncpus: '64'
  DBG: runhpc: 'configpp -c dumpmacros --define chip=sparc'
$
 

The %dumpmacros directive is new with SPEChpc 2021.

VII. Output files - and how they relate to your config file

This section describes how the location and contents of several kinds of output files are influenced by your config file.

VII.A. Help, I've got too many config files!

It was mentioned above that the HASH section of the config file is written automatically by the tools. Each time your config file is updated, a backup copy is made. Thus your config directory may soon come to look like this:

$ cd $SPEC/config
$ ls tune.cfg*
tune.cfg                        tune.cfg.2021-02-05T124831      tune.cfg.2021-02-05T125733
tune.cfg.2021-02-05T120242      tune.cfg.2021-02-05T125557      tune.cfg.2021-02-05T125738
tune.cfg.2021-02-05T122021      tune.cfg.2021-02-05T125603      tune.cfg.2021-02-05T125744
tune.cfg.2021-02-05T122026      tune.cfg.2021-02-05T125608      tune.cfg.2021-02-05T125749
tune.cfg.2021-02-05T124215      tune.cfg.2021-02-05T125614      tune.cfg.2021-02-05T125756
tune.cfg.2021-02-05T124222      tune.cfg.2021-02-05T125620      tune.cfg.2021-02-05T125802
tune.cfg.2021-02-05T124728      tune.cfg.2021-02-05T125626      tune.cfg.2021-02-05T125807
tune.cfg.2021-02-05T124739      tune.cfg.2021-02-05T125632      tune.cfg.2021-02-05T125812
tune.cfg.2021-02-05T124821      tune.cfg.2021-02-05T125727
$  

If this feels like too much clutter, you can disable the backup mechanism, as described under backup_config. Note that doing so may leave you with a risk of losing the config file in case of a filesystem overflow or system crash. A better idea may be to periodically remove just portions of the clutter, with selective removal of older version; or sweep them to a wastebasket every now and then:

$ cd $SPEC/config
$ mkdir wastebasket
$ mv *cfg.2021* wastebasket/
$

VII.B. The log file and verbosity levels

$SPEC/result (Unix) or %SPEC%\result (Windows) contains reports and log files. When you are doing a build, you will probably find that you want to pay close attention to the log files such as hpc2021.001.log. Depending on the verbosity level that you have selected, it will contain detailed information about how your build went.

The SPEChpc 2021 tool suite provides for varying amounts of output about its actions during a run. These levels range from the bare minimum of output (level 0) to copious streams of information that are probably useful only to tools developers (level 99). Selecting one output level gives you the output from all lower levels, which may cause you to wade through more output than you might like.

VII.B.1. Useful Search Strings

When you are trying to find your way through a log file, you will probably find these (case-sensitive) search strings useful:

VII.B.2. About Temporary Debug Logs

There are also temporary debug logs, such as hpc2021.001.log.debug. A debug log contains very detailed debugging output from the SPEC tools, as if --verbose 99 had been specified.

For a successful run, the debug log will be removed automatically, unless you specify "--keeptmp" on the command line, or "keeptmp=yes" in your config file.

For a failed run, the debug log is kept. The debug log may seem overwhelmingly wordy, repetitive, detailed, redundant, repetitive, and long-winded, and therefore useless. Suggestion: after a failure, try looking in the regular log first, which has a default verbosity level of 5. If your regular log doesn't have as much detail as you wish, then you can examine the additional detail in the debug log.

If you file a support request, you may be asked to send in the debug log.

VII.B.3. Verbosity levels

The 'level' referred to in the table below is selected either in the config file verbose option or in the runhpc command as in 'runhpc --verbose n'.

Levels higher than 99 are special; they are always output to your log file. You can also see them on the screen if you set verbosity to the specified level minus 100. For example, the default log level is 5. This means that on your screen you will get messages at levels 0 through 5, and 100 through 105. In your log file, you'll find the same messages, plus the messages at levels 106 through 199.

VII.D. Help, I've got too many log files!

If you do many builds and runs, you may find that your result directory gets too cluttered. Within a result directory, all output formats other than .rsf can be regenerated from your .rsf files. Therefore, you could reduce clutter by deleting HTML, PDF, and other reports. You can delete old .debug logs unless you plan to submit a support request.

Still feel cluttered? A simple solution is to move your result directory aside, giving it a new name. Don't worry about creating a new directory; runhpc will do so automatically. You should be careful to ensure no surprises for any currently-running users. If you move result directories, it is a good idea to also clean temporary directories at the same time.
Example:

cd $SPEC
mv result old-result
rm -Rf tmp/
cd output_root     # (If you use an output_root)
rm -Rf tmp/

VII.E. Finding the build directory

As described under "About Disk Usage" in runhpc.html, the SPEChpc 2021 tools do the actual builds and runs in newly created directories. The benchmark sources are never modified in the src directory.

The build directories for a benchmark are located underneath that benchmarks' top-level directory, typically $SPEC/benchspec/HPC/nnn.benchmark/build (Unix)

(If you are using the output_root feature, then the first part of that path will change to be your requested root instead of SPEC.)

The build directories have logical names, typically of the form build_<tune>_<label>.0000.

If the directory build_<tune>_<label>.0000 already exists when a new build is attempted for the same tuning and label, the directory will be re-used, unless:

• You have set the environment variable SPEC_NO_RUNDIR_DEL, or
• The directory has been left in a "locked" state by an aborted previous build.

In such cases, the 0000 will be incremented until a name is generated that is available. You can find locked directories by searching for lock=yes in the file $SPEC/benchspec/HPC/<nnn.benchmark>/run/list (Unix)

When more than one build directory has been created for a given tuning and label, you may need to trace the directory back to the specific build attempt that created it. You can do so by searching for the directory name in the log files:

$ grep build_peak_blue271.0000 *log | grep Building
hpc2021.838.log:  Building 605.lbm_s peak blue271: (build_peak_blue271.0000) [2021-02-14 05:26:09]
$  

In the above example, the grep command locates log #838 as the log that corresponds to this run directory.

VII.F. Files in the build directory

A variety of files are output to the build directory. Here are some of the key files which can usefully be examined:

VII.G. For more information

For more information about how the run directories work, see the descriptions of specinvoke, specmake, and specdiff in utility.html.

Appendix A. Troubleshooting

When something goes wrong, here are some things to check:

1. Are there any obvious clues in the log file? Search for the word "Building". Keep searching until you hit the next benchmark AFTER the one that you are interested in. Now scroll backward one screen's worth of text.

2. Did your desired switches get applied? Go to the build directory, and look at options*out.

3. Did the tools or your compilers report any errors? Look in the build directory at *err.

4. What happens if you try the build by hand? See the section on specmake in utility.html.

5. If an actual run fails, what happens if you invoke the run by hand? See the information about "specinvoke -n" in utility.html

6. Do you understand what is in your path, and why? Sometimes confusion can be greatly reduced by ensuring that you have only what is needed, avoiding, in particular, experimental and non-standard versions of standard utilities.

   Note: on Windows systems, SPEC recommends that Windows/Unix compatibility products should be removed from the %PATH% prior to invoking runhpc, in order to reduce the probability of certain difficult-to-diagnose error messages.

7. Try asking the tools to leave more clues behind, with keeptmp.

Appendix B. Obsolete/removed items