Building the SPEC OMP2012 Tool Suite

Last updated: $Date: 2012-10-12 09:29:24 -0400 (Fri, 12 Oct 2012) $ by $Author: BrianWhitney $

(To check for possible updates to this document, please see )


1. Introduction

a. What are the tools?

b. Pre-compiled binaries

c. When to build the tools yourself

2. How to build the tools

a. Notes specific to Unix systems

b. Notes specific to Windows systems

3. How to verify that your build succeeded

4. Describing your newly-built tools

a. Toolset presentation

1. Excluding otherwise valid candidates

2. Affecting presentation order

5. Packagetools

a. Example: use packagetools

b. Example: test the newly packaged tools

6. Troubleshooting: What to do if something goes wrong

a. Try it by hand

b. Known problems

c. Executing just part of buildtools


1. Introduction

SPEC supplies various tools that are used to ensure consistent operation of benchmarks across a variety of platforms. In order to generate a valid result file which can be submitted to SPEC, you must use the SPEC supplied tools.

1.a. What are the tools?

The tools include (but are not limited to):

specdiff examines results to see if the correct answer was obtained
specinvoke invokes benchmarks for OMP2012
specmake GNU make (Calling it "specmake" avoids possible conflicts with versions of make that may already be on your system. SPEC requires that published results use the versions of the tools that it supplies, so that if SPEC applies patches or extensions from time to time, all users run with a consistent tool set. Similar considerations apply to other tools in this list.)
specxz The xz compression program
specmd5sum md5sum from GNU coreutils, with enhancements
spectar GNU tar
specperl Perl
Various Perl modules such as Algorithm::Diff, GD, HTML::Parser, PDF::API2, URI, XML::SAX, etc.

Many of these tools are based on the freely available programs of the same name.

Note: In this document, "Unix" is used to describe Unix and all Unix-like systems such as Linux.

1.b. Pre-compiled binaries

You will find pre-compiled binaries for the tools in the directories:


The term $SPEC designates the top directory where you installed the benchmark suite. (On a Windows system, it would be called %SPEC%.)

The precompiled binaries will be automatically installed when you run (Unix) or install.bat (Windows).

1.c. When to build the tools yourself

Sometimes it may be necessary to rebuild the tools, for example if a change in an operating system renders a precompiled binary inoperable, or if you are the first person to add support for a new architecture.

It is generally not possible to do cross-compilation of Perl, as Perl's build process involves running binaries generated during the build. Therefore, getting a toolset onto an embedded platform without a native compiler is most likely impossible.

NOTICE: If you are adding support for a new architecture, and intend to submit results to SPEC, you must ask SPEC to review your tool build. Please turn on your operating system's session recorder (e.g. in Unix, typically the script command) prior to doing buildtools. In Windows, you might have to do repetitious tedious repeated cut/paste clicking and clicking (so make that command window as big as possible). Also, you will be expected to provide the output from runspec -V and runspec --test on a system OTHER than the build system, where you have installed your new tools build.

2. How to build the tools

If the directory $SPEC/tools/src (Unix) or %SPEC%\tools\src (Windows) does not yet exist, use xz and tar to expand install_archives/omp2012.tar.xz into a writable directly local to the build system. (Building tools on networked filesystems can work, but it's more likely to make some of GNU tar's tests fail, so it's safer to build on a local filesystem.)

xz -dc install_archives/omp2012.tar.xz | tar -xf - tools/src

The scripts $SPEC/tools/src/buildtools (Unix) or %SPEC%\tools\src\buildtools.bat (Windows) will build the tools. But you may need to invoke the buildtools script with appropriate environment variables set first - see the sections immediately following that provide notes about specific platforms.

You will need a C99 compiler. Most of the tools are C++ or C89, but XZ is C99.

If everything goes right, you won't have to do very much at all, other than watch build commands fly by.

2.a. Notes specific to Unix systems

When building the tools under Unix, the following warnings appear to be harmless as of May 2012:

You may find the following flags useful on the listed systems:

Check out those who have gone before. If the system for which you're attempting to build tools is similar to one for which a tools build exists, you may gain insight by looking at the top of the build logs in tools/src/buildtools.log. Even if your system is very unusual and very different than previous systems, it is probably still worth your while to look over the previous logs. You don't have to read all 20,000 lines; just check out the environment variables and path settings and so forth from the top of previous builds.

2.b. Notes specific to Windows systems

Because XZ is written in C99, no version of Visual Studio (as of May 2012) can be used to build the tools. Instead, the Windows version of buildtools is designed to use the MinGW GCC compiler. It has been tested successfully with both 4.3 and 4.5 versions of GCC; the latest version available should be used. The tools are not yet ported to be built with a 64-bit compiler; use the 32-bit tools instead.

The builds use makefiles to be processed with either mingw-make.exe (which is GNU make) or dmake.exe (which is built in the process of building tools). The full MSYS environment is not used, so all of the makefiles are pre-generated; the GNU autoconf system is not used. Compiler flags are defined in those pre-generated makefiles.
Building tools under Cygwin is not supported and probably won't work.

If it is necessary to unzip and untar the sources of the tools in Windows, open a Command-Prompt window, cd to %SPEC% (i.e. the top directory of SPEC). Then execute:

  cd tools
  bin\windows-i386\specxz.exe -d install_archives/omp2012.tar.xz
  ooo\windows-i386\spectar.exe -xvf install_archives/omp2012.tar

Then go ahead and execute


You can ignore warnings about objects that are not found, especially at the beginning of each tool build, when cleanup is attempted from previous builds. Some tools may not know how to make 'clean', 'distclean', or 'realclean'; don't worry about it.

In Windows XP with Service Pack 2 or later, there may be a Security Alert, depending on the Firewall settings. "Windows Firewall has blocked this program from accepting connections from the Internet or a network." You may ignore this warning.

All compiler warnings seen as of May 2012 appear to be harmless.

3. How to verify that your build succeeded

After a tool build, you should:

  cd $SPEC (Unix) or %SPEC% (Windows)

  shrc.bat (Windows)
  . ./shrc (Unix, if you are in an sh-compatible shell.  
                  If not, start one!)  

  See if you can at least get as far as asking the major tools
  to identify themselves:

       runspec -V 

  Or you can ask individual tools about themselves:

       specmake -v 
       specxz -h
       specperl -v
       specdiff -h
       runspec -h 
       specinvoke -h 

Running runspec --test will run more comprehensive tests on specperl.

4. Describing your newly-built tools

As a courtesy to others who might like to know which systems you intended your new tools to work on, and to describe the environment in which they were built, you may write a short description file. The contents of this file will be displayed when is run. Windows users only have one choice, and thus no description is printed. The file is $SPEC/tools/bin/<archname>/description. If that directory does not exist, you should create it. (If the directory does not exist, packagetools will create it, but in order for the description to be packaged, it must be present before packagetools is run.) See the "Packagetools" section below for guidance about picking "<archname>".

The format is very simple; in order to avoid wrapping on an 80-column screen, the first line must be no longer than 50 characters. In order to line up properly, subsequent lines should begin with 30 spaces and be no longer than 80 columns total.

For example, given $SPEC/tools/bin/turboblaster-m68k/description with the following contents

For TurboBlaster v2.1+ systems running on M68010
                             Built on TurboBlaster v2.0 with GCC 1.81.

A user using a TurboBlaster system when running would see in the list of toolset choices, something that looks approximately like

turboblaster-m68k            For TurboBlaster v2.1+ systems running on M68010
                             Built on TurboBlaster v2.0 with GCC 1.81.

There are plenty of examples on the install media in tools/bin/*/description.

4.a. Toolset presentation

If the toolset you're building isn't the only one that will work on a particular system, you may want to adjust where in the list of toolsets it appears, or even whether it appears at all. By default toolsets are presented in lexical ASCII order. However, it is possible to both adjust the position of a toolset in the list of candidates, as well as keeping it from appearing at all.

Let's assume that the TurboBlaster systems mentioned in the last section have migrated to PowerPC and have an emulation layer so that they can run old M68K binaries. Given two toolsets called turboblaster-m68k and turboblaster-ppc, normally the turboblaster-m68k would be presented and attempted first. And the emulation layer is so good that it would work, but it'd be slower than it needs to be.

4.a.1. Excluding otherwise valid candidates

There are several files that can be used to prevent from even attempting to use a toolset. The exclusion is silent, and for all intents and purposes makes the affected toolset behave just as one from another OS or architecture — it doesn't even show up on the candidate list.

To effect this exclusion, looks for signatures of the current system in files in each toolset's directory. A toolset is excluded from consideration if any of the following are true:

Output from... Appears in...
uname -m tools/bin/<archname>/excludearch
uname -s tools/bin/<archname>/excludearch
uname -r tools/bin/<archname>/excluderev
uname -p tools/bin/<archname>/excludeproc

On a PPC-based TurboBlaster system, uname -m returns "PowerPC". Putting "PowerPC" into tools/bin/turboblaster-m68k/excludearch would ensure that TurboBlaster/PPC users never even saw the TurboBlaster/M68K tools.

This is most useful for preventing cross-OS confusion. FreeBSD, for example, used to try to execute Solaris binaries. Putting "FreeBSD" (the output from uname -s on a FreeBSD system) into tools/bin/solaris-x86/excludearch fixed that problem and kept the unmaskable and alarming failure messages from appearing during installation.

4.a.2. Affecting presentation order

In the case where there may be two equally valid toolsets (as might be the case when there's a 32-bit and 64-bit build for the same OS), it's possible to change the order in which they're presented. Each toolset has a file named order in its directory. If it doesn't exist, packagetools will create this file with contents that are simply:


(See the packagetools section for guidelines on picking <archname>.) The "priority" of a toolset decreases as that number increases. Continuing with the TurboBlaster example, if we wanted the PowerPC toolset to be preferred, we'd put


into tools/bin/turboblaster-m68k/order. That would ensure that turboblaster-ppc would appear first in the menu, as the number in its order file is "1". Since it appears first in the menu, it's installed before turboblaster-m68k.

5. Packagetools

If everything has succeeded, and you intend to submit results using your new tools, you should submit the tools to SPEC. To do so:

     cd $SPEC (Unix) or cd %SPEC% (Windows)
     packagetools <archname>

Pick an architecture name that other users will recognize. Check on the install media in tools/bin for some examples.

The packagetools script will create:


Having created a large tarfile with everything in it, packagetools will then proceed to create an even larger tarfile with specxz and spectar it in too. This even bigger file is known as:


and is to be submitted to SPEC. (See also the example below.)

You can optionally add components to your platform's toolset. For example, if you would like $SPEC/config/default.cfg to be set in an appropriate way, you can add the relative path to default.cfg as a parameter to packagetools:

(on non-Windows systems)
     cd $SPEC
     packagetools <archname> config/default.cfg

More than one file may be specified in this way.

Operation on Windows is substantially similar; just provide the relative paths with backslashes instead of forward slashes.

Please submit the resulting tarfile to SPEC for review, along with the recording of your tool build session. SPEC will review your tools, and assuming that they pass review, will add the tools you have built to its patch library, for possible distribution to future users of your interesting new architecture.

NOTE 1: If your operating system is unable to execute the packagetools script, please have a look at what the script does and enter the corresponding commands by hand. Again, you will need to submit the results to SPEC.

NOTE 2: Be sure to test your packaged tools on a different system, preferably one with a different disk layout. If the destination system is unable to invoke, check that exists in one of the locations where shrc expects to find it.

5.a. Example: use packagetools

Here is an example use of packagetools. In the example below, notice that:

  1  $ . ./shrc
  2  $ which runspec
  3  /spec/omp2012/kit117/bin/runspec
  4  $ 
  5  $ export MYTOOLS=myOS-myHW
  6  $ mkdir tools/bin/$MYTOOLS
  7  $ echo \
  8     "For systems running myOS.  Note: works only with V12 or later" \
  9     tools/bin/$MYTOOLS/description
 10  $ 
 11  $ cp typescript $MYTOOLS.buildlog.txt
 12  $ runspec -V > $MYTOOLS.runspec-V.txt 2>&1
 13  $ runspec --test > $MYTOOLS.runspec--test.txt 2>&1
 14  $ 
 15  $ packagetools $MYTOOLS $MYTOOLS.buildlog.txt $MYTOOLS.runsp*txt 
 16  Preparing /spec/omp2012/kit117/tools/bin/myOS-myHW
 17  Generating MD5 checksums for the tools
 18  Generating the tools installation tarball
 19   100 %         12.0 MiB / 80.1 MiB = 0.149   1.0 MiB/s       1:17             
 20  Making the distribution tarball
 21  tools/bin/myOS-myHW/
 22  tools/bin/myOS-myHW/order
 23  tools/bin/myOS-myHW/tools-myOS-myHW.tar.xz
 24  tools/bin/myOS-myHW/description
 25  tools/bin/myOS-myHW/unbundled
 26  tools/bin/myOS-myHW/specxz
 27  tools/bin/myOS-myHW/specmd5sum
 28  tools/bin/myOS-myHW/spectar
 30  *******************************************************************
 32  The tarball to submit for inclusion in the distribution is
 34  myOS-myHW-117.tar
 36  ABSOLUTELY DO NOT submit the tarball in tools/bin/myOS-myHW
 37  as it is not complete.
 39  *******************************************************************
 41  $ specmd5sum myOS-myHW-117.tar > myOS-myHW-117.tar.md5

5.b. Example: test the newly packaged tools

Testing a newly-built toolset on the system where it was built is not enough to ensure basic sanity of the tools. Test for unintended dependencies by installing on an entirely different system. For example:

  1  $ cd /spec/omp2012
  2  $ cp /net/(buildsys)/spec/omp2012/kit117/myOS-myHW-117.tar* .
  3  $ cp /net/(buildsys)/spec/omp2012/kit117/bin/spectar .
  4  $ cp /net/(buildsys)/spec/omp2012/kit117/bin/specmd5sum .
  5  $ ls
  6  myOS-myHW-117.tar      myOS-myHW-117.tar.md5  specmd5sum             spectar
  7  $ ./specmd5sum -c myOS-myHW-117.tar.md5 
  8  myOS-myHW-117.tar: OK
  9  $ ./spectar -xvf myOS-myHW-117.tar
 10  tools/bin/myOS-myHW/
 11  tools/bin/myOS-myHW/spectar
 12  tools/bin/myOS-myHW/tools-myOS-myHW.tar.xz
 13  tools/bin/myOS-myHW/unbundled
 14  tools/bin/myOS-myHW/description
 15  tools/bin/myOS-myHW/specxz
 16  tools/bin/myOS-myHW/order
 17  tools/bin/myOS-myHW/specmd5sum
 18  $
 19  $ -u myOS-myHW
 20  .
 21  .
 22  .
 23  Installation successful.  Source the shrc or cshrc in
 24  /spec/omp2012/kit117
 25  to set up your environment for the benchmark.
 28  $ . ./shrc
 29  $ runspec -V > newsys.runspec-V.txt 2>&1
 30  $ runspec --test > newsys.runspec--test.txt 2>&1
 31  $
 32  $ grep -c "...ok" *runspec--test*
 33  myOS-myHW.runspec--test.txt:215
 34  newsys.runspec--test.txt:215
 35  $                                                                     
 36  $ spectar cvf - newsys.runspec* | specxz > newsys.tests.tar.xz 

6. Troubleshooting: What to do if something goes wrong

If something goes wrong, unfortunately, you're probably just going to have to take it apart and figure out what. Here are some hints on how to go about doing that.

6.a. Try it by hand

If something goes wrong, you probably do NOT want to make some random adjustment (like: reinstall a compiler, fix an environment variable, or adjust your path) and start all over again. That's going to be painful and take a lot of your time. Instead, you should temporarily abandon the buildtools script at that point and just try to build the offending tool, until you understand exactly why that particular tool is failing.

Consider turning on verbose diagnostics if your system has a way to do that. Make a huge terminal window (e.g. 200 columns wide by 84 lines tall, with 9999 lines recorded off the top), so you can see what is going on.

Read what buildtools (or buildtools.bat) does for you, then cd to tools/src/<tool> and try the commands by hand. For example, you might do something like this:

    cd $SPEC/tools/src/<toolname>
    make (or or whatever you note buildtools would have done)

Now, try fixing that environment variable or reinstalling that compiler, and rebuild the single tool. Does it look better?

If not, have a close look at the error messages and the Makefile. Does the Makefile use a feature that is not present in your version of make? If so, can you get it to work with GNU make?

Note that for GNU configure based tools (everything except Perl and its modules) you may specify your compiler by setting the CC environment variable. For compiler flags, set CFLAGS.

When building perl, note that:

Many of the build logs in tools/src/buildtools.log have examples of settings used for PERLFLAGS.

If you want to see more about what buildtools is doing for you, turn on your shell's verbose mode. For example:

   sh -x ./buildtools

6.b. Known problems

Try doing a web search to see if there are known problems with the tool on your architecture.

If SPEC supplies Version X.Y of a tool and it just won't build on your operating system, you might check whether there is a new Version X.Y+1 available. If so, download the new version to a scratch directory outside of the SPEC tree and try building it there. If that version succeeds, try to deduce why. Narrow it down to a one-line fix, won't you please? Then tell SPEC that you'd like the same one-line fix applied to its variant of the tool. Or, if you just can't narrow the fix down, ask SPEC whether it will approve use of Version X.Y+1 instead of X.Y on your system.

6.c. Executing just part of buildtools

Once you believe that you understand how to fix the problem tool, and can build it by hand, see whether the buildtools script can build it. You can execute just a portion of buildtools by defining environment variables. Please read the script itself to see what variables are allowed; the following are just some examples:

  SKIPALL      - turns off everything.  If you like, set this, then
                   turn individual phases on by setting them.
  DOTOOLSRM    - Remove previously installed tools
  DOCLEAN      - Run 'make clean', 'make distclean', and
                   'make realclean' in all source directories.
  DOMAKE       - build make
  DOXZ         - build xz
  DOTAR        - build tar
  DOMD5        - build specmd5sum
  DOSPECINVOKE - build specinvoke
  DORXP        - build rxp
  DOEXPAT      - build the expat XML parser, used by XML::SAX::ExpatXS
  DOPERL       - build perl
  DOPERL2      - build perl modules
  DOCOPY       - copy the results to $SPEC/bin and fix shbangs

It doesn't matter what you set the environment variables to -- any non-zero-length string will do. In all cases, "DO" can be replaced with "SKIP" to reverse the sense of the setting.

If you are using Windows, be sure to note the WARNING in buildtools.bat about how the variables work.

If you can now build the (formerly misbehaving) tool using just part of buildtools, then you can try building the whole set (i.e. a build of all tools from start to finish). It should work now, so be sure to turn on your logging facility.

7. Support

You'll find information about technical support in techsupport.html, either in the same directory where you are found the document you are reading now, or at

Copyright 1999-2012 Standard Performance Evaluation Corporation
All Rights Reserved