Runspec Avoidance

Using the SPEC MPI2007 benchmarks while making minimal use of SPEC's tool set

Updated for SPEC MPI2007 (new features are highlighted)

Last updated: 27-Nov-2009 bl/jlh/tre/cp
(To check for possible updates to this document, please see http://www.spec.org/mpi2007/Docs/.)

Contents

Introduction

Environment

Steps

Review two rules

Install

Pick a benchmark

Pick a config file

Fake it

Find the log

Find the build dir

Copy the build dir (triple only)

Build it

Rename the binary

Place the binary in the run dir

Copy the run dir

Run it

Save your work

Repeat

Validation

Introduction

This document is for those who prefer to avoid using some of the SPEC-supplied tools, typically because of a need for more direct access to the benchmarks. For example:

If the above describes you, here is a suggested path which should lead quickly to your desired state. This document shows you how to use SPEC's tools for the minimal purpose of just generating work directories, for use as a private sandbox. Note, however, that you cannot do formal, "reportable" runs without using SPEC's toolset.

Environments

Three different environments are referenced in this document, using these labels:

Steps

  1. Review two rules: Please read just one page from runrules.html, namely "4.5 Research and Academic usage of MPI2007" and "4.6 Required Disclosures"

    Note: links to SPEC MPI2007 documents on this web page assume that you are reading the page from a directory that also contains the other SPEC MPI2007 documents. If by some chance you are reading this web page from a location where the links do not work, try accessing the referenced documents at one of the following locations:

    These run rules sections acknowledge that the suite may be used in ways other than the formal environment that the tools help to enforce; but they warn that if you plan to publish your results, you should be able to state HOW your usage of the suite differs from the standard usage.

    So even if you skip over the tools and the run rules today, you should plan a time to come back and learn them later.

  2. Install: Get through a successful installation, even if it is on a different system than the one that you care about. Yes, we are about to teach you how to mostly bypass the tools, but there will still be some minimal use. So you need a working toolset and a valid installation. If you have troubles with the install procedures described in install-guide-unix.html or install-guide-windows.html, please see techsupport.html and we'll try to help you.

  3. Pick a benchmark: Pick a benchmark that will be your starting point.

    Choose one benchmark from the MPI2007 suite that you'd like to start with. For example, you might start with 107.leslie3d (Fortran) or 104.milc (C). These are two of the shortest benchmarks for lines of code, and therefore relatively easy to understand.

  4. Pick a config file: Pick a config file for an environment that resembles your environment. You'll find a variety of config files in the directory $SPEC/config/ on Unix systems, or %SPEC%\config\ on Windows, or at www.spec.org/mpi2007 with the submitted MPI2007 results. Don't worry if the config file you pick doesn't exactly match your environment; you're just looking for a somewhat reasonable starting point.

  5. Fake it: Execute a "fake" run to set up run directories, including a build directory for source code, for the benchmark.

    For example, let's suppose that you want to work with 107.leslie3d and your environment is at least partially similar to the environment described in the comments for $SPEC/config/example-qlogic-linux-x86_64-pathscale-infinipath.cfg

    [Other Unix-like systems should behave similarly to the example in this section.]

    For example, you could enter the following commands:

      % sh      
          ... if you're not already in a Bourne-compatible shell
      $ cd  /myspecdisk/tools/mpi2007/
          ... or wherever you installed the SPEC MPI2007 tree
      $ . ./shrc      (that's dot-space-dot-slash-shrc)*
      $ cd config
      $ cp example-qlogic-linux-x86_64-pathscale-infinipath.cfg my_test.cfg
      $ go          (to the $SPEC directory)
    
      Copy or edit a hostfile (as needed) based on the format accepted by your 
      MPI job launcher.  The following is an example for four 4-core hosts. 
      $ cat hostfile
      iqa-41:4
      iqa-42:4
      iqa-43:4
      iqa-44:4
    
      $ runspec --fake --loose --size mtest --tune base --config my_test --define hosts=$SPEC/hostfile --ranks=16 leslie3d 
    

    This command should report a success for the build, run and validation phases of the mtest case, but the actual commands have not been run. It is only a report of what would be run according to the config file that you have supplied. The --fake option above is new with MPI2007.

  6. Find the log: Near the bottom of the output from the previous step, notice the location of the log file for this run. The log file contains a record of the commands as reported by the "fake" run. You can find the commands by searching for "%%".

  7. Find the build dir: To find the build directory that was set up in the fake run go to the build directory for that benchmark. For example:,

      $ cd $SPEC/benchspec/MPI2007/107.leslie3d/build  ( or 'go 107 build' )
      $ ls -ltd *build*
    

    The above command prints the names of each build subdirectory, with the most recent first. (On Windows, the analogous comamnd would use reversed slashes, and "$SPEC" is spelt "%SPEC%". Instead of "ls -ltd", you would say something like "dir build*/o:d".) If this is your first time here, there will be only one directory listed, e.g. build_base_infinipath.0000

    You can work in this build directory, make source code changes, and try other build commands without affecting the original sources.

  8. Copy the build dir (triple only): If you are using a unified or cross-compile environment, you can skip to the next step. But if you are using a triple environment, then you will want to package up the build directory with a program such as tar -- a handy copy is in the bin directory of your SPEC installation, as spectar. Then, you will move the package off to whatever system has compilers.

    For example, you might say something like this:

      $ spectar -cvf  -  build_base_infinipath.0000 | specbzip2 > mybuild.tar.bz2
      $ ftp
      ftp> op buildsys
      Connected to buildsys
      Name: whoever
      Password:
      ftp> bin
      ftp> put mybuild.tar.bz2
    

    Note that the above example assumes that you have versions of bzip2 and tar available on the system that has compilers, which you will use to unpack the compressed tarfile, typically with something like this:

    bzip2 -dc mybuild.tar.bz2 | tar -xvf -

  9. Build it: Generate an executable using the build directory. If you are using a unified or cross-compile environment, then you can say commands such as these:

      $ cd build_base_infinipath.0000
            ...assuming only one such build directory
      $ specmake clean
            ...only if there is a previous build 
      $ specmake
      /usr/bin/mpif90 -f90=pathf90 -c -o modules.o -O3 modules.f
      ... 
    

    You can also carry out a dry run of the build, which will display the build commands without attempting to run them, by adding -n to the specmake command line. You might find it useful to capture the output of specmake -n to a file, so it can easily be edited, and used as a script.

    Re-build with (for example) different optimization flags:

      $ vi Makefile.spec
            ...edit the values of the FOPTIMIZE and perhaps EXTRA_LDFLAGS variables
      $ specmake clean
      $ specmake
      /usr/bin/mpif90 -f90=pathf90 -c -o modules.o -O3 -OPT:Ofast modules.f
      ... 
    

    In SPEC MPI2007 v2.0, There is one benchmark, 121.pop2, that needs a different build procedure from the above since the same build directory can be used to build two POP executables, one for the Large, and one for the Medium suites. Here is the procedure for the Large suite:

      $ vi Makefile.pop2lref.spec
            ...edit the values of the FOPTIMIZE and perhaps EXTRA_LDFLAGS variables
      $ specmake clean TARGET=pop2lref
      $ specmake TARGET=pop2lref   (if you have a multi-core system, you may want to do 
                                    'specmake -j 4 ...' to speed the build by using 4 cores)
      specperl /...mpi2007/bin/specpp  -DLREF -DSPEC_MPI -DNDEBUG -DSPEC_MPI_CASE_FLAG   POP.F90 -o POP.fppized.f90
      ... 
    
    If you are building 121.pop2 for the medium suite of v2.0, substitute pop2mref for the string pop2lref in the commands immediately above.

    If you are trying to debug a new system, you can prototype changes to Makefile.spec or even to the benchmark sources.

    If you are using a triple environment, then presumably it's because you don't have specmake working on the system where the compiler resides. But fear not: specmake is just GNU make under another name, so whatever make you have handy on the target system might work fine with the above commands. If not, then you'll need to extract the build commands from the log and try them on the system that has the compilers.

  10. Rename the binary: Once you have succeeded in building an executable image, rename it as

    <exename>_<tuning>.<extension>

    where:

    For example, if you are working with 107.leslie3d and your config file has the line

    ext=infinipath

    you would rename your executable to leslie3d_base.infinipath

  11. Place the binary in the run dir: Using techniques similar to those used to find the build directory, find the run directory established above, and place the binary into it. If you are using a unified or cross-compile environment, you can copy the binary directly into the run directory; if you are using a triple environment, then you'll have to retrieve the binary from the compilation system using whatever program (such as ftp) you use to communicate between systems.

    In a unified environment, the commands might look something like this:

      $ go 107 run
      $ ls -td run* 
      run_base_test_infinipath.0000
      $ cd run_base_test_infinipath.0000
      $ cp ../build_base_infinipath.0000/leslie3d_base.infinipath  . 
    
  12. Copy the run dir: If you are using a unified environment, you can skip this step. Otherwise, you'll need to package up the run directory and transport it to the system where you want to run the benchmark. For example:

      $ cd $SPEC/benchspec/MPI2007/107.leslie3d/run
      $ spectar cvf  -  run_base_test_infinipath.0000 | specbzip2 > myrun.tar.bz2
      $ ftp
      ftp> op runsys
      Connected to runsys
      Name: whoever
      Password:
      ftp> bin
      ftp> put myrun.tar.bz2
    

    Note that the above example assumes that you have versions of bzip2 and tar available on the run time system, which you will use to unpack the compressed tarfile, typically with something like this:

    bzip2 -dc myrun.tar.bz2 | tar -xvf -

  13. Run it: If you are using a unified environment, you are now ready to try "specinvoke":

      $ specinvoke -n
            ... shows the command line that executes the benchmark, in this example 
    
      timer ticks over every 1000 ns
      # Use another -n on the command line to see chdir commands
      # Starting run for copy #0
      /usr/bin/mpirun -m $SPEC/hostfile -np 16 \
          ../run_base_test_infinipath.0000/leslie3d_base.infinipath > \
          leslie3d.out 2>> leslie3d.err
    
      $ specinvoke -h
            ... explains other options to specinvoke, including '-i #' 
                for running multiple iterations
      $ specinvoke
            ... executes that command line
    

    If you are using a cross-compile or triple environment, then you won't be able to use specinvoke. Instead, you'll need to extract the run commands from the log and enter them by hand.

  14. Save your work: Important: if you are at all interested in saving your work, move the run/build* and run/run* directories to some safer location. That way, your work areas will not be accidentally deleted the next time someone comes along and uses one of runspec cleanup actions..

  15. Repeat: Admittedly, the large number of steps that it took to get here may seem like a lot of trouble. But that's why you started with a simple benchmark and the simplest workload (--size mtest in the fake step). Now that you've got the pattern down, it is hoped that it will be straightforward to repeat the process for the other available workloads: --size=mtrain and --size=mref, and then for additional benchmarks.

    But if you're finding it tedious... then maybe this is an opportunity to sell you on the notion of using runspec after all, which automates all this tedium. If the reason you came here was because runspec doesn't work on your brand-new environment, then perhaps you'll want to try to get it built, using the hints in tools-build.html.

Validation

Note that this document has only discussed getting the benchmarks built and running. Presumably at some point you'd like to know whether your system got the correct answer. At that point, you can use specdiff, which is explained in utility.html.


* dot-space-dot-slash-shrc is the safe way to type it. If you use Linux/bash, or if you have the current directory in your path, you can probably type ". shrc" (dot-space-shrc) with no problems.


Copyright © 1999-2010 Standard Performance Evaluation Corporation

All Rights Reserved