Skip navigation

Standard Performance Evaluation Corporation

Facebook logo LinkedIn logo Twitter logo
 

Chauffeur WDK - Frequently Asked Questions (FAQ) at 2017-03-29

Many common questions are addressed in this FAQ or in the Chauffeur WDK documentation on the Chauffeur WDK Information (https://www.spec.org/chauffeur-wdk) web page.

Section A: About the Chauffeur WDK

  • A1: Is this a Benchmark? Is Chauffeur WDK replacing SPECpower_ssj2008 or SERT?

    No. This is a development tool for creating new worklets for the Chauffeur framework

  • A2: Can I use Chauffeur WDK to make numeric comparisons?

    No. New worklets developed with Chauffeur WDK can be used for research purposes. The license prohibits competitive comparisons that promote one product over another.

  • A3: Where can I find the documentation?

    Documentation is provided with the Chauffeur WDK distribution in the "doc" directory.

  • A4: Are updates to documents available?

    Yes, the most recent documents are on the SPEC website on the Chauffeur WDK Information (https://www.spec.org/chauffeur-wdk) web page.

  • A5: Can I "turn off" compliance checking?

    Compliance checks will always be performed at the end of a run. When re-generating a report manually, you can use the "-s" parameter on the Reporter to skip validation checks. In this case, the run will automatically be marked invalid, but no additional validation will be performed.

  • A6: What is a "graduated workload"?

    A graduated workload operates at several levels, usually stepping up or down by some regular value.

  • A7: Why did SPEC choose to implement a graduated workload?

    Server power consumption has been shown to be significantly different at low load levels versus high load levels, and server loads vary widely from hour to hour and day to day.

    Since no definitive surveys exist, and all data centers are different, SPEC has chosen to provide graduated loads across a set of load levels to portray performance and power consumption in several reasonable ranges.

  • A8: Does Chauffeur WDK support measuring systems with DC-DC power supplies? (DC=Direct Current)

    No.

    DC powered servers can not be compared to AC (Alternating Current) powered servers because the AC to DC power conversion cost (energy loss) is not present in DC powered servers.

    Although physically possible, SPEC has neither developed nor tested methods for DC system measurement.

    The Chauffeur WDK can run on DC powered servers, but does not support power measurements on these servers.

  • A9: Is there a "User Guide" that can walk me through set up and running Chauffeur WDK?

    Yes. That document is the "Chauffeur WDK User Guide". See below

  • A10: Can I get documentation without purchasing Chauffeur WDK?

    Yes. The User Guide and other documents are available on the SPEC web site on the Chauffeur WDK Information (https://www.spec.org/chauffeur-wdk) web page.

  • A11: What are the key documents I should read before I get started?

    1. SPEC License
    2. SPEC Power and Performance Methodology
    3. Chauffeur WDK User Guide
    4. Measurement Setup Guide
    5. SERT Design Documents
    6. FAQ (this document)
    7. Run and Reporting Rules
  • A12: Does Chauffeur WDK include the latest version of SPEC Power and Temperature Daemon (PTDaemon)?

    Each Chauffeur WDK release includes the latest PTD version available at that time. Please check to see if a newer PTD is available here: https://www.spec.org/power/docs/SPECpower-Device_List.html

  • A13: Are updates to Chauffeur WDK and PTDaemon provided free of charge?

    SPEC provides minor bug fix updates without charge. See https://www.spec.org/chauffeur-wdk for information regarding any available updates. Licensees of Chauffeur WDK 1.x are eligible to receive a free upgrade until October 31, 2017.

Section B: Hardware Requirements

  • B1: What is the minimum hardware required?

    The Chauffeur WDK 2 requires a controller system and the Server Under Test (SUT), which may consist of one or more physical servers. The controller and the SUT must be connected via TCP/IP.

    A power analyzer is required for tests that include power measurements. A temperature sensor is required if temperature will be measured. A list of supported power analyzers and temperature sensors is available on the SPEC web site at: https://www.spec.org/power/docs/SPECpower-Device_List.html

    Power analyzers and temperature sensors are not required for workload development. Dummy mode for PTD will allow for development testing.

    The minimum hardware for the SUT depends on the workload being measured. The recommended minimum requirements include at least one processor, 4 GB RAM, and 1 GB free storage space.

    Hardware requirements are described in more detail in section 1.4 of the User Guide.

Section C: Software Requirements

  • C1: What software is required to run the Chauffeur WDK?

    The controller system and SUT require a Java SE 7 (or higher) runtime environment. The Chauffeur WDK has been tested on a variety of operating systems including Microsoft Windows Server 2008 R2 SP1, Windows Server 2012 R2, Red Hat Enterprise Linux 7.3, AIX 7.1, and Solaris 11. Other operating systems may also work, but some platform-specific features may not be available.

    Software requirements are described in more detail in section 1.4 of the Users Guide.

  • C2: Note for Windows systems with more than 64 logical processors

    For systems with more than 64 logical processors running Windows 2008 R2, be advised that a Microsoft hotfix addresses an issue where logical processors are unevenly allocated to K-Groups. This can cause performance issues and high run to run variation with the Chauffeur WDK. The hotfix to address this issue is found here: http://support.microsoft.com/kb/2510206

Section D: Power Analyzers

  • D1: Is a power analyzer required to run the Chauffeur WDK?

    No. The Chauffeur WDK may be used for performance tests without a power analyzer. A power analyzer is required for measuring energy efficiency.

  • D2: Can the Chauffeur WDK support multiple power analyzers?

    Yes. Multiple power analyzers and multiple temperature sensors are supported.

  • D3: What power analyzers have been tested with the Chauffeur WDK?

    A list of SPEC Accepted (supported and compliant) power measuring devices are documented on the SPEC web site at: Accepted Measurement Devices (https://www.spec.org/power/docs/SPECpower-Device_List.html)

    The PTDaemon program can be used to display a list of power and temperature measuring devices along with their device codes.

    For SPEC Accepted devices: Run the PTDaemon with the -h parameter, or zero parameters.

  • D4: Will my power analyzer work with the SPEC PTDaemon even though my device does not appear on the accepted devices list shown on the SPEC web site at Accepted Measurement Devices (https://www.spec.org/power/docs/SPECpower-Device_List.html)?

    No, only devices on the SPEC Accepted Device List are expected to produce valid results.

    If your device is not present on the devices list, it is unlikely that PTDaemon will successfully read data.

    Communication protocols are quite different among the various manufacturers and models of power analyzers.

  • D5: Will the SPEC PTDaemon connect to and read my power analyzer even though my device is not on the tested list?

    It is unlikely that PTDaemon will successfully read a meter not on the "tested" list.

    See the responses to the FAQ just above.

  • D6: How can I get my power analyzer included in the SPEC accepted devices list?

    Basically, documentation must show that the device has sufficient accuracy, code must be written to enable SPEC's PTDaemon software to read measured values from the device, and then testing must be performed to show that the device does meet SPEC's requirements in actual usage.

  • D7: No data cable came with my power analyzer (RS-232), what now?

    Each brand/model power analyzer may require a different cable. Some experimentation might be required.

    Some power analyzers will work with a "null modem" cable.

    One combination that has worked with some devices is a standard "Modem" cable with the addition of a "NULL MODEM" adapter.

    The above combination can also be plugged into a RS-232 to USB converter.

    Some data cable information can be found in the Measurement Setup Guide (https://www.spec.org/power/docs/SPEC-Power_Measurement_Setup_Guide.pdf)

  • D8: My power analyzer supports only RS-232 for data logging; my control system has no serial ports; neither DB-9 nor DB-25 -- What do I do?

    There are devices that convert RS-232 to USB, that cost $10-14 (US$).

    Several brands of serial to USB converters have been found to work well.

    With Windows, install the software per manufacturer instructions.

    Then, use the device manager to see what "COM" port number was assigned.

    Edit the runpower.bat file; change the default DEVICE_PORT number.

    Note that the COM port assignment on your system may change if you disconnect and re-connect a device or change USB connections.

  • D9: The PTDaemon software can't connect to my power analyzer. What is wrong?

    First, ensure the analyzer is communicating on the data cable with the host machine.

    The best way to verify this is to first use the vendor supplied software to read the analyzer.

    Most problems, such as cable not connected, wrong cable, incorrect device settings and so on, can be resolved with this method.

    If the vendor supplied software shows the correct readings, ensure that the following are correct:

    Ensure the following are set correctly in runpower.bat/runtemp.bat in the PTDaemon subdirectory:

    • NETWORK_PORT set correctly: 8888 for power analyzer 1, 8889 for temp sensor.
    • DEVICE set to a value that reflects your connected Power Analyzer (run the ptd executable for a device list).
    • DEVICE_PORT is set correctly (COM1, COM2, etc.)

    More details on device setup can be found in the Measurement Setup Guide (https://www.spec.org/power/docs/SPEC-Power_Measurement_Setup_Guide.pdf)

  • D10: Why does my power analyzer display unexpected watts readings?

    This is likely to be traced back to how the analyzer is connected or the device settings.

    More details on device setup can be found in the Measurement Setup Guide (https://www.spec.org/power/docs/SPEC-Power_Measurement_Setup_Guide.pdf)

    Instructions will also be found in the power analyzer documentation or on the vendor web site. You may also contact the device vendor for further technical support.

  • D11: How do I run without a power analyzer?

    The PTDaemon has a sensor type of "dummy".

    This is the best temporary fix if you will add or use a sensor in other runs.

    The device code for "dummy" power analyzer is 1000.

    Alternatively, edit listeners.xml, find each PowerAnalyzerListener and set <listener enabled="false">

Section E: Temperature Sensors

  • E1: How do I run without a temperature sensor?

    The PTDaemon has a sensor type of "dummy". This is the best temporary fix if you will add or use a sensor in other runs. The device code for "dummy" temperature sensor is 1001.

    Alternatively, edit listeners.xml, find each TemperatureSensorListener and set <listener enabled="false">

  • E2: What temperature sensors have been tested with the Chauffeur WDK?

    A list of successfully tested and compliant temperature measuring devices are listed on the SPEC web site on the Accepted Measurement Devices (https://www.spec.org/power/docs/SPECpower-Device_List.html) page.

  • E3: How do I find the "device code" for a temperature sensor?

    Run the PTDaemon program with zero command line parameters or -h. A list of temperature sensors and power analyzers will be displayed, along with the 'help' information.

Section F: General Troubleshooting

  • F1: I am having problems running the correct JVM on my SUT or controller. How do I ensure that the Chauffeur WDK is using a specific JVM?

    By default, the host.bat/.sh (SUT) or director.bat/.sh (Controller) will expect to find the Java executable in the system PATH. If a suitable Java version is not found, the test will fail.

    Some Java versions in Microsoft Windows may not automatically insert themselves into the system PATH during installation.

    Linux systems normally require an explicit path to the Java executable in the host.sh file or in the system PATH environmental variable.

    Please check the host.bat/.sh or director.bat/.sh file on the problem system, and ensure that the set JAVA=java line is modified to point directly to the JVM executable you intend to run on the SUT.

    When switching between multiple JVM vendors or versions, modifying the set JAVA= line to point to the correct version is the preferred method.

    • Example modification in host.bat on a Microsoft Windows system:
      set JAVA=c:\Hotspot-1.7.0_u15\bin\java.exe
    • Example modification in host.sh on a Linux system:
      set JAVA=/usr/local/IBM-J9/jre/bin/java
  • F2: For Chauffeur WDK configurations running more than 99 JVM clients on the SUT the following error messages will show up in "client.<#>.out" files for # greater than 99 and no "log.<#>.txt" files are generated:

    Can't get load log handler "java.util.logging.FileHandler"
    java.io.IOException: Couldn't get lock for log.%u.txt

    These error messages can be ignored. All client processes will proceed and finish their tasks correctly including those, which generate above error messages. For clients 100 and above you won't find the "log.<#>.txt" files but "client.<#>.out" files only.

  • F3: Why am I getting invalid results with messages that relate to my power analyzer?

    INVALID: samples with unknown uncertainty exceeding the threshold
    INVALID: The voltage reported was above the maximum allowed voltage
    INVALID: The calibration date for the power analyzer is over 1 year before the test date

    Uncertainty and voltage errors relate to how the power analyzer ranges are set for the test. Please refer to the below links for instructions:

  • F4: Why do I only get a results.xml.partial file in the results directory after a full run?

    If the user attempts to view the results before final processing has been completed, this can occur. It could also be caused by a failure during post-processing of the results file. Please review the file director.log on the controller. The error messages often point to the location in the results file where the error occurred.

  • F5: What is the "coefficient of variation between clients"?

    Coefficient of variation is a mathematical term that measures the variability of a set (defined as the standard deviation divided by the mean). Each Java instance that is run on the SUT is referred to as a client. Chauffeur WDK worklets have different limits on the maximum coefficient of variation between clients, as a way of ensuring that the test is run in a stable and repeatable way.

Section G: Development Troubleshooting

  • G1: On Solaris, when using the HotSpot JVM, the Host fails with an "Exception establishing connection to client". Examination of the client.0.out file reveals that the client JVM is failing with the message "Invalid initial heap size: -Xms8192m. The specified size exceeds the maximum representable size."

    A HotSpot installation on Solaris that supports both 32- and 64-bit operation defaults to 32-bit operation. On systems with more than 4GB per client JVM, the Host will select heap sizes that the 32-bit JVM can't handle. The workaround is to arrange for the client JVMs to run with the -d64 flag. To do that, edit the config.xml in the top-level directory and change the line that reads <option-set/> so that it is:

      <option-set>
        <parameter>-d64</parameter>
      </option-set>
    

    If you're going to be running worklets in an IDE, you'll need to make the identical change to the src/Chauffeur/ChauffeurTest/config.xml file.

Acquiring the Chauffeur WDK

Contacting SPEC

  • I have a question or problem that isn't covered in the the documentation.

    For Chauffeur WDK licensees, please use the support forum: https://www.spec.org/forums/index.php?board=2.0.

    For non-Chauffeur WDK licensees, please e-mail your question to info@spec.org.

Disclaimer

Product and service names mentioned herein may be the trademarks of their respective owners.
Copyright ©1988-2017 Standard Performance Evaluation Corporation (SPEC). All rights reserved.