SPEC
SERT&tm; - Frequently Asked Questions (FAQ)

SERT™ - Frequently Asked Questions (FAQ) at 14th November, 2019

Many common questions are addressed in this FAQ or in the SERT documentation on the SERT Information (http://www.spec.org/sert) web page.


Section A: About the SERT™

  • A1: Is this a Benchmark? Is SERT replacing SPECpower_ssj2008?
    • No. This is a tool for the EPA and other global energy efficiency governing bodies to utilize for their server power efficiency programs. The use of SERT data for Marketing purposes is specifically disallowed.
  • A2: Can I use the SERT to make numeric comparisons?
    • No. The use of SERT data for numeric comparisons (marketing or otherwise) is not allowed except for research purposes.
  • A3: Where can I find the documentation?
    • Documentation is provided with the SERT release in the "Documentation" directory.
  • A4: Are updates to documents available?
  • A5: What is a "compliant" run of SERT?
    • A compliant run meets all the requirements of the SERT run and reporting rules for a valid run.
    • In addition to the run and report rules, several validation and tolerance checks are built-in to the SERT.
  • A6: What is a "FDR"?
    • FDR is an acronym for "Full Disclosure Report".
    • The FDR is the document that represents a full run of the SERT
  • A7: Can I "turn off" compliance checking? My results are only non-compliant because I do not have a temperature sensor.
    • No. This is to prevent a Full Disclosure report being used that is or might be non-compliant.
  • A8: What is a "graduated workload"?
    • A graduated workload operates at several levels, usually stepping up or down by some regular value.
  • A9: Why did SPEC choose to implement a graduated workload?
    • Server power consumption has been shown to be significantly different at low load levels versus high load levels, and server loads vary widely from hour to hour and day to day.
    • Since no definitive surveys exist, and all data centers are different, SPEC has chosen to provide graduated loads across a set of load levels to portray performance and power consumption in several reasonable ranges.
  • A10: Can I acquire and run the SERT in my own facility?
    • Yes. See "Acquiring the SERT" below.
  • A11: Does the SERT support measuring systems with DC-DC power supplies? (DC=Direct Current)
    • No.
    • DC powered servers can not be compared to AC (Alternating Current) powered servers because the AC to DC power conversion cost (energy loss) is not present in DC powered servers.
    • Although physically possible, SPEC has neither developed nor tested methods for DC system measurement.
  • A12: Is there a "Users Guide" that walks me through set up, run and results interpretation?
    • Yes, that document is the "SERT Users Guide". See below
  • A13: Can I get documentation without purchasing the SERT?
  • A14: What are the key documents I should read before I get started?
    1. SPEC License
    2. SPEC Power and Performance Methodology
    3. SERT Users Guide
    4. Measurement Setup Guide
    5. SERT Design Documents
    6. FAQ (this document)
    7. Run and Reporting Rules
  • A15: Does my company qualify for reduced or non-profit pricing for the SERT product?
    • Reduced and non-profit pricing is reserved for organizations that qualify under IRS guidelines.
  • A16: Does SERT include the latest version of PTDaemon®?
  • A17: Are updates to SERT and PTDaemon provided free of charge?
    • This has yet to be determined. SPEC expects to provide minor bug fix updates without charge.

Section B: Hardware Requirements

  • B1: What is the minimum hardware required?
    • At minimum, you will need two network connected systems and one accepted power analyzer and accepted temperature sensor for fully compliant runs.
    • The minimum SUT (Server Under Test) hardware requirements are descipted in the SERT User Guide

Section C: Software Requirements

  • C1: Where can I obtain software listed as a supported option for SERT?
    • Please contact the vendor or provider of the software for further assistance if you are unable to locate it yourself.

Section D: Power Analyzers

  • D1: Is a power analyzer required to run the SERT?
    • For compliant runs, yes.
    • For any other runs, the SPEC PTDaemon® (Power Temperature Daemon) has a device type "dummy" for these occasions.
  • D2: Can the SERT support multiple power analyzers?
    • Yes. Multiple power analyzers and multiple temperature sensors are supported. Nevertheless, currenly not though the GUI.
  • D3: What power analyzers have been tested with the SERT?
    • A list of SPEC Accepted (supported and compliant) power measuring devices are documented on the SPEC web site at: Accepted Measurement Devices (http://www.spec.org/power/docs/SPECpower-Device_List.html)
    • The PTDaemon program can be used to display a list of power and temperature measuring devices along with their device codes;
    • For SPEC Accepted devices: Run the PTDaemon with the -h parameter, or zero parameters.
      • Listed devices (except 0/1000. Dummy) are valid for compliant runs of the SERT.
  • D4: Will my power analyzer work with the SPECpower PTDaemon even though my device does not appear on the accepted devices list shown on the SPEC web site at Accepted Measurement Devices (http://www.spec.org/power/docs/SPECpower-Device_List.html)?
    • No, only devices on the SPEC Accepted Device List are expected to produce valid results.
    • If your device is not present on the devices list, it is unlikely that PTDaemon will successfully read data.
    • Communication protocols are quite different among the various manufacturers and models of power analyzers.
  • D5: Will the SPECpower PTDaemon (Power Temperature Daemon) connect to and read my power analyzer even though my device is not on the tested list?
    • It is unlikely that PTDaemon will successfully read a meter not on the "tested" list.
    • See the responses to the FAQ just above.
  • D6: How can I get my power analyzer included in the SPEC accepted devices list?
    • Basically, documentation must show that the device has sufficient accuracy, code must be written to enable SPEC's PTDaemon software to read measured values from the device, and then testing must be performed to show that the device does meet SPEC's requirements in actual usage.
  • D7: Will a power analyzer calibration certificate for one device apply to other power analyzers of exactly the same model?
    • No. The calibration certificate issued is valid only for the specific device (one serial number) calibrated.
    • A recently calibrated power analyzer is required to produce and report compliant results. "Recent" means that the calibration certificate must be dated within one year of the test date.
  • D8: Is the test certificate I received with my instrument sufficient to meet the calibration requirements of the power analyzer requirements section of the run rules?
    • If the certificate is entitled "Certificate of Calibration" or "Certificate of Traceable Calibration" and includes a unique calibration number or sticker, lab accreditation to ISO or other standards, and test equipment certification with traceability to a national standards institute, it probably meets the requirements. National standards institutes include NIST (USA), PTB (Germany), and NML (ROC - Taiwan) among others.
  • D9: No data cable came with my power analyzer (RS-232), what now?
    • Each brand/model power analyzer may require a different cable. some experimentation might be required.
    • Some power analyzers will work with a "null modem" cable.
    • One combination that has worked with some devices is a standard "Modem" cable with the addition of a "NULL MODEM" adapter.
    • The above combination can also be plugged into a RS-232 to USB converter.
    • Some data cable information can be found in the Measurement Setup Guide (http://www.spec.org/power/docs/SPEC-Power_Measurement_Setup_Guide.pdf)
  • D10: My power analyzer supports only RS-232 for data logging; my control system has no serial ports; neither DB-9 nor DB-25. - What do I do?
    • There are devices that convert RS-232 to USB, that cost $10-14 (US$).
    • Several brands of serial to USB converters have been found to work well.
      • With Windows, install the software per manufacturer instructions.
        • Then, use the device manager to see what "COM" port number was assigned.
        • Edit the runpower.bat file; change the default DEVICE_PORT number.
        • Note that the COM port assignment on your system may change if you disconnect and re-connect a device or change USB connections.
  • D11: The PTDaemon software can't connect to my power analyzer.
    - What is wrong?
    • First, assure the analyzer is communicating on the data cable with the host machine.
      • The best way to verify this is to first use the vendor supplied software to read the analyzer.
    • Most problems can be resolved with this method, problems such as cable not connected, wrong cable, incorrect device settings and so on.
    • If the vendor supplied software shows the correct readings, assure that the following are correct:
      • In the SERT GUI PTDaemon Configuration, ensure the following options are set correctly:
        • Device Enabled checkbox is checked.
        • Listening port is 8888 (increment port for additional power or temperature devices.
        • Measurement Device is set to your supported Power Analyzer name
        • Interface is set correctly (Serial, GPIB, USB, or Ethernet).
        • Interface Address is set to the correct port for your configuration (COM1, COM2, etc.)
        • Finally, ensure you click the "Test Connection" button before proceeding.
        • Repeat these steps for your Temperature Sensor.
      • If not using the SERT GUI, ensure the following are set correctly in runpower.bat/runtemp.bat in the PTDaemon subdirectory:
        • NETWORK_PORT set correctly: 8888 for power analyzer 1, 8889 for temp sensor.
        • DEVICE set to a value that reflects your connected Power Analyzer (run the ptd exectable for a device list).
        • DEVICE_PORT is set correctly (COM1, COM2, etc.)
      • More details on device setup can be found in the Measurement Setup Guide (http://www.spec.org/power/docs/SPEC-Power_Measurement_Setup_Guide.pdf)
      • If your problems persist, please fill out a SPECpower support ticket at this link: http://www.spec.org/sert/feedback/issuereport.html.
  • D12: Why does my power analyzer display unexpected watts readings?
  • D13: I am testing at the 115V specified in the EPA EnergyStar document, and when I specify 115 for the voltage in the LineStandard section of test-environment.xml I get messages that 115V is not a known standard. What do I do?
    • Specify 120 for the voltage value. The LineStandard values refer to a country's standardized voltage under which the tests are being performed; 120V is the nominal standard voltage for the U.S.A. 115V is not a standardized voltage but rather a utilization voltage that the EPA has chosen. As long as the voltage measured at the SUT remains above 114V (95% of 120V) you will not get any errors or warnings.

Section E: Temperature Sensors

  • E1: How do I run without a temperature sensor?
    • The PTDaemon (Power Temperature Daemon) has a sensor type of "dummy".
      • this is the best temporary fix if you will add or use a sensor in other runs.
      • The device code for "dummy" temperature sensor is 1001.
    • You may also remove the temperature sensor data source from the ccs.props file.
      • comment the lines that define the temperature sensor (# as first character)
  • E2: What temperature sensors have been tested with the SERT?
  • E3: How do I find the "device code" for a temperature sensor?
    • Run the PTDaemon program with zero command line parameters or -h.
    • A list of temperature sensors and power analyzers will be displayed, along with the 'help' information.

Section F: General Troubleshooting

  • F1: What if the PTDaemon can't connect to the power analyzer?
    • See FAQ in the Power Analyzer section
    • Check the COM port and the analyzer type passed to the power daemon (ptd_xxxx.exe)
    • Assure the analyzer is connected via serial, GPIB or USB cable and the analyzer is powered on.
  • F2: What if I don't see the operating environment combination (processor architecture, operating system, JVM) that I need in the SERT-JVM_Options.html file?
    • First, check the most current file at http://www.spec.org/sert/SERT-JVM_Options.html. There is a link to this URL in the version of that file that is shipped with the product. It is possible that the desired environment has been included in the latest web version. Also, be aware that the options listed in the file are intended to be used for submissions to energy efficiency programs, such as the US EPA's ENERGY STAR program. If you are using the SERT to conduct research for purposes other than submission for such a program, you have more flexibility in the options that you choose to use.
    • For submission of results using hardware, operating system and JVMs that are related to those that are listed in the options file, but are not specifically listed in the desired combination (For example, a processor family that is older than those listed in the options file), SPEC recommends using the H1 or J1 option sets. If this delivers unsatisfactory results, the user can submit a service request to ask for the development of a better set of JVM options.
    • For submission of results using any of A) a hardware architecture that is not represented in the options file or B) an operating system that is not represented in the options file or C) a JVM that is not represented in the options file, the user is encouraged to contact the suppliers of the component(s) to request that they work with SPEC to complete the necessary testing to have their product be included in the list of accepted operating environments.
  • F3: How do I know which specific Oracle Hotspot or IBM J9 JVM version to choose?
    • The SERT is designed to run with 64-bit Java JRE 1.7.0. Using the latest release from Oracle or IBM is strongly encouraged. Consult these vendors' websites for access to the latest versions.
  • F4: I am having problems running the correct JVM on my SUT or controller. How do I ensure that the SERT is using a specific JVM?
    • By default, the serthost.bat/.sh (SUT) or director.bat/.sh (Controller) will expect to find the Java executable in the system PATH. If a suitable Java version is not found, the test will fail.
    • Some Java versions in Microsoft Windows may not automatically insert themselves into the system PATH during installation.
    • Linux systems normally require explicit pathing to the java executable in the serthost.sh file or in the system PATH environmental variable.
    • Please check the serthost.bat/.sh or director.bat/.sh file on the problem system, and ensure that the set JAVA=java line is modified to point directly to the JVM executable you intend to run on the SUT.
    • When switching between multiple JVM vendors or versions, modifying the set JAVA= line to point to the correct version is the preferred method.
      • Example modification in serthost.bat on a Microsoft Windows system: set JAVA=c:\Hotspot-1.7.0_u15\bin\java.exe
      • Example modification in serthost.sh on a Linux system: set JAVA=/usr/local/IBM-J9/jre/bin/java
  • F5: I am encountering unexpected hangs and exits in the GUI after inserting non-ASCII characters (e.g. German Umlaut Characters) into the test-environment.xml file.
    • The current release of the SERT does not support non-ASCII characters in the test-environment.xml file.
  • F6: What do I do if the SERT run has completed in the GUI, I click "Next", and the "View" button doesn't work?
    • The system is likely to be generating all of the html result files, which can sometimes take as long as 45 minutes to complete for very large SUT configurations. Wait for the "View" button to work, and you can view the results file at that point.
  • F7: For SERT configurations running more than 100 JVM clients on the SUT the following error messages will show up in "client.<#>.out" files for # greater than 99 and no "log.<#>.txt" files are generated:
    • Can't get load log handler "java.util.logging.FileHandler"
    • java.io.IOException: Couldn't get lock for log.%u.txt
    • These error messages can be ignored. All client processes will proceed and finish their tasks correctly including those, which generate above error messages. For clients 100 and above you won't find the "log.<#>.txt" files but "client.<#>.out" files only.
  • F8: Why am I getting invalid results with messages that relate to my power analyzer?
  • F9: Why do I only get a results.xml.partial file in the results directory after a full run?
    • If the user attempts to view the results before final processing has been completed, this can occur. It could also be caused by a failure during post-processing of the results file. Please review the file director.log on the controller. The error messages often point to the location in the results file where the error occurred.
  • F10: What is the "coefficient of variation between clients"?
    • Coefficient of variation is a mathematical term that measures the variability of a set (defined as the standard deviation divided by the mean). Each Java instance that is run on the SUT is referred to as a client. SERT worklets have different limits on the maximum coefficient of variation between clients, as a way of ensuring that the test is run in a stable and repeatable way.
  • F11: Why do I get WARNING messages about coefficient of variation between clients on storage worklets?
    • A single client is run for each physical disk during storage worklets. If the results for the same worklet between clients show high variability, the warning is produced. If the disks are of different types this message is to be expected. If the disks are identical, this message usually indicates inconsistent configuration between disks.
  • F12: Why do I get INVALID messages about coefficient of variation between clients on memory worklets?
    • This usually indicates an unbalanced memory configuration between nodes on a multi-processor system. If channels, speeds and capacities are not identical on each node, it is unlikely that memory worklets can run successfully.
  • F13: My run terminates early on a SUT running Windows Server 2008 R2
    • There are several common causes for this issue, all documented in the User Guide and JVM options list. You must install Service Pack 1 for Windows Server 2008 R2. You must enable "lock pages in memory". See the User Guide for more information.
  • F14: During the installation process on recent Windows OSes, the SERT installed can't create the SERT install directory off of my C:\ drive.
    • This is a known permission issue on some recent Windows Operating Systems. As a workaround, you can manually create the SERT installation path (C:\SERT-1.0.2) prior to invoking the installer.
  • F15: I am seeing the Storage worklets get invalid messages with USB Hard Drives
    • USB Hard drives are not supported and will not work with the SERT tool.
  • F16: I am seeing more invalids show up in system configurations with smaller memory sizes.
    • It is strongly recommended that you reboot your SUT in between runs with smaller memory configurations (less than 1GB per HW thread).
  • F17: I am seeing java.io.IOException error messages when I run reporter.bat against a path that contains a space while trying to reformat a SERT report.
    • Change your path to results to be reformatted to not contain any spaces.

Section G: Invalid Results

  • G1: Can I submit an Invalid Result?
    • No
  • G2: Why am I getting invalid results with messages that relate to my power analyzer?
  • G3: Why do I get INVALID messages about coefficient of variation between clients on memory worklets?
    • This usually indicates an unbalanced memory configuration between nodes on a multi-processor system. If channels, speeds and capacities are not identical on each node, it is unlikely that memory worklets can run successfully.
  • G4: I am seeing the Storage worklets get invalid messages with USB Hard Drives
    • USB Hard drives are not supported and will not work with the SERT tool.
  • G5: I am seeing more invalids show up in system configurations with smaller memory sizes.
    • It is strongly recommended that you reboot your SUT in between runs with smaller memory configurations (less than 1GB per HW thread).

Section H: Note for Systems with > 64 logical processors

  • H1: For systems with more than 64 logical processors running Windows 2008 R2, be advised that a Microsoft hotfix addresses an issue where logical processors are unevenly allocated to K-Groups. This can cause performance issues and high run to run variation with the SERT. The hotfix to address this issue is found here: http://support.microsoft.com/kb/2510206

Acquiring the SERT™

Contacting SPEC

  • I have a question or problem that isn't covered in the the documentation.
    • Disclaimer

    SPEC, the SPEC logo, and the name PTDaemon are registered trademarks of the Standard Performance Evaluation Corporation (SPEC) and SERT is a trademark of SPEC. Additional product and service names mentioned herein may be the trademarks of their respective owners. Copyright ® 1998-2019 Standard Performance Evaluation Corporation (SPEC). All rights reserved.