SPECvirt_sc2013 Technical Support

Version 1.1-03 - December 8, 2016



                                                NOTE:
To ease workload setup and tuning, review recent submissions and
download the accompanying supporting documentation archive (TGZ).
It contains the submission's files with non-default settings for OS,
network, memory, database, and application server tuning.

Example VM Environment

Q. Are there any scripts or pre-loaded VMs available to help me set up the benchmark?

A. The SPEC Virtualization subcommittee provides scripts that assist you in rapidly creating a SPECvirt_sc2013 example test environment. You can create a SPECvirt_sc2013 tile consisting of the five different workload VMs plus a database server as well as a client VM. The guide and scripts are also useful when building from scratch. Sign up at the SPECvirt Forum, then review the first topic for instructions on downloading the scripts and their accompany documentation.

NOTE: The versions of software referenced in the ExampleVM Setup Guide represent a known working combination of software and are used to build a working environment for example purposes only. SPEC does not endorse the software used in this example, and the fact that these versions are used in this document does not mean they would be compliant for submitting an official SPECvirt_sc2013 result. Please consult the SPECvirt_sc2013 Run and Reporting Rules to check on software version compliance.

Harness

Q. How do I increase error logging during a test?

A. In Control.config, the default for logging is 1 (DEBUG_LEVEL=1). You can increase this to 2 (mid-level logging) or 3 (maximum logging).

Q. How do I resolve the following Java connection problem:

java.rmi.ServerException: RemoteException occurred in server thread; nested exception is:
java.rmi.RemoteException: Connection refused (Connection refused); nested exception is:
java.lang.Exception: Connection refused (Connection refused)

A. Make sure the SPECpoll process is running on each workload VM (not the client). On each workload VM, check the contents of /tmp/pollme.out (for example, webserver):

cat /tmp/pollme.out
Creating RMI listener using RMI Registry port 8001
webserver/10.4.3.61:8001 ready...
2015-10-27 19:20:07:588 contacting host 'infraserver'...
webserver: 1445944807646
infraserver: 1445944807645

If you don't see similar output, start SPECpoll on the workload VM (for example on webserver):

ssh webserver '. .bash_profile; java -jar /opt/SPECpoll/pollme.jar -n `hostname` -p 8001 > /tmp/pollme.out 2>&1&'

Q. I've completed a run that passes all validation tests and would like to submit the result, but some (or all) of the testbed information in the disclosure report .html needs to be updated. How do I regenerate updated/corrected .sub and .html files for submission?

A. The recommended way to do this is to update all of the information in the Testbed.config file (located in the SPECvirt directory). Then, copy and insert the contents of Testbed.config into the .raw file at the line just above "END SPECvirt_sc2013". Finally, regenerate the .sub file with the usual syntax:

java -jar reporter.jar -r name-of-rawfile

Q. What version of Java do I need?

A. You need to run a supported version of Java 6 or greater Compatible software. The clients, appservers, and batchservers require Java 6 or greater JDK. All other VMs require Java 6 or greater JRE.

Q. How long does a compliant test run?

A. A compliant test must run for 7200 seconds. If POLL_INTERVAL_SEC in Control.config is greater than 7200 seconds, the test continues to run, but the run is non-compliant. At the end of the test, the harness reports a compliance error:

* Compliance Errors:

o POLL_INTERVAL_SEC value is: 3600; should be 7200 or greater.

Webserver/Infraserver

Q. How do I fix the following webserver SSL or TLS errors:

2015-05-06 09:25:51:414 SPECweb_Support: [ERROR] STATE 0; makeHttpRequest() failed.
2015-05-06 09:25:51:475 SslConnection: [ERROR] IOException during SSL handshake: javax.net.ssl.SSLHandshake
Exception: No appropriate protocol (protocol is disabled or cipher suites are inappropriate)

...

2015-05-06 09:26:16:544 SslConnection: [ERROR] IOException during SSL handshake: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
2015-05-06 09:26:22:535 Connection: [ERROR] Write to socket failed! IOException was: javax.net.ssl.SSLException: Connection has been shutdown: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure

A. SSLv3 is disabled by default starting with JDK 8u31 and JDK 7u75. Refer to this Oracle note for details.

To enable SSLv3, log into the client and:

  1. Remove SSLv3 and MD5withRSA from the jdk.tls.disabledAlgorithms property in the $JDK_HOME/jre/lib/security/java.security file (SPECvirt assumes an MD5-based SSLv3 cipher).
  2. Add SSL_RSA_WITH_RC4_128_MD5 to the jdk.tls.legacyAlgorithms list in the $JDK_HOME/jre/lib/security/java.security file.
  3. Remove !MD5 from the SSLCipherSuite list in /etc/httpd/conf.d/ssl.conf file.

Q. If I want to use TLS, which cipher suites do you recommend?

A. Many Java 1.7 JRE versions do not support TLS_RSA_WITH_AES_128_GCM_SHA256. We found that the following two cipher suites perform similarly to SSLv3 and are available options for Java 1.7 JREs:

TLS_RSA_WITH_AES_128_CBC_SHA
TLS_RSA_WITH_AES_256_CBC_SHA

Note that the CPU consumption on the SPECweb prime client driver will likely be noticeably higher. Please review the documentation for your version of Java 1.7 to verify which cipher suites are supported. Also, some cipher suites must be enabled explicitly in the JRE settings.

Alternatively, Java 1.8 offers a larger range of TLS cipher suites (namely, the TLS GCM varieties). If Java 1.7 cipher options are too limited, you may want to consider upgrading to Java 1.8 to enable and experiment with the additional cipher options.

Q. How do I fix webserver validation errors?

A. This can occur if there are Besim processes (e.g. besim_fcgi.fcgi) on the infraserver VM that retain the data from a prior run. This is most likely to occur when a test is aborted and restarted but the httpd and associate Besim daemons are not restarted on the infraserver VMs. To avoid this condition, delete the old besim_support.globals file and restart the httpd on the infraserver:

ssh infraserver /etc/init.d/httpd stop
ssh infraserver rm -f /tmp/besim_support.globals
ssh infraserver /etc/init.d/httpd start

You also can configure the harness to run an initialization script that contains these commands. For example, the Control.config contains: PRIME_HOST_INIT_SCRIPT[1] = "webInit.sh" and the script webInit.sh would include the commands above.

Q. When running webserver, the test seems to be running and possibly gives one or two lines of polling data when the test ends. Examination of Clientmgr logs shows one or more SPECweb tests have large numbers of errors similar to:

Messages from: 10.17.181.118:1191

-> 2009-10-02 10:06:43:115 Creating specwebclient using RMI Registry port 1191
-> 2009-10-02 10:06:43:144 perf2.company.com:1191 ready...
-> 2009-10-02 10:09:04:082 LoadGenerator: Setting sleep delay.
-> 2009-10-02 10:09:07:373 specwebclient: setting tile ID to 1
-> 2009-10-02 10:09:08:383 LoadGenerator: Ramping up for 60 seconds.
-> 2009-10-02 10:09:08:383 LoadGenerator: Starting 500 threads.
-> 2009-10-02 10:10:08:385 LoadGenerator: All threads running, test starting
-> 2009-10-02 10:10:59:731 Connection: SocketTimeoutException waiting for end-of-header
-> 2009-10-02 10:10:59:731 SPECweb_Support: [ERROR] SocketTimeoutException encountered during run!
-> 2009-10-02 10:11:01:264 SPECweb_Support: [ERROR] Invalid title tag, expecting 'SPECweb2005: Support Category' but got 'SPECweb2005: Support Home' was found
-> 2009-10-02 10:11:01:264 SPECweb_Support: [ERROR] Expected 13 catalog listings, 7 found.
-> 2009-10-02 10:11:01:264 SPECweb_Support: [ERROR] STATE 2; Expected 27500 bytes appended to response but found 51700
-> 2009-10-02 10:11:01:264 SPECweb_Support: [ERROR] STATE 2; RESPONSE INVALID on GET /support/catalog.php?id=3936 HTTP/1.1
-> 2009-10-02 10:11:01:264 SPECweb_Support: [ERROR] Response was:
-> http/1.1 200 ok

A. Check the open file limit on the clients:

ssh client2 "ulimit -n"

If you get a number such as 1024, then the ulimit has not been set. Check the /etc/sysctl.conf file to make sure that you have a line similar to:

fs.file-max = 500000

Then check /etc/bashrc to make sure the following is included:

# Overide open files limit of 1024
ulimit -n 100000 > /dev/null 2>&1

Then reboot the affected client(s) and rerun the test.

Q. How do I resolve HttpRequestSched errors?

A. Closely look at the network tunings for your VMs, especially for the workload that is reporting failures. The webserver does the largest amount of network traffic with the client and is usually the first to require network tuning. Looking at the published results for applied tunings in those configurations provides a good idea of where and how to start tuning.

Q. Webserver fails, and examination of Clientmgr logs shows errors similar to:

Messages from: 10.17.181.118:1191

-> 2014-07-29 15:11:52:602 SPECweb_Support: [ERROR] STATE 4; makeHttpRequest() failed.
-> 2014-07-29 15:11:52:602 SPECweb_Support: [ERROR] STATE 4; makeHttpRequest() failed.
-> 2014-07-29 15:11:50:603 SslConnection: [ERROR] IOException creating socket. Exception is:
-> java.net.ConnectException: Connection refused
-> hostServer is: webserver; port is: 443

A. Issue the following commands on the webserver:

netstat –ant|grep 443
ps –ef|grep httpd

If the netstat command returns nothing and the ps -ef command shows no httpd processes running, then you need to enable SSL or TLS. Shut down httpd and add to /etc/httpd/conf/httpd.conf:

Include conf.d/ssl.conf

Add to /etc/httpd/conf.d/ssl.conf:

ServerName webserver:443

Then restart httpd.

Appserver/Dbserver

Q. What parameters do I use to configure open source databases to meet Atomicity, Consistency, Isolation, and Durability (ACID) requirements as described in the SPECvirt Run and Reporting Rules?

A. For PostgreSQL, fsync=on and synchronous_commit=on are required. For MySQL and MariaDB, innodb_flush_log_at_trx_commit=1 is required. Depending on your configuration and products used, there may be other parameters needed as well. Please consult with your product's documentation on strict ACID configuration.

Q. How do I resolve FAILED transaction validation errors such as:

Example 1

0-0-0 = Work Order Transaction validation FAILED
0-0-1 = Largeorder Vehicle Purchase Rate FAILED
0-0-2 = Regular Vehicle Purchase Rate FAILED

Example 2 (from appserver audit report):

Work Order Transaction validation
Condition : New Work Order TxCount <= New Work Order DB Count
Work Order Tx Count 96472
Work Order DB Count 96450
Work Order Transaction validation FAILED

Example 3 (from html results file):

Validation Errors:

A. This usually means that the application server VM and client are configured properly but require tuning. Try tuning heap size and the network between the clients and VM. Review recent submissions and download the accompanying supporting documentation archive (TGZ) for application and database server settings.

In the case of GlassFish, try tuning the following domain.xml parameters:

To find a sample domain.xml file, extract the archive and look under the Virtualization folder (VIRT) and select any tile. Then navigate through the appserver folder to software (SOFTW) then spec2004-01 then config where domain.xml resides.

Q. How do I fix the Unsuccessful switchlog appserver error:

Messages from: 10.17.181.114:1091
-> 2010-08-18 15:58:16:893 Creating jappclient using RMI Registry port 1091
-> 2010-08-18 15:58:16:931 perf2.company.com:1091 ready...
-> Driver Host: perf2company.com Tile Number:0
-> 2010-08-18 16:02:19:457 matchOut() messages set ...
-> 2010-08-18 16:02:19:458 Starting rmiregistry; bindWait = 45000
-> 2010-08-18 16:03:04:466 Starting Controller; bindWait = 45000
-> 2010-08-18 16:03:04:473 waiting for: Binding controller to /
-> rec'd notifyInterrupt(101) call
-> Binding controller to //perf2company.com:2098/Controller
-> Launcher: done in waitMatch(0)
-> 2010-08-18 16:03:49:611 Starting Agents
-> ---------------//perf2.company.com:2098/Controller
-> Calling switchLog as master
-> url[0] is : http://specemulator:8080/Emulator/EmulatorServlet?cmd=switchlog
-> url[1] is : http://specdelivery:8000/Supplier/DeliveryServlet?cmd=switchlog
-> 2010-08-18 16:04:34:775 Exception encountered executing runScript(): java.io. IOException: Unsuccessful switchlog
-> java.io.IOException: Unsuccessful switchlog
-> at org.spec.jappserver.launcher.Driver.switchLog(Driver.java:705)
-> at org.spec.jappserver.launcher.Driver.runScript(Driver.java:531)
-> at org.spec.jappserver.launcher.jappclient.setup(jappclient.java:146)
-> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

A. If the following steps are successful, it is very likely that the AppServer VM and the Database VM are configured properly. The switchlog or any other client side error could be a misconfiguration on client(s):

Verify that emulator is accessible. Otherwise check the server log where emulator application is deployed:

curl http://specemulator:8080/Emulator/EmulatorServlet?cmd=switchlog
200 OK

Verify that SPECjAppServer application is available on AppServer:

curl http://specdelivery:8000/Supplier/DeliveryServlet?cmd=switchlog
200 OK

For any other response span style="font-family: Courier New,Courier,monospace;", such as 400 Error, check AppServer server log and fix the issue. For example:

/opt/glassfish/domains/spec2004-1/generated/jsp/j2ee-apps/SPECjAppServer/supplier_war/Delivery_log1.err
(Permission denied)

ssh specdelivery
ls -l /opt/glassfish/domains/spec2004-1/generated/jsp/j2ee-apps/SPECjAppServer/supplier_war
-rw-r--r-- 1 root root 42 Aug 16 20:43 Delivery_log0.err
-rw-r--r-- 1 root root 42 Aug 16 20:43 Delivery_log1.err
-rw-rw-r-- 1 spec spec 44 Dec 17 2009 tldCache.ser

Change their permissions:

chmod -R 775 /opt/glassfish

Verify that database is properly configured and accessible by the SPECjAppServer application on AppServer:

curl http://specdelivery:8000/SPECjAppServer/app?action=atomicityTests

The resulting text should show PASSED for the three Atomicity tests.

Q. How do I fix the NullPointerException appserver error in Clientmgrn_1088.out:

-> Error: Connection: UnknownHostException?: specdelivery
-> java.lang.NullPointerException?
-> at org.spec.jappserver.driver.http.HttpDealer?.login(HttpDealer?.java:651)
-> at org.spec.jappserver.driver.DealerEntry?.logIn(DealerEntry?.java:1193)
-> at org.spec.jappserver.driver.DealerEntry?.loginCtl(DealerEntry?.java:1134)
-> at org.spec.jappserver.driver.DealerEntry?.doBrowse(DealerEntry?.java:1046)
-> at org.spec.jappserver.driver.DealerEntry?.run(DealerEntry?.java:256)

A. Verify that specdelivery is correctly specified in hosts and the following succeeds:

ping specdelivery

Check the open file limit on the clients:

ssh client2 "ulimit -n"

If you get a number such as 1024, then the ulimit has not been set. Check the /etc/sysctl.conf file to make sure that you have a line similar to:

fs.file-max = 500000

Then check /etc/bashrc to make sure the following is included:

# Overide open files limit of 1024
ulimit -n 100000 > /dev/null 2>&1

Then reboot the affected client(s) and rerun the test.

Q. How do I build SPECjAppServer.ear using GlassFish?

A. Building SPECjAppServer.ear for any application server requires vendor-specific deployment descriptors. For GlassFish, build_glassfish.xml is the modified version of the generic build.xml to include GlassFish deployment descriptors from src/deploy/glassfish. To build SPECjAppServer.ear with GlassFish, update config/glassfish.env as per your environment and run ant from the SPECjAppServer2004 directory as follows:

ant -f build_glassfish.xml

Mailserver

Q. How do I resolve continued IMAP timeout errors during the course of mailserver runs:

Warning: .doMyLongCommand received an SocketTimeoutException exception java.net.SocketTimeoutException: Read timed out

A. You can create a "warmed up" mailstore. This is a one-time task. Run a two- or three-hour mailserver test and ignore the IMAP timeout errors during the first 20 or 30 minutes. When the test stops, shutdown the mailserver software and use tar to create an archive of the warmed-up mailstore for subsequent test runs. For example, if you use Dovecot:

ssh mailserver
/etc/init.d/dovecot stop
cd /mailstore/dovecot; tar -czf /root/BACKUPS/dovecot_500users_warmed.tgz /mailstore/dovecot
/etc/init.d/dovecot start

Then restore the mailstore using this archive between test runs.

Q. How to I restore the mailstore between test runs?

A. Use a mailserver initialization script in Control.config to restore the mailstore. Using the example in the previous question, after you back up the mailstore archive to /root/BACKUPS/dovecot_500users_warmed.tgz on the mailserver, the mailserver init script that Control.config calls on the prime client might contain:

ssh mailserver "/etc/init.d/dovecot stop"
sleep 10
ssh mailserver "cd /mailstore/dovecot; rm -rf maildir"
ssh mailserver "cd /mailstore/dovecot; tar -zxf /root/BACKUPS/dovecot_500users_warmed.tgz"
sleep 10
ssh mailserver "/etc/init.d/dovecot start"
sleep 10

Q. How do I resolve the following repetitive validation errors:

UID_STORE_NUM_UNSET_FLAGS_SEEN Mix
Pct Failed. Expected: 30.000, Achieved: 34.912
FETCH_NUM_RFC822HEADER Mix Pct Failed. Expected: 515.000, Achieved: 627.550

A. Restore the mailstore to its original state and rerun.

Q. Does the mailstore needed to be restored to its initial state after each benchmark run?

A. It is optional whether or not you restore the mailstore between test runs. Some submissions restored the mailstore before starting the test, while others didn't.

Q. How do I resolve the following message count errors:

2013-07-02 00:04:59:167
LoadGenerator: Threads started, test in progress...
test1 Expected 1969 messages, found 2002
test1 FATAL ERROR - Aborting run, and EXITING.
test11 Expected 1969 messages, found 2003
test11 FATAL ERROR - Aborting run, and EXITING.

A. Finding more messages than expected suggests that the mailstore needs to be reloaded or restored from a previously warmed up backup, if available.

Q. How do I resolve the following mailstore loading errors:

BAD Error in IMAP command APPEND:
Invalid arguments

A. This usually occurs when the SPECimap/IMAP*.config files are in DOS format instead of UNIX format. Edit the config files with vi and do the following to convert to UNIX format:

:set ff=unix
:w
:q

Batchserver

There are no known issues at this time.

Submissions

Q. How do I submit results?

A. Information on how and when to submit SPECvirt results is available at http://www.spec.org/osg/submitting_results.html. Specific SPECvirt information on review cycle scheduling is at http://www.spec.org/osg/calendar/2014_osg-virt.html. Additional details on the SPECvirt submission and review process are available at:

SPEC does not post the email aliases for submissions or the ftp upload information. For this information, the Benchmarker must contact info@spec.org. Benchmark license numbers may be requested.

Note that benchmarkers must follow all Run and Reporting Rules when testing and generating results for submission and publication. Benchmarkers can submit anytime prior to the weekly review cycle deadline as the results are held in a locked directory. Benchmarkers can request that results can be withdrawn during the review cycle. Minor updates such as adding missing information or making editorial corrections are allowed, but the expectation is that the submission packet is essentially complete as detailed in the Run and Reporting Rules.

Next Steps

If you haven't solved your problem with the information above and need to contact SPECvirt_sc2013 Support, below is a list of the information to collect, review, and submit with your problem report:

You should also consider using an example collection script from a published archive TGZ as a template to collect potentially useful debugging information. Include all key files and logs with your problem report to give SPECvirt technical support volunteers the best chance to help you resolve your problem.


Product and service names mentioned herein may be the trademarks of their respective owners.

Copyright © 2013-2016 Standard Performance Evaluation Corporation (SPEC).

All rights reserved.