Performance Summary:

Performance Details:

Validation Errors:

•   No Validation Errors Found

Physical Configuration:

Virtual Configuration:

Notes:

Physical System Notes

• BIOS settings used:
• Minimum Processor Idle Power Core State set to C3 State
• Power Profile set to Custom
• Thermal Configuration set to Increased Cooling

Storage Notes

• Each port of the SUT's QLogic QMH2562 8Gb Fibre Channel HBA was connected to
• a Brocade 8Gb SAN Switch for HP BladeSystem c-Class module. Each SAN Switch
• module was connected via one 8Gb Fibre Channel link to 1 x StorageWorks
• 8/24 SAN switch. 7 x StorageWorks P2000 storage enclosures were connected
• via 2 x 8Gb Fibre Channel links to the StorageWorks SAN switch. All P2000's
• contained 24 x 146 GB SFF 15 K SAS drives and were partitioned into 3 x volumes.
• Each volume was configured as RAID10 with 8 drives and contained one tile's VMs.
• One volume was not used on the seventh P2000.

Virtualization Software Notes

• Settings made to drivers via esxcfg-module in ESX:
• for qla2xxx driver: ql2xexecution_throttle=255, ql2xmaxqdepth=128, ql2xintrdelaytimer=1
• All VMs' for a given tile, except for the appserver VM, were assigned to the virtual switch attached to one of the 10GbE ports. Each
• 10 GbE port was assigned to 10 tiles except for the second which supported 9 tiles. The appserver
• VMs ran on the third 10 GbE port. The second virtual NIC for the appserver, dbserver, infraserver
• and the webserver were attached to an internal virtual switch unique to their tile. These internal
• switches were set to use jumbo frames by using the command: esxcfg-vswitch -m 9000 vSwitch[x].
• All VMs started from ESX vSphere Client
• Advanced ESX Software settings used:
• /adv/BufferCache/SoftMaxDirty = 75
• /adv/Cpu/HaltingIdleMsecPenalty = 0
• /adv/Disk/SchedNumReqOutstanding = 128
• /adv/Irq/RoutingPolicy = 0
• /adv/Net/MaxNetifRxQueueLen = 1000
• /adv/Net/MaxPortRxQueueLen = 500
• /adv/Net/VmxnetTxCopySize = 1500
• /adv/Net/vmxnetThroughputWeight = 128
• /adv/Misc/TimerMinHardPeriod = 2000
• /sched/freeMemoryState/minFreePct = 2
• For the Db, App, Web, and Infra VMs of even tiles sched.cpu.affinity set to 0-19 and sched.mem.affinity set to numa node 0
• For the Db, App, Web, and Infra VMs of odd tiles sched.cpu.affinity set to 20-39 and sched.mem.affinity set to numa node 1

Web Server VM Notes

• Boot parameters set: divider=10, selinux=0, audit=0 elevator=deadline.
• SELINUX=disabled was set in /etc/selinux/config.
• Both Virtual NICs used VMXNET 3 driver.
• Internal NIC set to use jumbo frames by adding 'MTU=9000' to /etc/sysconfig/network-scripts/ifconfig-eth1
• VM setting CPU shares was set to 'high' via vConsole
• VM setting Memory Reservation was set to '18944' via vConsole
• Following items were added to /etc/sysctl.conf:
• fs.file-max = 500000
• net.core.optmem_max = 1000000
• net.core.rmem_default = 1000000
• net.core.rmem_max = 1000000
• net.core.wmem_default = 1000000
• net.core.wmem_max = 1000000
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.tcp_mem = 10000000 10000000 10000000
• net.ipv4.tcp_rmem = 10000000 10000000 10000000
• net.ipv4.tcp_wmem = 10000000 10000000 10000000
• Add line in httpd.conf to 'Include conf/extra/httpd-default.conf'
• Add line in httpd.conf to 'Include conf/extra/httpd-mpm.conf'
• Change values in extra/httpd-default.conf:
• MaxKeepAliveRequest 0
• Change values in extra/httpd-mpm.conf:
• StartServers 20
• MinSpareServers 20
• MaxSpareServers 20
• ServerLimit 1536
• MaxClients 1536
• MaxRequestsPerChild 4000

Application Server VM Notes

• Boot parameters set: divider=10, selinux=0, audit=0 elevator=deadline.
• SELINUX=disabled was set in /etc/selinux/config.
• Both virtual NICs used VMXNET 3 driver
• Internal NIC set to use jumbo frames by adding 'MTU=9000' to /etc/sysconfig/network-scripts/ifconfig-eth1
• VM setting CPU shares was set to '3600' via vConsole
• VM setting Memory Reservation was set to '4096' via vConsole
• The following lines were added to /etc/sysctl.conf:
• fs.file-max = 500000
• net.core.optmem_max = 1000000
• net.core.rmem_default = 1000000
• net.core.rmem_max = 1000000
• net.core.wmem_default = 1000000
• net.core.wmem_max = 1000000
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.tcp_mem = 10000000 10000000 10000000
• net.ipv4.tcp_rmem = 10000000 10000000 10000000
• net.ipv4.tcp_wmem = 10000000 10000000 10000000
• Huge pages were mounted by using the following lines in /etc/rc.local:
• echo 1536 > /proc/sys/vm/nr_hugepages
• echo 500 > /proc/sys/vm/hugetlb_shm_group
• mkdir /dev/hugepages
• mount -t hugetlbfs hugetlbfs /dev/hugepages
• The following lines were added to /etc/security/limits.conf:
• soft memlock 3072000
• hard memlock 3072000
• The following lines were added to /opt/glassfish/domains/spec2004-1/domain.xml:
• To jdbc-connection-pool line parameter 'max-pool-size = 64' was set
• To request-processing line parameter 'initial-thread-count = 90' was set
• To ejb-container line, parameter 'max-pool-size=64' was set
• To mdb-container line, parameter 'max-pool-size = 64' was set
• jvm-option '-XX:+AggressiveHeap' was set
• jvm-option '-Xmn2048m' was set
• jvm-option '-Xmx2560m' was set
• jvm-option '-Xms2560m' was set
• jvm-option '-XX:PermSize=192m' was set
• jvm-option '-Xss128k' was set
• jvm-option '-XX:+DisableExplicitGC' was set
• jvm-option '-XX:+UseParallelOldGC' was set
• jvm-option '-XX:+UseLargePages' was set
• jvm-option '-XX:LargePageSizeInBytes=2m' was set
• jvm-option '-XX:-UseBiasedLocking' was set
• jvm-option '-XX:CompileThreshold=1000' was set
• jvm-option '-XX:+AggressiveOpts' was set
• jvm-option '-XX:+UseCompressedOops' was set
• jvm-option '-XX:+UseFastAccessorMethods' was set
• jvm-option '-XX:+UseStringCache' was set
• jvm-option '-XX:+OptimizeStringConcat' was set
• The appserver workload application (Tomcat) and configuration files are located in /opt/SUNWjAppServer2004.

Mail Server VM Notes

• Boot parameters set: divider=10, selinux=0, audit=0 elevator=deadline.
• SELINUX=disabled was set in /etc/selinux/config.
• Virtual NIC used VMXNET 3 driver
• VM setting CPU shares was set to '450' via vConsole
• The following lines were added to /etc/sysctl.conf:
• fs.file-max = 500000
• net.core.optmem_max = 1000000
• net.core.rmem_default = 1000000
• net.core.rmem_max = 1000000
• net.core.wmem_default = 1000000
• net.core.wmem_max = 1000000
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.tcp_mem = 10000000 10000000 10000000
• net.ipv4.tcp_rmem = 10000000 10000000 10000000
• net.ipv4.tcp_wmem = 10000000 10000000 10000000
• dovecot was upgraded to 2.0.4-1_119.el5.x84_64. The dbox format was used.
• Contents of dovecot.conf file:
• base_dir = /var/run/dovecot/
• disable_plaintext_auth = no
• listen = *, ::
• log_path = /var/log/dovecot
• mail_debug = no
• mail_location = dbox:/mailstore/dovecot/maildir/%u
• mdbox_rotate_size = 2048
• passdb {
• args = /etc/passwd.dovecot
• driver = passwd-file
• }
• protocols = imap
• service auth {
• user = root
• }
• service imap-login {
• inet_listener imap {
• port = 143
• }
• process_min_avail = 10
• }
• ssl = no
• userdb {
• driver = passwd
• }
• protocol lda {
• postmaster_address = postmaster@example.com
• mail_fsync = optimized
• }

Database Server VM Notes

• Boot parameters set: divider=10, selinux=0, audit=0 elevator=deadline.
• SELINUX=disabled was set in /etc/selinux/config.
• Both virtual NICs used VMXNET 3 driver
• Internal NIC set to use jumbo frames by adding 'MTU=9000' to /etc/sysconfig/network-scripts/ifconfig-eth1
• VM setting CPU shares was set to '2400' via vConsole
• VM setting Memory Reservation was set to '512' via vConsole
• The following lines were added to /etc/sysctl.conf:
• fs.file-max = 500000
• net.core.optmem_max = 1000000
• net.core.rmem_default = 1000000
• net.core.rmem_max = 1000000
• net.core.wmem_default = 1000000
• net.core.wmem_max = 1000000
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.tcp_mem = 10000000 10000000 10000000
• net.ipv4.tcp_rmem = 10000000 10000000 10000000
• net.ipv4.tcp_wmem = 10000000 10000000 10000000
• Database tuning options in /var/lib/pgsql/9.0/data/postgresql.conf
• wal_buffers = 20MB
• checkpoint_completion_target = 0.9
• fsync = off
• synchronous_commit = off
• wal_sync_method = fdatasync
• effective_cache_size = 1200MB
• shared_buffers = 1000MB
• work_mem = 64MB
• maintenance_work_mem = 1MB
• max_connections = 250
• ssl = off
• full_page_writes = off
• debug_assertions = false
• log_min_duration_statement = 1000
• default_statistics_target = 100
• log_rotation_size = 16MB
• log_destination = syslog

Infraserver VM Notes

• Boot parameters set: divider=10, selinux=0, audit=0 elevator=deadline.
• SELINUX=disabled was set in /etc/selinux/config.
• Both Virtual NICs used VMXNET 3 driver
• Internal NIC set use jumbo frames by adding 'MTU=9000' to /etc/sysconfig/network-scripts/ifconfig-eth1
• VM setting Memory Reservation was set to '512' via vConsole
• The following lines were added to sysctl.conf:
• fs.file-max = 500000
• net.core.optmem_max = 1000000
• net.core.rmem_default = 1000000
• net.core.rmem_max = 1000000
• net.core.wmem_default = 1000000
• net.core.wmem_max = 1000000
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.tcp_mem = 10000000 10000000 10000000
• net.ipv4.tcp_rmem = 10000000 10000000 10000000
• net.ipv4.tcp_wmem = 10000000 10000000 10000000

Idle Server VM Notes

• Boot parameters set: divider=10, selinux=0, audit=0 elevator=deadline.
• SELINUX=disabled was set in /etc/selinux/config.
• Virtual NIC used VMXNET 3 driver
• VM setting CPU shares was set to '700' via vConsole
• VM setting Memory shares was set to 'low' via vConsole
• VM setting Storage shares was set to 'low' via vConsole
• The following lines were added to sysctl.conf:
• fs.file-max = 500000
• net.core.optmem_max = 1000000
• net.core.rmem_default = 1000000
• net.core.rmem_max = 1000000
• net.core.wmem_default = 1000000
• net.core.wmem_max = 1000000
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.tcp_mem = 10000000 10000000 10000000
• net.ipv4.tcp_rmem = 10000000 10000000 10000000
• net.ipv4.tcp_wmem = 10000000 10000000 10000000

Client Driver Notes

• Each physical client runs 3 virtual clients. Each virtual client uses 1 x vCPU, 2 GB memory, and
• 1 x 1Gb NIC port (one of the NC360m NICs is not used). The first virtual client on the first
• physical client is dedicated as the prime controller. One virtual client was used for each tile.

Other Notes

• A single ProCurve 2910al 48-port switch was used to connect clients to SUT.
• The 10GbE ports were attached to a dual port 10 GbE SFP+ expansion module (J9008A)
• 3 of the 4 of the 10Gb NIC ports were used for networking for the benchmarking. The
• fourth port used to access the vConsole for administration.
• 1 x StorageWorks 8/24 SAN Fibre switch was used to connect the SUT to the external storage.