Performance Summary:

Power Summary:

Power Details:

Validation Errors:

•   No Validation Errors Found

Power Measurement Configuration:

Physical Configuration:

Virtual Configuration:

Notes:

Physical System Notes

• Firmware version PSE121H-1.50 [Lenovo internal version] (Public equivalent: PSE122H-1.50)
• Firmware settings used:
• Operating Mode: Maximum Performance
• Processor Uncore Frequency Scaling: Disable
• Memory Patrol Scrub: Disable
• The test sponsor attests, as of date of publication, that CVE-2017-5754 (Meltdown) is mitigated in the system as tested and documented:
• Yes: hosts, guests
• The test sponsor attests, as of date of publication, that CVE-2017-5753 (Spectre variant 1) is mitigated in the system as tested and documented:
• Yes: hosts, guests
• The test sponsor attests, as of date of publication, that CVE-2017-5715 (Spectre variant 2) is mitigated in the system as tested and documented:
• Yes: hosts, guests
• The test sponsor attests, as of date of publication, that CVE-2018-3620 and CVE-2018-3646 (Foreshadow) are mitigated in the system as tested and documented:
• Yes: hosts, guests

Storage Notes

• The System Under Test has 24 800GB internal SSDs configured as two RAID5 arrays with twelve disks each.
• One of the arrays contains the logical volume for the hypervisor OS that was booted.
• Each RAID disk has a large partition that was made into a volume group. The disks for the VMs were each allocated as a logical volume from one of the volume groups.

Virtualization Software Notes

• The following kernel boot options were used:
• audit=0 elevator=noop hpet=disable clocksource=tsc tsc=reliable highres=off cgroup_disable=cpu,cpuacct,cpuset,memory,blkio,hugetlb intel_iommu=on iommu=pt pci=realloc intel_idle.max_cstate=0 processor.max_cstate=1 mce=ignore_ce nowatchdog ipv6.disable=1 default_hugepagesz=1G megaraid_sas.smp_affinity_enable=0
• The following services were disabled:
• cron, kdump-early, kdump, irqbalance, postfix, smartd, SuSEfirewall2, SuSEfirewall2_init
• The following items were added in /usr/lib/sysctl.d/99-specvirt.conf:
• fs.aio-max-nr = 1572864
• kernel.perf_event_max_sample_rate = 250
• kernel.sched_min_granularity_ns = 30000000
• kernel.sched_wakeup_granularity_ns = 30000000
• kernel.sched_latency_ns = 100000
• kernel.sched_migration_cost_ns = 13000
• kernel.sched_rt_runtime_us = 960000
• kernel.numa_balancing = 0
• vm.dirty_background_ratio = 5
• vm.dirty_ratio = 10
• vm.swappiness = 0
• Configuration file /etc/modprobe.d/i40e.conf includes:
• blacklist i40evf
• options i40e max_vfs=0,0,64,64,64,64,64,64,64,64,64,64,64,64,64,64,64,64
• Network configuration was as follows:
• Ethernet devices eth4, eth5, eth6, eth7, eth8, eth9, eth10, eth11, eth12, eth13, eth14, eth15, eth16, eth17, eth18, and eth19 were the physical functions that were used by the host to provide physical transport for the virtual functions.
• Kernel module i40e does not have a max_vfs option and ignores it. Instead, a systemd initialization script reads the max_vfs option from /etc/modprobe.d/i40e.conf and allocates the virtual functions.
• 672 virtual functions were used for all the VMs.
• All VMs except those on NUMA node 3 were assigned a VF that is routed to the NUMA node to which the VM's VCPUs were pinned. VMs on NUMA node 3 were assigned a VF that is routed to NUMA node 7.
• After the system booted, a script was run that allocated 728 1GB huge pages on each of the NUMA nodes.
• After the system booted, a kernel scheduler option was set manually with the command: echo NO_WAKEUP_PREEMPTION >/sys/kernel/debug/sched_features
• VM configurations were as follows:
• All VMs were managed with libvirt and created using the virsh utility.
• All VMs were created with the following XML template:
  • <domain type='kvm'>
  •   <name>$vm_name</name>
  •   <memory unit='MiB'>$vm_memory</memory>
  •   <vcpu cpuset='$host-cpu-list'>$vm_vcpus</vcpu>
  •   <os>
  •     <type arch='x86_64' machine='pc'>hvm</type>
  •     <boot dev='hd'/>
  •   </os>
  •   <features>
  •     <acpi/>
  •   </features>
  •   <cpu mode='host-passthrough'>
  •     <cache mode='passthrough'/>
  •     <topology sockets='1' cores='$vm_vcpus' threads='1'/>
  •   </cpu>
  •   <clock offset='utc'>
  •     <timer name='pit' tickpolicy='delay'/>
  •     <timer name='rtc' tickpolicy='catchup'/>
  •     <timer name='hpet' present='no'/>
  •   </clock>
  •   <on_poweroff>destroy</on_poweroff>
  •   <on_reboot>restart</on_reboot>
  •   <on_crash>restart</on_crash>
  •   <devices>
  •     <emulator>/usr/bin/qemu-kvm</emulator>
  •     <disk type='block' device='disk'>
  •       <driver name='qemu' type='raw' cache='none' io='native'/>
  •       <source dev='$vm_disk'/>
  •       <target dev='vda' bus='virtio'/>
  •     </disk>
  •     <interface type='hostdev' managed='yes'>
  •       <mac address='$vm_mac'/>
  •       <source>
  •         <address type='pci' domain='0x0000' bus='$bus' slot='$slot' function='$function'/>
  •       </source>
  •     </interface>
  •     <serial type='pty'>
  •       <target port='0'/>
  •     </serial>
  •     <console type='pty'>
  •       <target type='serial' port='0'/>
  •     </console>
  •     <input type='mouse' bus='ps2'/>
  •     <graphics type='vnc' port='-1' autoport='yes' keymap='en-us'/>
  •     <memballoon model='none'/>
  •   </devices>
  • </domain>
• App, Db, Web, and Batch VMs set unit='GiB' in the <memory> tag and have these additional lines of XML:
  •   <memoryBacking>
  •     <hugepages>
  •       <page size='1' unit='G'/>
  •     </hugepages>
  •   </memoryBacking>
• Db and Mail VMs have these additional lines of XML for their data disk:
  •   <disk type='block' device='disk'>
  •     <driver name='qemu' type='raw' cache='none' io='native'/>
  •     <source dev='$vm_data_disk'/>
  •     <target dev='vdb' bus='virtio'/>
  •   </disk>
• VMs were bound to host CPUs via their XML configuration in the following manner:
  • All VMs of tiles 008, 015, 022, 029, 036, 043, 050, 057, 072, 079, 086, 093, 100, 107, 114, and 121 were bound to NUMA node 0.
  • All VMs of tiles 001, 016, 023, 030, 037, 044, 051, 058, 065, 080, 087, 094, 101, 108, 115, and 122 were bound to NUMA node 1.
  • All VMs of tiles 002, 009, 024, 031, 038, 045, 052, 059, 066, 073, 088, 095, 102, 109, 116, and 123 were bound to NUMA node 2.
  • All VMs of tiles 003, 010, 017, 032, 039, 046, 053, 060, 067, 074, 081, 096, 103, 110, 117, and 124 were bound to NUMA node 3.
  • All VMs of tiles 004, 011, 018, 025, 040, 047, 054, 061, 068, 075, 082, 089, 104, 111, 118, and 125 were bound to NUMA node 4.
  • All VMs of tiles 005, 012, 019, 026, 033, 048, 055, 062, 069, 076, 083, 090, 097, 112, 119, and 126 were bound to NUMA node 5.
  • All VMs of tiles 006, 013, 020, 027, 034, 041, 056, 063, 070, 077, 084, 091, 098, 105, 120, and 127 were bound to NUMA node 6.
  • All VMs of tiles 007, 014, 021, 028, 035, 042, 049, 064, 071, 078, 085, 092, 099, 106, 113, and 128 were bound to NUMA node 7.
• App VMs were reniced 0.
• Db VMs were reniced -4.
• Web VMs were reniced -3.
• Infra VMs were reniced 19.
• Mail VMs were reniced 19.
• Batch VMs were reniced -6.
• After the VMs were started, the command /usr/sbin/irqbalance -o was run to balance the interrupts across the NUMA nodes.

Web Server VM Notes

• The following kernel boot options were used:
• selinux=0 audit=0 elevator=deadline clocksource=tsc tsc=reliable highres=off ipv6.disable=1 nowatchdog intel_idle.max_cstate=0 processor.max_cstate=1 cgroup_disable=memory,cpu,blkio spec_store_bypass_disable=off
• The following services were disabled:
• auditd crond ip6tables iptables kdump messagebus netconsole nfs nginx php-fpm postfix rdisc restorecond rhnsd rhsmcertd rngd rpcsvcgssd saslauthd
• root file system mount options: noatime,barrier=0
• Created /sbin/ifup-local to run the commands:
• ethtool -G eth0 rx 4096 tx 4096
• ethtool -C eth0 adaptive-rx off adaptive-tx off rx-usecs 2500
• The tuned package was installed and activated with the virtual-guest profile.
• The following items were added in /etc/sysctl.conf:
• fs.file-max = 500000
• kernel.sched_features = 7275
• net.core.netdev_budget = 3000
• net.core.netdev_max_backlog = 30000
• net.core.somaxconn = 81920
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.ipv4.tcp_fin_timeout = 20
• net.ipv4.tcp_keepalive_intvl = 30
• net.ipv4.tcp_keepalive_probes = 4
• net.ipv4.tcp_keepalive_time = 120
• net.ipv4.tcp_max_syn_backlog = 20000
• net.ipv4.tcp_max_tw_buckets = 1500000
• net.ipv4.tcp_timestamps = 0
• net.ipv4.tcp_tw_recycle = 1
• net.ipv4.tcp_tw_reuse = 1
• vm.swappiness = 0
• The following lines were added to /etc/security/limits.conf:
• * soft memlock unlimited
• * hard memlock unlimited
• The following commands were run before the benchmark was started:
• ulimit -n 1048576
• ulimit -u 1000000
• The php-fpm package was installed. /etc/php-fpm.d/www.conf changes:
• user = nginx
• group = nginx
• pm.max_children = 700
• pm.start_servers = 100
• pm.min_spare_servers = 100
• pm.max_spare_servers = 200
• /etc/nginx/nginx.conf contains:
• user                 nginx;
• worker_processes     4;
• worker_cpu_affinity  0001 0001 0010 0010;
• worker_rlimit_nofile 65535;
• pid         /var/run/nginx.pid;
• events {
•   use                epoll;
•   worker_connections 65535;
•   multi_accept       on;
•   accept_mutex       off;
• }
• http {
•   include            /etc/nginx/mime.types;
•   default_type       application/octet-stream;
•   log_format common '$remote_addr - $remote_user [$time_local] $request'
•                     '"$status" $body_bytes_sent ';
•   access_log        /var/log/nginx/access.log.gz common gzip buffer=4m flush=1m;
•   sendfile           on;
•   sendfile_max_chunk 1m;
•   tcp_nopush         on;
•   tcp_nodelay        on;
•   keepalive_timeout  120;
•   keepalive_requests 10000;
•   open_file_cache          max=204800 inactive=20s;
•   open_file_cache_min_uses 1;
•   open_file_cache_valid    30s;
•   open_file_cache_errors   on;
•   read_ahead               4096;
•   charset utf-8;
•   server_names_hash_bucket_size 128;
•   client_header_buffer_size     2k;
•   large_client_header_buffers   4 4k;
•   include /etc/nginx/conf.d/*.conf;
• }
• /etc/nginx/conf.d/default.conf contains:
• upstream fastcgi_backend {
•   server 127.0.0.1:9000;
•   keepalive 128;
• }
• server {
•   listen 80 backlog=81920;
•   server_name  localhost;
•   location / {
•     root   /var/www/html;
•     index  index.html index.htm index.php;
•   }
•   error_page   500 502 503 504  /50x.html;
•   location = /50x.html {
•     root   /usr/share/nginx/html;
•   }
•   location = /error.log {
•     root   /var/log/nginx;
•   }
•   location ~ .*\.(html|htm|gif|jpg|jpeg|bmp|png|ico|txt|js|css)$ {
•     root     /var/www/html;
•     expires  100d;
•   }
•   location ~ ^/images/ {
•     root     /var/www/html/support;
•     expires  100d;
•   }
•   location ~ \.php$ {
•     root           html;
•     fastcgi_pass   fastcgi_backend;
•     fastcgi_keep_conn on;
•     fastcgi_index  index.php;
•     fastcgi_param  SCRIPT_FILENAME /var/www/html$fastcgi_script_name;
•     include        fastcgi_params;
•   }
• }
• /etc/nginx/conf.d/ssl.conf contains:
• server {
•   listen       443 ssl backlog=81920;
•   ssl_certificate      /etc/pki/tls/certs/localhost.crt;
•   ssl_certificate_key  /etc/pki/tls/private/localhost.key;
•   ssl_session_cache    shared:SSL:1m;
•   ssl_session_timeout  5m;
•   ssl_protocols SSLv3 TLSv1 TLSv1.1 TLSv1.2;
•   ssl_ciphers   ALL:!ADH:!EXPORT:!SSLv2:RC4+RSA:+HIGH:+MEDIUM:+LOW;
•   ssl_prefer_server_ciphers   on;
•   location / {
•     root   /var/www/html;
•     index  index.html index.htm index.php;
•   }
•   location ~ .*\.(html|htm|gif|jpg|jpeg|bmp|png|ico|txt|js|css)$ {
•     root     /var/www/html;
•     expires  100d;
•   }
•   location ~ ^/images/ {
•     root    /var/www/html/support;
•     expires 100d;
•   }
•   location ~ \.php$ {
•     root           html;
•     fastcgi_pass   fastcgi_backend;
•     fastcgi_keep_conn on;
•     fastcgi_index  index.php;
•     fastcgi_param  SCRIPT_FILENAME /var/www/html$fastcgi_script_name;
•     include        fastcgi_params;
•   }
• }
• Web VM used NFS to access files from infra VM.
• After the Infra VM was booted and its NFS service was up, the Web VM mounted the NFS file system via test automation software.
• The datastore was mounted on /var/www/html/support/downloads
• NFS mount options used: noatime,actimeo=14400,nocto

Application Server VM Notes

• The following kernel boot options were used:
• selinux=0 audit=0 elevator=deadline clocksource=tsc tsc=reliable highres=off ipv6.disable=1 nowatchdog intel_idle.max_cstate=0 processor.max_cstate=1 cgroup_disable=memory,cpu,blkio spec_store_bypass_disable=off
• The following services were disabled:
• auditd crond ip6tables iptables kdump messagebus netconsole postfix rdisc restorecond rhnsd rhsmcertd rngd saslauthd svnserve
• root file system mount options: noatime,barrier=0
• Created /sbin/ifup-local to run the commands:
• ethtool -G $1 rx 4096 tx 4096
• ethtool -C eth0 adaptive-rx off adaptive-tx off rx-usecs 200
• The tuned package was installed and activated with the virtual-guest profile.
• The following items were added in /etc/sysctl.conf:
• fs.file-max = 500000
• kernel.sched_features = 7275
• net.core.somaxconn = 8192
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.ipv4.tcp_max_tw_buckets = 500000
• vm.swappiness = 0
• The following lines were added to /etc/security/limits.conf:
• * soft memlock unlimited
• * hard memlock unlimited
• The following commands were run before the benchmark was started:
• ulimit -n 1048576
• ulimit -u 1000000
• There were two WebSphere profiles, AppSrv01 for SPECjAppServer2004 and AppSrv02 for SPEC Emulator.
• Heap size for AppSrv01 was changed to 2880 MB.

Mail Server VM Notes

• The following kernel boot options were used:
• selinux=0 audit=0 elevator=deadline clocksource=tsc tsc=reliable highres=off ipv6.disable=1 nowatchdog intel_idle.max_cstate=0 processor.max_cstate=1 cgroup_disable=memory,cpu,blkio spec_store_bypass_disable=off
• The following services were disabled:
• auditd crond ip6tables iptables kdump mdmonitor messagebus netconsole postfix rdisc restorecond rhnsd rhsmcertd rngd saslauthd
• root file system mount options: noatime,barrier=0
• Mail store file system mount options: noatime,barrier=0,data=writeback
• Created /sbin/ifup-local to run the command:
• ethtool -C eth0 adaptive-rx off adaptive-tx off rx-usecs 1000
• The tuned package was installed and activated with the virtual-guest profile.
• The following items were added in /etc/sysctl.conf:
• fs.file-max = 500000
• kernel.sched_features = 7275
• net.core.somaxconn = 8192
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.ipv4.tcp_max_tw_buckets = 500000
• vm.swappiness = 0
• The following lines were added to /etc/security/limits.conf:
• * soft memlock unlimited
• * hard memlock unlimited
• The following commands were run before the benchmark was started:
• ulimit -n 1048576
• ulimit -u 1000000
• Dovecot source code was configured with the command:
• ./configure --with-passwd-file --with-passwd --without-shadow --without-pam --without-pop3d --prefix=$dovecotdir
• Dovecot source code was compiled with default options.
• dovecot.conf contains:
• base_dir = /var/run/dovecot/
• ssl = no
• ssl_cert =
• ssl_key =
• disable_plaintext_auth = no
• listen = *
• log_path = /var/log/dovecot.log
• mail_debug = no
• mail_location = dbox:/mailstore/dovecot/maildir/%u
• mdbox_rotate_size = 8192
• maildir_very_dirty_syncs = yes
• mbox_very_dirty_syncs = yes
• mbox_write_locks = fcntl
• mmap_disable = yes
• mail_fsync = never
• mailbox_list_index = yes
• mailbox_list_index_very_dirty_syncs=yes
• auth_cache_size = 5000
• passdb {
•   args = scheme=plain /etc/passwd.dovecot
•   driver = passwd-file
• }
• protocols = imap
• service auth {
•   user = root
• }
• service imap-login {
•   inet_listener imap {
•     port = 143
•   }
•   process_min_avail = 10
•   service_count = 0
• }
• service imap {
•   process_min_avail = 10
• }
• userdb {
•   driver = passwd
• }
• dict {
• }

Database Server VM Notes

• The following kernel boot options were used:
• selinux=0 audit=0 elevator=deadline clocksource=tsc tsc=reliable highres=off ipv6.disable=1 nowatchdog intel_idle.max_cstate=0 processor.max_cstate=1 cgroup_disable=memory,cpu,blkio spec_store_bypass_disable=off
• The following services were disabled:
• auditd crond ip6tables iptables kdump messagebus netconsole postfix rdisc restorecond rhnsd rhsmcertd rngd saslauthd
• root file system mount options: noatime,barrier=0
• DB storage file system mount options: noatime,barrier=0,data=writeback
• Created /sbin/ifup-local to run the commands:
• ethtool -G eth0 rx 4096 tx 4096
• ethtool -C eth0 adaptive-rx off adaptive-tx off rx-usecs 250
• The tuned package was installed and activated with the virtual-guest profile.
• The following items were added in /etc/sysctl.conf:
• fs.file-max = 500000
• kernel.randomize_va_space = 0
• kernel.sched_features = 7275
• net.core.somaxconn = 32384
• net.ipv4.tcp_max_tw_buckets = 1000000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• vm.swappiness = 0
• The following lines were added to /etc/security/limits.conf:
• * soft memlock unlimited
• * hard memlock unlimited
• The following commands were run before the benchmark was started:
• ulimit -n 1048576
• ulimit -u 1000000
• 13312 2MB huge pages were allocated prior to starting the database instance.
• Database tuning options:
• db2 update db cfg for specdb using DATABASE_MEMORY 6553600
• db2set DB2_LARGE_PAGE_MEM=DB
• db2set DB2_PINNED_BP=ON
• db2 alter bufferpool IBMDEFAULTBP size 5898240
• db2 alter tablespace roll_space NO FILE SYSTEM CACHING
• db2 alter tablespace C_cust_space NO FILE SYSTEM CACHING
• db2 alter tablespace C_supp_space NO FILE SYSTEM CACHING
• db2 alter tablespace C_site_space NO FILE SYSTEM CACHING
• db2 alter tablespace C_parts_space NO FILE SYSTEM CACHING
• db2 alter tablespace C_custinv_space NO FILE SYSTEM CACHING
• db2 alter tablespace M_parts_space NO FILE SYSTEM CACHING
• db2 alter tablespace M_bom_space NO FILE SYSTEM CACHING
• db2 alter tablespace M_wo_space NO FILE SYSTEM CACHING
• db2 alter tablespace M_lo_space NO FILE SYSTEM CACHING
• db2 alter tablespace M_inv_space NO FILE SYSTEM CACHING
• db2 alter tablespace O_cust_space NO FILE SYSTEM CACHING
• db2 alter tablespace O_ords_space NO FILE SYSTEM CACHING
• db2 alter tablespace O_ordl_space NO FILE SYSTEM CACHING
• db2 alter tablespace O_item_space NO FILE SYSTEM CACHING
• db2 alter tablespace S_comp_space NO FILE SYSTEM CACHING
• db2 alter tablespace S_sc_space NO FILE SYSTEM CACHING
• db2 alter tablespace S_po_space NO FILE SYSTEM CACHING
• db2 alter tablespace S_site_space NO FILE SYSTEM CACHING
• db2 alter tablespace S_supp_space NO FILE SYSTEM CACHING
• db2 alter tablespace U_space NO FILE SYSTEM CACHING
• db2 update database config for specdb using LOGFILSIZ 65536
• db2 update database config for specdb using logprimary 64 logsecond 4

Infraserver VM Notes

• The following kernel boot options were used:
• selinux=0 audit=0 elevator=deadline clocksource=tsc tsc=reliable highres=off ipv6.disable=1 watchdog=0 intel_idle.max_cstate=1 processor.max_cstate=1 cgroup_disable=memory,cpu,blkio spec_store_bypass_disable=off
• The following services were disabled:
• auditd crond htcacheclean httpd ip6tables iptables kdump mdmonitor messagebus netconsole nfs postfix rdisc restorecond rhnsd rhsmcertd rngd rpcsvcgssd saslauthd
• root file system mount options: noatime,barrier=0,data=writeback
• Created /sbin/ifup-local to run the command:
• ethtool -C eth0 adaptive-rx off adaptive-tx off rx-usecs 333
• The tuned package was installed and activated with the virtual-guest profile.
• The following items were added in /etc/sysctl.conf:
• fs.file-max = 500000
• kernel.sched_features = 7275
• net.core.somaxconn = 81920
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.ipv4.tcp_max_tw_buckets = 1500000
• net.ipv4.tcp_tw_reuse = 1
• net.ipv4.tcp_keepalive_time = 120
• net.ipv4.tcp_keepalive_intvl = 30
• net.ipv4.tcp_timestamps = 0
• net.ipv4.tcp_syn_retries = 4
• net.ipv4.tcp_retries2 = 7
• vm.swappiness = 0
• The following lines were added to /etc/security/limits.conf:
• * soft memlock unlimited
• * hard memlock unlimited
• The following commands were run before the benchmark was started:
• ulimit -n 1048576
• ulimit -u 1000000
• httpd.conf changes:
• ServerLimit 10240
• MaxClients 10240
• Listen 81
• Include conf/fcgi.conf
• fcgi package was built from Fedora 14 source RPM, which includes patch to compile with GCC 4.4.
• Created /etc/httpd/conf/fcgi.conf that contains:
• LoadModule fastcgi_module modules/mod_fastcgi.so
• ScriptAlias /fcgi-bin/ "/var/www/fcgi-bin/"
• FastCgiIpcDir /etc/httpd/fastcgi
• <Directory "/var/www/fcgi-bin">
•   AllowOverride None
•   Options +ExecCGI -Includes
•   SetHandler fastcgi-script
•   Order allow,deny
•   Allow from all
• </Directory>
• AddHandler fastcgi-script fcgi
• FastCgiServer /var/www/fcgi-bin/besim_fcgi.fcgi -processes 50
• Standard Linux NFS server was enabled to serve files to the Web VM.
• /etc/exports file contains:
• /support-download *(rw,async,no_subtree_check,crossmnt,no_root_squash)

Batch Server VM Notes

• The following kernel boot options were used:
• selinux=0 audit=0 elevator=deadline clocksource=tsc tsc=reliable highres=off ipv6.disable=1 nowatchdog intel_idle.max_cstate=0 processor.max_cstate=1 cgroup_disable=memory,cpu,blkio spec_store_bypass_disable=off
• The following services were disabled:
• auditd crond ip6tables iptables kdump mdmonitor messagebus netconsole postfix rdisc restorecond rhnsd rhsmcertd rngd saslauthd
• root file system mount options: noatime,barrier=0
• The tuned package was installed and activated with the virtual-guest profile.
• The following items were added in /etc/sysctl.conf:
• fs.file-max = 500000
• kernel.sched_features = 7275
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• vm.swappiness = 0
• The following lines were added to /etc/security/limits.conf:
• * soft memlock unlimited
• * hard memlock unlimited
• The following commands were run before the benchmark was started:
• ulimit -n 1048576
• ulimit -u 1000000

Client Driver Notes

• Two client systems each host ten prime client VMs. Eleven client systems each host nine prime client VMs. One client system hosts nine prime client VMs plus the prime controller VM.
• Each prime client VM has 8 VCPUs, 10 GB memory, and a 16 GB disk. 1 ixgbevf and 1 virtio_blk device were used in all client VMs.
• All prime client VMs run Red Hat Enterprise Linux 6.9.

Power Configuration Notes

• The SUT has four power supplies.
• The two power meters were each connected to two of the SUT's power supplies via a Y power cable.
• There was no power meter for external storage because there was no external storage.

Other Notes

• An external 10Gb Lenovo RackSwitch G8264 switch was used to connect the client systems to the system under test (SUT).
• The SUT used the eight dual-port 10Gb PCIe NICs for communication with the VMs.
• The SUT NICs and the external client systems were connected to the switch using 10Gb SFP+ DAC cables.
• An external 1Gb Lenovo RackSwitch G8052 switch was used to manage all the systems.
• All systems were connected to the 1Gb switch via a separate cable plugged into a 1Gb port on the system.

Performance Details: