Performance Summary:

Validation Errors:

•   No Validation Errors Found

Physical Configuration:

Virtual Configuration:

Notes:

Physical System Notes

• BIOS version: 2.00 (06/27/2019)
• BIOS settings used:
• Workload Profile: Virtualization - Max Performance
• Thermal Configuration : Maximum Cooling
• Performance Determinism: Power Deterministic
• NUMA memory domains per socket: Four memory domains per socket
• Last-Level Cache (LLC) as NUMA: Enabled
• Memory Patrol Scrubbing: Disable
• The test sponsor attests, as of date of publication, that CVE-2017-5754 (Meltdown) is mitigated in the system as tested and documented:
• Yes: hosts, guests
• The test sponsor attests, as of date of publication, that CVE-2017-5753 (Spectre variant 1) is mitigated in the system as tested and documented:
• Yes: hosts, guests
• The test sponsor attests, as of date of publication, that CVE-2017-5715 (Spectre variant 2) is mitigated in the system as tested and documented:
• Yes: hosts, guests
• The test sponsor attests, as of date of publication, that CVE-2018-3620 and CVE-2018-3646 (Foreshadow) are mitigated in the system as tested and documented:
• Yes: hosts, guests

Storage Notes

• 8 x 1.6TB internal SAS drives were connected to the P408i-a and P408i-p SR Controllers
• Created 2 x RAID 5 array each containing 4 x SSD drives, on P408i-a and P408i-p
• Created 28 x logical disks on each array
• 2 x 800GB internal SAS drives were connected to the P408i-a SR Controller.
• Created 1 x RAID 1 array with 1 x logical disk for OS.

Virtualization Software Notes

• Boot parameters set: splash=silent quiet showopts crashkernel=154M,high crashkernel=72M,low audit=0 elevator=deadline hpet=disable clocksource=tsc tsc=reliable nohz=on highres=off cgroup_disable=cpu,cpuacct,memory,blkio,hugetlb amd_iommu=on pci=realloc pci=pcie_bus_safe nosoftlockup ipv6.disable=1 mce=ignore_ce nmi_watchdog=0 kvm-amd.avic=1
• The following services were disabled:
• auditd, autoyast-initscripts, blk-availability, ca-certificate, chorny-wait, chronyd, console-getty, cron, cups, debug-shell, dmraid-activation, dnsmasq, ebtables, grub2-once, irqbalance iscsid, issue-add-ssh-keys, kdump-early, kdump, kexec-load, libvirt-guests, lvm2-monitor, multipathd, nfs-blkmap, nfs-server, nfs, nfsserver, numad, postfix, radvd, raw, rpcbind, rpmconfigcheck, serial-getty@, sysstat, systemd-timesyncd, udisks2, wickedd-auto4, wickedd-dhcp4, wickedd-dhcp6, wickedd-nanny
• /usr/lib/sysctl.d/specvirt.conf was updated with the following lines:
• fs.aio-max-nr = 1048576
• kernel.sched_min_granularity_ns = 20000000
• kernel.sched_wakeup_granularity_ns = 25000000
• kernel.sched_latency_ns = 100000
• kernel.sched_migration_cost_ns = 50000
• /etc/sysctl.conf was updated with the following lines:
• vm.min_free_kbytes = 4096
• vm.dirty_background_ratio = 5
• vm.dirty_ratio = 10
• vm.swappiness = 0
• kernel.numa_balancing = 0
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.core.netdev_max_backlog = 500000
• /etc/modprobe.d/kvm-amd.conf was updated with the following line:
• options kvm-amd npt=1
• Network configuration is as follows:
• One 331i Adapter NIC port(eno1) was used for host access
• The 8 x 562FLR-SFP ports were assigned to bridges (eth4,eth5,eth6,eth7,eth8,eth9,eth10,eth11)
• 45 virtual functions were configured for each port
• Ethernet devices eth4,eth5,eth6,eth7,eth8,eth9,eth10 and eth11 are the physical functions that are used by the host to provide physical transport for the virtual functions.
• Add scripts for i40evf settings in /etc/profile.d/sriov.sh as follows,
• echo 48 > /sys/bus/pci/devices/0000:43:00.0/sriov_numvfs
• echo 48 > /sys/bus/pci/devices/0000:43:00.1/sriov_numvfs
• echo 48 > /sys/bus/pci/devices/0000:63:00.0/sriov_numvfs
• echo 48 > /sys/bus/pci/devices/0000:63:00.1/sriov_numvfs
• echo 48 > /sys/bus/pci/devices/0000:a3:00.0/sriov_numvfs
• echo 48 > /sys/bus/pci/devices/0000:a3:00.1/sriov_numvfs
• echo 48 > /sys/bus/pci/devices/0000:e3:00.0/sriov_numvfs
• echo 48 > /sys/bus/pci/devices/0000:e3:00.1/sriov_numvfs
• All the guest VMs use Virtual Functions
• Each VM is given a virtual function that is routed to the NUMA node to which the VM's VCPUs are pinned.
• VM configurations were as follows:
• All VMs were managed with libvirt
• All VMs were created with the following XML template:
• <domain type='kvm'>
• <name>$vm_name;/name>
• <memory>$vm_memory</memory>
• <vcpu placement='static' current=$host_cpu_list</vcpu>
• <os>
• <numatune'>
• <memory mode ='preferred' nodeset=$host_node_list/>
• </numatune>
• <type arch='x86_64' machine='pc-i440fx-2.9'>hvm</type>
• <boot dev='hd'/>
• </os>
• <features>
• <acpi/>
• <acic/>
• <pae/>
• </features>
• <cpu mode='host-passthrough' check='none'>
• <topology sockets='1' cores=$ threads='1'/>
• </cpu>
• <clock offset='utc'>
• <timer name='pit' tickpolicy='delay'/>
• </clock>
• <on_poweroff>destroy</on_poweroff>
• <on_reboot>restart</on_reboot>
• <on_crash>destroy</on_crash>
• <devices>
• <emulator>/usr/bin/qemu-kvm</emulator>
• <disk type='block' device='disk'>
• <driver name='qemu' type='raw' cache='none' io='native'/>
• <source dev='$vdisk'/>
• <target dev='vda' bus='virtio'/>
• <address type='pci' domain='0x0000' bus='$bus' slot='$slot' function='$function'/>
• </disk>
• <controller type='usb' index='0' model='piix3-uhci'>
• <address type='pci' domain='0x0000' bus='$bus' slot='$slot' function='$function'/>
• </controller>
• <controller type='pci' index='0' model='pci-root'/>
• <interface type='hostdev' managed='yes'>
• <mac address='$vmac'/>
• <source>
• <address type='pci' domain='0x0000' bus='$bus' slot='$slot' function='$function'/>
• </source>
• <address type='pci' domain='0x0000' bus='$bus' slot='$slot' function='$function'/>
• </interface>
• <input type='mouse' bus='ps2'/>
• <input type='keyboard' bus='ps2'/>
• <graphics type='vnc' port='-1' autoport='yes'>
• <listen type='address'/>
• </graphics>
• <sound model='ich6'>
• <address type='pci' domain='0x0000' bus='$bus' slot='$slot' function='$function'/>
• </sound>
• <video>
• <model type='cirrus' vram='16384' heads='1' primary='yes'/>
• <address type='pci' domain='0x0000' bus='$bus' slot='$slot' function='$function'/>
• </video>
• <memballoon model='none'/>
• </devices>
• </domain>
• VMs' nice value was set after startup via the renice command to the following settings:
• appservers = -19
• dbserver = -19
• webserver = -19
• infraserver = 8
• mailserver = 10
• batchserver = 19
• mailserver6 VM was bound to NUMA node 0
• All VMs of tile 1 and mailserver2 were bound to NUMA node 1
• All VMs of tiles 2, 3, and 4, except mailserver2, were bound to NUMA node 2
• All VMs of tiles 6, 7, and 8, except mailserver6, were bound to NUMA node 3
• All VMs of tiles 5 were bound to NUMA node 4
• All VMs of tiles 10, 11, and 12, except mailserver10, were bound to NUMA node 5
• All VMs of tile 9 and mailserver10 were bound to NUMA node 6
• All VMs of tiles 14 and 15 were bound to NUMA node 7
• All VMs of tiles 13 were bound to NUMA node 8
• All VMs of tiles 16, 18, and 19, except mailserver18, were bound to NUMA node 9
• All VMs of tile 17 and mailserver18 were bound to NUMA node 10
• All VMs of tile 20 and 22 were bound to NUMA node 11
• All VMs of tile 21 and mailserver23 were bound to NUMA node 12
• All VMs of tiles 23, 24, and 26, except mailserver23, were bound to NUMA node 13
• All VMs of tiles 25 and mailserver27 were bound to NUMA node 14
• All VMs of tiles 27, 28, and 58, except mailserver 27 and appserver58, were bound to NUMA node 15
• All VMs of tiles 29 were bound to NUMA node 16
• All VMs of tiles 30, and 31 were bound to NUMA node 17
• All VMs of tile 33 and mailserver34 were bound to NUMA node 18
• All VMs of tiles 32, 34, and 35, except mailserver 34, were bound to NUMA node 19
• All VMs of tiles 36, 38, and 39, except mailserver38 were bound to NUMA node 20
• All VMs of tiles 37 and mailserver38 were bound to NUMA node 21
• All VMs of tiles 40 and 42 were bound to NUMA node 22
• All VMs of tile 41 were bound to NUMA node 23
• All VMs of tiles 43, 44, and 46, except mailserver46, were bound to NUMA node 24
• All VMs of tile 45 and mailserver46 were bound to NUMA node 25
• All VMs of tiles 47 and 48 and appserver58 were bound to NUMA node 26
• All VMs of tile 49 were bound to NUMA node 27
• All VMs of tiles 50, 51, and 52, except mailserver50, were bound to NUMA node 28
• All VMs of tiles 53 and mailserver50 were bound to NUMA node 29
• All VMs of tiles 54, 55, and 56, except mailserver54, were bound to NUMA node 30
• All VMs of tile 57 and mailserver54 were bound to NUMA node 31

Web Server VM Notes

• The following boot options were used:
• selinux=0 audit=0 elevator=deadline
• The following services were disabled:
• abrt-ccpp abrt-oops abrtd atd auditd autofs cpuspeed cups haldaemon ip6tables iptables irqbalance kdump messagebus nfs nfslock portreserve postfix rhnsd rhsmcertd rngd rpcgssd rpcsvcgssd sysstat
• tuned virtual-guest profile applied to VM
• Following items were added in /etc/sysctl.conf
• fs.file-max = 500000
• net.core.somaxconn = 81920
• net.ipv4.tcp_max_tw_buckets = 1500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.ipv4.tcp_tw_reuse=1
• net.ipv4.tcp_keepalive_time = 120
• net.ipv4.tcp_keepalive_intvl = 30
• net.ipv4.tcp_timestamps = 0
• vm.swappiness=0
• root file-system options: barrier=0,noatime
• Contents of nginx configuration file /etc/nginx/nginx.conf
• user nginx;
• worker_processes 6;
• worker_cpu_affinity 00000001 00000001 00000001 00000010 00000010 00000010;
• worker_rlimit_nofile 65535;
• pid /var/run/nginx.pid;
• events {
• use epoll;
• worker_connections 65535;
• multi_accept on;
• accept_mutex off;
• }
• http {
• include /etc/nginx/mime.types;
• default_type application/octet-stream;
• log_format common '$remote_addr - $remote_user [$time_local] $request'
• '"$status" $body_bytes_sent ';
• access_log /var/log/access.log common gzip buffer=4m flush=1m;
• sendfile on;
• sendfile_max_chunk 1m;
• tcp_nopush on;
• tcp_nodelay on;
• keepalive_timeout 120;
• keepalive_requests 200;
• open_file_cache max=204800 inactive=20s;
• open_file_cache_min_uses 1;
• open_file_cache_valid 30s;
• ssl_protocols SSLv3 TLSv1 TLSv1.1 TLSv1.2;
• charset utf-8;
• server_names_hash_bucket_size 128;
• client_header_buffer_size 2k;
• large_client_header_buffers 4 4k;
• include /etc/nginx/conf.d/*.conf;
• }
• Contents of nginx configuration file /etc/nginx/conf.d/default.conf
• server {
• listen 80 backlog=8192;
• server_name localhost;
• location / {
• root /var/www/html;
• index index.html index.htm index.php;
• }
• error_page 500 502 503 504 /50x.html;
• location = /50x.html {
• root /usr/share/nginx/html;
• }
• location ~ .*.(html|htm|gif|jpg|jpeg|bmp|png|ico|txt|js|css)$
• {
• root /var/www/html;
• expires 100d;
• }
• location ~ ^/images/ {
• root /var/www/html/support;
• expires 100d;
• }
• location ~ .php$ {
• root html;
• fastcgi_pass unix:/dev/shm/php-cgi.sock;
• fastcgi_index index.php;
• fastcgi_param SCRIPT_FILENAME /var/www/html$fastcgi_script_name;
• include fastcgi_params;
• }
• }
• Contents of nginx configuration file /etc/nginx/conf.d/example_ssl.conf
• server {
• listen 443 ssl;
• server_name localhost;
• ssl on;
• ssl_certificate /etc/pki/tls/certs/localhost.crt;
• ssl_certificate_key /etc/pki/tls/private/localhost.key;
• ssl_session_cache shared:SSL:1m;
• ssl_session_timeout 5m;
• ssl_ciphers ALL:!kEDH!ADH:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP;
• ssl_prefer_server_ciphers on;
• location / {
• root /var/www/html;
• index index.html index.htm index.php;
• }
• location ~ .*\.(html|htm|gif|jpg|jpeg|bmp|png|ico|txt|js|css)$
• {
• root /var/www/html;
• expires 100d;
• }
• location ~ ^/images/ {
• root /var/www/html/support;
• expires 100d;
• }
• location ~ \ .php$ {
• root html;
• fastcgi_pass unix:/dev/shm/php-cgi.sock;
• fastcgi_index index.php;
• fastcgi_param SCRIPT_FILENAME /var/www/html$fastcgi_script_name;
• include fastcgi_params;
• }
• }
• The following lines were changed in /etc/php-fpm.d/www.conf:
• listen = /dev/shm/php-cgi.sock
• listen.owner = nginx
• listen.group = nginx
• user = nginx
• group = nginx
• pm.max_children = 500
• pm.start_servers = 200
• pm.min_spare_servers = 10
• pm.max_spare_servers = 200
• request_terminate_timeout = 300
• listen.backlog = 81920
• The following lines were added to /etc/security/limits.conf:
• * soft nofile 800000
• * hard nofile 800000
• * soft nproc 102400
• * hard nproc 102400

Application Server VM Notes

• Boot parameters set: selinux=0 audit=0 elevator=deadline
• The following services were disabled:
• abrt-ccpp abrt-oops abrtd atd auditd autofs cpuspeed crond cups haldaemon ip6tables iptables irqbalance kdump lvm2-monitor mdmonitor messagebus nfs nfslock portreserve postfix rhnsd rhsmcertd rngd rpcgssd rpcsvcgssd sysstat
• tuned virtual-guest profile applied to VM
• Following items were added in /etc/sysctl.conf
• fs.file-max = 500000
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• vm.swappiness = 0
• /opt file-system options: barrier=0
• Following items were set in /opt/common_scripts/ostune_misc.sh and executed by /etc/rc.d/rc.local
• echo tsc > /sys/devices/system/clocksource/clocksource0/current_clocksource
• There are two WebSphere profiles, AppSrv01 for SPECjAppServer2004 and AppSrv02 for SPEC Emulator
• Heap size for AppSrv01 has been changed to 2048 MB
• The following lines were added to /etc/security/limits.conf:
• * soft nofile 130172
• * hard nofile 130172
• * soft nproc 130172
• * hard nproc 130172
• spec soft nproc 1000000
• spec hard nproc 1000000
• * soft memlock 8388608
• * hard memlock 8388608

Mail Server VM Notes

• Boot parameters set: selinux=0 audit=0 elevator=deadline
• The following services were disabled:
• abrt-ccpp abrt-oops abrtd atd auditd autofs cpuspeed cups haldaemon ip6tables iptables irqbalance kdump messagebus nfs nfslock portreserve postfix rhnsd rhsmcertd rngd rpcgssd rpcsvcgssd sysstat
• tuned virtual-guest profile applied to VM
• Following items were added in /etc/sysctl.conf
• fs.file-max = 500000
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• root file-system options: barrier=0
• datastore file-system options: noatime,barrier=0,data=writeback
• Contents of dovecot configuration file /etc/dovecot/dovecot.conf:
• dict {
• }
• !include conf.d/*.conf
• base_dir = /var/run/dovecot/
• disable_plaintext_auth = no
• log_path = /var/log/dovecot.log
• mail_location = dbox:/mailstore/dovecot/maildir/%u
• mail_fsync=never
• maildir_very_dirty_syncs = yes
• mbox_very_dirty_syncs = yes
• mbox_write_locks = fcntl
• passdb {
• args = scheme=plain /etc/dovecot/passwd.dovecot
• driver = passwd-file
• }
• postmaster_address = postmaster@example.com
• service auth {
• user = root
• }
• service imap-login {
• inet_listener imap {
• port = 143
• }
• process_min_avail = 10
• service_count = 0
• }
• service imap {
• process_limit = 1024
• }
• ssl = no
• ssl_cert = </etc/pki/dovecot/certs/dovecot.pem
• ssl_key = </etc/pki/dovecot/private/dovecot.pem
• userdb {
• driver = passwd
• }
• The following lines were added to /etc/security/limits.conf:
• * soft nofile 102400
• * hard nofile 102400
• * soft nproc 102400
• * hard nproc 102400
• dovecot soft nproc 1000000
• dovecot hard nproc 1000000
• dovenull soft nproc 1000000
• dovenull hard nproc 1000000

Database Server VM Notes

• Boot parameters set: selinux=0 audit=0 elevator=deadline cgroup_disable=memory,cpu,blkio
• The following services were disabled:
• abrt-ccpp abrt-oops abrtd atd auditd autofs cpuspeed cups haldaemon ip6tables iptables irqbalance kdump messagebus nfs nfslock portreserve postfix rhnsd rhsmcertd rngd rpcgssd rpcsvcgssd sysstat
• tuned virtual-guest profile applied to VM
• Following items were added in /etc/sysctl.conf
• fs.file-max = 500000
• net.core.somaxconn = 32384
• net.ipv4.tcp_max_tw_buckets = 1000000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• vm.swappiness = 0
• kernel.shmmax = 137438953472
• kernel.shmmni = 11118096
• kernel.shmall = 8294967296
• kernel.sched_migration_cost_ns = 150000
• root file-system options: barrier=0
• datastore file-system options: noatime,barrier=0,data=writeback
• Following items were set in /opt/common_scripts/ostune_misc.sh and executed by /etc/rc.d/rc.local
• echo tsc > /sys/devices/system/clocksource/clocksource0/current_clocksource
• sysctl -w kernel.randomize_va_space=0
• sysctl -w vm.nr_hugepages=0
• sysctl -w vm.nr_hugepages=12288
• sysctl -w vm.nr_overcommit_hugepages=512
• Database tuning options:
• db2 update db cfg for specdb using DATABASE_MEMORY 6100000
• db2set DB2_LARGE_PAGE_MEM=DB; db2set DB2_PINNED_BP=ON
• alter bufferpool IBMDEFAULTBP size 5500000
• alter tablespace roll_space NO FILE SYSTEM CACHING
• alter tablespace C_cust_space NO FILE SYSTEM CACHING
• alter tablespace C_supp_space NO FILE SYSTEM CACHING
• alter tablespace C_site_space NO FILE SYSTEM CACHING

Infraserver VM Notes

• Boot parameters set: selinux=0 audit=0 elevator=deadline
• The following services were disabled:
• abrt-ccpp abrt-oops abrtd atd auditd autofs cpuspeed cups haldaemon ip6tables iptables irqbalance kdump messagebus nfslock portreserve postfix rhnsd rhsmcertd rngd rpcgssd rpcsvcgssd sysstat
• tuned virtual-guest profile applied to VM
• Following items were added in /etc/sysctl.conf
• fs.file-max = 500000
• net.core.somaxconn = 81920
• net.ipv4.tcp_max_tw_buckets = 1500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.ipv4.tcp_tw_reuse=1
• net.ipv4.tcp_keepalive_time = 120
• net.ipv4.tcp_keepalive_intvl = 30
• net.ipv4.tcp_timestamps = 0
• net.ipv4.tcp_syn_retries = 4
• net.ipv4.tcp_retries2 = 7
• vm.swappiness=0
• root file-system options: barrier=0,noatime
• datastore file-system options: noatime,barrier=0,data=writeback
• httpd.conf changes
• Timeout 20
• ## KeepAliveTimeout 15
• ##Include conf.d/*.conf
• #MaxKeepAliveRequests 100
• Listen 81
• ServerName besim:80
• <IfModule prefork.c>
• StartServers 10
• MinSpareServers 10
• MaxSpareServers 10
• ServerLimit 4096
• MaxClients 4096
• MaxRequestsPerChild 640000
• </IfModule>
• CustomLog logs/access_log common
• LoadModule fastcgi_module modules/mod_fastcgi.so
• ScriptAlias /fcgi-bin/ "/var/www/fcgi-bin/"
• FastCgiIpcDir /etc/httpd/fastcgi
• <Directory "/var/www/fcgi-bin">
• AllowOverride None
• Options +ExecCGI -Includes
• SetHandler fastcgi-script
• Order allow,deny
• Allow from all
• </Directory>
• AddHandler fastcgi-script fcgi
• FastCgiServer /var/www/fcgi-bin/besim_fcgi.fcgi -processes 50
• NFS export options:
• rw,async,no_subtree_check,no_root_squash
• The following lines were added to /etc/security/limits.conf:
• * soft nofile 102400
• * hard nofile 102400
• * soft nproc 102400
• * hard nproc 102400
• apache soft nproc 1000000
• apache hard nproc 1000000

Batch Server VM Notes

• Boot parameters set: selinux=0 audit=0 elevator=deadline
• The following services were disabled:
• abrt-ccpp abrt-oops abrtd atd auditd autofs cpuspeed cups haldaemon ip6tables iptables irqbalance kdump messagebus nfs nfslock portreserve postfix rhnsd rhsmcertd rngd rpcgssd rpcsvcgssd sysstat
• tuned virtual-guest profile applied to VM
• Following items were added in /etc/sysctl.conf
• fs.file-max = 500000
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• The following lines were added to /etc/security/limits.conf:
• * soft nproc 102400
• * hard nproc 102400
• * soft nofile 102400
• * hard nofile 102400

Client Driver Notes

• All prime clients were VMs running Red Hat Enterprise Linux Server 7.3 KVM
• The prime controller ran on the first vClient and one vClient was used for each tile.
• Each vClient has 6 vCPUs, 12GB memory, 16GB disk, 1 virtio_blk device and 1 i40evf device
• The first vClient acted as prime controller therefore had one more virtio_net device for management
• All physical clients have 2 x 10 Gbps ports, which were configured to have 10 VFs each. Each vClient used one VF.
• The client VM images were provided by the system local disk
• Five physical clients each hosted 10 vClients, and one physical client hosted 8 vClients for a total of 58 vClients.

Other Notes

• 1 x HP H3C S5820-24XG-SFP+ switch was used to connect the clients and the SUT

Performance Details: