Performance Summary:

Validation Errors:

•   No Validation Errors Found

Physical Configuration:

Virtual Configuration:

Notes:

Physical System Notes

• BIOS settings used:
• Power Profile set to High Performance
• The test sponsor attests, as of date of publication, that CVE-2017-5753 (Spectre Variant 1) is mitigated in the system as tested and documented:
• Yes: hosts, No: guests
• The test sponsor attests, as of date of publication, that CVE-2017-5715 (Spectre Variant 2) is mitigated in the system as tested and documented:
• Yes: hosts, No: guests
• The test sponsor attests, as of date of publication, that CVE-2018-3639 (Variant 4) is mitigated in the system as tested and documented:
• Yes: hosts, No: guests

Storage Notes

• The System Under Test has 2 3.84TB SATA SSDs configured as one RAID1 arrays in BIOS by RS242-18i 4G PM8236 raid controller the hypervisor OS that was booted.
• The System Under Test has 4 3.84TB SATA SSDs configured as one RAID10 arrays in BIOS by RS242-18i 4G PM8236 raid controller for vm images and datastores.

Virtualization Software Notes

• Boot parameters set: default_hugepagesz=1G hugepagesz=1G hugepages=750
• The following services were disabled:
• iptables, ip6tables, ksm, firewalld
• Selinux was set to disabled.
• Swap partition was disabled.
• /etc/sysctl.conf was updated with the following lines:
• kernel.sysrq=0
• net.ipv4.ip_forward=0
• net.ipv4.conf.all.send_redirects=0
• net.ipv4.conf.default.send_redirects=0
• net.ipv4.conf.all.accept_source_route=0
• net.ipv4.conf.default.accept_source_route=0
• net.ipv4.conf.all.accept_redirects=0
• net.ipv4.conf.default.accept_redirects=0
• net.ipv4.conf.all.secure_redirects=0
• net.ipv4.conf.default.secure_redirects=0
• net.ipv4.icmp_echo_ignore_broadcasts=1
• net.ipv4.icmp_ignore_bogus_error_responses=1
• net.ipv4.conf.all.rp_filter=1
• net.ipv4.conf.default.rp_filter=1
• net.ipv4.tcp_syncookies=1
• kernel.dmesg_restrict=1
• net.ipv6.conf.all.accept_redirects=0
• net.ipv6.conf.default.accept_redirects=0
• In /etc/security/limits.conf
• *soft memlock unlimited
• *hard memlock unlimited
• Tuned service configuration:
• tuned-adm profile network-throughput
• Network configuration is as follows:
• 2 x Mellanox Technologies MT2892 Family [ConnectX-6 Dx], 68 virtual functions were configured for each port.
• 2 x Mellanox Technologies MT27710 Family [ConnectX-4 Lx],34 virtual functions configured on a single port.
• Web, Infra, App and DB VMs are allocated 2 VFs each.
• Mail and Batch VMs are allocated 1 VF each.
• VM configurations are as follows:
• VMs are managed with libvirt and virsh utility.
• VMs are created with the following XML template:
• <domain type='kvm'>
• <name>$vm_name</name>
• <memory>$vm_memory</memory>
• <currentMemory>$vm_memory</currentMemory>
• <vcpu cpuset='$host_cpu_list'>$vm_cpus</vcpu>
• <numatune>
• <memory mode='strict' nodeset='$vm_node'/>
• </numatune>
• <os>
• <type arch='aarch64' machine='virt-6.2'>hvm</type>
• <boot dev='hd'/>
• </os>
• <features>
• <acpi/>
• <apic/>
• <pae/>
• </features>
• <cpu mode='host-passthrough' check='none'>
• <topology sockets='1' cores='$vm_vcpus' threads='1'/>
• </cpu>
• <clock offset='utc'>
• </clock>
• <on_poweroff>destroy</on_poweroff>
• <on_reboot>restart</on_reboot>
• <on_crash>restart</on_crash>
• <devices>
• <emulator>/usr/bin/qemu-system-aarch64</emulator>
• <disk type='file' device='disk'>
• <driver name='qemu' type='qcow2' cache='none'/>
• <source dev='$vm_disk'/>
• <target dev='vda' bus='virtio'/>
• </disk>
• <interface type='hostdev' managed='yes'>
• <mac address='$mac_address'/>
• <driver name='vfio'/>
• <label>'ethx'</label>
• <source>
• <address type='pci' domain='0x0000' bus='$bus' slot='$slot' function='$function'/>
• </source>
• </interface>
• </devices>
• </domain>
• The following section is added for app and db VMS:
• <memoryBacking>
• <hugepages>
• <page size='1048576' unit='KiB'/>
• </hugepages>
• </memoryBacking>
• VMs' nice value was set after startup via the renice command to the following settings:
• app = -20
• db = -12
• web = -8
• infra = 19
• mail = 19
• batch = 19
• VMs are bound to host CPUs via their XML configuration in the following manner:
• All VMs of tiles 01, 02, 03, 04, 05, 06, 07 and 08 were bound to NUMA node 0.
• All VMs of tiles 09, 10, 11, 12, 13, 14, 15 and 16 were bound to NUMA node 1.
• All VMs of tiles 25, 26, 27, 28, 37, 38, 39 and 40 were bound to NUMA node 2.
• All VMs of tiles 17, 18, 19, 20, 21, 22, 23 and 24 were bound to NUMA node 3.
• All(Except Web and Infra) VMs of tile 29, 30, 31 and 32 were bound to NUMA node 1.
• All(Except Web and Infra) VMs of tile 33, 34, 35 and 36 were bound to NUMA node 2.
• Web and Infra VMs of tile 33, 34, 35 and 36 were bound to NUMA node 3.
• Web and Infra VMs of tile 29, 30, 31 and 32 were bound to NUMA node 0.

Web Server VM Notes

• Boot parameters set: audit=0 ipv6.disable=1
• The following services were disabled:
• ksm, iptables, irqbalance, abrtd, smartd, firewalld, multipathd
• The tuned package was installed and activated with the virtual-guest profile.
• /etc/sysctl.conf changes:
• fs.file-max = 500000
• net.core.somaxconn = 81920
• net.ipv4.tcp_max_tw_buckets = 1500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.ipv4.tcp_tw_reuse=1
• net.ipv4.tcp_keepalive_time = 120
• net.ipv4.tcp_keepalive_intvl = 30
• net.ipv4.tcp_timestamps = 0
• vm.swappiness=0
• net.core.net_dev_budget = 3000
• net.core.net_dev_max_backlog = 30000
• net.ipv4.tcp_fin_timeout = 20
• net.ipv4.tcp_keepalive_probes = 4
• net.ipv4.tcp_max_syn_backlog = 20000
• net.ipv4.tcp_tw_recycle = 1
• /etc/security/limits.conf changes:
• *soft memlock unlimited
• *hard memlock unlimited
• *hard nofile 1000000
• *soft nofile 1000000
• *hard nproc 100000
• *soft nproc 100000
• The following lines were changed in /usr/local/etc/php-fpm.d/www.conf:
• request_terminate_timeout = 300s
• pm = dynamic
• pm.max_children = 700
• pm.start_servers = 250
• pm.min_spare_servers = 100
• pm.max_spare_servers = 300
• /etc/nginx/nginx.conf contains:
• user apache apache;
• worker_processes 4;
• worker_cpu_affinity 0001 0001 0010 0010;
• worker_rlimit_nofile 65535;
• pid /var/run/nginx.pid;
• events {
• use epoll;
• worker_connections 65535;
• multi_accept on;
• accept_mutex off;
• }
• http {
• include /etc/nginx/mime.types;
• default_type application/octet-stream;
• log_format common '$remote_addr - $remote_user [$time_local] $request'
• '"$status" $body_bytes_sent ';
• access_log off;
• sendfile on;
• sendfile_max_chunk 1m;
• tcp_nopush on;
• tcp_nodelay on;
• keepalive_timeout 120;
• keepalive_requests 10000;
• open_file_cache max=204800 inactive=20s;
• open_file_cache_min_uses 1;
• open_file_cache_valid 30s;
• read_ahead 4096;
• charset utf-8;
• server_names_hash_bucket_size 128;
• client_header_buffer_size 2k;
• large_client_header_buffers 4 4k;
• include /etc/nginx/conf.d/*.conf;
• fastcgi_read_timeout 300s;
• }
• /etc/nginx/conf.d/default.conf contains:
• upstream fastcgi_backend {
• server 127.0.0.1:9000;
• keepalive 128;
• }
• server {
• listen 80 backlog=81920;
• server_name localhost;
• location / {
• root /var/www/html;
• index index.html index.htm index.php;
• }
• error_page 500 502 503 504 /50x.html;
• location = /50x.html {
• root /usr/share/nginx/html;
• }
• location = /error.log {
• root /var/log/nginx;
• }
• location ~ .*.(html|htm|gif|jpg|jpeg|bmp|png|ico|txt|js|css)$ {
• root /var/www/html;
• expires 100d;
• }
• location ~ ^/images/ {
• root /var/www/html/support;
• expires 100d;
• }
• location ~ .php$ {
• root html;
• fastcgi_pass fastcgi_backend;
• fastcgi_keep_conn on;
• fastcgi_index index.php;
• fastcgi_param SCRIPT_FILENAME /var/www/html$fastcgi_script_name;
• include fastcgi_params;
• }
• }
• /etc/nginx/conf.d/ssl.conf contains:
• server {
• listen 443 ssl backlog=81920;
• server_name webserver:443;
• ssl_certificate /etc/pki/tls/certs/hostname.crt;
• ssl_certificate_key /etc/pki/tls/private/hostname.key;
• ssl_session_cache shared:SSL:1m;
• ssl_session_timeout 5m;
• ssl_protocols SSLv3 TLSv1 TLSv1.1 TLSv1.2;
• ssl_ciphers ALL:!ADH:!EXPORT:!SSLv2:RC4+RSA:+HIGH:+MEDIUM:+LOW;
• ssl_prefer_server_ciphers on;
• location / {
• root /var/www/html;
• index index.html index.htm index.php;
• }
• location ~ .*.(html|htm|gif|jpg|jpeg|bmp|png|ico|txt|js|css)$ {
• root /var/www/html;
• expires 100d;
• }
• location ~ ^/images/ {
• root /var/www/html/support;
• expires 100d;
• }
• location ~ .php$ {
• root html;
• fastcgi_pass fastcgi_backend;
• fastcgi_keep_conn on;
• fastcgi_index index.php;
• fastcgi_param SCRIPT_FILENAME /var/www/html$fastcgi_script_name;
• include fastcgi_params;
• }
• }
• Selinux was set to disabled.
• Swap partition was disabled.

Application Server VM Notes

• Boot parameters set: audit=0 ipv6.disable=1 default_hugepagesz=2m hugepagesz=2m
• The following services were disabled:
• ksm, iptables, irqbalance, abrtd, smartd, firewalld, multipathd
• The tuned package was installed and activated with the virtual-guest profile.
• /etc/sysctl.conf changes:
• net.ipv4.tcp_timestamps = 0
• net.ipv4.tcp_max_tw_buckets = 1500000
• net.ipv4.tcp_rmem = 1048576 1048576 41943040
• net.ipv4.tcp_wmem = 1048576 1048576 41943040
• net.ipv4.tcp_mem = 1048576 1048576 41943040
• net.ipv4.tcp_window_scaling = 0
• net.ipv4.tcp_tso_win_divisor = 8
• net.core.rmem_max = 41943040
• net.core.wmem_max = 41943040
• net.core.rmem_default = 1048576
• net.core.wmem_default = 1048576
• net.core.optmem_max = 41943040
• net.core.netdev_max_backlog = 81920
• net.core.somaxconn = 81920
• net.ipv4.tcp_max_syn_backlog = 8192
• net.ipv4.conf.all.arp_filter=1
• net.ipv4.ip_local_port_range = 10000 65000
• fs.file-max = 500000
• vm.min_free_kbytes = 65536
• net.ipv4.tcp_tw_reuse=1
• vm.nr_hugepages = 3500
• vm.swappiness = 0
• net.ipv4.ip_forward = 0
• net.ipv4.conf.default.rp_filter = 1
• net.ipv4.conf.default.accept_source_route = 0
• kernel.sysrq = 0
• kernel.core_uses_pid = 1
• nnet.ipv4.tcp_syncookies = 1
• net.bridge.bridge-nf-call-ip6tables = 0
• net.bridge.bridge-nf-call-iptables = 0
• net.bridge.bridge-nf-call-arptables = 0
• kernel.msgmnb = 65536
• kernel.msgmax = 65536
• kernel.shmmax = 68719476736
• kernel.shmall = 4294967296
• /etc/security/limits.conf changes:
• root hard memlock 3072000
• root soft memlock 3072000
• root hard nofile 130172
• root soft nofile 130172
• root hard nproc 130172
• root soft nproc 130172
• The following lines were added to glassfish domain.xml:
• Set the max-pool-size attribute value to 512 in the jdbc-connection-pool element
• Set the steady-pool-size attribute value to 16 in the jdbc-connection-pool element
• Set the max-thread-pool-size attribute to 50 for the thread-pool element with the name http-thread-pool
• Set the max-thread-pool-size attribute to 100 for the thread-pool element with the name admin-thread-pool
• Selinux was set to disabled.
• Swap partition was disabled.

Mail Server VM Notes

• Boot parameters set: ipv6.disable=1
• The following services were disabled:
• ksm, iptables, irqbalance, abrtd, smartd, firewalld, multipathd
• The tuned package was installed and activated with the virtual-guest profile.
• /etc/security/limits.conf changes:
• *soft memlock unlimited
• *hard memlock unlimited
• Contents of dovecot configuration file /etc/dovecot/dovecot.conf:
•   dict {
•   }
•   !include conf.d/*.conf
•   !include_try local.conf
•   disable_plaintext_auth = no
•   log_path = /var/log/dovecot.log
•   mail_location = dbox:/mailstore/dovecot/maildir/%u
•   passdb {
•   args = scheme=plain /etc/dovecot/passwd.dovecot
•   driver = passwd-file
•   }
•   service auth {
•   user = root
•   }
•   service imap-login {
•   inet_listener imap {
•   port = 143
•   }
•   process_min_avail = 10
•   service_count = 0
•   }
•   service imap {
•   process_limit = 1024
•   }
•   ssl = no
•   ssl_cert = </etc/pki/dovecot/certs/dovecot.pem
•   ssl_key = </etc/pki/dovecot/private/dovecot.pem
•   userdb {
•   driver = passwd
•   }
• listen = *
• Selinux was set to disabled.
• Swap partition was disabled.
• Mail storage file system mount options: defaults,noatime,barrier=0.

Database Server VM Notes

• Boot parameters set: audit=0 ipv6.disable=1
• The following services were disabled:
• ksm, iptables, irqbalance, abrtd, smartd, firewalld, multipathd
• Selinux was set to disabled.
• Swap partition was disabled.
• The tuned package was installed and activated with the virtual-guest profile.
• /etc/sysctl.conf changes:
• kernel.sem = 500 64000 400 512
• net.ipv4.tcp_timestamps = 0
• net.ipv4.tcp_max_tw_buckets = 1500000
• net.ipv4.tcp_rmem = 1048576 1048576 41943040
• net.ipv4.tcp_wmem = 1048576 1048576 41943040
• net.ipv4.tcp_mem = 1048576 1048576 41943040
• net.ipv4.tcp_window_scaling = 0
• net.ipv4.tcp_tso_win_divisor = 8
• net.core.rmem_max = 41943040
• net.core.wmem_max = 41943040
• net.core.rmem_default = 1048576
• net.core.wmem_default = 1048576
• net.core.optmem_max = 41943040
• net.core.netdev_max_backlog = 81920
• net.core.somaxconn = 81920
• net.ipv4.tcp_max_syn_backlog = 8192
• net.ipv4.conf.all.arp_filter=1
• net.ipv4.ip_local_port_range = 10000 65000
• fs.file-max = 500000
• vm.min_free_kbytes = 65536
• net.ipv4.tcp_tw_reuse=1
• /etc/security/limits.conf changes:
• * soft nproc 578528
• * hard nproc 578528
• * soft memlock 33554432
• * hard memlock 33554432
• postgres soft nproc 578528
• postgres hard nproc 578528
• /var/lib/pgsql/data/postgresql.conf changes:
• logging_collector = off
• track_activities = off
• track_counts = off
• listen_addresses = '*'
• port = 5432
• max_connections = 2200
• ssl = off
• shared_buffers=7000MB
• temp_buffers = 2000
• max_prepared_transactions = 1000
• work_mem=200MB
• maintenance_work_mem = 1025MB
• vacuum_cost_delay = 50
• bgwriter_delay = 200ms
• bgwriter_lru_maxpages=100
• fsync = on
• wal_sync_method = fdatasync
• full_page_writes = off
• wal_buffers = 2500
• commit_delay = 10
• checkpoint_segments = 256
• checkpoint_timeout = 300
• checkpoint_warning = 99
• enable_seqscan = off
• random_page_cost = 1.5
• cpu_tuple_cost = 0.001
• cpu_index_tuple_cost = 0.0005
• cpu_operator_cost = 0.00025
• effective_cache_size = 40GB
• default_statistics_target = 100
• bgwriter_lru_maxpages=100
• log_min_duration_statement = 1500
• log_statement = 'none'
• default_transaction_isolation = 'read committed'
• lc_messages = 'C'
• lc_monetary = 'C'
• lc_numeric = 'C'
• lc_time = 'C'
• deadlock_timeout = 1000
• Using ramdisk on VM to store database logs:
• Add in /etc/fstab:tmpfs /mnt/ramdisk tmpfs rw,size=10G 0 0

Infraserver VM Notes

• Boot parameters set: ipv6.disable=1
• The following services were disabled:
• ksm, iptables, irqbalance, abrtd, smartd, firewalld, multipathd
• The tuned package was installed and activated with the virtual-guest profile.
• /etc/sysctl.conf changes:
• fs.file-max = 500000
• net.core.somaxconn = 81920
• net.ipv4.tcp_max_tw_buckets = 1500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.ipv4.tcp_tw_reuse=1
• net.ipv4.tcp_keepalive_time = 120
• net.ipv4.tcp_keepalive_intvl = 30
• net.ipv4.tcp_timestamps = 0
• net.ipv4.tcp_syn_retries = 4
• net.ipv4.tcp_retries2 = 7
• vm.swappiness=0
• /etc/security/limits.conf changes:
• *soft memlock unlimited
• *hard memlock unlimited
• /etc/httpd/conf/httpd.conf changes:
• Listen 81
• <IfModule prefork.c>
• StartServers 5
• MinSpareServers 5
• MaxSpareServers 20
• ServerLimit 10240
• MaxClients 10240
• MaxRequestsPerChild 10000
• </IfModule>
• <IfModule worker.c>
• StartServers 4
• MaxClients 65535
• MinSpareThreads 25
• MaxSpareThreads 75
• ThreadsPerChild 25
• MaxRequestsPerChild 10000
• </IfModule>
• LoadModule fcgid_module modules/mod_fastcgi.so
• ScriptAlias /fcgi-bin/
• FcgidIPCDir /etc/httpd/fastcgi
• <Directory
•     AllowOverride None
•     Options +ExecCGI -Includes
•     SetHandler fcgid-script
•     Order allow,deny
•     Allow from all
• </Directory>
• AddHandler fcgid-script fcgi
• FastCgiServer /var/www/fcgi-bin/besim_fcgi.fcgi -processes 50
• Selinux was set to disabled.
• Swap partition was disabled.
• Standard Linux NFS server was enabled to serve files to the Web VM.
• /etc/exports file contains:
• /home/webfiles/downloads webserver${index}-int(rw,no_root_squash)

Batch Server VM Notes

• Boot parameters set: ipv6.disable=1
• The following services were disabled:
• ksm, iptables, irqbalance, abrtd, smartd, firewalld, multipathd
• The tuned package was installed and activated with the virtual-guest profile.
• /etc/security/limits.conf changes:
• *soft memlock unlimited
• *hard memlock unlimited
• Selinux was set to disabled.
• Swap partition was disabled.
• The following items were added to /etc/bashrc:
• ulimit -n 500000 > /dev/null 2>&1
• ulimit -u unlimited
• ulimit -s unlimited

Client Driver Notes

• All prime client VMs run Red Hat Enterprise Linux Server 7.4.
• Two virtual clients were used for each tile,One virtual client per tile ran the webserver workload and the other virtual client per tile ran the other workloads.
• The first virtual client is the prime controller.
• Each prime client VM has 6 VCPUs, 20 GB memory, and a 40 GB disk. 2 vfio_pci and 1 virtio_blk device were used in all client VMs.
• Each web client VM has 5 VCPUs, 20 GB memory, and a 40 GB disk. 2 vfio_pci and 1 virtio_blk device were used in all client VMs.

Other Notes

• The SUT and client systems are directly connected using fiber optic cables.
• The client1 and client2 systems are directly connected using fiber optic cables.

Performance Details: