Performance Summary:

Performance Details:

Validation Errors:

•   No Validation Errors Found

Physical Configuration:

Virtual Configuration:

Notes:

Physical System Notes

• Firmware settings used:
• TurboModeEnable=Enable
• TurboBoost=Traditional
• HardwarePrefetcher=Disable

Storage Notes

• All storage accessed with 2 x 4 Gb Fibre Channel links, through Qlogic 5600 Switch
• to 2 x DS3400 storage. Each DS3400 controller houses 12 73 GB SAS drives and connects
• to 3 EXP3000 SAS enclosures, each enclosure with 12 73 GB SAS drives. Total drive count
• is 96. There are 12 RAID5 arrays, each consisting of 8 drives. Each array has one
• drive defined for one Tile. The first array has one extra logical drive defined for
• the system's boot disk. Each drive defined for a Tile is divided into 12 logical volumes
• (2 per VM) using LVM

Virtualization Software Notes

• The following was run after each boot:
•   echo 69067 >/proc/sys/vm/nr_hugepages
•   mkdir /dev/hugepages
•   mount -t hugetlbfs none /dev/hugepages
•   for i in `seq 1 24`; do echo performance >/sys/devices/system/cpu/cpu$i/cpufreq/scaling_governor; done
• The following boot options were used:
•   elevator=deadline intel_iommu=on iommu=pt pci=assign-busses
• Network configuration is as follows:
•   Two software bridges are created, br0, br1, and the two Broadcom
•   Ethernet ports are assigned to a bridge (eth0 to br0, eth1 to br1)
•   For every VM, their first virtIO network device is assigned to one
•   of these bridges, even numbered tiles to br0, odd numbered tiles to br1
•   For each tile, there is another bridge created, name t01....tN
•   For all Web and Infra VMs, their second virtIO network device is
•   assigned to a bridge corresponding to the tile number.
•   Kernel module for Intel Gigabit ET Server adapter is loaded with
•   these options to create 36 virtual functions: max_vfs=6,6,6,6,6,6
•   These virtual functions are assigned (one per VM) to the Web, App
•   and DB VMs. When a virtual function is present on a VM, a script is
•   executed to shut down the first virtio_net device and migrate the IP
•   settings to the virtual function device. This is done to facilitate
•   easy IP/dns management when using a mix of VMs with and without
•   virtual functions. The scripts, convert_to_eth[1|2].sh are
•   available in the disclosure archive.
•   On the host, Ethernet devices eth2 - eth7 are the 6 physical functions
•   for the Intel Ethernet ports. These are not used by the host, except
•   to provide physical transport for the virtual functions.
• All VMs are launched with nice utility (-n 19) to adjust their scheduling priority
• The following VMs <VM-type>t<Tile-number> are bound to NUMA node 0 and have a preferred memory
• allocation node of 0:
•   appt02,appt04,appt06,appt08,appt10,appt12,dbt02,dbt04,dbt06,dbt08,dbt10,dbt12
•   infrat02,infrat04,infrat06,infrat08,infrat10,infrat12,mailt01,mailt02,mailt03
•   mailt04,mailt06,mailt08,mailt10,mailt12,webt02,webt04,webt06,webt08,webt10,webt12
• The following VMs <VM-type>t<Tile-number> are bound to NUMA node 1 and have a preferred memory
• allocation node of 1:
•   appt01,appt03,appt05,appt07,appt09,appt11,dbt01,dbt03,dbt05,dbt07,dbt09,dbt11
•   infrat01,infrat03,infrat05,infrat07,infrat09,infrat11,mailt05,mailt07,mailt09
•   mailt11,webt01,webt03,webt05,webt07,webt09,webt11
• All VMs' memory but idle VMs are fulfilled with huge (2MB) pages.

Web Server VM Notes

• Disabled the following services via chkconfig: anacron,atd,bluetooth,cpuspeed,crond,cups,dc_client,dc_server,firstboot,gpm,httpd,ip6tables,iptables,irqbalance,isdn,postgresql,rhnsd,sendmail,smartd,squid,tux,xfs,yum-updatesd
• nofile and noproc set to 102400 in /etc/security/limits.conf
• httpd.conf changes:
•   KeepAlive on
•   MaxKeepAliveRequests 0 (default 100)
•   KeepAliveTimeout 5 (default 15)
•   prefork MPM StartServers=20 (default 8)
•   prefork MPM MinSpareServers=20 (default 5)
•   prefork MPM MaxSpareServers=20 (default 20)
•   prefork MPM ServerLimit=1536 (default 256)
• Boot options: divider=10
•   prefork MPM MaxRequestsPerChild=1000 (default 4000)
•   CustomLog logs/access_log common
• Web VM used NFS to access files from infra VM
• No options were specified in the NFS mount
• File-system /data-disk (ext3) mounted with noatime and data=writeback options

Application Server VM Notes

• Disabled the following services via chkconfig: anacron,atd,bluetooth,cpuspeed,crond,cups,dc_client,dc_server,firstboot,gpm,httpd,ip6tables,iptables,irqbalance,isdn,postgresql,rhnsd,sendmail,smartd,squid,tux,xfs,yum-updatesd
• Heap size for Application server is 1280 MB, using huge pages
• The following was run after each boot:
•   echo 2818572288 >/proc/sys/kernel/shmmax
•   echo 688128 >/proc/sys/kernel/shmall
• Boot options: divider=10
• File-system /data-disk (ext3) mounted with noatime and data=writeback options

Mail Server VM Notes

• Disabled the following services via chkconfig: anacron,atd,bluetooth,cpuspeed,crond,cups,dc_client,dc_server,firstboot,gpm,httpd,ip6tables,iptables,irqbalance,isdn,postgresql,rhnsd,sendmail,smartd,squid,tux,xfs,yum-updatesd
• dovecot.conf:
•   base_dir = /var/run/dovecot/
•   protocols = imap
•   listen = *:143
•   disable_plaintext_auth = no
•   log_path = /var/log/dovecot
•   ssl = no
•   login_processes_count = 10
•   mail_location = dbox:/mailstore/dovecot/maildir/%u
•   mail_debug = yes
•   maildir_copy_with_hardlinks = yes
•   dbox_rotate_size = 2048
•   dbox_rotate_min_size = 16
•   dbox_rotate_days = 0
•   protocol lda {
•    postmaster_address = postmaster@example.com
•   }
•   auth default {
•    mechanisms = plain
•    passdb passwd-file {
•     args = /etc/passwd.dovecot
•    }
•    userdb passwd {
•    }
•   user = root
•   }
•   dict {
•   }
•   plugin {
•   }
• Boot options: divider=10
• File-system /data-disk (ext3) mounted with noatime and data=writeback options

Database Server VM Notes

• Disabled the following services via chkconfig: anacron,atd,bluetooth,cpuspeed,crond,cups,dc_client,dc_server,firstboot,gpm,httpd,ip6tables,iptables,irqbalance,isdn,postgresql,rhnsd,sendmail,smartd,squid,tux,xfs,yum-updatesd
• Databse tuning options:
•   db2 update db cfg for specdb using DATABASE_MEMORY 200000
•   db2set DB2_LARGE_PAGE_MEM=DB; db2set DB2_PINNED_BP=ON
•   alter bufferpool IBMDEFAULTBP size 160000
•   alter tablespace roll_space NO FILE SYSTEM CACHING
•   alter tablespace C_cust_space NO FILE SYSTEM CACHING
•   alter tablespace C_supp_space NO FILE SYSTEM CACHING
•   alter tablespace C_site_space NO FILE SYSTEM CACHING
•   alter tablespace C_parts_space NO FILE SYSTEM CACHING
•   alter tablespace C_custinv_space NO FILE SYSTEM CACHING
•   alter tablespace M_parts_space NO FILE SYSTEM CACHING
•   alter tablespace M_bom_space NO FILE SYSTEM CACHING
•   alter tablespace M_wo_space NO FILE SYSTEM CACHING
•   alter tablespace M_lo_space NO FILE SYSTEM CACHING
•   alter tablespace M_inv_space NO FILE SYSTEM CACHING
•   alter tablespace O_cust_space NO FILE SYSTEM CACHING
•   alter tablespace O_ords_space NO FILE SYSTEM CACHING
•   alter tablespace O_ordl_space NO FILE SYSTEM CACHING
•   alter tablespace O_item_space NO FILE SYSTEM CACHING
•   alter tablespace S_comp_space NO FILE SYSTEM CACHING
•   alter tablespace S_sc_space NO FILE SYSTEM CACHING
•   alter tablespace S_po_space NO FILE SYSTEM CACHING
•   alter tablespace S_site_space NO FILE SYSTEM CACHING
•   alter tablespace S_supp_space NO FILE SYSTEM CACHING
•   alter tablespace U_space NO FILE SYSTEM CACHING
•   update db cfg for specdb using logprimary 64
•   reorgchk update statistics on table all
• Boot options: divider=10 hugepages=416
• File-system /data-disk (ext3) mounted with noatime and data=writeback options

Infraserver VM Notes

• Disabled the following services via chkconfig: anacron,atd,bluetooth,cpuspeed,crond,cups,dc_client,dc_server,firstboot,gpm,httpd,ip6tables,iptables,irqbalance,isdn,postgresql,rhnsd,sendmail,smartd,squid,tux,xfs,yum-updatesd
• nofile and noproc set to 102400 in /etc/security/limits.conf
• httpd.conf changes:
•   Listen port to 81 (default 80)
•   prefork MPM StartServers=100 (default 8)
•   prefork MPM MinSpareServers=100 (default 5)
•   prefork MPM MaxSpareServers=100 (default 20)
•   prefork MPM ServerLimit=1024 (default 256)
•   prefork MPM MaxClients=1024 (default 256)
•   prefork MPM MaxRequestsPerChild=16000 (default 4000)
•   LoadModule fastcgi_module modules/mod_fastcgi.so
•   ServerName besim:80
•   CustomLog logs/access_log common
•   FastCgiServer /var/www/fcgi-bin/besim_fcgi.fcgi -processes 50
• Boot options: divider=10
• Standard Linux NFS server is enabled to serve files to Web VM
• /etc/exports file contains:
•  /support-download *(rw,no_subtree_check,crossmnt,no_root_squash)
• File-system /data-disk (ext3) mounted with noatime and data=writeback options

Idle Server VM Notes

• Disabled the following services via chkconfig: anacron,atd,bluetooth,cpuspeed,crond,cups,dc_client,dc_server,firstboot,gpm,httpd,ip6tables,iptables,irqbalance,isdn,postgresql,rhnsd,sendmail,smartd,squid,tux,xfs,yum-updatesd
• Boot options: divider=10
• File-system /data-disk (ext3) mounted with noatime and data=writeback options

Client Driver Notes

• 1 client system is dedicated for prime controller
• 4 remaining clients have 3 VMs each. Each VM
• is running the prime clients for 1 tile.
• Paravirtual clock is used for client VMs.
• Each VM is assigned 2 vCPUs, 6 GB memory,
• 1 virtio_net device, and 1 virtio_blk device.
• There are 4 Broadcom 1Gbps Ethernet ports
• on each physical client. The first is dedicated
• for host access. The remaining 3 each have
• a unique software bridge assigned (br1-br3),
• and 1 VM is assigned per bridge.
• Red Hat Enterprise Linux 5.5 (64-bit, KVM) is
• used for hosting VMs. All VMs use Red Hat
• Enterprise Linux 5.5 (64-bit). Each VM uses
• divider=10 boot option.

Other Notes

• A single Netgear switch, model GSM7352S, is used for all
• physical Ethernet connections, and a Qlogic 5600 switch
• for all fibre channel connections.