Performance Summary:

Performance Details:

Validation Errors:

•   No Validation Errors Found

Physical Configuration:

Virtual Configuration:

Notes:

Physical System Notes

• ILOM settings used:
• In the web interface, remove the limits at menu path
• Power Management / Limit
• At the OS level in all Domains:
• poweradm set administrative-authority=none

Storage Notes

• OS installed on internal 6x 600GB disks. Primary LDOM and 2 guest LDOMs installed on raid-1 disk pairs.
• All Sun Server X3-2L Storage Servers running Solaris 11.1 and COMSTAR software with following tunings
• - Sun Server X3-2L BIOS - disable uncore frequency scaling
• - Solaris 11.1 /etc/system - set idle_cpu_no_deep_c=1
• - Solaris 11.1 /kernel/drv/qlt.conf -
• - bucketcnt2k=32768; bucketcnt8k=8192; bucketcnt64k=4096;
• - bucketcnt128k=16384; bucketcnt256k=128
• - FC-AL Port A connected to Brocade switch to LDOM1
• - FC-AL Port B connected to Brocade switch to LDOM2
• 1x Sun Server X3-2L Storage Servers (2x 2.9GHz, 128GB mem, 12x 600GB(10k) disks, 4x 100GB SSDs) configured with zone root pools and JMS log storage.
• - Provided 2 x 6, 600GB disk RAID1 Solaris Volume Manager (SVM) mirrors, interlace=64k 1 exported on port A, 1 exported on port B used for the zone root pools
• - Provided 2 x 2, 100GB Solid State SATA Drive RAID1 Solaris Volume Manager (SVM) mirrors 1 exported on port A, 1 exported on port B used for the JMS logs
• 2x Sun Server X3-2L Storage Servers (2x 2.4GHz, 64GB mem, 12x 3TB disks, 4x Sun Flash Accelerator F40 PCIe Cards) configured with web target file set attached to Infra zones and web and infra log files.
• - Per unit provided 2 x 2 Sun Flash Accelerator F40 PCIe cards (4 flash devices per card) RAID 1 Solaris Volume Manager (SVM) mirror, interlace=16k 1 exported on port A, 1 exported on port B used for the web target file set pool attached to Infra zones
• - Per unit provided 6x 2, 3TB disk RAID1 Solaris Volume Manager (SVM) mirrors, 3 exported on port A, 3 exported on port B used for the Web and Infra zones log pools (per port - 1 unit per log type)
• 1x Sun Server X3-2L Storage Servers (2x 2.4GHz, 96GB mem, 6x 600GB(15k) disks, 4x Sun Flash Accelerator F40 PCIe Cards) configured with web target file set attached to Infra zones.
• - Per unit provided 2 x 2 Sun Flash Accelerator F40 PCIe cards (4 flash devices per card) RAID 1 Solaris Volume Manager (SVM) mirror, interlace=16k 1 exported on port A, 1 exported on port B used for the web target file set pool attached to Infra zones. 600GB drives not used.
• 2x Sun Server X3-2L Storage Servers (2x 2.4GHz, 96GB mem, 6x 600GB(15k) disks, 4x Sun Flash Accelerator F40 PCIe Cards) configured with Mail zone mail store, mail logs, DB redo logs.
• - Per unit provided 2 x 2 Sun Flash Accelerator F40 PCIe cards (4 flash devices per card) RAID 1 Solaris Volume Manager (SVM) mirror, 1 exported on port A, 1 exported on port B used for the mail stores
• - Per unit provided 2x 6, 600GB disk RAID1 Virtual disk, 1 exported on port A, 1 exported on port B. One unit used for the mail log other unit used for DB redo logs.
• 3x Sun Server X3-2L Storage Servers (2x 2.9GHz, 32GB mem, 12x 600GB(10K) disks configured with DB data.
• - Per unit provided 2x 6, 600GB disk RAID1 SVM mirrors, interlace=64k 1 exported on port A, 1 exported on port B used for the db data
• ZFS file system used to create zpools created in the global zone for the web target file set, mail store, and DB data.
• The default recordsize was used for all but the DB data pool which used 8K.
• Filesystems were created for each tile and mounted exclusively in the zone.
• SVM volumes used to create web/infra/mail/JMS logs and per zone soft partitions created with the default interlace.

Virtualization Software Notes

• Two cores/32GB mem from processor 0 ran the primary domain.
• Fifteen cores/480GB mem from processor 0 ran first guest domain (LDOM1).
• Fifteen cores/480GB mem from processor 1 ran second guest domain (LDOM2).
• Guest domain LDOM1 held VMs for odd tiles.
• Guest domain LDOM2 held VMs for even tiles.
• Tuning for Primary Domain:
• Oracle Solaris 11.1.14.0.5
• /etc/system:
• set autoup = 345600
• set maxphys=1048576
• set rlim_fd_max=100000
• set rlim_fd_cur=100000
• stop intrd
• Tuning for guest domains LDOM1 and LDOM2:
• swap size: 404GB
• Oracle Solaris 11.1.14.0.5
• /etc/system:
• set autoup = 345600
• set maxphys=1048576
• set rlim_fd_max=100000
• set rlim_fd_cur=100000
• set zfs:zfs_arc_max=137438953472
• stop intrd
• Network Tuning:
• /etc/system/rc2.d/S99spec
• ndd -set /dev/tcp tcp_conn_req_max_q 819200
• ndd -set /dev/tcp tcp_conn_req_max_q0 819200
• ndd -set /dev/tcp tcp_xmit_hiwat 524288
• ndd -set /dev/tcp tcp_recv_hiwat 524288
• ndd -set /dev/tcp tcp_smallest_anon_port 2048
• ndd -set /dev/tcp tcp_naglim_def 1
• ndd -set /dev/tcp tcp_max_buf 4194304
• ndd -set /dev/tcp tcp_cwnd_max 2097152
• 4 virtual switches (etherstubs) created for inter-zone communications
• VNIC0 (odd tiles) and VNIC1 (even tiles) for non-benchmark inter-VM communication
• VNIC2 for web/besim communication
• VNIC3 for japp/DB communication
• 2x 10 Gbit NIC port (1 per PCI card) are virtualized and used for client/VM communication.
• Tuning in /kernel/drv/ixgbe.conf:
• ixgbe[0/2]: intr_throttling=2000
• ixgbe[1/3]: intr_throttling=1000
• LDOM1's ixgbe0 handles tiles
• LDOM1's ixgbe2 handles tiles
• LDOM2's ixgbe0 handles tiles
• LDOM2's ixgbe2 handles tiles
• Interrupts from the FC-AL and Network controllers were configured across the threads domain using the /usr/sbin/pcitool utility.
• LDOM1 Configuration Notes:
• All zones in the LDOM share the available 480 GB of memory
• psrset used to created
• - pset 0 - CPU 0-25 - OS
• - pset 1 - CPU 26-118 - bound web/infra/japp/db/mail VMs
• - pset 2 - CPU 119 - bound idle
• Interrupts to CPU binding:
• - ixgbe0 to 5/6/7/8, ixgbe2 to 1/2/3/4
• - ixgbe1 (not used) to 5, ixgbe3 (not used) to 1
• - emlxs0 to 9, emlxs1 to 10,emlxs2 to 11, emlxs3 to 12
• LDOM2 Configuration Notes:
• All zones in the LDOM share the available 480 GB of memory
• psrset used to created
• - pset 0 - CPU 0-25 - OS
• - pset 1 - CPU 26-118 - bound web/infra/japp/db/mail VMs
• - pset 2 - CPU 119 - bound idle
• Interrupts to CPU binding:
• - ixgbe0 to 5/6/7/8, ixgbe2 to 1/2/3/4
• - ixgbe1 (not used) to 5, ixgbe3 (not used) to 1
• - emlxs0 to 9, emlxs1 to 10,emlxs2 to 11, emlxs3 to 12
• VM configurations are as follows:
• All VMs are managed with zonadm and created using zone cfg utility
• All VMs are created using the following XML template with VM specific exceptions noted below:
• <?xml version="1.0" encoding="UTF-8"?>
• <!DOCTYPE zone PUBLIC "-//Sun Microsystems Inc//DTD Zones//EN" "file:///usr/share/lib/xml/dtd/zonecfg.dtd.1">
• <zone name="<"zone name> zonepath="<zone root dir>" autoboot="false" brand="solaris" ip-type="exclusive" limitpriv="default,dtrace_proc,dtrace_user">
• <network physical="<administrative network (not used for BM)"> configure-allowed-address="true"/>
• DB VM modifications
• <network physical="<vnic to japp VM>" configure-allowed-address="true"/>
• <dataset name="<db data file system>" alias="data"/>
• <dataset name="<db backup file system>" alias="DB_BACKUP"/>
• <device match="<block redo1>"/>
• <device match="<raw redo1>"/>
• <device match="<block redo2>"/>
• <device match="<raw redo2>"/>
• <device match="<block redo1 backup>"/>
• <device match="<raw redo1 backup>"/>
• <device match="<block redo2 backup>"/>
• <device match="<raw redo2 backup>"/>
• Idle VM modifications
• <network physical="<ixgbe-vnic to client>" configure-allowed-address="true"/>
• Infra VM modifications
• <network physical="<ixgbe-vnic to client>" configure-allowed-address="true"/>
• <network physical="<vnic to web VM>" configure-allowed-address="true"/>
• <dataset name="<target file set>" alias="data_cmstr"/>
• <filesystem special="<infra log file system>" raw="<infra log file system>" directory="/http_logs" type="ufs">
• <fsoption name="nologging"/>
• <fsoption name="nosuid"/>
• <fsoption name="noatime"/>
• </filesystem>
• jApp VM modifications
• <network physical="ixgbe-vnic to client" configure-allowed-address="true"/>
• <network physical="vnic to db VM" configure-allowed-address="true"/>
• <filesystem special="<japp log file system raw>"="<japp log file system>" directory="/jms" type="ufs">
• <fsoption name="nologging"/>
• <fsoption name="nosuid"/>
• <fsoption name="noatime"/>
• </filesystem>
• Mail VM modifications
• <network physical="ixgbe-vnic" configure-allowed-address="true"/>
• <dataset name="mail store disk" alias="data_cmstr"/>
• <filesystem special="<mail log file system>" raw="<mail log file system>" directory="/mail_logs" type="ufs">
• <fsoption name="nologging"/>
• <fsoption name="nosuid"/>
• <fsoption name="noatime"/>
• </filesystem>
• <dataset name="mail store backup" alias="mail_backup"/>
• Web VM modifications
• <network physical="ixgbe-vnic to client" configure-allowed-address="true"/>
• <network physical="vnic to infra VM" configure-allowed-address="true"/>
• <filesystem special="<web log file system>" raw="<web log file system>" directory="/http_logs" type="ufs">
• <fsoption name="nologging"/>
• <fsoption name="nosuid"/>
• <fsoption name="noatime"/>
• </filesystem>
• Network setting within the web, infra, idle, mail japp zones, the ixgbe's rxfanout set to 1.

Web Server VM Notes

• ulimit -n 65536
• executed using the priocntl -c FX -e
• magnus.conf file changes:
• remove - Init fn="load-modules" shlib="libj2eeplugin.so
• add - Init fn="load-modules" shlib="libfastcgi.so
• obj.conf file changes:
• remove the following lines:
• NameTrans fn="ntrans-j2ee" name="j2ee
• PathCheck fn="find-index-j2ee
• PathCheck fn="find-index" index-names="index.html,home.html,index.jsp
• ObjectType fn="type-j2ee
• Error fn="error-j2ee
• add the following lines:
• NameTrans fn="pfx2dir" from="/support" dir="/export/home/docs/support" name="es-internal
• PathCheck fn="find-index" index-names="index.html,home.html,index.php
• <If -f $path>
• Service type="magnus-internal/php
• fn="responder-fastcgi
• app-path="/usr/php/5.3/bin/php-cgi
• bind-path="localhost:3101
• app-env="PHPRC=/etc/php/5.3
• app-env="PHP_INI_SCANDIR=/etc/php/5.3/conf.d
• app-env="PHP_FCGI_CHILDREN=4
• app-env="PHP_FCGI_MAX_REQUESTS=2000
• app-env="FCGI_WEB_SERVER_ADDRS=127.0.0.1
• bucket="php-bucket
• </If>
• <Else>
• Service type="magnus-internal/php" fn="set-variable" error="404
• </Else>
• server.xml file changes:
• log-file: /http_logs/errors
• log-stdout: false
• log-stderr: false
• log-to-console: false
• log-level: failure
• platform:32
• keep-alive
• threads: 8
• max-connections: 17500
• timeout: 120
• poll-interval: 0.05
• thread-pool
• queue-size: 17500
• min-threads: 69
• max-threads: 69
• stack-size: 131072
• file-cache
• max-age: 3600
• max-entries: 10000
• max-open-files: 10000
• sendfile: true
• sendfile-size: 262144
• replacement: false
• max-heap-file-size: 32768
• max-heap-space: 100000000
• max-mmap-file-size: 16000001
• max-mmap-space: 1
• cache-content: true
• http
• false
• stats
• enabled: false
• interval: 15
• access-log: /http_logs/access
• http-listener
• acceptor-threads: 2
• listen-queue-size: 8192
• document-root: /opt/oracle/webserver7/https-web401/docs</document-root>
• virtual-server
• document-root: /export/home/docs

Application Server VM Notes

• application server:
• JAVA_OPTIONS= -server -Xms1700m -Xmx1700m -Xmn1000m -Xss128k -XX:+AggressiveOpts -XX:+UseParallelGC -XX:ParallelGCThreads=8 -XX:PermSize=128m -XX:LargePageSizeInBytes=4m -XX:+UseParallelOldGC -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
• WL_OPTIONS: -Dweblogic.management.discover=false -Dweblogic.SocketReaders=2 -Dweblogic.StuckThreadMaxTime=900 -Dweblogic.diagnostics.debug.DebugLogger.DISABLED=true -Dweblogic.management.allowClearTextPasswords=true
• emulator
• JAVA_OPTIONS= -server -Xms1700m -Xmx1700m -Xmn1g -Xss128k -XX:+AggressiveOpts -XX:+UseParallelGC -XX:ParallelGCThreads=16 -XX:LargePageSizeInBytes=4m -XX:MaxTenuringThreshold=3 -XX:SurvivorRatio=20 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:-TraceClassUnloading -XX:PermSize=96m
• WL_OPTIONS: -Dweblogic.management.discover=false -Dweblogic.SocketReaders=4 -Dsun.net.inetaddr.ttl=0 -Dnetworkaddress.cache.ttl=0

Mail Server VM Notes

• ulimit -n 65536
• executed using the priocntl -c FX -e
• Dovecot was compiled with default options. The dovecot.conf was used:
• dict {
• }
• #!include conf.d/*.conf
• base_dir = /export/home/dovecot/
• default_login_user = dovecot
• disable_plaintext_auth = no
• log_path = /mail_logs/dovecot.log
• info_log_path = /mail_logs/debug_info_dovecot.log
• mail_debug = yes
• auth_verbose=yes
• auth_debug=yes
• auth_debug_passwords=yes
• verbose_ssl=yes
• mail_location = dbox:/export/home/vmail/%u
• mbox_write_locks = fcntl
• mail_fsync = optimized
• passdb {
• args = scheme=plain username_format=%u /export/home/dovecot/etc/passwd.dovecot
• driver = passwd-file
• }
• postmaster_address = postmaster@example.com
• protocols = imap
• service auth {
• user = root
• }
• service imap-login {
• inet_listener imap {
• port = 143
• }
• process_min_avail = 10
• service_count = 0
• }
• service imap {
• process_limit = 1024
• }
• ssl = no
• userdb {
• driver = passwd
• }

Database Server VM Notes

• db_name = specdb
• control_files = /data/cntrlspecdb
• compatible = 11.2.0.3.0
• cpu_count = 4
• db_cache_size = 256M
• db_8k_cache_size = 100M
• db_4k_cache_size = 100M
• db_file_multiblock_read_count = 128
• db_block_size = 2048
• db_files = 256
• db_file_multiblock_read_count = 128
• db_writer_processes = 8
• dml_locks = 300
• filesystemio_options = setall
• log_checkpoint_interval = 0
• log_checkpoints_to_alert = TRUE
• processes = 300
• open_cursors = 2000
• shared_pool_size = 200M
• sessions = 400
• transactions = 400
• transactions_per_rollback_segment = 1
• parallel_max_servers = 100
• undo_management = AUTO
• undo_retention = 5
• undo_tablespace = undo_ts
• DB Datastore contains 55 GB for DB data and 16 GB for transaction redo logs.

Infraserver VM Notes

• executed using the priocntl -c FX -e
• magnus.conf file changes:
• remove - Init fn="load-modules" shlib="libj2eeplugin.so
• add -
• Init fn="pool-init" block-size="65536" num-blocks="1
• Init fn="load-modules" shlib=
• Init fn="init_besim" LateInit="no
• MaxProcs 2
• obj.conf file changes:
• remove -
• AuthTrans fn="match-browser" browser="*MSIE*" ssl-unclean-shutdown="true'
• NameTrans fn="ntrans-j2ee" name="j2ee
• NameTrans fn="pfx2dir" from="/mc-icons" dir="/opt/oracle/webserver7/lib/icons" name="es-internal
• PathCheck fn="uri-clean
• PathCheck fn="check-acl" acl="default
• PathCheck fn="find-pathinfo
• PathCheck fn="find-index-j2ee
• PathCheck fn="find-index" index-names="index.html,home.html,index.jsp
• ObjectType fn="type-j2ee
• ObjectType fn="type-by-extension
• ObjectType fn="force-type" type="text/plain
• Service method="(GET|HEAD)" type=
• Service method="(GET|HEAD|POST)" type="*~magnus-internal/*" fn="send-file
• Service method="TRACE" fn="service-trace
• Error fn="error-j2ee
• add -
• NameTrans fn="assign-name" from="/BeSim/specweb2005_besim" name="besim" nostat="/BeSim/specweb2005_besim" find-pathinfo-forward=true
• <Object name="besim">
• ObjectType fn="force-type" type="magnus-internal/besim
• Service fn="service_besim" method="(GET|POST)" type="magnus-internal/besim
• </Object>
• server.xml file changes:
• log-file: /http_logs/errors
• log-stdout: false
• log-stderr: false
• log-to-console: false
• log-level: failure
• platform:32
• user: root
• keep-alive
• threads: 4
• max-connections: 8192
• timeout: 3600
• thread-pool
• queue-size: 8192
• min-threads: 65
• max-threads: 65
• stack-size: 131072
• stats:
• enabled: false
• qos
• enabled: false
• ssl-session-cache
• enabled: false
• access-log: /http_logs/access
• http-listener
• port: 81
• listen-queue-size: 8192

Idle Server VM Notes

• none

Client Driver Notes

• set sq_max_size=0
• set rlim_fd_max=32768
• set rlim_fd_cur=32768
• set ddi_msix_alloc_limit=8 (not used in the T3-4 clients)
• set ip:ip_soft_rings_10gig_cnt=16
• ndd -set /dev/tcp tcp_conn_req_max_q 40000
• ndd -set /dev/tcp tcp_conn_req_max_q0 40000
• ndd -set /dev/tcp tcp_xmit_hiwat 524288
• ndd -set /dev/tcp tcp_recv_hiwat 524288
• ndd -set /dev/tcp tcp_naglim_def 1
• ndd -set /dev/tcp tcp_fin_wait_2_flush_interval 85000
• ndd -set /dev/tcp tcp_smallest_anon_port 1200

Other Notes

• Two 24 port Arista 7124S 10 GbE Switches. One switch per LDOM. This switch's End of Sale date was Nov 1, 2011. It is supported until Nov 1. 2014.
• Two 48 port Brocade 6510 Fiber Switches. One switch per LDOM.