If cloud under test is baremetal or instances take a long time to provision, the tester can adjust the update_attempts in osgcloud_rules.yaml:
vm_defaults: update_attempts: 60 update_frequency: 5
The values for update_attempts and update_frequency determine whether benchmark considers the provisioning to have failed. A tester can set them to a small value to force provisioning failures for testing.
During a compliant run, the value for update_attempts is automatically calculated based on the max of the average AI provisioning time of YCSB and KMeans.
The tester can configure which workloads to run during testing of elasticity + scalability phase by modifying the following parameter:
workloads: [ ycsb, kmeans ]
The tester can set the maximum AIs to be provisioned to a specific value or number of AIs from which results is received to a specific value, and ignore the stopping conditions during testing of elasticity + scalability phase. These parameters are controlled as follows:
maximum_ais: 24 reported_ais: 14 ignore_qos_when_max_is_set: false
Elasticity + scalability phase is run as follows. It assumes that CBTOOL is already running and is connected with your cloud.:
python osgcloud_elasticity.py --exp_id SPECRUN
If you run the elasticity + scalability phase as:
python osgcloud_elasticity.py --exp_id SPECRUNID
It assumes that a baseline_SPECRUNID.yaml files exists in ~/results/SPECRUNID/perf .
The results and logs of elasticity + scalability phase are present in the following directory.:
Following files and directories are generated as a result of a successful elasticity + scalability phase run:
elasticity_SPECRUNID.yaml osgcloud_elasticity_SPECRUNID-20150811234204UTC.log SPECRUNIDELASTICITY20150811234204UTC
Where the timestamp in file and directory names will change based on the date/time the elasticity + scalability phase was run.
The tester must set appropriate values in osgcloud_environment.yaml file. The key/value pairs from this file are dumped into the submission file.
The program osgcloud_fdr.py operates on the performance data stored in ~/results/SPECRUN/perf , and the environment file.
The tester should fill out the details of its cloud environment (e.g., physical machines for whitebox cloud) in osgcloud_environment.yaml . The file should be present in ~/results/SPECRUN/perf .
The submission file generation program is then run as follows (assuming your SPECRUN id was SPECRUNID):
python osgcloud_fdr.py --baseline_yaml ~/results/SPECRUNID/perf/baseline_SPECRUNID.yaml --elasticity_results_path ~/results/SPECRUNID/perf/SPECRUNIDELASTICITY*** --exp_id SPECRUNID --environment_yaml osgcloud_environment.yaml
Where *** indicates the timestamp that forms of the elasticity results directory name.
The output of this phase is as the following:
osgcloud_fdr_SPECRUNID-20150811234502UTC.log fdr_ai_SPECRUNID.yaml sub_file_SPECRUNID.txt
Assuming the results directory was ``~/result``s , the submission file is present in the following directory:
If the elasticity experiment was run successfully, the HTML report can be generated as:
cd ~/osgcloud/driver python osgcloud_fdr_html.py --exp_id EXPID
For whitebox clouds, the tester must include an architecture diagram of the cloud in the PNG format. It can be included in the final HTML report as follows:
python osgcloud_fdr_html.py --exp_id EXPID --networkfile arch.png
The resulting file is generated in:
The name of the file is:
Before starting elasticity + scalability phase, it is best to restart CBTOOL if another elasticity + scalability phase was run. There is no need to restart CBTOOL if only baseline phases were run.
When testing elasticity + scalability phase, it is best to start with small number of AIs. Set the number of AIs to 6 or 8.
It is not necessary to run baseline phase every time before elasticity + scalability phase. However, elasticity + scalability phase needs the output of baseline phase (as a YAML) file to determine the stopping condition.
Assuming the experiment ID was EXPID and the result directory was ~/results , and the baseline was already run, the tested can do the following to repeat the elasticity + scalability phase:
cd ~/results/EXPID/perf/ rm -rf *ELASTICITY*; rm -f *elasticity*.log
The instance provisioning time of a cloud under test may be long. The time until CBTOOL waits to check for instance provisioning can be adjusted using update_attempts parameter in osgcloud_rules.yaml file. It should be set to a value such that ( update_attemps x update_frequency ) never exceeds three times the maximum baselne AI provisioning time of YCSB or KMeans workloads. For a compliant run, it must be set before the start of the experiment.
If any errors are encountered during an elasticity + scalability phase run, they can be checked in CBTOOL log:
cd /var/log/cloudbench grep ERROR *
The tester can check for errors for specific AI by searching for an AI number.
The errors can also occur during an AI_run for a workload. Examples of errors include:
Cassandra: Create, Remove, List keyspace fail, or seeds fail to form a cluster. YCSB: Data generation fails Hadoop: Hadoop slaves fail to form a cluster KMeans: Data generation fails
Instance names also include AI numbers. This can come handy when you are debugging elasticity + scalability phase in your cloud.
The tester can check for failed AIs by running the following command at the CBTOOL prompt:
MYSIMCLOUD> ailist failed
To test that instance supporting evidence collection works properly, please follow these steps:
Start CBTOOL and ensure that it can connect to your cloud and launch instances.
Ensure that workload images have been created.
Launch an application instance of Cassandra or Hadoop:
aiattach cassandra_ycsb aiattach hadoop
Determine the Linux username and SSH key path for instances. Assuming the Linux user name is cbuser and SSH key path is ~/osgcloud/cbtool/credentials/cbtool_rsa
Test supporting evidence collection for an instance. Run the supporting evidence instance script on CBTOOL machine to collect supporting evidence for an instance.
Create a directory where the results are stored:
mkdir /tmp/instance -p
Run the supporting evidence collection script:
cd ~/osgcloud/driver/support_script/ ./collect_support_data.sh remote_vm_sysinfo 10.146.5.41 cbuser ~/osgcloud/cbtool/credentials/cbtool_rsa /tmp/instance/ SCRIPT INSTANCE/CASSANDA/HADOOP IPADDR IPOFINSTANCE LINUXUSER SSHKEYPATH TARGETDIR OUTPUT: tree /tmp/instance |-- date.txt |-- df.txt |-- dpkg.txt |-- etc | |-- fstab | |-- hosts | |-- iproute2 | | |-- ematch_map | | |-- group | | |-- rt_dsfield | | |-- rt_protos | | |-- rt_realms | | |-- rt_scopes | | `-- rt_tables | |-- nsswitch.conf | |-- security | | `-- limits.conf | `-- sysctl.conf |-- hostname.txt |-- ifconfig.txt |-- lspci.txt |-- mount.txt |-- netstat.txt |-- ntp.conf |-- proc | |-- cmdline | |-- cpuinfo | |-- devices | |-- meminfo | |-- modules | |-- partitions | |-- swaps | `-- version |-- route.txt `-- var `-- log `-- dmesg
Test supporting evidence collection for YCSB and Cassandra.
Find the IP address of instance with YCSB role from CBTOOL (by typing vmlist on CBTOOL CLI). Then run the following commands:
mkdir /tmp/ycsb -p ./collect_support_data.sh remote_vm_software 10.146.5.41 cbuser ~/osgcloud/cbtool/credentials/cbtool_rsa /tmp/cassandra cassandra_ycsb OUTPUT from machine with YCSB role: $ tree /tmp/ycsb /tmp/ycsb/ |-- javaVersion.out |-- role `-- YCSB |-- custom_workload.dat `-- workloads |-- workloada |-- workloadb |-- workloadc |-- workloadd |-- workloade |-- workloadf `-- workload_template
Find the IP address of an instance with SEED role from CBTOOL (by typing vmlist on CBTOOL CLI). Then run the following commands:
mkdir /tmp/seed -p ./collect_support_data.sh remote_vm_software 10.146.5.41 cbuser ~/osgcloud/cbtool/credentials/cbtool_rsa /tmp/seed cassandra_ycsb OUTPUT: $ tree /tmp/cassandra /tmp/ |-- cassandra | |-- du_datadir | |-- du_datadir_cassandra | |-- du_datadir_cassandra_usertable | |-- nodetool_cfstats | `-- nodetool_status |-- cassandra_conf | |-- cassandra-env.sh | |-- cassandra-rackdc.properties | |-- cassandra-topology.properties | |-- cassandra-topology.yaml | |-- cassandra.yaml | |-- commitlog_archiving.properties | |-- logback-tools.xml | |-- logback.xml | `-- triggers | `-- README.txt |-- javaVersion.out `-- role
Testing supporting evidence collection from Hadoop.
Find the IP address of instance with HADOOPMASTER role from CBTOOL (by typing vmlist on CBTOOL CLI). Then run the following commands:
mkdir /tmp/hadoop -p ./collect_support_data.sh remote_vm_software 10.146.5.41 cbuser ~/osgcloud/cbtool/credentials/cbtool_rsa /tmp/hadoop hadoop OUTPUT from machine with HADOOPMASTER role: tree /tmp/hadoop/ /tmp/hadoop/ |-- hadoop | |-- datahdfs | |-- dfsadmin_report | |-- du_datanodedir | |-- du_namenodedir | |-- input_hdfs_size | |-- output_hdfs_size | `-- version |-- hadoop_conf | |-- capacity-scheduler.xml | |-- configuration.xsl | |-- container-executor.cfg | |-- core-site.xml | |-- hadoop-env.cmd | |-- hadoop-env.sh | |-- hadoop-metrics2.properties | |-- hadoop-metrics.properties | |-- hadoop-policy.xml | |-- hdfs-site.xml | |-- httpfs-env.sh | |-- httpfs-log4j.properties | |-- httpfs-signature.secret | |-- httpfs-site.xml | |-- kms-acls.xml | |-- kms-env.sh | |-- kms-log4j.properties | |-- kms-site.xml | |-- log4j.properties | |-- mapred-env.cmd | |-- mapred-env.sh | |-- mapred-queues.xml.template | |-- mapred-site.xml | |-- mapred-site.xml.template | |-- masters | |-- slaves | |-- ssl-client.xml.example | |-- ssl-server.xml.example | |-- yarn-env.cmd | |-- yarn-env.sh | `-- yarn-site.xml |-- javaVersion.out `-- role