A. Two: The FABAN harness master and a load generator can run on one machine, and another must be used as the SUT. This could also be done on one machine with two concurrently running VMs with separate IP addresses. However, this would just be to try out the benchmark. Nn practice, you would want to have several machines to make sure the load generators are not the performance bottleneck and ensure that you test the SUT capacity correctly.
A. Yes they can.
A: There is no preset limit from Faban on the number of clients that it can manage, but we are limited by network bandwidth and the processing capability of the Faban master. SPEC SIP subcommittee members have tested up to 20 clients.
A: 32 bit and 64 bit Intel Xeon, 64 bit AMD Opteron, UltraSPARC T2
A: SunOS: Solaris10, OpenSolaris; Linux: Red Hat Enterprise Linux 4 and 5, Ubuntu 8.04, Fedora 10
A: We will try to help you as much as possible but cannot guarantee the benchmark will work on these platforms. If you wish to contribute developer resources to make this happen we will be glad to accept the help. Source code is available to SPECsip_Infrastructure2011 licensees to aid in porting efforts.
A. Java 1.6 is required. Some 1.6 classes are required for the user authentication.
A: The minimum required time is: ~2 hours: 10 min pre-registration, 45 min warm up time, 60 min steady state run, and 1 min cool down. However, you can do a shorter, non-compliant run to test that the system is working properly. A reasonable short run would be 1 minute pre-registration, 5 minutes warm-up time, and 10 minute steady-state run time.
A. There has been no known standards-based SIP benchmark. There is a well known de-facto benchmark called SIPstone (www.sipstone.org), which is a micro-benchmark. The main difference in the SPEC SIP Infrastructure 2011 benchmark is that it is a full system benchmark: it includes a user behavior model, authentication (on per-transaction basis), and device registration. Under the workload, device registration contributes the bulk of SIP workload, as is found in VoIP environments. In addition, because of the authentication requirement, the benchmark strongly exercises the back-end interface from the SIP server to a subscriber database or an authentication server.
A:The amount of time to populate the user location database before the first call made. Recommended minimal time is 10 minutes since this the time interval between successful device registrations at steady state run but it is allowed to use longer time to ensure the successful population of user locations.
A.It should be long enough so that each device is successfully registered. If the time is set too short, the registration arrival rate to the SUT may be high enough to cause SIP messages to drop. One can monitor the interface and socket drop counters (such as using netstat) to see if there is packet drop during the registration, or use snoop or tcpdump to observe if SUT is keeping up with the registration rate.
A: There are four stages of a "completed" test run: 1) Initial Registration, 2) Warm Up period, 3) Steady State, and 4) Cool Down. The statistics are collected only during Steady State. However, if during Initial Registration there are devices failing to register, the test can also fail during Steady State.
A:The amount of time for the test rig to get into the steady state. The load drivers will send the same load during the warm up time and during the steady state run but statistics are not collected during the warm up time. There is a minimal Warm Up time required for a conforming run. Refer to the Run Rules for the requirement.
A:The interval over which the transaction and call statistics are collected.
A:The device registration is generated by a separate sipp process so the registration still happen when you are in a call.
A:The additional time the load drives maintain their load after the end of the steady state. This makes sure that the load remains constant until the last SIPp instance finishes collecting run statistics.
A:The benchmark run completes the four stages of a run, without being interrupted because of errors.
A: No. Think of the 1 hour steady state run period as a time window. During this window, there are three potential scenarios for calls. First is simple: calls can start and end during the window. Second is more subtle: calls can start during the window but complete after the window. From the point of view of the monitoring software, these look like calls that didn't finish. Third and finally, calls start before the window (during the warm-up phase) and end during the window. These look like mystery calls that completed but never started. The software tracks both started calls and completed calls. A completed call percentage over 100 percent simply means there were more calls of the 3rd case than of the 2nd case during the run.
A: From the Master, ping the host name or IP address of each client. If ping works then there is connectivity between the master and the client.
A: From the SUT, ping each IP address (or host name) of the client, that are on the same subnet as the SUT. Then run a network performance micro-benchmark such as uperf, iperf or netperf to make sure there is adequate bandwidth between the SUT and client.
A: By default, for SunOS the Faban master talks to its clients using rsh. Since the master is SunOS, you'll need to edit the cmdmap.xml in the SunOS directory to map rsh to ssh.
A. The installer cannot connect to the X server on your desktop. Make sure you have your X display settings set correctly. Test them with a simple application such as xclock.
A: Reread the User's Guide and make sure that you've followed all the steps and have run through the check lists in section 3. If you still haven't resolved your problem, review this FAQ to see if there is anything similar to the condition you are seeing.
A.Before launch, the benchmark validation checks consistency of the input parameters. If there is inconsistency such as number of UAC control interfaces, number of UAC data interfaces, and number of UAC data ports are not identical, the validation check will fail and the run stops. The benchmark also checks connectivity by sending ping message to the clients. If any such ping is not successful, the run also stops. During the four stages of a run, at any time a SIPp process exits (because of unrecoverable error) the benchmark will stop.
A: To detect whether the clients are bottleneck, one may monitor network interface statistics and cpu load on the clients. When there is packet loss, or high CPU load, it is an indication of client overload.
A: Increase the number of clients, or use clients with more powerful CPU and better network capabilities.
A. The recipient of the call has closed down the state regarding this call. The message has arrived too late.
A. A SIP UDP retransmission timed out after multiple retries. Perhaps the SUT was over capacity (subjected to too much load). One end of the call has terminated the call. This will not terminate the benchmark run. In other words, these messages are normal.
A. The yumi-07 interface has gone dormant (perhaps due to power saving mode) and needs to be waken up by the first ping packet. Rerunning the benchmark solves the issue.
A. Yes, the message is for information only. SIPp logs such messages to stderr.
A. This is first message in the sipp error log. By itself it does not indicate an issue, but Faban will display this message when it disrupts the benchmark run due to other reasons. Scroll back the Faban Run Log to look for other signs of trouble, such as the message discussed next.
A. An earlier instance of the SIPp load driver is present, and owns the IP address and UDP port so that the new SIPp instance cannot be launched. To debug, log into the client machine to look for lingering sipp processes and kill them if found.
A. The BYE message arrived not conforming to the scenarios defined in the Design Document.
A. A CANCEL message arrived when the UAS was expecting a different message or call scenario.
A. A message was misrouted to the wrong client, which did not expect to receive this message from the sender IP address. Due to the SIPp UAC-UAS control protocol, SIPp will exit immediately upon receiving such a message. This error commonly happen if the SIP server on the SUT was not restarted before a run, combined with the fact that not all registration transactions during the Initial Registration were successful (so there were stale location entries in the SIP server).
A. The load driver client has run out of CPU, memory, or networking resources and cannot sustain that amount of load generation. You need to switch to more powerful clients, or use more clients to share the load generation.
A. Check if the Benchmark Configuration page is filled properly. The benchmark has a few validation checks before it starts a run. First, the sum of load distribution needs to be 100.0. Second, the benchmark check if the number of UAC, UAS and UDE clients are the same. For example, if there are 3 UAC clients specified, then number of UAS and number UDE clients should also be three. This applies to the Control Interfaces, Data Interfaces, and Data Ports. Make sure a single space is used to separate the fields, not multiple spaces.
A. The SUT was likely overloaded and could not respond to SIP transactions.
A. The Java being used on the clients is not the same as on the master. Syncing them may solve some problems.
A. No. SIPp sends this message to stderr for some reason and FABAN picks it up as a warning. It can be ignored.
A: No. You have error logging turned on and there were no errors. The master could not find an error logging file since it doesn't exist.
Copyright © 2009-2011 Standard Performance Evaluation Corporation. All rights reserved. Java® is a registered trademark of Sun Microsystems.