besim_design_0523

SPECweb2005 Backend Simulator (BeSim)

Design Document

Overview

SPECweb2005 includes new workloads to represent web server performance for applications such as online banking and e-commerce. In real life the web servers supporting these applications would need to communicate with backend application and database servers. The current literature on these Internet applications show an architecture where the web server or web server farm interfaces with an application server or servers via web services protocols (e.g. SOAP over HTTP); the application server connects to the database server. Since the WEB subcommittee's focus is on measuring the performance of the web server, the SPECweb2005 benchmark needs to include a lightweight and efficient Backend Simulator (BeSim) to stand in for the Application and Database server portion of the solution architecture. The Backend Simulator must be able to behave like a perfectly sized backend system for the SPECweb2005 web server under test (SUT) and not become a bottleneck that would effect the measurement of the SUT.

The Backend Simulator runs on one or more systems that are accessible by the system under test (SUT) and by the prime client. The workload specific web server API code interfaces with BeSim via HTTP requests to keep the simulation more true to life and keep the processing required to something familiar to people who've worked with web servers and SPECweb99. The workload specific API code sends requests to BeSim to obtain data that is typically acquired from backend systems such as a user's bank balance or a registered user's shopping preferences. The data in the BeSim response is then used by the API code to complete the page to be returned as the response to a client's request.

Basic Goals and Functions for BeSim

BeSim must be portable to the same platforms used for client load generators.
BeSim should use a common code base.
BeSim should allow new workloads to be easily supported.
BeSim functions and processing must be keep as simple as possible to ensure that its does not become a bottleneck for the web server and effect the measurement of the SUT.

The total time spent in the API code setting up the request to BeSim plus the time spent parsing the BeSim response and formatting the page to be returned to the client should be an order of magnitude larger than any time spent in BeSim processing a request. So while there is additional load on the server due to traffic between the SUT and BeSim, traffic to BeSim must not become backed up and delay its response beyond that expected for a given request.

BeSim must be synchronized by the prime client. The prime client supplies the required test parameters to BeSim during the setup phase so that BeSim can generate the required responses to match the specified workload.
BeSim logs (via web server access logging) sufficient information on the requests it receives so that it can be determined whether the number of requests handled by BeSim was within the expected range for the test.
BeSim and the prime client may exchange some unique pieces of data (such as a small random number and a larger random number) that the SUT would not share.

BeSim could use this data to position and set some portion of the response to the API code. The API code would include that somewhere in its response to back to the client. The client's validation code could know what to look for and verify that the data had been obtained from BeSim.

BeSim listens on a specified port (preferably a non-standard port, i.e. not 80 ) for HTTP requests from the prime client or the SUT.

Typically, these requests would be a modified form of the original dynamic request sent by the client. The API code may append additional parameters to the query string as needed.

BeSim returns an HTTP response; the data being returned may be encoded in the entity body of the response. For. example:

  - HTTP/1.1 200 OK

    Connection: close

    Content-Length: 481

    Content-Type: text/html

    <html>
    <head><title>SPECweb2005 BESIM</title></head>
    <body>
    <p>SERVER_SOFTWARE = Apache/2.0.53 
    <p>REMOTE_ADDR = 192.168.1.4
    <p>SCRIPT_NAME = /fcgi-bin/besim_fcgi.fcgi
    <p>QUERY_STRING = 1&4&58&3
    <pre>
    0
    0000002058&/www/bank/images/user0000000058/CIF000003&\
    /www/bank/images/user0000000058/CIB000003
    </pre>
    /body></html>

BeSim supports init commands sent by the prime client for each workload. For Example:

To initialize BeSim for the workload and set values for validation of response:

GET besim?<workload#>&<Command#>[&Parameter_n=Value...]\
[&Unique1=Value1&Unique2=Value2]

Example: B
GET besim?1&0&1097157010&1&2000&200&/www/bank/images&0

BeSim Implementation

The key requirements for BeSim's implementation are portability, easy of maintenance, and easy of extension to support new workloads. To meet these requirements, BeSim makes use of existing HTTP software and is implemented as a dynamic API module. This approach leverages already developed HTTP software that knows all about handling multiple HTTP requests and allows for rapid prototyping for new workloads.

BeSim consists of a common set of core code that is used within one of several common web server API's. If a faster BeSim is needed, users can either tune the web server software or select a faster web server.

How BeSim Works

The BeSim system is connected to the Prime Client and the SUT, when a SPECweb2005 test is started, the Prime Client sends an initialization message in the form of a dynamic HTTP request with a query string containing all the parameters the BeSim software needs for the test.

Besim initializes its data structures and writes out a small file (i.e. /tmp/besim_<workload>.globals) so if there are multiple copies of BeSim (such as multiple fast-cgi instances) they can synchronize.

A value exchanged during initialization is used by BeSim to construct some of the data in the responses. The clients, which also have this value use it to validate the dynamic data returned by the SUT in its responses to the clients dynamic requests.

Most dynamic HTTP/HTTPS requests sent by the clients to the SUT require that the SUT send a request to BeSim to acquire one or more pieces of data. The SUT uses this data to construct a response to the client's request.   The lightweight nature of BeSim allows it to respond quickly to these requests so that the overhead does not significantly impact the SUT.

A standardized request and response format is used between the SUT and BeSim. All requests sent to BeSim are in the form:

        GET /<api-dir>/<besim_api>?<Workload_Type>&<Command_Type>&<param=value>[&<param=value>]

Responses returned by BeSim are in the form:

        HTTP 200 OK
        Cache-control: no-store
        Content-type: text/html
        Content-Length: [length of all return text - excluding headers]

        <html>
        <head><title>SPECweb2005 BESIM</title></head>
        <body>
        <p>SERVER_SOFTWARE = [ServerSoftware]
        <p>REMOTE_ADDR = [RemoteAddr]
        <p>SCRIPT_NAME = [ScriptName]
        <p>QUERY_STRING = [QueryString]
        <pre>
        [BeSIM MessageText]
        </pre>
        </body></html>

The BeSim MessageText includes a status value of "0\n" for OK and "1\n" for ERROR. If BeSim returns an Error it also returns an Error Message to indicate why it failed. If the request was OK, the rest of the Message Text contains the data for the response. The workload specific BeSim request-response sequences are included in the appendixes below.

Building BeSim

To build BeSim, the tester must select and install web server software on the system to server as the BeSim Box. They must also select a compatible BeSim API. SPECweb2005 supplies Fast-CGI, ISAPI, and NSAPI implementations. The directions for building each API are included with the BeSim software in the Make_Readme and in the User documentation. The kit also includes a set of perl scripts to allow simple testing of BeSim operations.

Appendix A. BeSim For Banking Request-Response Sequences

Command (# - Name) Script Mapping DB Op	Request Parameters	Response
0 - Reset Prime Client Only NA	N_TIME&\ N_MinUserID&\ N_MaxUserID&Load&\ Check_URL_Base&\ Num_check_subdirs	S_Status (DONE)
1 - LoginPasswd Login Select	N_UserID	S_Status N_Password
2 - AcctBalances Login, Order_check, Transfer Select	N_UserID	S_Status N_NumberOfAccts Lines: N_Acct_#&N_Type&D_Balance
3 -AcctSummary Account_summary Select	N_UserID	S_Status N_NumberOfAcctsLines: N_Acct_#&N_Type&D_Balance&\ D_TotalDeposit&D_AvgDeposit&\ D_TotalWitdrawal&D_AvgWithdrawal
4 - CheckDetail1 Check_detail_html and Check_detail_image Select	N_UserID&N_Check	S_Status N_Acct&S_FrontURL&S_BackURL Note: URL's for check images are created using the base URL path info supplied with the Reset (0) Command and appending: "/user<10char_UID>/ <CIF\|CIB><6digit_Check_No.>" (where CIF stands for check image front and CIB stands for check image back)
5 - Bill Payment Bill_pay Select	N_UserID	S_Status N_NumberOfPayees Lines: N_PayeeId&D_Payment&N_Date
6 - PostPayee Post_payee Insert	N_UserID&N_PayeeID&\ S_Name&S_Address&\ S_City&S_State&\ N_Zip&S_Phone	S_Status (DONE) N_Confirmation
7 - QuickPay Quick_pay Insert	N_UserID&N_PayeeId&\ N_Date&D_Amount	S_Status (DONE) N_Confirmation
8 - ReviewBillPay Bill_pay_status_output Select	N_UserID&N_StartDate&\ N_EndDate	S_Status N_NumberOfPayees Lines: N_PayeeId&N_Date&D_Amount
9 - Profile Profile Select	N_userID	S_Status S_Address&S_Email&S_Phone
10 - ChangeProfile Change_profile Update	N_UserID& \ S_Address&\ S_Email&S_Phone	S_Status (DONE) N_Confirmation
11 - PlaceChkOrder Place_check_order Select,Update,Insert	N_UserID&N_Acct#&\ N_Date&D_Price	S_Status (DONE) N_Confirmation
12 - PostTransfer Post_transfer Select,Update,Insert,Insert,Select	N_UserID&N_Acct1&\ D_Amount&\ N_Acct2,N_Date	N_NumberOfAccts Linesi(2): N_Acct_# &D_Balance

Data Formats

Userid - char[10]
Password - char[32] (encrypted using MD5 is returned by BeSim to webserver)
(Client passes char[10] to webserver)
Account number - char[10]
Account Type - int
Account Balance - numeric 10.2
Total Deposit - numeric 10.2
Avg Deposit - numeric 10.2
Total withdrawal - numeric 10.2
Avg withdrawal - numeric 10.2
Front image location - var char[100] (path)
Back image location - var char[100]
Payee id - char[20]
Payment amount - numeric 10.2
Date - in format yyyymmdd
Payee name - var char[100]
Street Address - var char[100] – use %20 for blank character in request strings
City - var char[20]
Zip - char[5]
Phone number - char[20]
email address - char[30] – use %40 for @ character in request strings
S_Status – int string ["0\n"=OK, non-zero=error]

Note: "\" line continuation, not part of actual string

Appendix B. BeSim For Ecommerce Request-Response Sequences

Command (# - Name) Script Mapping DB Op	Request Parameters	Response
0 - Reset Prime Client Only NA	N_TIME&N_MinRegion&N_MaxRegion	S_Status (DONE)
1 - Search Search Select	N_Region&S_Segment&\ S_Keyword1+S_Keyword2...	S_Status N_Results Lines: S_Segment&S_Title&\ S_URL&S_Overview
2 - ProductLines browse Select	N_Region&S_CustType	S_Status N_NumberOfProductLines (scales with N_Load): S_ProductLine1 S_ProductLine2...
3 - ProductModels browse_productline Select	N_Region&S_CustType&\ S_ProductLine	S_Status \N_ProductModelLines (scales with N_Load): S_ProductName&\ S_ProductHighlights&\ S_Feature1&S_FeatureDesc1&\ S_Feature2&S_FeatureDesc2&\ S_Feature3&S_FeatureDesc3...
4 - ProductDetails productdetail Select	N_Region&S_CustType&\ S_Product	S_Status N_ProductDetail Lines: S_ProductDetail1&\ S_ProductDetail2...
5 - CustomizationChoices customize Select	N_Region&S_CustType&\ S_ProductModel &N_Stage	S_Status N_NumberOfComponents Lines (varies with N_Stage): S_ComponentType&\ S_ComponentName&\ S_ComponentID&\ D_Price
6 - GetPrice cart & customize Select	N_Region&S_Item&\ S_ComponentID1&\ S_ComponentID2...	S_Status D_Price S_Currency
7 - CheckoutLogin login Select	N_Region&S_UserID	S_Status S_Password
8 - CheckoutRegister login Insert	N_Region&\ S_FirstName&S_LastName&\ S_Email&S_Password	S_Status N_Confirmation
9 - SaveCart cart Insert	N_Region&S_Email&\ N_Items&\ S_Item1&\ N_Quantity1&\ N_Components&\ S_ComponentID1&\ S_ComponentID2&\ S_Item2&\ N_Quantity2&\ N_Components&\ S_ComponentID1&\ S_ComponentID2...	S_Status N_Confirmation
10 - SubmitOrder confirm Insert	N_Region&S_Email&S_ShipFirstName&\ S_ShipLastName&S_ShipAddress&\ S_ShipCity&S_ShipState&N_ShipZip&\ S_ShipPhone&S_ShipMethod&\ S_BillFirstName&S_BillLastName&\ S_BillAddress&S_BillCity&S_BillState&\ N_BillZip&S_BillPhone&S_CardType&\ N_CardNo&N_ExpMo&N_ExpYr&\ N_Items&\ S_Item1&N_Quantity1&\ N_Components&\ S_ComponentIDn&\ S_Item2&N_Quantity2&\ N_Components&\ S_ComponentIDn.	S_Status S_ShipDate&N_Confirmation
11 - GetRegions cart & customize Select	N_Region_flag (where 0 or any value returns list of all regions)	S_Status N_NumberRegions Lines: S_Region

Data Formats

Region – char[8], i.e. “region1”-“region80”
Segment – var char[11], specifies the search segment, i.e. “techsupport”, “all”
KeywordN – var char[128] – search keywords (user input)
Title – char[16] – brief title of a document (could be random characters)
URL – char[128] – URL to the document (starts with http://, could be random after that)
Overview – char[256] – a brief description of the document (could be random)
CustType – char[4], i.e. “home”, “mlb”
ProductLine – var char[16], i.e. “desktop”, “server”, “storage”
ProductModels – int
ProductModel – var char [16], i.e. “desktop5”, “server16”
Stage – int, described below:

Stage #	Stage Type	Avg. # component types returned	Avg. # BeSim selections returned (per component)
1	DetailedComponents	20	5
2	ServiceSupport	6	4
3	Accessories	7	6

Component Type – var char[128], i.e. “CPU”, “Memory”, “OS”
Component Name – var char[128], i.e. “Dual Processor Intel Xeon, 2.0GHz/1MB Cache”
Price – int, i.e. 199 for $199
UserID – email address - char[120] – use %40 for @ character in request strings
Password - char[32], since this is encrypted using MD5 is returned by BeSim to webserver
(Client passes char[10] to webserver)
FirstName – var char[14]
LastName – var char[15]
Address – var char[50]
City – var char[30]
State – char[2]
Zip – char[5]
Phone – char[15]
ShipMethod – char[5]
CardName – char[80]
CardType – char[16]
CardNo – long
ExpMo – int, 1-12
ExpYr – int, i.e. 2004
ShipDate - in format MM/DD/YYYY
Confirmation – long
S_Status – int strine ["0\n"=OK, non-zero=error]

Note: "\" line continuation, not part of actual string

Appendix C. BeSim For Support Request-Response Sequences

Command (# - Name) Script Mapping DB Op	Request Parameters	Response
0 - Reset Prime Client Only NA	N_TIME&N_Load	S_Status (DONE)
1 - Category Listing Index Select	(none)	S_Status N_Results: N_ProductCategoryID1&S_ProductCategory1 N_ProductCategoryID2&S_ProductCategory2...
2 - Product Listing Catalog Select	N_ProductCategoryID	S_Status N_Results: N_ProductID1&S_Product1 N_ProductID2&S_Product2...
3 - Search Search Select	S_keyword1+\ S_keyword2+\ S_keyword3...	S_Status N_Results: N_ProductID1&S_Product1 N_ProductID2&S_Product2...
4 - Download Categories Product Select	N_ProductID	S_Status N_Results: S_DownloadCategoryID1 S_DownloadCategoryID2...
5 - Languages Product Select	(none)	S_Status N_Results: S_Language1 S_Language2...
6 - Operating Systems Product Select	N_ProductID	S_Status N_Results: S_OS1 S_OS2...
7 - Files FileCatalog Select	S_ProductID&\ S_DownloadCategory&\ S_Language&S_OS	S_Status N_Results: N_FileID1&S_FileName1&S_Date&N_Size1 T_Description1 N_FileID2&S_FileName2&S_Date&N_Size2 T_Description2....
8 - File File Select	N_ProductID&\ S_DownloadCategory&\ S_Language&S_OS	S_Status S_FileName&S_Date&N_Size&\ S_DownloadURL T_Description T_AdditionalInfo

Data Formats

All N_*ID – unsigned int
S_Status – "0\n" if OK, non-zero if not
S_ProductCategory – char[25], i.e. “Servers”
S_Product – char[50], i.e. “ProEdge Ultra 1500”
S_DownloadCategory – char[25], i.e. “BIOS Updates”
S_Language – char[25], i.e. “Brazilian Portuguese”
S_OS – char[50], i.e. “Operating System Y 2004 SP2”
S_FileName – varchar(255), i.e. “DisplayDriver1_4423.exe”, “web2004-1.00.tgz”
T_Description – varchar(255), i.e. “Vendor XYZ Driver Update 44.23 (Service Release 2)”
T_AdditionalInfoN – text block; usually installation instructions, which can be in the 1KB-2KB range
S_DownloadURL – varchar(255), starts with ‘http://’
S_Date – datetime
N_Size – bigint, size in bytes

Note: "\" line continuation, not part of actual string

SPECweb(r) is a trademark of the Standard Performance Evaluation Corporation (SPEC) .

Copyright (c) 2005 Standard Performance Evaluation Corporation

Revision date: May 27, 2005