Defect identified in SPECjbb®2013
affects comparability of results
December 9, 2014 - The Standard Performance Evaluation Corporation (SPEC) has
identified a defect in the SPECjbb®2013 benchmark
suite. SPEC has suspended sales of the SPECjbb®2013
benchmark and is no longer accepting new submissions of SPECjbb®2013
results for publication on SPEC's website (www.spec.org).
SPEC is advising SPECjbb®2013 licensees and users of the
SPECjbb®2013 metrics that a defect recently uncovered impacts
the comparability of results. This flaw can significantly reduce the amount of work done during
the measurement period, resulting in an inflated SPECjbb®2013
metric. SPEC recommends that users not utilize SPECjbb®2013
results for system comparisons without a full understanding of the impact of this defect on each benchmark result.
SPECjbb®2013 results published on SPEC's website
have been marked Code Defect (CD) to alert readers to this issue. The SPEC OSG Java
subcommittee is working to revise the benchmark to correct this defect, add additional
validation safeguards, and release a new version as soon as possible. Current licensees
will receive a free copy of the new version when it becomes available.
Below is a summary of the SPECjbb®2013 defect and its secondary
effects. Please be sure to review this before utilizing any SPECjbb®2013
data or running the SPECjbb®2013 benchmark for system comparisons.
Primary defect: Partial Purchase Requests result in a workload that can vary significantly
Purchase transactions are the main transactions executed in SPECjbb®2013. The Transaction
Injector (TxI) will initialize a purchase request to one of the available Supermarkets. Each
purchase request will try to purchase an average of 60 items before checking out at the
register. As load increases on large systems and latencies get longer, the store may have
run out of inventory and the purchase request will not be able to purchase all intended items.
This partial transaction occurs because the replenish request triggered when the inventory for
a given item goes below 10% is not completed in time and the Supermarkets inventory reaches
zero for that item. The threshold for a successful purchase was set too low (~1% of original
target) and, in some circumstances, purchases with as few as a single item are declared
successful. Since this flaw can affect results to varying degrees, the metrics obtained from
these results are not comparable.
Secondary effects of above defect
- Lightweight HQ Datamining Requests:
As part of a Purchase Request, a receipt is sent to the HQ for the Supermarket.
Due to the effect described above with partial transaction receipts having fewer
than 60 items purchased, the HQ datamining requests iterating through the partially
fulfilled requests are lighter weight than expected, resulting in further inflated
- SPECjbb®2013 v1.0 compared to v1.01:
Due to a bug fix in SPECjbb®2013, the above defect resulted in further performance inflation.
Due to this, SPECjbb®2013 v1.0 and v1.01 results cannot be compared.
For more information, contact SPEC.