Skip navigation

Standard Performance Evaluation Corporation


2008 SPEC Benchmark Workshop

January 27, 2008
Westin San Francisco Airport

The 2008 SPEC Benchmark Workshop was held on January 27, 2008, in conjunction with the Annual Meeting of the Standard Performance Evaluation Corporation (SPEC). The workshop took place at the Westin San Francisco Airport in Millbrae, California and brought together benchmark developers and users in industry and academia. The program featured a series of talks on:

  • available benchmark suites,
  • the use of benchmarks in industry, academia, and government organizations, and
  • benchmarking methodologies.

Workshop Schedule

08:30-09:00 Continental Breakfast

Keynote Talk

09:00-10:00 Pitfalls and Methodologies for Evaluating Managed Languages
Kathryn McKinley, The University of Texas at Austin
Powerpoint presentation  Slides (.ppt)

Programmers are increasingly turning to managed languages, such as Java and C#, because they decrease development time and the resulting programs have fewer defects. However, many architects and systems researchers do not yet evaluate or design new approaches for these workloads, retarding innovation. One reason may be that measuring managed languages has a number of challenges, such as how to factor in and out dynamic compilation and garbage collection. This talk shows the pitfalls of ignoring these effects, and recommends methodologies for measuring managed languages in a meaningful way.

Kathryn McKinley is a professor at The University of Texas at Austin. She received her PhD, MS, and BA from Rice University. Her research interests include compiler optimization, architecture, memory management, and software engineering. She has published over 75 refereed articles. Her honors include ACM Distinguished Scientist and she currently serves as the co-Editor-in-Chief of TOPLAS. She has graduated eight PhD students and is currently supervising eight graduate students.

10:00-10:30 Coffee Break

Session 1 Chair: John R. Mashey - Techviser
10:30-11:00 SPECweb2005 in the Real World: Using IIS and PHP
Samuel R. Warner, Intel Corporation
James S. Worley, IBM Corporation
Word document   Paper (.doc)
Powerpoint presentation   Slides (.ppt)
11:00-11:30 Working Differently with Parallel Workflows - The New Standard Workstation Benchmark
Frank Jensen - Intel Corporation
Word document   Paper (.doc)
Powerpoint presentation   Slides (.ppt)
11:30-12:00 Characterization of SPECpower_ssj2008 benchmark
Anil Kumar, Larry D. Gray, Harry H. Li, Christopher B. Jorgensen - Intel Corporation
PDF document   Paper (.pdf)
Powerpoint presentation   Slides (.ppt)
PDF document   Slides (.pdf)

12:00-01:30 Lunch

Invited Talks Chair: Rudi Eigenmann - Purdue University
01:30-02:30 Convergence of Emerging Workloads
Pradeep K. Dubey, Intel Corporation

Continued growth in compute power coupled with the growth in massive online data has enabled more pervasive use of effective modeling and simulation of events, objects, and concepts in real-time. We find such processing needs at the heart of many emerging applications and services. This processing can be decomposed three fundamental computing capabilities: recognition, mining, and synthesis (RMS). A set of RMS workloads is examined in terms of usage, mathematical models, numerical algorithms, and underlying data structures. Our analysis suggests a workload convergence that has implications in computing platforms. A diverse set of RMS applications is emerging from market segments such as graphics, gaming, media-mining, unstructured information management, financial analyses, and interactive virtual communities. These applications present a relatively focused and highly overlapping set of common platform challenges and an opportunity for an effective representation of these emerging workloads in future benchmark and performance suites.

Pradeep Dubey is a senior principal engineer and manager of Innovative Platform Architecture (IPA) in the Microprocessor Technology Lab, part of the Corporate Technology Group at Intel. His research focus is computer architectures to efficiently handle new application paradigms for the future computing environment. Dubey previously worked at IBM's T.J. Watson Research Center, and Broadcom Corporation. He was one of the principal architects of the AltiVec* multimedia extension to Power PC* architecture. He also worked on the design, architecture, and performance issues of various microprocessors, including Intel's i386TM, i486TM, and Pentium processors. He holds over 25 patents and has published extensively. Dr. Dubey received a BS in electronics and communication engineering from Birla Institute of Technology, India, an MSEE from the University of Massachusetts at Amherst, and a PhD in electrical engineering from Purdue University. He is a Fellow of IEEE.

02:30-03:00 Application-Based High Performance Computing System Performance Metrics
Larry P. Davis, Roy L. Campbell Jr., and William A. Ward Jr., Department of Defense
Powerpoint presentation   Slides (.ppt)

The Department of Defense High Performance Computing Modernization Program (HPCMP) has carefully prepared a comprehensive application benchmarking suite based on the workload of its user community. It has updated that application benchmarking suite annually over the past eight years to determine the performance of offered systems during the program's yearly acquisition of HPC capability. As part of that process, each HPC system in the program's current inventory is benchmarked with that year's suite, and an overall capability is determined for each system based on a weighted average over all test cases in the suite. The performance of the overall set of HPCMP systems is tracked over time as a program performance metric. The evolution of the application benchmarking suite and the historical track of HPCMP system performance using these benchmarked-based metrics will be discussed.

Larry P. Davis has more than 30 years experience in science and technology. He is one of the founders and currently the Deputy Director of the DoD High Performance Computing Modernization Program (HPCMP) in the Institute for Defense Analyses. Dr. Davis also is responsible for HPCMP's benchmarking and performance modeling activities. Davis has published over twenty refereed journal articles in addition to numerous technical reports.

03:00-03:30 Coffee Break

Session 2 Chair: Rema Hariharan - AMD
03:30-04:00 On the Object Orientedness of C++ Programs in SPEC CPU 2006
Ciji Isen, Lizy John, The University of Texas at Austin
PDF document   Paper (.pdf)
Powerpoint presentation   Slides (.ppt)
04:00-04:30 A Component-based Definition of Spatial Locality
Xiaoming Gu, Chen Ding - University of Rochester
Chengliang Zhang - Microsoft Corporation
PDF document   Paper (.pdf)
Powerpoint presentation   Slides (.ppt)
04:30-05:00 The Return of Synthetic Benchmarks
Ajay M. Joshi, Lieven Eeckhout, Lizy K. John - The University of Texas at Austin
PDF document   Paper (.pdf)
Powerpoint presentation   Slides (.ppt)

05:00-7:30 Social Gathering and Award Presentation