SPECmail2001 Frequently Asked Questions
Q1: What is SPECmail2001?
A1: SPECmail2001 is an industry standard benchmark designed to measure a systemís ability to act as a mail server compliant with the Internet standards Simple Mail Transfer Protocol (SMTP) and Post Office Protocol ĖVersion 3 (POP3). The benchmark models consumer users of an Internet Service Provider (ISP) by simulating a real world workload. The goal of SPECmail2001 is to enable objective comparisons of mail server products.
Q2: What is the user profile for SPECmail2001?
A2: A consortium of experts within SPEC has defined the following user profile:
Q3: What is the performance metric for SPECmail2001?
A3: SPECmail2001 expresses performance in terms of SPECmail2001 messages per minute (MPM). For example:
Q4: Where can I find published results for SPECmail2001?
A4: SPECmail2001 results are available via SPECís Web site: http://www.spec.org/.
Q5: Who developed SPECmail2001?
A5: SPECmail2001 was developed by the Standard Performance Evaluation Corporation (SPEC), a consortium of leading hardware and software vendors and university representatives. The Mail Server subcommittee within SPEC included participants from major mail server vendors.
Q6: What are the limitations of SPECmail2001?
A6: The first release of SPECmail2001 does not support Internet Message Access Protocol (IMAP) or Webmail (email accessible via a browser). The plan is to include IMAP in a future SPEC mail server benchmark. Currently, there are no plans to address Webmail.
Q7: Why are message attachments not measured in SPECmail2001?
A7: While ISP POP consumers do receive message attachments (i.e. MIME, Multipurpose Internet Mail Extensions), these attachments are a client-side issue under the POP3 and SMTP protocols. Since the email client is responsible for parsing MIME attachments within a message, there is no impact on the mail server except for message size. SPECmail2001's message size distribution includes a consideration for attachments.
Q8: Can I use SPECmail2001 to determine the size of the mail server I need?
A8: SPECmail2001 cannot be used to size a mail server configuration, because it is based on a specific workload. There are numerous assumptions made about the workload, which may or may not apply to other user models. SPECmail2001 is a tool that provides a level playing field for comparing mail server products that are targeted for an ISP POP consumer environment. Expert users of the tool can use the benchmark for internal stress testing, with the understanding that the test results are for internal use only.
Q9: How do I obtain the benchmark?
A9: It can be purchased on CD-ROM. A discount price is available for universities and other non-profit organizations. To place an order, please contact SPEC at: http://www.spec.org/spec/contact.html
Q10: What is contained on the SPECmail2001 CD?
A10: The SPECmail2001 CD contains: Benchmark software including Java source code, documentation, and configuration files.
Q11: What additional software is required to run SPECmail2001?
A11: SPECmail2001 requires a mail server that supports POP3 and SMTP. A Java Runtime Environment (JRE) version 1.1.8 or later must be installed on the client machines.
Q12: What hardware is required to run the benchmark?
A12: In addition to the hardware for the system under test, one or more client machines are required. The number and size of client machines required by the benchmark will depend on the size of the workload.
Q13: What if I have questions about running the SPECmail2001 benchmark?
A13: Refer to the Trouble-Shooting section of SPECmail2001 User Guide or send questions to firstname.lastname@example.org.
Q14: How can I submit SPECmail2001 results?
A14: SPEC members can submit for free and non-members pay a fee. For more information about submitting results, please contact SPEC.
Q15: Where can I go for more information?
A15: SPECmail2001 documentation consists of four documents: User Guide, Run Rules, Architecture White Paper and this FAQ. The documents can be found on SPECís Web site: http://www.spec.org/.
Copyright © 2001 Standard Performance Evaluation Corporation