Benchmark (computing)

Lore & Background

As computer architecture advanced, it became more difficult to compare the performance of various computer systems simply by looking at their specifications. Therefore, tests were developed that allowed comparison of different architectures. For example, Pentium 4 processors generally operated at a higher clock frequency than Athlon XP or PowerPC processors, which did not necessarily translate to more computational power; a processor with a slower clock frequency might perform as well as or even better than a processor operating at a higher frequency. This gave rise to concepts like BogoMips and the megahertz myth.

Benchmarks are designed to mimic a particular type of workload on a component or system. Synthetic benchmarks do this by specially created programs that impose the workload on the component, while application benchmarks run real-world programs on the system. Application benchmarks usually give a much better measure of real-world performance, but synthetic benchmarks are useful for testing individual components like a hard disk or networking device. Since around 1995, the SPEC collection of benchmarks have become widely used.

Computer companies are known to tune their systems to improve performance on benchmark tests that are not representative of real world usage. Benchmarks have been improved to more closely mimic real world usage, so that any tuning carries over to application performance. Software vendors also use benchmarks in their marketing, such as the 'benchmark wars' between rival relational database makers in the 1980s and 1990s. Companies commonly report only those benchmarks that show their products in the best light, and have been known to mis-represent the significance of benchmarks.

Reader's Guide

Benchmarking is not easy and often involves several iterative rounds to arrive at predictable, useful conclusions. Interpretation of benchmarking data is extraordinarily difficult. Vendors tend to tune their products specifically for industry-standard benchmarks, and some have been accused of 'cheating' by designing systems that give much higher benchmark numbers but are not as effective at actual likely workloads. Many benchmarks focus entirely on speed of computational performance, neglecting other important features such as security, availability, reliability, execution integrity, serviceability, scalability, and total cost of ownership. The Transaction Processing Performance Council Benchmark specifications partially address these concerns by specifying ACID property tests, database scalability rules, and service level requirements, as well as requiring a price/performance metric using a simplified TCO formula. However, vendors have been known to price specifically for the benchmark, designing a highly specific 'benchmark special' configuration with an artificially low price. Benchmarks also tend to ignore requirements for development, test, and disaster recovery computing capacity, and have trouble adapting to widely distributed servers and grid computing. Users often have different perceptions of performance than benchmarks suggest, valuing predictability and low worst-case response times over mean scores. Many server architectures degrade dramatically at high levels of usage, yet vendors tend to publish benchmarks at about 80% usage without documenting what happens when demand spikes. In Software Engineering Research, benchmarking is recognized as one of the main empirical methods, used to evaluate new methods and tools and compare them against the state of the art.

Did You Know?

• Benchmarking as part of continuous integration is called Continuous Benchmarking.
• The SPEC collection of benchmarks became widely used around 1995.
• Some vendors have been accused of 'cheating' at benchmarks by designing systems that give higher numbers but are less effective at actual workloads.