It is sometimes all too easy to focus too much on the results of a particular benchmark and let this "tunnel vision" take over all performance work.

It is very important that a wide selection of standard industry benchmarks exists, as they will be used by anyone from hardware vendors to undergraduate researchers, for ultimately increasing performance for various runtimes and applications. Conclusions drawn, given a benchmark setup, can have wide-ranging implications.

One danger is, of course, if a mainstream benchmark gets too well adopted. Classic examples here are the SPECjvm benchmark suite and later the SPECjbb benchmark.

To put it bluntly, if a graduate student can run a benchmark from his workstation with a simple command line, the problems it addresses will get much love and research. If not, they won't. SPECjbb has a simple command line, SPECjAppServer doesn't. SPECjAppServer, in itself, is an excellent benchmark that pretty much stresses any desired portion...