Towards a Useful Synthetic Graphics Benchmark
By Loyd Case on August 18, 2003 at 1:05 pm
The programmability of graphics hardware is becoming more and more sophisticated over time, to the point where they almost look like general array processors, rather than fixed-function parts. We need to start thinking of these pieces of silicon in a different way.
The SPEC Example
Currently, one of the more respected synthetic benchmark suites for processors is the SPEC benchmarks. The SPEC tests are divided up into SPECfp and SPECint (floating point and integer). Further granularity is achieved by dividing up the tests into base and peak scores. The SPEC rate scores exercise throughput rather than raw computation.
The concept of base and peak are quite interesting. The base scores are those benchmarks generated by fixed source code that cannot be altered by the CPU maker. Here are the definitions of the SPECint and SPEC_int_base benchmarks from the SPEC FAQ for their CPU2000 benchmark suite.
"SPECint2000: The geometric mean of twelve normalized ratios (one for each integer benchmark) when compiled with aggressive optimization for each benchmark."
"SPECint_base2000: The geometric mean of twelve normalized ratios when compiled with conservative optimization for each benchmark."
Note that the standard SPECint2000 score represents a score generated by source code compiled with "aggressive" optimizations, while SPECint_base2000 scores are generated by code compiled with "conservative" optimizations.
We're now at the stage where DirectX shaders are beginning to stabilize, and OpenGL 1.5 has started to define the pixel and vertex shaders that will ultimately comprise OpenGL 2.0. Now is a good time for the industry to be thinking about synthetic benchmarks that are well-thought out, in the manner of the SPEC benchmark suites.
Here's my modest proposal for a good synthetic benchmark suite for programmable graphics hardware:
Have separate suites for pixel and vertex shaders. I realize this isn't easy (many pixel shader operations are often made possible by earlier vertex shader operations).
The above benchmarks should reflect common operations used by programmers, but each test doesn't have to look like it's generated from game content. A good example of how this sort of thing was done in the past was the 3D Winbench inspection tests.
Have each suite be run twice: one without optimizations, and one with optimizations. In the "aggressive" case, any optimization would be allowed as long as the numerical result at the end was unchanged. This can also be somewhat challenging, given differences in floating-point precision used by different graphics parts.
The "conservative" case should consist of code that's digitally signed or otherwise encrypted in a manner that prevents in-place optimization.
I realize this is no easy task, and may require *gasp* co-operation between competitors. But the SPEC model could work well here . The other main problem is that shader languages are still evolving, so any benchmark will only be a snapshot in time of capabilities. But given the value that OEM hardware manufacturers and end users place on benchmarks, this type of synthetic benchmark would go a long way towards creating confidence in the whole benchmarking process. It would also be a terrific supplement to applications tests.
It's likely that neither Nvidia nor ATI would be terribly excited by this type of testing. Each prefers reviewers to use "real games" – by which each company means "titles optimized for our product". But an organization like SPEC – comprised of some of their biggest customers – should have the leverage to get the companies to cooperate with the process. The initial definition of the benchmark suites would no doubt be arduous and contentious. In the end, though, the industry and customers would benefit from an open process for benchmarking similar to that used today by SPEC. Such a consortium could even hand the actual development task to people like Futuremark, who already has a development team in place. Since all the major players would be involved, policing potential cheating becomes easier, too.