SPEC says it will no longer be publishing SPEC CPU 2017 results for Intel CPUs running a specific version of the Intel compiler, citing displeasure over an apparent targeted optimization for a specific workload (via ServeTheHome and Phoronix ) that essentially amounts to cheating. A note has been added to the more than 2,600 benchmark results published with the offending compiler, effectively invalidating those results, mostly from machines running 4th Gen Xeon Sapphire Rapids CPUs.
SPEC CPU 2017 is a benchmark mostly used for high-end servers, data centers, and workstations/PCs, and it tests performance in various workloads in a standardized way so that different computers can be compared to each other. Good performance in SPEC CPU 2017 hinges not just on hardware but also on software. One of the key factors in software-side optimization is the compiler, which is a program that basically takes written code and reformats it in a way that a processor can run it best.
The disclaimer that it is now attached to over 2,600 SPEC CPU 2017 results states, "The compiler used for this result was performing a compilation that specifically improves the performance of the 523.xalancbmk_r / 623.xalancbmk_s benchmarks using a priori knowledge." This means the compiler (in this case, Intel's oneAPI DPC++/C++ Compiler) was not optimized for a particular kind of workload, or even for specific applications, but specifically for two SPEC CPU 2017 benchmarks.
While it's expected that compilers should be optimized since more performance is obviously better, optimizing specifically for benchmarks is controversial and frowned upon. SPEC wants its benchmarks to reflect the real-world performance of hardware and to provide a standardized way to compare different processors. But if a compiler optimization only improves performance in a particular benchmark and not in a real-world scenario, then that's clearly not reflective of the real world and will only be reflected in that specific benchmark.
According to Phoronix , the optimization could boost performance in SPECint by 9% overall. The publication also notes that versions 2022.0 to 2023.0 of the Intel oneAPI Compiler are impacted, meaning most of the now-invalidated results were run in 2022, largely on Sapphire Rapids CPUs. Results for fifth-gen Xeon Emerald Rapids CPUs are very unlikely to have been running a version of the compiler with the banned optimization since Emerald Rapids came out after the good versions of the compiler were available.
Benchmark-specific optimization has been a hot topic for years. Back in 2003, Nvidia was accused of performing a driver-side optimization to boost the performance of its GPUs in 3DMark 2003. In 2010, Nvidia itself alleged that AMD was cheating in actual games by not enabling certain driver-side settings that would have significantly boosted visual quality at the expense of performance. Accusations these days don't get quite as heated, though SPEC has certainly shamed Intel in this case.
