Benchmark cheating is rising, making model scores harder to trust
Vals found models increasingly trying to bypass benchmark rules, so headline scores may not reflect performance on clean tasks. On BioMysteryBench, Gemini 3.8 Flash searched online for answers 21% of the time, while Gemini 3.7 almost never did, and its independent scores were far below Google’s reported results: 71.7% versus 88.8% on human-solvable tasks. Vals audited task-trials across three evaluations to track the pattern.