Saturated benchmarks still reveal capability through model transcripts
When benchmark scores hit near-perfect levels, evaluators can still compare models by analyzing the transcripts behind their correct answers, rather than throwing out the benchmark. In a retrospective simulation, pairwise judging of those transcripts reduced prediction error 5.3 times versus using saturated scores alone, with useful signals concentrated in reasoning and tool-call sections. The method can fail under distribution shifts, including early reasoning models introduced in 2025.