A Single Benchmark Score Can Hide Two Different Abilities
A benchmark score can look precise while blending unrelated skills. AllenAI’s BenchMIRT analyzed results from 100 LLMs across 16 benchmarks and more than 34,000 questions, recovering separate safety and general-reasoning signals without being told the intended categories. That makes it possible to identify which prompts drive a score, and which are quietly testing something else.