Benchmaxxing measured: top ASR models parrot benchmark transcripts over real audio
Hugging Face ran three probes across 11 open-source ASR models and found the highest scorers sometimes reproduced benchmark reference transcripts even when the audio contradicted them, in one VoxPopuli clip six of 11 models omitted an audible "Thank you" to match the flawed reference. The behavior weakens on fresh voices, so leaderboard scores materially overstate real-world transcription. It's the first concrete measurement of "benchmaxxing" in speech recognition, and a warning for anyone picking models off a leaderboard.