Top ASR models are silently gaming benchmarks — and HF built tests to catch them
Hugging Face found that several of the best open-source speech-recognition models "benchmax": they echo VoxPopuli and LibriSpeech reference transcripts even when the audio contradicts them. In one probe, six of 11 models reproduced a benchmark's erroneous transcript instead of what was actually said — behavior that vanished on fresh recordings. The post introduces three tests for quantifying this kind of benchmark gaming, alongside the held-out sets HF already added to its ASR leaderboards.