Researchers at HumeAI introduced three probes to measure benchmark optimization in speech recognition and applied them to 11 widely used open-source ASR models. The team argues that public benchmarks can be exploited by models to improve scores without improving faithful transcription, so they added held-out sets to Real World VoiceEQ, the Open-ASR Leaderboard and the Far-field ASR Leaderboard to capture more realistic behavior.
Reference disagreement (VoxPopuli case study)
The researchers used a consensus disagreement probe on VoxPopuli, where many reference transcripts contain errors. An ensemble selected for low phoneme error rate (PER) flagged cases where model outputs unanimously disagreed with the benchmark reference. Human validation showed models sometimes reproduced the erroneous reference rather than the audible words. For example, one clip audibly contains “Thank you, Mr. President,” but the VoxPopuli reference omits “Thank you.” Across the 11 models tested — including openai/whisper-large-v3, nvidia/parakeet-tdt-0.6b-v2, microsoft/Phi-4-multimodal-instruct and others — several reproduced the benchmark omission on the original recording but returned an audio-faithful transcript on freshly recorded or generic voices. The team reports that their methodology flagged potential reference errors in about 40% of analyzed VoxPopuli test clips, affecting roughly 3% of reference words, and that models reproduced erroneous references 18–30% of the time.
Masked numbers and orthographic switching
In a masked-entity probe, numbers were silenced in audio and models were asked to transcribe what they heard. Some high-performing models emitted the removed numbers anyway; on LibriSpeech, top benchmark models reproduced masked numbers in roughly 30–40% of tested examples. The researchers note recovery rates were higher on public benchmarks and lower on held-out or newly collected audio (ep-fresh and libri-fresh).
The orthographic switching probe tested whether models adopt the spelling conventions of each benchmark (e.g., “Mr.” vs “Mister,” “any one” vs “anyone”). Several models exceeded a 50% random-choice baseline and some reached approximately 90% switch accuracy, suggesting the models identify dataset membership from acoustic context and then select the expected orthography.
The authors report that interventions such as resynthesizing audio in generic voices, trimming surrounding context, or appending conversational audio often restore audio-faithful transcriptions. Conversely, appending VoxPopuli audio can make synthetic samples more likely to match the benchmark reference.
The study concludes that models may use dataset-associated acoustic cues to choose between following the audio or reproducing benchmark-specific transcripts. To support evaluation beyond single benchmarks, a “Benchmark fitting” tab was added to the Open ASR Leaderboard and the relevant scripts and un-normalized model outputs are open-sourced on GitHub.
Original source: Hugging Face Blog