Word Error Rate Fell 3.5 Points When Someone Fixed the Transcript

The answer key is an input too

Word error rate is a subtraction. You take the model output, you take the reference transcript, you count the edits between them. Both halves are inputs to the score. In practice only one of them is ever audited.

Artificial Analysis re-transcribed 628 English clips from VoxPopuli by hand and published the result as VoxPopuli-Cleaned-AA under Apache-2.0: roughly 120 minutes of European Parliament audio in clips of 5 to 38 seconds, corrected for wrong words, missed words and partial stutters. The dataset card reports that average model word error rate on VoxPopuli fell by 3.5 percentage points after the cleaning. No weights changed. The scores moved because the answer key moved.

That is the pattern. Its second half costs more: while the answer key was wrong, the models scoring best were the ones that had learned to reproduce the wrong answer.

Today's instance

A paper from Hume AI, posted to arXiv on 20 August and written up on the Hugging Face blog the day after, puts numbers on that second half across eleven open speech models, Whisper large-v3, Canary-Qwen 2.5b, Parakeet TDT 0.6b v2, Voxtral Mini 3B and Kimi-Audio 7B Instruct among them.

Three probes, each built on audio that does not fully determine the transcript.

Reference disagreement. A phoneme-error-rate ensemble flags spans where the reference and the audio disagree. Forty percent of VoxPopuli test clips are flagged, covering about 3% of all reference words. Models showing the behavior reproduce the erroneous span 18 to 30% of the time. On one clip where the benchmark transcript drops a spoken greeting, six of the eleven models drop it too.

Masked-number recovery. Silence a number in the audio and a faithful model has nothing to emit. Some of the strongest benchmark performers emit it anyway on roughly 30 to 40% of LibriSpeech examples, and recovery runs highest on the public benchmarks, lower on newly collected audio.

Orthographic switching. VoxPopuli abbreviates a title where LibriSpeech spells it out, and the two are acoustically identical. Several models beat the 50% coin-flip baseline at matching whichever convention the current dataset uses, some reaching around 90%.

The held-out sets are the load-bearing part. ep-fresh is European Parliament audio recorded after the models' training cutoffs; libri-fresh is LibriVox material from newly active narrators. Same domains, same acoustics, no reference anywhere in training. On the greeting clip, all but one model transcribes what was actually said once the audio is fresh. The paper also reports the behavior can be pushed both directions by editing context: trimming the surrounding benchmark audio or appending ordinary conversational audio moves models back toward the audio, and appending VoxPopuli audio moves them away from it.

That is the falsifiable version of the claim. If a model's advantage on a public benchmark were capability, it would survive fresh audio from the same domain. Show me a model that keeps its margin on ep-fresh and I will accept that its VoxPopuli number was real.

The instrument has a boundary, and the code says where

The scorers now live in the Open ASR Leaderboard repository under benchmark_fitting, and reading them is more useful than reading the write-up. Section 1 scores every disagreement span as ref, consensus or excluded, and reports rate = ref / (ref + consensus).

Look at what excluded covers. A span is dropped when the two references differ only by a normalization artifact, when the model reproduces fewer than floor(0.5 x reference words) of the span, when the hypothesis runs past three times the reference length, or when the model produces a third reading matching neither. Every one of those is a way of being bad at transcription.

So the denominator holds only models that already got the surrounding words right. The metric is conditional on competence, which means it cannot rank a weak model as honest, only decline to score it. A model that garbles the clip does not show up as faithful; it disappears. Treat a low rate from a low-accuracy model as missing data.

The README states its own limits: the rates cover disagreement spans rather than whole datasets, small differences should not be read as rankings, VoxPopuli is currently the only leaderboard dataset carrying the reference pairing the method needs, and agreement is not proof of where it came from. The Hume repository shows the same thing in its own quickstart, which runs five models over 150 VoxPopuli clips and 31 detected discrepancies, and the reference-error acceptance rate comes out at 0.000 for every one of them. The effect is real and it is not everywhere. That repository is eight commits old with six stars: a reference implementation, not a dependency.

Why teams keep walking into this

Your internal eval set has the property VoxPopuli had. Someone labeled it once, from a sample of production traffic, and it has not been re-labeled since. Every model you have shipped after that was selected, tuned and sometimes prompted against it. The set stopped being a measurement and became a target, and the errors inside it are part of the target.

This is the mechanism under a good share of why enterprise AI doesn't ship: the number that justified the project measured agreement with a frozen artifact rather than behavior on new inputs, and nobody found out until the inputs were new. It is the same shape, with a larger bill, behind most of what goes wrong between a proof of concept and production.

The cheapest way out

Three moves, in ascending cost.

Hold back a fresh slice. Not a random split of the same labeling batch, which shares that batch's errors: new data, collected after the last time you tuned anything, labeled by someone who has not seen the outputs. ep-fresh and libri-fresh are exactly this, built from parliamentary sessions and narrator recordings that postdate the models.

Set the gap threshold before you look. Decide now what difference between the golden set and the fresh slice you would call fitting rather than noise. On VoxPopuli the reference alone was worth 3.5 points of word error rate, so a gap that size on your own data is not sampling variance.

Score the disagreements, not the average. If you keep both an original reference and a correction, you can compute the same ratio the leaderboard now publishes:

pip install -r requirements/requirements_jobs.txt
python benchmark_fitting/score_voxpopuli_ref_errors.py --preds_dir results

Read n_eligible before you read rate. A ratio computed over a handful of spans is not a finding, and given those exclusion rules it is a handful more often than you would guess.