Open ASR Leaderboard on Hugging Face: Ranks 1 and 2 Swap on a Common Mean, and 8 of 65 Rows Skip TED-LIUM

TL;DR

• On November 7, 2026 the Hugging Face Open ASR Leaderboard board lists 65 rows from 65 repos, all lower-is-better word error rate (WER), 0 verified. 45 rows come from pull requests that are still open, all opened by one account, and 20 sit on a repo's main branch.
• The board's mean_wer is not the same average for every row. 54 rows average 8 datasets. 8 rows, including ranks 2, 3 and 4, have no TED-LIUM score and average 7. TED-LIUM is the easiest column for all 54 rows that report it, so leaving it out raises a row's mean by about 0.3 to 0.6 points.
• On the 7 datasets every row has, ranks 1 and 2 swap. The board gives 4.76 to ARK-ASR-3B and 4.87 to MOSS-Transcribe-preview-2B. On the common 7 the numbers are 5.04 and 4.87.
• The board's rank 1 is first in only 2 of 8 columns, and five different checkpoints lead the eight columns. Ranks 3 and 4 are one measurement stored twice.
• One row's mean does not match its own parts (7.12 listed, 6.94 recomputed), the rank 6 file sits in a gated repo I could not open, and two rows carry only a mean.

Blue bars are the numbers on the board. Orange bars are my recomputation over AMI, Earnings-22, GigaSpeech, LibriSpeech clean, LibriSpeech other, SPGISpeech and VoxPopuli, the seven columns that all 62 rows with per-dataset values share. For the rows that already average those seven, the bars are equal. The axis starts at 4, so lengths exaggerate the gaps; read the numbers.

What does the Open ASR board on Hugging Face contain?

The Open ASR Leaderboard scores speech recognition models on a fixed set of public English test sets and reports word error rate, the share of reference words a transcript gets wrong. Hugging Face lists hf-audio/open-asr-leaderboard as a benchmark dataset, so a model repo can publish results in .eval_results/*.yaml and the board collects them.

I pulled the board on November 7, 2026 (KST) from the JSON behind the leaderboard view. It has 65 rows, one per repo, and verified is false on every one. Source labels: 61 rows say open-asr-leaderboard, two say moondream, and two cite their own model card.

The dataset's eval.yaml names 12 task ids: mean_wer, rtfx, and ten WER sets (LibriSpeech clean and other, Common Voice, VoxPopuli, TED-LIUM, GigaSpeech, SPGISpeech, Earnings-22, AMI, and a cleaned VoxPopuli). It does not say which of those feed mean_wer, and it does not say that all of them must be present. No readable file reports Common Voice or the cleaned VoxPopuli, so the working definition is a mean over up to eight sets that nobody wrote down in one place.

Where do the 65 rows come from? 45 are open pull requests, every one opened by the same account (SaylorTwift). 43 of those were opened in April 2026 and two on October 28. The other 20 are YAML files already on the model repo's main branch. A row from an open pull request is a claim that the repo owner has not yet accepted, and the board shows it next to accepted ones with no marker.

Is every mean_wer an average of the same datasets?

No. I fetched the YAML behind every row, from the pull request ref when the row has one and from main otherwise. 64 of 65 files opened. The rank 6 file is in a gated repo and answered HTTP 401 without accepting its terms, so I could not check it. Of the 64:

The eight rows without TED-LIUM sit at ranks 2, 3, 4, 8, 20, 23, 45 and 48. Their mean_wer is the average of the seven they have. That is a fair way to report a model, but it is not the number the other 54 rows carry, and the board sorts them in one column.

Does the difference matter? TED-LIUM is the easiest column. For every one of the 54 rows that report it, the TED-LIUM WER is lower than that row's own mean over the other seven. The median TED-LIUM score is 4.1. Adding an easy set pulls a mean down, so a row that skips it starts about 0.56 points behind at the median, and about 0.3 to 0.5 points behind among the top rows. The rank 1 row moves from 5.04 to 4.76 when its TED-LIUM score (2.79) joins the average.

moondream's own file header makes the same point from the other side. It says its card figure of 5.80 "is the mean of the other seven" and the board value of 5.32 averages all eight. My recomputation of that row gives 5.803 over seven, which matches its note.

Do ranks 1 and 2 survive a common mean?

Not on the numbers I can compute. The board's top two:

On the board, rank 1 leads by 0.11. On the seven columns both rows have, rank 2 leads by 0.18. Across all 62 rows with per-dataset values, the common-7 order starts with the MOSS preview model, then ARK-ASR-3B, then the MOSS Diarize pair, then Zipformer-cr-ctc-transducer-XL-290M, then moondream's parakeet-ultra. The board has parakeet-ultra at rank 5 and the Zipformer at rank 8, so those two swap as well.

I do not claim rank 2 is the better model. Here is the fair reading. If the MOSS preview model had a TED-LIUM score of at most 4.04, its 8-set mean would stay below ARK-ASR-3B's 4.76. The rows ranked 1, 5, 7 and 9 report TED-LIUM scores of 2.79, 1.94, 2.45 and 2.28, so a model this strong would probably clear that bar and also be first on eight sets. But that score is not on the board, and the board's order of the top two comes from comparing a 7-set mean with an 8-set mean. So the board cannot say which is first.

Who actually leads each column?

The board's rank 1 is a good model by any column. It is just not first everywhere. Among the 62 rows with per-dataset values, ARK-ASR-3B ranks 4th on AMI, 3rd on Earnings-22, 2nd on GigaSpeech, 1st on both LibriSpeech sets, 14th on SPGISpeech, and 6th on TED-LIUM and VoxPopuli. Its SPGISpeech WER is 2.46 against a best of 1.63.

Five different checkpoints lead the eight columns:

A single mean hides this, which is its job. Ranks 1 to 9 on the board fit inside 1.0 point, 4.76 to 5.76, while the per-column leaders are spread across that whole group. Which model to pick depends on whether your audio looks like meetings (AMI), earnings calls, read audiobooks or parliamentary speech. The board sorts by mean_wer; the per-dataset columns you would need are only in each repo's YAML file.

The board publishes no per-utterance output, so I cannot put an interval on any of these gaps. I can say that TED-LIUM has 1,155 test utterances and VoxPopuli 1,842, so the smaller columns are the noisier ones. A 0.11 gap in a mean of eight is a small thing to hang a rank on.

Why are ranks 3 and 4 the same row twice?

OpenMOSS-Team/MOSS-Transcribe-Diarize and hf-audio/MOSS-Transcribe-Diarize-HF have the same parameter count (908,513,280), the same date (2026-07-25), and identical values in every field of their YAML files: the mean, RTFx and each per-dataset WER. The second repo sits under the hf-audio organization, the same one that hosts the dataset, and looks like a converted copy of the first. Publishing a copy is fair. On the board it is two rows, and it moves everything below it down one place. If you read "top 5" as five models, you are reading four.

What do self-reported rows change?

Two rows come from moondream (ranks 5 and 12), and they say so. Their YAML header reads "Self-reported", says the scores use the Open ASR Leaderboard's own pipeline with its normalizers as of September 2026, and carries a second note about TED-LIUM: the leaderboard's TED-LIUM split "is no longer distributed", so tedlium_wer scores the eleven TED-LIUM 3 test talks whole. That is a different test from the leaderboard's own TED-LIUM split, which the dataset card lists at 1,155 utterances and which should sit behind the other 52 TED-LIUM values. So the eighth column of those two rows measures something else, and the column that is easiest for every row is exactly the one that differs.

The header discloses this well, yet the value still lands as a plain number in a column that sorts it against the others.

The two LiquidAI rows (ranks 13 and 15) are the opposite case. Each file holds a single mean_wer (6.23 and 6.36), sourced from the model card. The card describes it as the average WER "on the ESB / Open ASR benchmark suite" and does not list the datasets. There are no per-dataset values to check, and no RTFx.

Does the mean match its parts, and is the speed column usable?

For the 62 rows with per-dataset values, I recomputed the mean from the parts. 61 match their own mean_wer to within 0.011. One does not: nvidia/canary-180m-flash lists 7.12, and its eight values average 6.94. If the parts are right, that row belongs at rank 21 instead of 25. I do not know which number is stale. The pull request that added it has been open since April 7.

The board also carries RTFx, the real-time factor, for 59 rows. It ranges from 42 to 6,399. Four rows are on the accuracy and speed frontier, meaning no other row is both faster and lower in WER: ARK-ASR-3B (4.76 at 491), parakeet-tdt-0.6b-v2 (6.05 at 3,386), parakeet-tdt_ctc-110m (7.49 at 5,345) and stt_en_fastconformer_ctc_large (8.97 at 6,399). Moving from rank 1 to parakeet-tdt-0.6b-v2 costs 1.29 points of WER and buys a 6.9 times faster row.

Neither the YAML files I read nor the dataset's README says what hardware RTFx was measured on, and five of the 64 readable files have no RTFx at all. A speed column without a stated machine is a hint, not a measurement.

How to reproduce this audit

The script below pulls the board, reads each row's file, and recomputes both the common-7 mean and the all-sets mean. It skips gated repos and mean-only rows and prints the five best rows on the common 7 plus any row whose mean disagrees with its parts. I ran it on November 7, 2026, and it printed the MOSS preview model first at 4.866, ARK-ASR-3B second at 5.044, and canary-180m-flash as the one mismatch.

import json, urllib.request, yaml def get(url): req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"}) return urllib.request.urlopen(req, timeout=120).read().decode() SEVEN = ["ami", "earnings22", "gigaspeech", "librispeech_clean", "librispeech_other", "spgispeech", "voxpopuli"] board = json.loads(get("https://huggingface.co/api/datasets/hf-audio/open-asr-leaderboard/leaderboard")) rows = [] for r in board: pr = r.get("pullRequest") ref = f"refs%2Fpr%2F{pr}" if pr else "main" try: text = get(f"https://huggingface.co/{r['modelId']}/raw/{ref}/{r['filename']}") except Exception: continue # gated repos answer 401  v = {e["dataset"]["task_id"]: e["value"] for e in yaml.safe_load(text)} if not all(f"{k}_wer" in v for k in SEVEN): continue # mean-only rows cannot be checked  m7 = sum(v[f"{k}_wer"] for k in SEVEN) / 7 sets = [k for k in v if k.endswith("_wer") and k != "mean_wer"] m_all = sum(v[k] for k in sets) / len(sets) rows.append((r["rank"], r["modelId"], r["value"], round(m_all, 2), round(m7, 3), len(sets))) print("board rank | repo | board mean | recomputed | common-7 | sets") for x in sorted(rows, key=lambda x: x[4])[:5]: print(x) print("mean does not match its parts:", [x[:4] for x in rows if abs(x[2] - x[3]) > 0.011]) 

What should a reader do with this board?

A few habits that cost nothing:

1. Read the number of sets next to the mean. A mean over 7 and a mean over 8 are different measurements, and on this board the gap is larger than the gap between several neighbouring ranks.
2. Sort by the column you care about. Meetings, earnings calls, read speech and parliamentary speech rank the same models differently.
3. Treat a tie as one measurement. Identical values under two repo names are a copy, not a replication.
4. Check who opened the pull request and whether it is still open. 45 of 65 rows have not been accepted by the repo owner.
5. Ask what RTFx was measured on before you trade accuracy for speed.

For the board itself, three small changes would remove most of the ambiguity: define which tasks feed mean_wer in eval.yaml, show the number of sets beside the mean, and mark rows that are not comparable on a column, such as a different TED-LIUM split.

FAQ

What is the Open ASR Leaderboard on Hugging Face?
It is a benchmark dataset page that collects word error rate results for speech recognition models on public English test sets such as LibriSpeech, AMI, GigaSpeech and TED-LIUM. Model repos publish results as YAML files and the board lists them.

Is the board's mean_wer the average of the same datasets for every model?
Not on November 7, 2026. 54 rows average eight datasets, and eight rows average seven because they have no TED-LIUM score. Two more rows list only a mean.

Which model is first on the Open ASR board?
The board lists Edge0/ARK-ASR-3B first with 4.76. On the seven datasets all rows share, OpenMOSS-Team/MOSS-Transcribe-preview-2B is lower at 4.866 against 5.044. The board cannot settle the order without the MOSS model's TED-LIUM score.

Why does leaving out TED-LIUM change a mean?
TED-LIUM is the easiest column. For all 54 rows that report it, its WER is lower than the row's own mean over the other seven sets. A mean without it is higher by about 0.3 to 0.6 points.

Does verified mean the scores were reproduced?
On this board, verified is false on all 65 rows. The scores are whatever each YAML file says, whether it arrived through a pull request or a commit on the repo's main branch.

Can I trust the RTFx column?
Use it as a rough guide. The files do not state the hardware, and five of the 64 readable files have no RTFx.

Further reading

• How Hugging Face Official Benchmark Leaderboards Actually Work: .eval_results YAML, the base_model Field and More
• When a 0.4-Point Lead Means Nothing: Reading Open-Model Leaderboard Margins Like a Statistician

Data pulled November 7, 2026 (KST) from the board JSON and the YAML files named above. All numbers are the files' own values or my arithmetic on them. The common-7 mean is my construction, not a leaderboard metric.