Verified Is Not Reproduced

Hugging Face published results on 13 August from a community challenge that tried to reproduce a scientific conference. Between 15 July and 2 August, 1,221 participants published 6,816 reproduction logbooks covering 2,226 of the 6,341 accepted ICML 2026 papers, and an automated judge issued verdicts on 35,908 extracted claims. ICML 2026 itself ran 6 to 11 July in Seoul, so the challenge began four days after the conference closed.

The headline numbers: 51% of attempted papers, 1,103 of them, had at least one claim independently verified. 23%, or 496, had at least one claim falsified or contested.

Both figures are real and neither is the one to carry into a design review. That number is 266.

What a percentage counts

266 papers were fully reproduced — every extracted claim verified. Another 632 had partial verification with no falsification. So of 2,226 papers that somebody actually attempted, 266 came out the other side with nothing outstanding, roughly one in eight.

The gap between 1,103 and 266 is the gap between a paper having a claim that survived and a paper surviving. A paper enters the 51% by having one claim confirmed while the rest sit unresolved. If you are reading a paper to decide whether to build on its method, the claim you care about is the specific one your design depends on, and the aggregate says nothing about whether that one was among the 3,978 confirmed.

The redundancy in the effort makes this concrete. 6,816 logbooks against 2,226 papers means the average paper was attempted more than three times, and 242 papers ended with independent teams reaching opposite verdicts. The verdicts dataset is keyed per logbook, not per paper, for exactly that reason — a paper does not have a verdict, it has a set of them.

Where the measurement stops

502 papers produced toy-scale evidence only. Another 280 established nothing either way. That is not a property of those papers. It is the shape of the budget.

The challenge dataset card tells participating agents to run at least one scaled experiment on a Hugging Face GPU Job and to record the job URL, GPU type, configuration, and the scale relative to the paper. It also states the cost model plainly: the job timeout is the hard cost cap, with maximum cost equal to timeout multiplied by the hardware flavor rate, across flavors from CPU up through A10G, A100 and H200. Participants received $20 in compute credits, and 2,962 jobs were launched in total.

So the measurement has a boundary and the boundary is priced. Above it — claims checkable inside a timeout somebody was willing to pay for — you get verified or falsified. Below it you get toy-scale or inconclusive. A claim that only appears at full pretraining scale is not reachable inside that cap, and it does not surface as a warning either. It surfaces as one of the 502.

This is the same failure mode that eats AI projects internally: a result that holds at demo scale is treated as a result, and the cost of establishing it at real scale never gets scoped. The literature now has its own version of it, at a measured 502 papers. It is the reason the production phase deserves its own budget rather than being treated as a formality after the prototype works.

The direction of the error

2,226 of 6,341 papers were attempted. The other 4,115 have no verdict at all, and whatever selected the 2,226, it was not randomisation.

Here is the falsifiable part. I expect the full-verification rate across those 4,115 to come out at or below the 266-in-2,226 observed here, not above it, because the papers a volunteer can finish inside a timeout are the papers with released code, small models, and cheap claims — the conditions that also make a paper likelier to hold up. Draw a random sample from the un-attempted set, run it under the same protocol, and if the fully-verified fraction lands at or above one in eight, this reading is wrong and the 51% is closer to an estimate than a ceiling.

One more caveat the write-up is honest about and a summary would drop: the judge is a model. GLM-5.2 assigned every verdict by reading logbooks that agents wrote, under an instruction to treat each logbook's self-assessment as untrusted. That instruction is correct and it does not convert the output into a measurement. A falsified verdict here is one model's reading of another model's write-up of an experiment a person steered. The report's own conclusion is that pure agent execution hit real limits and that the results it trusts most came from runs where a human was steering.

Looking your paper up

The write-up says the verdicts were frozen in a public dataset. They were. The repository ICML-2026-agent-repro/verdicts holds verdicts.json and verdicts_final.json, both 27.6 MB, the second committed with a message naming it the final frozen snapshot at challenge close on 3 August — a day after the end date the write-up gives.

What the announcement does not mention is that you cannot query it where it sits. The Hub dataset viewer — the ordinary way to page through a dataset without downloading it — fails on that repository with a parse error, reporting that it expected bytes and received a list, in a column named after a logbook. The companion challenge dataset, MIT licensed and 17.5 MB, fails its own viewer with a cast error. Checking a paper means pulling the file down and searching it yourself:

curl -L -o verdicts_final.json 'https://huggingface.co/datasets/ICML-2026-agent-repro/verdicts/resolve/main/verdicts_final.json?download=true'

Four rules for using it this quarter:

1. Search the snapshot for the paper's title before you schedule work that assumes its result. The repository recorded 63,259 downloads last month against a viewer that does not load, so budget for parsing 27.6 MB of JSON rather than for a lookup API.
2. Treat a paper as unverified unless every claim carries a verified status. One confirmed claim out of many is what the 51% measures.
3. If the paper is one of the 4,115 nobody attempted, there is no verdict. An absent falsification is not a clean bill.
4. If a verdict is falsified or contested, open the logbook before acting on it. With 242 papers showing teams in direct disagreement, the verdict is a pointer to evidence, not the evidence.