Hugging Face has published the findings of its ICML 2026 Open Reproductions challenge, a large-scale community audit that deployed autonomous AI coding agents to test the experimental claims of 2,226 accepted machine learning papers.
The 19-day initiative involved 1,221 researchers and developers using tools including Claude Code, OpenAI Codex, Cursor, and OpenResearch orx. Participants generated 6,816 publicly auditable reproduction logbooks and executed 2,962 cloud compute jobs, examining roughly 34% of the 6,352 papers accepted to ICML 2026 (which received 23,918 submissions).
Aggregate Audit Results
Each reproduction attempt published its code, artifacts, and execution traces to Hugging Face Spaces. An automated judge model running GLM-5.2 evaluated 35,908 extracted scientific claims, categorizing outcomes into verified, falsified, toy-scale evidence, or inconclusive:
- 51% of examined papers (1,103 papers) had at least one claim independently verified. Among these, 266 papers were fully reproduced across all claims, while 632 were partially confirmed without falsifications. Real experiments confirmed 3,978 individual claims.
- 23% of examined papers (496 papers) had at least one claim falsified or contested. This total includes 49 papers where all extracted claims failed reproduction and 242 papers where independent teams produced contradictory empirical results on the same claims.
- 502 papers yielded evidence only at toy or synthetic scale due to proprietary data or unreleased checkpoints.
- 280 papers produced inconclusive outcomes, predominantly caused by missing repository dependencies or incomplete setup documentation.
High-Profile Discrepancies and Proof Failures
The audit surfaced several high-profile discrepancies across accepted papers, including spotlights:
- Learning-Augmented Paging Proof Error: In "Towards Optimal Robustness in Learning-Augmented Paging," an accepted spotlight paper whose reviewer noted low confidence due to unchecked proofs, reproduction agents identified a mathematical failure in the bounding steps. The true robustness grew as
H_k + Theta(log k)rather than the claimedH_k + O(1), confirmed through nine-sigma empirical sweeps up tok = 1,024. - Finite-Horizon Proof Violations: In "Attention's forward pass and Frank-Wolfe," which claimed token particles collapse to the origin under convex hull conditions, agents generated exact rational arithmetic counterexamples showing violations emerging after step 224.
- Objective Function Mismatch: In "Self-Distillation Enables Continual Learning," the theoretical proofs analyzed reverse KL divergence, whereas the published codebase implemented forward KL, failing to replicate headline performance gains.
- Padding Token Inflation: In "Do Transformers Need Three Projections?", an evaluation flaw was uncovered where roughly 66% of evaluated tokens were EOS padding tokens with near-zero loss, deflating perplexity degradation from an actual 9.4% down to a reported 3.1%.
The Role of Agents in Scientific Verification
The scale of submissions to major AI conferences (doubling year-over-year) has created severe reviewer bandwidth constraints. While coding agents automated dependency setup, script execution, and sweep sweeps across thousands of claims, Hugging Face researchers noted that interpreting contradictory results, validating mathematical bounds, and distinguishing implementation bugs from genuine theoretical flaws required human oversight.
All reproduction logbooks, datasets, and execution traces have been released publicly on Hugging Face Spaces.
