Organizers of an open challenge say community teams reproduced 2,226 ICML 2026 papers between July 15 and August 2, 2026, producing 6,816 Trackio logbooks and judging 35,908 extracted claims. The event combined human steering with coding agents and an automated judge to audit papers at scale.
Scope and methods
Participants selected papers from the 6,341 accepted ICML 2026 submissions indexed by the organizers, extracted concrete claims, and ran reproductions using a variety of agents and tools, including Claude Code, Codex, Cursor, Pi and OpenResearch’s orx. Each run produced a Hugging Face Space containing code, artifacts and optional agent traces; an open-weights model, GLM-5.2, re-read every logbook to issue per-claim verdicts: verified, falsified, toy, or inconclusive. Organizers provided $20 in HF Jobs credits per participant; in total, 2,962 HF Jobs were launched and 274 full agent-trace datasets were published.
Key results
Aggregating claim-level verdicts, organizers report that 51% of examined papers (1,103) had at least one claim independently verified. Of those, 266 papers were fully reproduced and 632 were partially reproduced with no falsifications. In total, 3,978 individual claims were confirmed with real experiments.
Conversely, 23% of examined papers (496) had at least one claim falsified or contested; this includes 49 papers where all extracted claims were falsified and 242 papers where independent teams reached opposite verdicts on the same claims. The remainder included 502 papers with only toy-scale evidence and 280 papers where no verdict could be established, most often because artifacts were missing. All verdicts were frozen in a public dataset at challenge close.
Notable findings and human involvement
Several concrete issues were documented. For example, reproductions found that the paging paper “Towards Optimal Robustness in Learning-Augmented Paging” exhibited an additive term growing like 0.38 ln k rather than the claimed O(1); a re-implementation up to k = 1,024 confirmed growth at roughly nine sigma, suggesting true robustness H_k + Θ(log k). Independent teams found counterexamples to a collapse claim in “Attention’s forward pass and Frank-Wolfe” at horizons t = 224, ~3,800 and 6,416; the authors acknowledged the issue and are working on a fix. In “Self-Distillation Enables Continual Learning,” reproductions identified a mismatch between the paper’s reverse KL analysis and the released code’s forward KL default; the authors uploaded a clarified arXiv version. In “Do Transformers Need Three Projections?” participants found ~66% of evaluated positions were EOS padding tokens, changing a cited 3.1% quality cost to roughly 9.4% once corrected.
Organizers emphasize that agent execution has limits—agents missed scale-dependent behaviors and sometimes built falsifications on unit or arithmetic errors—and that human-in-the-loop workflows produced the most reliable results. All logbooks, verdicts, traces, and artifacts from the challenge were published in a public dataset by the organizers.
Original source: Hugging Face Blog