OpenAI reported on July 8, 2026 that a detailed audit of the SWE-Bench Pro coding benchmark revealed pervasive evaluation problems and estimated that roughly 30% of tasks are broken. The company said these flaws can distort measurements of model coding capabilities and advised model developers to inspect results carefully.
Audit process and scope
The evaluation used a multi-stage quality-assurance pipeline. An initial automated filter reviewed task instructions, model attempts, and the tests used to grade solutions, flagging 286 potentially broken tasks. A datapoint analysis pipeline then flagged 200 tasks (27.4%) as likely broken. In parallel, a human annotation campaign of experienced software engineers reviewed the flagged subset; that campaign identified 249 broken tasks (34.1%).
Audits combined Codex-based investigator agents that could run tests and inspect repositories with independent human review. Each flagged task underwent investigator-agent passes and was then reviewed by five engineers; disagreements were escalated for further investigation. OpenAI reported a 74% overlap between agent-flagged categories and human judgments, and noted human reviewers tended to mark more cases as broken or assign multiple issue labels.
Types of failures identified
OpenAI categorized the principal issues into four failure modes:
- Overly strict tests: unit tests enforce implementation details not stated in prompts, causing correct submissions to fail.
- Underspecified prompts: task descriptions omit requirements that hidden tests enforce and that are not reasonably inferable.
- Low-coverage tests: tests do not fully exercise the requested feature, allowing incomplete fixes to pass.
- Misleading prompt: prompts direct models toward the wrong behavior or contradict test expectations.
As context, OpenAI noted that on the 731-task public split of SWE-Bench Pro, frontier models’ pass rate rose from 23.3% to 80.3% over eight months—an improvement that increases the sensitivity of evaluations to such dataset flaws.
Given the audit findings and similar problems previously reported in SWE-bench Verified, OpenAI retracted its earlier recommendation to adopt SWE-Bench Pro and urged the evaluation community to develop benchmarks built by experienced software developers. OpenAI emphasized that valid benchmarks should provide meaningful, hard-to-game signal for deployment and safety decisions.
Original source: OpenAI News