OpenAI reports that 'approximately 30% of SWE-Bench Pro is broken,' highlighting numerous flaws in the benchmark used to measure AI coding capabilities.
On July 8, 2026, OpenAI reported that a detailed investigation of its AI coding capability benchmark, 'SWE-Bench Pro,' revealed that approximately 30% of the tasks had problems that prevented them from being evaluated.
Separating signal from noise in coding evaluations | OpenAI
SWE-Bench Pro gives AI software feature modification tasks and considers it a success if it passes new tests without breaking existing functionality. It's a practical benchmark to see if it can read existing codebases and make workable fixes, but according to OpenAI, there are many evaluation problems, such as 'the instructions say to 'put one space at the beginning of each line when converting to Markdown,' but the hidden test used for scoring asks for two spaces, so even if the AI writes the code as instructed, it will be marked as incorrect.'
The evaluation problems can be broadly categorized into four types: 'overly strict tests' that force implementation methods not mentioned in the instructions; 'inadequately explained questions' where the conditions required for hidden tests cannot be gleaned from the question; 'narrow scope of testing' where necessary functions cannot be adequately inspected and incomplete fixes may still pass; and 'misleading questions' where the question leads the AI in a direction unrelated to the test.
SWE-Bench Pro was originally designed to handle more realistic coding challenges that involve longer work processes than the traditional 'SWE-bench Verified.' OpenAI had previously recommended migrating to SWE-Bench Pro due to concerns about design flaws and potential contamination of training data in SWE-bench Verified.
OpenAI explains that the standard benchmark used to measure AI coding ability is 'no longer meaningful,' and that examining the problems that initially failed to solve revealed that the problems themselves were flawed - GIGAZINE
However, in the 731 publicly available tasks of SWE-Bench Pro, the pass rate for the most advanced model rose from 23.3% to 80.3% in eight months, making it necessary to verify whether the increase in scores truly reflects an improvement in AI capabilities.
Therefore, OpenAI created a quality assurance pipeline that examines the task's problem statement, AI-generated solutions, task metadata, and failure logs. The initial automated filter extracted 286 suspicious tasks, which were then verified using a combination of verification by an investigation agent based on the Codex, judgment by researchers, and review by five experienced software engineers.
As a result, OpenAI's pipeline determined that 27.4% of the tasks were broken. Meanwhile, human-based classification work identified problems with 34.1% of the tasks. Based on both results, OpenAI estimates that approximately 30% of the tasks in SWE-Bench Pro are broken.
OpenAI points out that GitHub issues and pull requests were originally designed for collaborative development among humans and were not created as independent tasks to accurately measure AI capabilities. As a result, there are cases where the problem statement, the actual implemented fixes, and the scoring tests do not perfectly align. Following this analysis, OpenAI has withdrawn its previous recommendation to use SWE-Bench Pro and states that new benchmarks designed by experienced software developers for AI evaluation are needed going forward.
Related Posts:
in AI, Posted by log1d_ts