SWE‑Bench Pro’s broken tasks undermine AI coding progress claims
OpenAI’s audit of SWE‑Bench Pro uncovered that over a third of its 731 public tasks are broken—tests are too strict, prompts underspecified, or coverage low. This calls into question recent claims of massive progress in AI coding, and suggests the community must rethink benchmark design before trusting model capabilities.