Why SWE-bench Verified No Longer Measures Frontier Coding Capabilities
Why SWE-bench Verified no longer measures frontier coding capabilities
OpenAI stopped reporting SWE-bench Verified scores and recommends SWE-bench Pro instead. It audited 138 tasks that o3 failed inconsistently across 64 runs and found 59.4% had test or prompt flaws. The key issue is contamination: tested frontier models reproduced some gold patches or task details.
Why it matters: OpenAI audited 138 problems o3 repeatedly failed and found at least 59.4% had overly rigid test cases or vague descriptions, which explains why it retired the benchmark.