Why SWE-bench Verified No Longer Measures Frontier Coding Capabilities

Why SWE-bench Verified no longer measures frontier coding capabilities

OpenAI stopped reporting SWE-bench Verified scores and recommends SWE-bench Pro instead. It audited 138 tasks that o3 failed inconsistently across 64 runs and found 59.4% had test or prompt flaws. The key issue is contamination: tested frontier models reproduced some gold patches or task details.

Why it matters: HKR-H/K/R all pass: OpenAI backs the SWE-bench Verified retirement with an audit and contamination evidence, then points to SWE-bench Pro. It affects coding-model evaluation, but it is not a model or major product launch, so it sits in 78–84.