Skip to content
Hacker News front page

Why SWE-bench Verified No Longer Measures Frontier Coding Capabilities

Why SWE-bench Verified no longer measures frontier coding capabilities

OpenAI stopped reporting SWE-bench Verified scores and recommends SWE-bench Pro instead. It audited 138 tasks that o3 failed inconsistently across 64 runs and found 59.4% had test or prompt flaws. The key issue is contamination: tested frontier models reproduced some gold patches or task details.

Why it matters: OpenAI audited 138 problems o3 repeatedly failed and found at least 59.4% had overly rigid test cases or vague descriptions, which explains why it retired the benchmark.

Read the original ↗Export Markdown