Skip to content
OpenAI News

Introducing SWE-bench Verified

OpenAI released SWE-bench Verified, a human-validated subset built with the benchmark’s authors to assess real software issue resolution more reliably. The post names 3 failure modes in SWE-bench: overly narrow tests, underspecified issue statements, and unreliable environment setup; as of Aug. 5, 2024, top agents scored about 20% on SWE-bench and 43% on SWE-bench Lite. The key point is that the original benchmark can systematically underestimate coding-agent ability.

Why it matters: OpenAI and the original SWE-bench authors built a human-verified subset, arguing the original systematically underestimates real coding ability due to rigid unit tests, vague descriptions and environment issues.

Read the original ↗Export Markdown