#评测/基准

2 today

Aug 13, 2024Tuesday

OpenAI NewsAug 13, 18:00

Introducing SWE-bench Verified

OpenAI released SWE-bench Verified, a human-validated subset built with the benchmark’s authors to assess real software issue resolution more reliably. The post names 3 failure modes in SWE-bench: overly narrow tests, underspecified issue statements, and unreliable environment setup; as of Aug. 5, 2024, top agents scored about 20% on SWE-bench and 43% on SWE-bench Lite. The key point is that the original benchmark can systematically underestimate coding-agent ability.

Why it matters: This is a strong benchmark release, not a routine post: OpenAI re-audited SWE-bench with the original authors, named 3 defect classes, and reported new score ceilings of 20% and 43%. HKR-H/K/R all pass because it changes how builders read code-agent leaderboards.

Take

OpenAI is pulling SWE-bench back from leaderboard theater into risk evaluation. The original set spans 12 Python repos, and the top agents sat around 20% on SWE-bench and 43% on Lite as of Aug. 5, 2024. That reads like weak coding agents. The post names three benchmark defects: narrow tests, underspecified issues, and unreliable environments. Those defects reject valid patches and mislabel harness failure as model failure.

I buy this move more than another model claiming a few extra points. SWE-bench Verified is a human-validated subset built with the benchmark authors, and it fits OpenAI’s Preparedness Framework for Model Autonomy. Cursor, Devin, and Claude Code live on long-loop debugging and environment recovery, not clean single-shot pass rates. The old SWE-bench was partly blind to that.

Jul 17, 2024Wednesday

OpenAI NewsJul 17, 18:00

Prover-Verifier Games improve legibility of language model outputs

OpenAI trained GPT-4-family prover-verifier games so stronger models write solutions weaker models can verify; under time-limited human review, correctness-only optimization led to nearly 2x more evaluation errors. The post says the large and small models differ by about 3 orders of magnitude in pretraining compute, and checkability training recovers about half the performance gain of correctness-only optimization; the full experimental numbers are not fully disclosed in the provided text.

Why it matters: This is a substantive OpenAI research release with HKR-H/K/R all present: novel setup, clear mechanism, and strong relevance to scalable oversight. The excerpt confirms the method and the human-evaluation effect, but not the full experimental tables, so it fits the 78–84 band, نه

Take

OpenAI’s useful admission here is that optimizing for correct answers can make reasoning hostile to review. In their GPT-4-family prover setup, correctness-only optimization made time-limited human evaluators commit nearly twice as many errors. Training the prover to write chains a weaker verifier can check recovered about half the performance gain from accuracy-only tuning.

I buy the direction because legibility becomes a training target, not a post-hoc “please explain” wrapper. The setup also has the right asymmetry: the large and small models differ by roughly three orders of magnitude in pretraining compute, which resembles strong systems writing for weaker auditors. The caveat is ugly: the provided article cuts off before the full experimental tables, so that “half the gain” claim is not yet enough for product decisions.