PaperBench: Evaluating AI’s Ability to Replicate AI Research

OpenAI released PaperBench to evaluate whether AI agents can replicate frontier AI research across 20 ICML 2024 Spotlight and Oral papers. The benchmark includes 8,316 gradable subtasks with author-co-developed rubrics; the best tested agent, Claude 3.5 Sonnet (New) with open-source scaffolding, scored 21.0% on average. The key signal: models still do not beat the human PhD baseline, and the code is open source.

Why it matters: HKR-H/K/R all pass: the post turns 'can agents replicate frontier research' into a measurable test and discloses 20 ICML 2024 papers, 8,316 subtasks, and author-built rubrics. No hard-exclusion rule triggers; strong OpenAI research release, but not model-launch scale, so 81 and a