Skip to content
OpenAI News

PaperBench: Evaluating AI’s Ability to Replicate AI Research

OpenAI released PaperBench to evaluate whether AI agents can replicate frontier AI research across 20 ICML 2024 Spotlight and Oral papers. The benchmark includes 8,316 gradable subtasks with author-co-developed rubrics; the best tested agent, Claude 3.5 Sonnet (New) with open-source scaffolding, scored 21.0% on average. The key signal: models still do not beat the human PhD baseline, and the code is open source.

Why it matters: PaperBench scores AI agents on replicating 20 ICML 2024 papers, with rubrics written with the original authors and 8,316 gradable subtasks.

Read the original ↗Export Markdown