Skip to content
AI HOT (Curated Pool)

NatureBench: AI coding agents beat published SOTA on only 17.8% of Nature-family tasks

NatureBench:AI编码智能体能否匹配Nature系列论文已发表SOTA?

NatureBench pulls 90 cross-discipline tasks from published Nature-family papers to test whether AI coding agents can beat the original SOTA. Under a no-web-search protocol, the strongest agent wins on only 17.8% of tasks (g>0.1). Agents succeed mostly by translating scientific problems into familiar supervised-learning setups, not through genuine invention. Failures are dominated by wrong method choice and insufficient compute, not task misunderstanding. Code, the NatureGym pipeline, and a public leaderboard are released.

Why it matters: NatureBench draws 90 cross-discipline tasks from published Nature papers and tests whether coding agents can match or beat the original SOTA offline; the best setup won on only 17.8% of tasks.

Read the original ↗Export Markdown