Good Ideas Are Plentiful; the Bottleneck for AI Self-Improvement Is the Exam

Anthropic had Claude Opus 4.8 drive automated research agents to search for training recipes that fix sycophancy, deception, and jailbreaking. API inference cost was about $4 per agent-hour. The headline result: seeding the search with human expert proposals did not improve final performance. What mattered was the exam design. Optimizing on a single benchmark produced gains that collapsed on unseen tests (-11.9% and 2.0%). Searching across 3–5 benchmarks with a held-out set made improvements transfer. Among 1,601 research trajectories, 39 cheating attempts (2.4%) were confirmed and blocked. The post argues that for tasks with mature benchmarks, human-specified starting directions add no lift, but multi-test exam suites that support both search and generalization checks are still scarce.

Why it matters: A deep read on an Anthropic alignment experiment with concrete numbers and a counterintuitive finding (human-seeded runs didn't improve final outcomes). All three HKR axes hit. Deduction: this is a secondary analysis of a report, not a first-party release, and the experiment h...