Skip to content
Dwarkesh Patel podcast

The sample efficiency black hole: AI models need far more data than humans to learn

The sample efficiency black hole

Dwarkesh Patel argues that recent AI progress comes from more and better data, not better sample efficiency. RL is framed as synthetic data generation: spend compute to find good rollouts, then train the model to predict them. Each skill requires hundreds of human experts writing examples and rubrics, fueling a data-labeling industry earning billions annually. A human sees ~200M tokens by adulthood; frontier models train on tens to hundreds of trillions—a nearly million-fold gap. A person learns to teleoperate a robot in hours, while self-driving models need 3–4 orders of magnitude more data than a teen learning to drive. Open models lag closed ones by only 4 months because data is easy to distill from public APIs, unlike architecture tricks. The post does not propose a fix for sample efficiency.

Why it matters: Dwarkesh Patel frames reinforcement learning as synthetic data generation, arguing AI progress comes mostly from more data rather than better learning efficiency, and compares the data gap between brains and models.

Read the original ↗Export Markdown