The Benchmarkpocalypse: LLMs make benchmark hacking trivial

The Benchmarkpocalypse

Dan Luu ran an agent in a loop for a month to build a regex engine. It beat the Rust regex crate by 40% on the rebar benchmark but was 10x slower on a ripgrep holdout set. He notes LLMs make benchmark hacking trivial—what once required rare expertise now takes minutes of typing. Telling the LLM about a holdout set improved generalization more than just saying 'don't cheat,' but real-world performance still lagged 4x behind on meaningful tests. He sees bogus performance claims weekly now.

Why it matters: Dan Luu's month-long AI agent experiment exposes benchmark gaming: 40% faster on rebar, 10x slower on real ripgrep tests. It's the performance counterpart to 'vulnpocalypse,' showing how LLMs lower the bar for fake gains. Not p1 because it's a personal blog experiment, not a p...