When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation
This ICML 2026 paper defines and measures benchmark saturation across 60 LM benchmarks. Nearly half are already saturated, and older benchmarks saturate faster. Expert curation helps resist saturation; keeping test data private does not. The post doesn't name the specific benchmarks but identifies 14 properties linked to saturation, offering a framework for building longer-lasting evaluations.
Why it matters: This ICML paper systematically analyzes 60 language model benchmarks, finds nearly half are saturated, and shows hiding test data does not delay saturation while expert-written questions do.