Allen AI's BenchMIRT uses psychometric IRT to reveal what LLM benchmarks actually measure

BenchMIRT: What are LLM benchmarks actually measuring?

Allen AI open-sourced BenchMIRT, a method that audits LLM benchmarks using multidimensional item response theory. It analyzed 100 models across 16 benchmarks and 34K+ questions, automatically recovering two dominant capability dimensions: safety and general reasoning. A BBQ question about a grandson and grandfather booking an Uber tests age bias but also requires reasoning. WildJailbreak's harmful and benign prompts map to safety and reasoning respectively—averaging them into one score hides that split. BenchMIRT identifies which questions best separate strong from weak models, enabling cleaner evaluation with fewer items. Code, data, and the tech report are public.

Why it matters: Allen AI open-sourced a method that uses item response theory to audit benchmarks, backed by 100 models, 16 benchmarks, and 34k questions. Score stays below 80 because it's a methodology tool rather than a shippable product update, but it hits all three HKR axes and is genuine...