Mercor study finds AI benchmarks cover too few real US work tasks
What happened
On October 17, Hacker News' front page featured Mercor research on AI benchmarks and how well they cover US occupational tasks. The study started from 29 AI benchmarks, mapped their prompts to O*NET occupational tasks, and set a threshold of at least 5 prompts for a task to count as covered. On that basis it estimates 90% of US work hours go to tasks these benchmarks cover little or not at all. Of 923 occupations, only 208 have any task meeting the coverage threshold, and 6 of 22 occupation categories have no qualifying occupation at all.
Written by AI from the coverage · updated 1 hour ago
Coverage
Follow the reports to see the story from different sides.
- Hacker News front page90% of US working hours are spent on tasks not covered well by AI benchmarks
Mercor 基于初选的 29 个 AI 基准估算，美国 90% 的工作时间用于这些基准覆盖很少或完全未覆盖的任务。研究将提示词映射到 O*NET 职业任务，以至少 5 个提示词作为覆盖门槛，923 个职业中仅 208 个有任务达标，22 个职业类别中有 6 个没有职业达标。
Heat over time
Not enough continuous observations to draw a trend yet.