Skip to content
OpenAI News

Introducing SimpleQA

OpenAI open-sourced SimpleQA, a 4,326-question benchmark for factual short-answer QA and model calibration. Two independent AI trainers verified each item; a 1,000-question audit showed 94.4% agreement and an estimated inherent error rate near 3%. The key signal: it is built to challenge frontier models, and the post says GPT-4o scores below 40%.

Why it matters: OpenAI released SimpleQA, a benchmark of 4,326 fact questions cross-verified by two trainers with 1,000 spot-checked and 94.4% agreement, for measuring how accurately models answer short factual questions.

Read the original ↗Export Markdown