Frontier models are much better at physics than benchmarks suggest—expert re-grading shows why

How good are frontier models at physics?

Researchers at Yale and other institutions had physics faculty and PhDs re-grade six widely used physics benchmarks. Most answers previously marked wrong turned out to be grader errors, incorrect reference solutions, or ambiguous questions. For GPT-5.6-Sol, corrected mean@4 jumped from 47.3% to 78.7% on HLE-Physics and from 61.0% to 87.2% on CMT-Benchmark. The near-saturation on these closed-ended tasks signals an urgent need for harder, expert-validated evaluations.

Why it matters: Yale physicists re-graded six popular physics benchmarks and found most 'wrong answers' were actually grading bugs or ambiguous questions. Corrected scores show GPT-5.6-Sol jumping from 47.3% to 78.7% on HLE-Physics — near saturation. A solid takedown of benchmark trustworthin...