Skip to content
Latent Space

Cognition launches FrontierCode: a coding benchmark that asks 'would you actually merge this?'

[AINews] FrontierCode: Benchmarking for Code Quality over Slop

Cognition built FrontierCode, a benchmark that scores code on mergeability and maintainability, not just passing unit tests. Tasks were designed with open-source maintainers, each taking 40+ hours, and evaluated on regression safety, cleanliness, scope, test correctness, and maintainability. The best model, Opus 4.8, hits only about 13% on the hardest tier—far below the 50%+ common on SWE-Bench-style evals. The post also notes METR found many SWE-bench-passing PRs wouldn't actually be merged, and FrontierCode directly measures that false-positive problem.

Why it matters: Cognition releases FrontierCode, a benchmark written by open source maintainers that scores whether a change adds new bugs, its scope and maintainability instead of just passing tests.

Read the original ↗Export Markdown