Computing Life · Share · YageJul 17, 2026, 08:00 UTC+8

BackendForge turns AI coding benchmarks from written exams into interviews, halving pass rates on the same code

BackendForge：AI 已经能写完整 App，评测也得学会追问

BackendForge starts with 7,250 tests across 56 backend tasks, then has agents hunt for gaps and add 640 more. Only +8.8% more tests, yet GPT-5.5's passing tasks drop from 31 to 16, and Claude Opus 4.7 from 33 to 10. Same code, sharper questions on permissions, dirty data, and cascading effects. The paper says materials will be released, but they aren't public yet—don't generalize these numbers yet.

Why it matters: BackendForge raises a sharp benchmarking methodology question: when agents can write full backends, tests must learn to probe permissions, dirty data, and operation ordering. The numbers are hard—GPT-5.5 and Claude Opus 4.7 saw their full-pass rates halved under the new test s...

Read the original ↗Export Markdown

Take

The reason to click: BackendForge flips the evaluation logic. Instead of making models do more tasks, it makes the test suite ask sharper questions. Researchers wrote 7,250 tests across 56 backend tasks, then had three agents hunt for gaps—permissions, dirty data, cascading effects that the first version missed. They added 640 tests, just +8.8%, but the same code's pass rate got cut in half. GPT-5.5 dropped from 31/56 tasks fully passing to 16; Claude Opus 4.7 fell from 33 to 10.

The idea isn't new—Microsoft's RESTler has been fuzzing API call sequences for years. BackendForge's twist is running this gap-hunt before the benchmark ships, customizing new tests per task based on what the reference implementation gets wrong, then freezing the final set. No per-model tampering, but sharper coverage of edge cases the first draft missed.

I'd discount the numbers a bit. The paper says materials will be released, but they aren't public yet, so no one can audit those 640 new tests. Don't generalize this pass-rate drop to other full-app benchmarks. Also, all 56 tasks are backend services—no frontend, full-stack, or mobile.

The useful signal: when AI starts building entire apps from scratch, benchmarks can't just scale up task size. The question design needs to evolve from a one-shot written exam into an interview that keeps asking follow-ups.

Source: Computing Life · Share · Yage · yage.ai
Published: Jul 17, 2026, 08:00 UTC+8 · Jul 17, 2026