Specific releases Real-SWE: benchmarking AI coding agents on private, real-world enterprise codebases
Specific tested 8 frontier models on real production tasks from 8 companies' private codebases. Anthropic Fable 5.1 with Claude Code leads at 38.8% resolution rate, followed by GPT-6 Astra at 33.8% and Gemini 3.8 Flash at 31.2%. Tasks involve real business consequences like fixing tax calculations and customer migrations, requiring models to navigate company-specific conventions. Even the best model fails on most tasks—38.8% is a long way from replacing engineers. The post doesn't disclose total task count or time limits per task.
Why it matters: Specific got access to 8 companies' private production repos and threw real business tasks — tax calc fixes, customer migrations — at frontier models. Fable 5.1 + Claude Code hit 38.8% solve rate; GPT-6 Astra is also on the board. This is the closest third-party benchmark to '...