Skip to content
Hacker News front page

Specific releases Real-SWE: benchmarking AI coding agents on private, real-world enterprise codebases

Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

Specific tested 8 frontier models on real production tasks from 8 companies' private codebases. Anthropic Fable 5.1 with Claude Code leads at 38.8% resolution rate, followed by GPT-6 Astra at 33.8% and Gemini 3.8 Flash at 31.2%. Tasks involve real business consequences like fixing tax calculations and customer migrations, requiring models to navigate company-specific conventions. Even the best model fails on most tasks—38.8% is a long way from replacing engineers. The post doesn't disclose total task count or time limits per task.

Why it matters: Specific built its benchmark from private production codebases licensed by eight companies, with billing, tax and customer-migration tasks; the best model solved 38.8%.

Read the original ↗Export Markdown