Why high SWE-bench scores don't translate to real-world Kotlin projects
为什么 SWE-bench 高分,在大厂 Kotlin 项目里依然会打转?
JetBrains released the Kotlin Benchmark on July 24, 2026, testing AI coding agents on 105 real-world tasks across 8 open-source projects. With the same Opus 4.7 model, Claude Code hit 85.71% and Junie 81.9%—a nearly 4-point gap driven by how each agent harness handles Gradle build logs. A good harness uses Language Server diagnostics to catch static errors locally, then runs full builds only at key checkpoints and extracts just the blocking lines from noisy output. The article argues that Python-based benchmarks like SWE-bench reward trial-and-error strategies that collapse under Kotlin's heavy build overhead. It recommends teams stop buying off public leaderboards and instead use the Harbor container spec to build private micro-eval matrices from their own historical PRs and issues, measuring Pass@k stability, token cost, and whether patches respect internal architectural constraints.
Why it matters: JetBrains ran 105 real tasks from 8 open source projects, and on the same model Claude Code and its own Junie differed by nearly 4 points in success rate.