Same model, 9 harnesses: cost per pass varies 17× in FrontierHarness Eval
Show HN: FrontierHarness Eval – 9 harness, same model, cost per pass varies 17x
Runta benchmarked 12 harness configs on Kimi K3 with identical cold-start environments across 360 runs. Codex led at 66.7% pass rate and $3.47 per task; Exo Harness was cheapest at $1.05 with 53.3% pass rate; Claude Code hit 63.3% but cost $18.34 per task. Cache hit rate doesn't equal savings—Claude Code had the lowest cache hit rate at 67.8% yet the highest cost per successful task at $0.288. The post doesn't disclose which specific software engineering tasks were used or their difficulty distribution.
Why it matters: Runta ran 9 harnesses in 12 configurations on the same Kimi K3 model with identical cold starts, and Claude Code cost $18.34 per pass against Codex's $3.47 at a lower pass rate.