Hacker News front pageSep 17, 2026, 06:10 UTC+8

Coding agent harnesses can 2× your cost with no real accuracy gain

HarnessTax: How Much Does the Harness Matter for Coding Agents?

UC Berkeley and Arena researchers tested 7 models across 3 harnesses—Claude Code, Codex CLI, and Pi—on 30 tasks each from SWE-bench Lite and Terminal-Bench 2.0, with 3 repetitions per pair. Harness choice barely moves success rates (±2–5%), but cost can vary up to 5×. Claude Fable 5 hits 97.8% in Claude Code at $1.33, and 96.7% in Pi at $0.67. Pi, a minimal open-source harness with just read, write, edit, and bash, reaches the Pareto frontier on both benchmarks. The post doesn't spell out the exact open-source licenses for Pi and Codex CLI, and doesn't link the full pricing sheet.

Why it matters: Systematic eval from UC Berkeley and Arena: 21 model–harness pairs on standard benchmarks yield a counterintuitive finding—harness barely moves success rate but swings cost 5x. Concrete numbers, clean experimental design, practical takeaway. Held at 78 because the body excerpt...

Read the original ↗Export Markdown

Take

This one's worth opening because it tests something everyone assumes: that Claude models belong in Claude Code. The UC Berkeley and Arena team ran 7 models across 3 harnesses—Claude Code, Codex CLI, and Pi—on 30 tasks each from SWE-bench Lite and Terminal-Bench 2.0, with 3 repetitions per pair. Harness choice barely budged success rates, but cost swung up to 5×.

The clearest number: Claude Fable 5 hits 97.8% in Claude Code at $1.33, and 96.7% in Pi at $0.67. Pi is a minimal open-source harness with just read, write, edit, and bash—and it lands on the Pareto frontier on both benchmarks, meaning it's the cheapest way to hit a given success rate.

I'd discount this a bit on two fronts. The post doesn't link the full pricing sheet or spell out the exact open-source licenses for Pi and Codex CLI, so you'll need to dig into the repos yourself. And 30 tasks with 3 reps per pair isn't a huge sample—the ±5% confidence intervals on Terminal-Bench aren't that tight.

But the core takeaway holds: if you're running batch tasks through Claude Code, swapping to a lighter harness could halve your token costs with almost no success-rate hit. For solo devs and small teams, that's faster money saved than waiting for the next model price cut.

Source: Hacker News front page · harnesstax.github.io
Published: Sep 17, 2026, 06:10 UTC+8 · 13 days ago