Computing Life · Share · YageJun 25, 2026, 08:00 UTC+8

KV Cache Hit Rate: The #1 Cost Lever for Agent Inference

Agent inference bills are dominated by prefill—re-reading the full context before every tool call—not by token generation. Spheron measured prefill at 85–95% of agent inference cost, with a 267:1 input-to-output token ratio. Raising KV cache hit rate from 0% to 90% can drop monthly GPU bills from $20K to $2K. Three engineering layers address this: compression (CompressKV retains only 3% of KV cache while keeping 97% LongBench QA performance, though FlashAttention kernels don't expose attention scores), routing (prefix-hash routing cut TTFT p90 from 92.5s to 0.54s vs. round-robin), and API-level prompt caching (Claude charges 0.1x for cached input, but Anthropic silently dropped the default TTL from 1 hour to 5 minutes in March 2026, causing 100x bill spikes). Teams running multi-turn agents should enable prompt caching before debating model choice and make cache hit rate the first dashboard metric.

Why it matters: Spheron's production measurements plus independent arXiv validation plus corroboration from Cockroach Labs and Manus make a solid case on agent inference cost structure. The 267:1 input-to-output ratio and 10x bill reduction at 90% cache hit rate are hard numbers. Scores lower...

Read the original ↗Export Markdown

Take

This piece is worth opening because it nails the bill structure of agent inference. A ReAct agent making 10 tool calls produces 500 output tokens but chews through 800K input tokens. Spheron measured prefill at 85–95% of agent inference cost, with a 267:1 input-to-output token ratio. The bulk of the bill isn't generating new content—it's making the model re-chew old information every round.

Raising KV cache hit rate from 0% to 90% drops monthly GPU bills from $20K to $2K. That number comes from independent verification in arXiv 2605.26297, not vendor hype. Of the three engineering layers, API-level prompt caching is the only one most developers can touch directly: Claude charges 0.1x for cached input, but Anthropic silently cut the default TTL from 1 hour to 5 minutes in March 2026, causing 100x bill spikes. You now need to explicitly set `"ttl": 3600` to avoid that trap.

The routing layer problem is also worth flagging: with default round-robin load balancing, identical-prefix requests land on different GPU replicas, zeroing out cache hit rate entirely. llm-d on 8 pods with 16 H100s crushed TTFT p90 from 92.5s to 0.54s—that's not an optimization gap, it's a usability gap.

The compression layer (CompressKV retains only 3% of KV cache while keeping 97% LongBench QA performance) is currently stuck on engineering blockers: FlashAttention kernels don't expose attention scores, and paged attention memory fragmentation means evicted space isn't actually freed. Tangram and UltraQuant are attacking this from different angles.

If you're running multi-turn agents, enable prompt caching first, make cache hit rate your first dashboard metric, then debate model choice.

Source: Computing Life · Share · Yage · yage.ai
Published: Jun 25, 2026, 08:00 UTC+8 · Jun 25, 2026