Computing Life · Share · YageSep 6, 2026, 08:00 UTC+8

tok/s is the most deceptive performance number in agentic scenarios

The author tested Cerebras-hosted Qwen 27B (claimed 1,500 tok/s) against a local instance (measured 102 tok/s) in real coding workflows. After accounting for prefill, actual effective throughput differed by only 3.5×, not 15×. Worse, the cloud session's context exploded from 11K to 99K in 65 seconds, hitting 865K total tokens against a 450K TPM cap—killed by a 429 error in three minutes with a $1.57 bill. The same task locally took 11 minutes and cost $0.017 in electricity, running stably for hours. The takeaway: ignore advertised tok/s for agentic workloads; measure end-to-end effective throughput and weigh context lifespan, cost, and rate limits. The query tool is open-sourced.

Why it matters: The author uses 30 days of real agentic workflow data to pull Cerebras' claimed 1500 tok/s down to an effective 357 tok/s — only 3.5x faster than local. Concrete numbers and methodology, not hand-waving. Not scored higher because the cloud sample is small (28 generations) and ...

Read the original ↗Export Markdown

Take

This piece is worth opening because the author ran a real coding workflow on Cerebras-hosted Qwen 27B (claimed 1,500 tok/s) against the same model locally (measured 102 tok/s), and the takeaway is blunt: stop using advertised tok/s to pick agentic models.

His measured effective throughput: 357 tok/s on Cerebras, 102 tok/s locally—only 3.5× faster, not 15×. The gap comes from prefill. Agentic loops re-read the entire history every turn; context exploded from 11K to 99K in 65 seconds, and prefill ate most of the cloud's decode advantage. Worse, faster generation spins the loop faster, pumping more tokens per minute into the API. The session hit Cerebras's 450K TPM cap in three minutes, threw a 429, and billed $1.57. The same task ran locally in 11 minutes for under 2 cents of electricity.

The author proposes a practical formula: usable = (intelligence ≥ threshold) × effective throughput × context lifespan × cost × reliability. Once Qwen 27B cleared the intelligence bar, the local setup won on context lifespan, cost, and reliability, losing only on throughput. That framework is more useful than any single speed number.

I'd discount this a bit: the data comes from one person's 30-day logs, and the P50 context range of 50K–120K may not match your workload. But the core logic—that prefill and rate limits are the real bottlenecks in agentic scenarios—holds up.

Source: Computing Life · Share · Yage · yage.ai
Published: Sep 6, 2026, 08:00 UTC+8 · 24 days ago