Hacker News front pageSep 19, 2026, 22:44 UTC+8

Brood War Bench: No model played beyond beginner level in StarCraft

Brood War Bench

Ben Swerdlow pitted 19 models against each other in StarCraft: Brood War. Codex Astra / xhigh went 18–0, but no model surpassed beginner level. Older models treated the RTS as turn-based and got destroyed while thinking; Grok 4.6 issued only 6 command batches in 43 minutes and never fielded a combat unit. Claude Fable earnestly climbed the tech tree but couldn't execute. Codex models favored early Probe harassment that paralyzed opponents. The post doesn't specify whether matches were pure AI vs AI or involved human input.

Why it matters: First-person benchmark with 19 models playing StarCraft against each other. Concrete data (win rates, APM, cost) and a clear finding: none surpass beginner level. Codex Astra won by early worker harass, not macro play — that detail carries signal. Not an 85 because it's more a...

Read the original ↗Export Markdown

Take

This one's worth opening because it drops models into an environment that demands continuous perception and fast decisions, and the results are brutally honest: nobody played above beginner level. Codex Astra / xhigh went 18-0, but not through macro play — it won by sending a single Probe to harass enemy workers, which caused opposing models to freeze up for dozens of seconds while they thought about what to do. That's less RTS skill and more exploiting a 'thinking paralysis' bug in the opponent.

Grok 4.6 was the worst: its xhigh run logged 11,138 reasoning tokens but issued only six command batches in 43 minutes and never fielded a combat unit. Older models treated the RTS as turn-based and got destroyed mid-thought. Claude Fable earnestly climbed the tech tree but couldn't execute.

I'd discount this a bit — the post doesn't specify whether matches were pure AI vs AI or involved human input, and there's no map pool or race distribution. But one detail I find genuinely interesting: Codex's lower-effort settings often had higher APM but lower win rates, which suggests overthinking might be the real bottleneck here.

Source: Hacker News front page · bw.swerdlow.dev
Published: Sep 19, 2026, 22:44 UTC+8 · 10 days ago