Recreating Minecraft Is Not a Benchmark
After GPT Astra launched, feeds filled with the same demo-benchmarks: one-prompt Minecraft clones, MS Paint, SVG animations. Kuber Mehta argues these tests are broken—visually impressive and easy to grasp, but trivial for labs to overfit by the next release, so they no longer measure real capability.
Why it matters: Author Kuber Mehta argues that demo-style benchmarks like recreating Minecraft have fixed targets, so labs can optimize against them before the next release, which makes them poor measures of real model capability.