The AI Benchmark Yardstick Moved Faster Than the Models

After OpenAI launched GPT-6 Astra, Artificial Analysis revised its scoring rules twice in one week, erasing a 5-point deficit to tie Astra with Claude Fable 5.1—without any model update. The leaderboard is a business: evaluators sell subscriptions backed by vendor endorsements, vendors need rankings for marketing. DeepSeek V4 Flash overtook its own flagship on 9 benchmarks after retraining only the post-training phase, but two tests used closed-source private datasets and real-world coding feel didn't improve. The same model scored 62.7% vs 99.9% on ARC-AGI-3 depending on the execution harness. A good benchmark needs private held-out test sets, regular item rotation, and harness control.

Why it matters: A well-sourced industry commentary with concrete version numbers and score shifts, exposing how a benchmark vendor rewrote its scoring rules twice in one week after GPT-6 Astra's release, flipping the ranking from a 5-point deficit to a tie for first. Hits all three HKR axes a...