Benchmarks

Anthropic's Claude Opus 5.5 is now available on Arena's Agent Arena, where users vote on rankings after the model runs real long-horizon agent tasks. The model can use web search, a file system, and a terminal; the leaderboard uses causal tracking to measure performance relative to the average model. The post doesn't spell out Battle Mode specifics or show example tasks.

Claude Opus 5.5 scored 58 on the Artificial Analysis Intelligence Index, the highest measured so far. It leads on 6 of 10 evaluations, including Humanity's Last Exam at 61.4% and SciCode at 66.9%, and matches GPT-6 Astra (xhigh) on Terminal-Bench 4.0 at 59.6%. On the agentic knowledge-work eval AA-Briefcase, it hit 1822 Elo—143 points above Fable 5.1—and surpassed GPT-5.6 Sol on both analytical quality and presentation. Pricing dropped to $4/$20 per 1M input/output tokens (from $5/$25), with cache reads down 60% to $0.20. Output tokens per task grew ~60% vs Opus 5, so cost per task stayed flat. Context window remains 1M tokens with image and text input.

Max Woolf spent over a year testing whether agentic LLMs can iteratively optimize Rust code with a hard pass/fail rule: each iteration must deliver at least a 5% speedup or roll back. With Claude Opus 4.5 and later models, he got 2–20× speedups on algorithms like UMAP versus mature libraries, all without unsafe code. The post includes the exact prompts and benchmark results. The caveat: these gains are measured on his specific benchmarks and may not translate directly to production workloads.

ZCode was caught silently packaging entire Git histories for cloud upload, with .git objects making up 86.6% of snapshots. HarnessTax benchmarked Claude Fable 5 across frameworks: Claude Code and minimal Pi achieved near-identical success rates but a 2x cost gap. A Microsoft architect replaced multi-model agent loops with a single model reading skill docs and calling tools directly—halving API calls but increasing total tokens by 22%. Each story unpacks one number and a reminder to check which layer a metric actually measures.

Xiaomi released two omni-modal models: MiMo-V2.6-Pro and Flash. The Pro version scored 1628 on Code Arena WebDev, up 153 points from MiMo-V2.5-Pro's 1475, landing around 10th overall and ~3rd among open-weight models under MIT license. The post doesn't disclose Flash's benchmark numbers or parameter counts.

Xiaomi released MiMo-V2.6-Pro, scoring 46 on the Artificial Analysis Intelligence Index—up from 26 for the previous V2.5-Pro. It's now the highest among open-weight models. The post doesn't disclose parameter count, architecture details, or a release timeline.

Xiaomi MiMo-V2.6-Pro scored 46 on the Artificial Analysis Intelligence Index, the highest among open-weight models. The previous MiMo-V2.5-Pro scored 26. The post doesn't disclose model size, training data, or release license, so hold for details.

Fireworks AI benchmarked 18 models on DeepSWE: picking the right model per task beats any single model. GPT-6 Astra alone scores 74.1% at $6.52/task. An oracle router across all 18 hits 97.6% at $1.88. Open-weight models alone reach 90.3% at $1.45. 94 of 113 tasks need a model under $3; the three priciest models are the best pick on only 3 tasks. FireRouter aims to make that per-task choice before the work starts—the post doesn't yet detail how.

Dan McKinley argues that prompt engineering is a distraction. Building consumer-facing agents taught him that even structured output fails on a fraction of requests—models will flood a field with nonsense. His fix was renaming a field from 'title' to 'heading,' which he calls deranged. The talk pushes for pass^k testing and evaluation pipelines to constrain behavior, since prompts alone can't tame the beast. The post is a slide deck; it names no specific eval frameworks or metrics.

Ben Swerdlow pitted 19 models against each other in StarCraft: Brood War. Codex Astra / xhigh went 18–0, but no model surpassed beginner level. Older models treated the RTS as turn-based and got destroyed while thinking; Grok 4.6 issued only 6 command batches in 43 minutes and never fielded a combat unit. Claude Fable earnestly climbed the tech tree but couldn't execute. Codex models favored early Probe harassment that paralyzed opponents. The post doesn't specify whether matches were pure AI vs AI or involved human input.

UC Berkeley and Arena researchers tested 7 models across 3 harnesses—Claude Code, Codex CLI, and Pi—on 30 tasks each from SWE-bench Lite and Terminal-Bench 2.0, with 3 repetitions per pair. Harness choice barely moves success rates (±2–5%), but cost can vary up to 5×. Claude Fable 5 hits 97.8% in Claude Code at $1.33, and 96.7% in Pi at $0.67. Pi, a minimal open-source harness with just read, write, edit, and bash, reaches the Pareto frontier on both benchmarks. The post doesn't spell out the exact open-source licenses for Pi and Codex CLI, and doesn't link the full pricing sheet.

Researchers at Yale and other institutions had physics faculty and PhDs re-grade six widely used physics benchmarks. Most answers previously marked wrong turned out to be grader errors, incorrect reference solutions, or ambiguous questions. For GPT-5.6-Sol, corrected mean@4 jumped from 47.3% to 78.7% on HLE-Physics and from 61.0% to 87.2% on CMT-Benchmark. The near-saturation on these closed-ended tasks signals an urgent need for harder, expert-validated evaluations.

Rohan Bansal post-trained a Qwen 4B model to beat Postgres's default query plans. After SFT distillation from 500 GPT-6 Astra trajectories and a custom GRPO variant for RL, the model achieved 44.7% latency reduction and 81% geometric mean speedup across 113 join-heavy queries. Training ran on a rented 2×H100 node with four Postgres containers on his desk for measurement. The post doesn't disclose total training time or per-inference latency.

AIUC announced a $40M Series A led by Ribbit Capital and First Harmonic. CEO Rune Kvist, Anthropic's first product hire, argues that trust and liability—not capability—will cap AI adoption. They built AIUC-1, a standard that stress-tests agents for jailbreaks, hallucinations, and data leaks, backed by real insurance. Cursor, Harvey, Lovable, and ElevenLabs are already working with them. The episode raises a sharp hypothetical: what happens when a $20 Cursor subscription contributes to a $200M plane crash. The post doesn't disclose specific premium or claims-handling details.

Arena added four new models to its Image-to-WebDev leaderboard. GPT-6 Astra (Max) leads at 1733, 129 points ahead of GPT-5.6 Sol (xHigh). Claude Fable 5.1 (Max) is third at 1710, Muse Spark 1.3 (Max) fourth at 1645, and GLM-5.3-Flash tenth at 1588. The post doesn't disclose evaluation tasks or sample size, so I'd take the gaps with a grain of salt.

Good Start Labs trained a 30B model on the railroad game 1830 and found that training design determines skill transfer. A multi-turn terminal agent version improved at financial research tasks—querying databases, writing Excel formulas, reasoning on the fly—while single-turn training did not. The company spun out of Every last October with $3.6M in funding, betting on verifiable game environments for RL-based skill teaching.

Michael Noukhovitch's blog walks through his new paper on the Matthew Effect in RL post-training for LLMs: as training progresses, the model samples easy problems more and hard problems less, because early successes on easy tasks dominate the reward signal. His proposed fix, Never Give Up (NGU), forces a minimum sampling ratio for hard problems so they don't get squeezed out. On Olmo 3.1 7B math training, NGU lifts AIME 2025 pass@1 from 26.7% to 33.3%; on code, LiveCodeBench pass@1 goes from 23.4% to 26.1%. The post also covers async RL staleness tricks and frames the Matthew Effect as a form of primacy bias. The body doesn't disclose NGU's specific hyperparameters or extra compute cost, so I'd discount the gains until those details surface.

今天最直观的教训来自 @搞仁义毛义仁：给 Astra 一个空白 C++ 仓库，代码写得一塌糊涂；把积累了大量 code review 经验的 GacUI 上下文导进去，质量立刻飙升。这说明模型不是不会写，是得用具体规则去“规训”。@今天群内信息量极大 实测了 Ollama Cloud 跑 DeepSeek V4.1 Flash，解码吞吐是官方 API ...

Fireworks ran a full benchmark suite on DeepSeek-V4.1-Flash. On DeepSWE, it scores 74.34% pass@1, in the same band as GPT-6 Astra at 74.12%, but costs $0.43 per task—15x cheaper. The model uses a 552B MoE with a split activation design: 8B active for input, 16B for output, plus KV cache optimizations. On Terminal-Bench 2.1 it trails Astra by 1 point while costing 12x less. The post mentions an HLE and oracle router eval but does not disclose the actual scores.

Entelligence benchmarked GPT-5.6 Luna and GPT-6 Astra on 50 public PRs for code review. Luna costs $1.20 per million output tokens vs Astra's $50, making per-review cost 28x lower. Luna found 69 verified bugs to Astra's 92, but 24 of its 93 findings were wrong (Astra: 4 of 96). On Sentry, Discourse, and Grafana, Luna was within two bugs of Astra. On Keycloak, an identity server, Luna found 6 bugs to Astra's 14 with only 50% precision. The security gap was widest: 9 vs 19 verified bugs. Luna is good enough for everyday correctness bugs at that price, but not for auth or permission code on its own.