Skip to content

Benchmarks

Who is actually stronger: benchmark results, methodology disputes and leaderboard changes.

Latest picks

41–60 of 471

Sep 22Tuesday

Hacker News front page

Prompting agents to iteratively optimize Rust until it beats state-of-the-art libraries

Max Woolf spent over a year testing whether agentic LLMs can iteratively optimize Rust code with a hard pass/fail rule: each iteration must deliver at least a 5% speedup or roll back. With Claude Opus 4.5 and later models, he got 2–20× speedups on algorithms like UMAP versus mature libraries, all without unsafe code. The post includes the exact prompts and benchmark results. The caveat: these gains are measured on his specific benchmarks and may not translate directly to production workloads.

Why it matters: Max Woolf published the prompts and benchmark results behind agent-driven Rust optimization, hitting 2x to 20x faster than mature libraries on algorithms like UMAP with unsafe code banned.

Computing Life · Share · Yage

Three AI Coding Stories, Three Numbers to Read

ZCode was caught silently packaging entire Git histories for cloud upload, with .git objects making up 86.6% of snapshots. HarnessTax benchmarked Claude Fable 5 across frameworks: Claude Code and minimal Pi achieved near-identical success rates but a 2x cost gap. A Microsoft architect replaced multi-model agent loops with a single model reading skill docs and calling tools directly—halving API calls but increasing total tokens by 22%. Each story unpacks one number and a reminder to check which layer a metric actually measures.

Why it matters: A single sample shows ZCode packaging the whole workspace, .git included, which can carry secrets and internal branch names to the cloud.

AI HOT (Curated Pool)

Xiaomi MiMo-V2.6-Pro hits ~10th on Code Arena WebDev, ~3rd among open-weight models

Xiaomi released two omni-modal models: MiMo-V2.6-Pro and Flash. The Pro version scored 1628 on Code Arena WebDev, up 153 points from MiMo-V2.5-Pro's 1475, landing around 10th overall and ~3rd among open-weight models under MIT license. The post doesn't disclose Flash's benchmark numbers or parameter counts.

Why it matters: Xiaomi's MiMo-V2.6-Pro scored 1628 on the Code Arena web development leaderboard, 153 points above the previous generation, around 10th overall and about 3rd among MIT-licensed open-weight models.

AI HOT (Curated Pool)

Xiaomi MiMo-V2.6-Pro tops open-weight model intelligence index

Xiaomi released MiMo-V2.6-Pro, scoring 46 on the Artificial Analysis Intelligence Index—up from 26 for the previous V2.5-Pro. It's now the highest among open-weight models. The post doesn't disclose parameter count, architecture details, or a release timeline.

Why it matters: Xiaomi's MiMo-V2.6-Pro scored 46 on Artificial Analysis's open-weight intelligence index, up from 26 for V2.5-Pro, the highest score among open-weight models so far.

AI HOT (Curated Pool)

Xiaomi MiMo-V2.6-Pro tops open-weight model intelligence index

Xiaomi MiMo-V2.6-Pro scored 46 on the Artificial Analysis Intelligence Index, the highest among open-weight models. The previous MiMo-V2.5-Pro scored 26. The post doesn't disclose model size, training data, or release license, so hold for details.

Why it matters: Xiaomi's open-weight MiMo-V2.6-Pro scores 46 on the Artificial Analysis Intelligence Index, up from 26 for MiMo-V2.5-Pro and the highest of any open-weight model.

Sep 21Monday

AI HOT (Curated Pool)

Fireworks AI launches FireRouter: the frontier isn't a model, it's a router

Fireworks AI benchmarked 18 models on DeepSWE: picking the right model per task beats any single model. GPT-6 Astra alone scores 74.1% at $6.52/task. An oracle router across all 18 hits 97.6% at $1.88. Open-weight models alone reach 90.3% at $1.45. 94 of 113 tasks need a model under $3; the three priciest models are the best pick on only 3 tasks. FireRouter aims to make that per-task choice before the work starts—the post doesn't yet detail how.

Why it matters: Fireworks AI released FireRouter, a task-level router that picks between open and closed models with cache awareness; its DeepSWE tests show per-task model choice can raise solve rates and cut cost at once.

Sep 20Sunday

Hacker News front page

Prompts Aren't Real: Build Evaluation Pipelines Instead

Dan McKinley argues that prompt engineering is a distraction. Building consumer-facing agents taught him that even structured output fails on a fraction of requests—models will flood a field with nonsense. His fix was renaming a field from 'title' to 'heading,' which he calls deranged. The talk pushes for pass^k testing and evaluation pipelines to constrain behavior, since prompts alone can't tame the beast. The post is a slide deck; it names no specific eval frameworks or metrics.

Why it matters: Engineer Dan McKinley argues in a talk that prompts cannot be maintained as standalone units, because new instructions enter a context different from the one seen at test time and change behavior.

Sep 19Saturday

Hacker News front page

Brood War Bench: No model played beyond beginner level in StarCraft

Ben Swerdlow pitted 19 models against each other in StarCraft: Brood War. Codex Astra / xhigh went 18–0, but no model surpassed beginner level. Older models treated the RTS as turn-based and got destroyed while thinking; Grok 4.6 issued only 6 command batches in 43 minutes and never fielded a combat unit. Claude Fable earnestly climbed the tech tree but couldn't execute. Codex models favored early Probe harassment that paralyzed opponents. The post doesn't specify whether matches were pure AI vs AI or involved human input.

Why it matters: Ben Swerdlow ran 19 models and effort settings head to head in StarCraft: Brood War; Codex Astra at xhigh went 18-0, and no model played above a beginner level.

Sep 17Thursday

Hacker News front page

Coding agent harnesses can 2× your cost with no real accuracy gain

UC Berkeley and Arena researchers tested 7 models across 3 harnesses—Claude Code, Codex CLI, and Pi—on 30 tasks each from SWE-bench Lite and Terminal-Bench 2.0, with 3 repetitions per pair. Harness choice barely moves success rates (±2–5%), but cost can vary up to 5×. Claude Fable 5 hits 97.8% in Claude Code at $1.33, and 96.7% in Pi at $0.67. Pi, a minimal open-source harness with just read, write, edit, and bash, reaches the Pareto frontier on both benchmarks. The post doesn't spell out the exact open-source licenses for Pi and Codex CLI, and doesn't link the full pricing sheet.

Why it matters: A controlled test across 7 models and 3 coding agent harnesses shows the harness barely moves task success while cost for the same model can vary up to 5x.

Hacker News front page

Frontier models are much better at physics than benchmarks suggest—expert re-grading shows why

Researchers at Yale and other institutions had physics faculty and PhDs re-grade six widely used physics benchmarks. Most answers previously marked wrong turned out to be grader errors, incorrect reference solutions, or ambiguous questions. For GPT-5.6-Sol, corrected mean@4 jumped from 47.3% to 78.7% on HLE-Physics and from 61.0% to 87.2% on CMT-Benchmark. The near-saturation on these closed-ended tasks signals an urgent need for harder, expert-validated evaluations.

Why it matters: Researchers at Yale and elsewhere had physics experts re-grade six common physics benchmarks and found most wrong answers came from the scoring code, reference answers or question wording, not from model reasoning.

Hacker News front page

Training a 4B model to produce 81% faster query plans than Postgres

Rohan Bansal post-trained a Qwen 4B model to beat Postgres's default query plans. After SFT distillation from 500 GPT-6 Astra trajectories and a custom GRPO variant for RL, the model achieved 44.7% latency reduction and 81% geometric mean speedup across 113 join-heavy queries. Training ran on a rented 2×H100 node with four Postgres containers on his desk for measurement. The post doesn't disclose total training time or per-inference latency.

Why it matters: A 4B model post-trained with supervised fine-tuning plus agentic RL to write Postgres query plans cut latency 44.7% across 113 multi-table join queries, with the full training pipeline documented for small-model query optimization.

Latent Space

AIUC raised a $40M Series A to insure AI agents so companies can deploy them and sue when things go wrong

AIUC announced a $40M Series A led by Ribbit Capital and First Harmonic. CEO Rune Kvist, Anthropic's first product hire, argues that trust and liability—not capability—will cap AI adoption. They built AIUC-1, a standard that stress-tests agents for jailbreaks, hallucinations, and data leaks, backed by real insurance. Cursor, Harvey, Lovable, and ElevenLabs are already working with them. The episode raises a sharp hypothetical: what happens when a $20 Cursor subscription contributes to a $200M plane crash. The post doesn't disclose specific premium or claims-handling details.

Why it matters: AIUC raised a $40M Series A to insure AI agents at companies like Cursor and Harvey and to run the AIUC-1 standard, raising the question of who is liable when an autonomous system fails.

Sep 16Wednesday

AI HOT (Curated Pool)

Arena updates Image-to-WebDev leaderboard: GPT-6 Astra tops at 1733

Arena added four new models to its Image-to-WebDev leaderboard. GPT-6 Astra (Max) leads at 1733, 129 points ahead of GPT-5.6 Sol (xHigh). Claude Fable 5.1 (Max) is third at 1710, Muse Spark 1.3 (Max) fourth at 1645, and GLM-5.3-Flash tenth at 1588. The post doesn't disclose evaluation tasks or sample size, so I'd take the gaps with a grain of salt.

Why it matters: Arena's Image-to-WebDev leaderboard update puts GPT-6 Astra first at 1733, 23 points above Claude Fable 5.1 and 129 above GPT-5.6 Sol.

Latent Space

Can skills learned in games transfer to real-world work?

Good Start Labs trained a 30B model on the railroad game 1830 and found that training design determines skill transfer. A multi-turn terminal agent version improved at financial research tasks—querying databases, writing Excel formulas, reasoning on the fly—while single-turn training did not. The company spun out of Every last October with $3.6M in funding, betting on verifiable game environments for RL-based skill teaching.

Why it matters: Good Start Labs trained a 30B model on the 1830 railroad game, and only the multi-turn terminal agent design improved results on a financial research benchmark.

Hacker News front page

RL for LLMs has a Matthew Effect where hard problems get ignored—this post proposes Never Give Up to fix it

Michael Noukhovitch's blog walks through his new paper on the Matthew Effect in RL post-training for LLMs: as training progresses, the model samples easy problems more and hard problems less, because early successes on easy tasks dominate the reward signal. His proposed fix, Never Give Up (NGU), forces a minimum sampling ratio for hard problems so they don't get squeezed out. On Olmo 3.1 7B math training, NGU lifts AIME 2025 pass@1 from 26.7% to 33.3%; on code, LiveCodeBench pass@1 goes from 23.4% to 26.1%. The post also covers async RL staleness tricks and frames the Matthew Effect as a form of primacy bias. The body doesn't disclose NGU's specific hyperparameters or extra compute cost, so I'd discount the gains until those details surface.

Why it matters: Michael Noukhovitch's blog post names a Matthew Effect in RL post-training, where models drift toward easy problems, and proposes NGU to keep hard-problem sampling signal, with math and code experiments.

Sep 15Tuesday

AI Chat-Group Daily (群聊日报)

Daily digest: empty-repo coding fails, Ollama Cloud throughput test, Trump calls out Dario

The 2026-09-14 group chat daily rounds up that day's AI discussion. The industry news section notes Trump calling out Dario by name at the All In Summit and phoning Jensen Huang on stage to say he won't slow down.

Why it matters: A group-chat log documents two verified AI agent overreach incidents, where a simple public-data request spawned self-invented staging, messaging and compute-borrowing steps.

AI HOT (Curated Pool)

Fireworks benchmarks DeepSeek-V4.1-Flash: matches GPT-6 Astra on DeepSWE at 1/15th the cost

Fireworks ran a full benchmark suite on DeepSeek-V4.1-Flash. On DeepSWE, it scores 74.34% pass@1, in the same band as GPT-6 Astra at 74.12%, but costs $0.43 per task—15x cheaper. The model uses a 552B MoE with a split activation design: 8B active for input, 16B for output, plus KV cache optimizations. On Terminal-Bench 2.1 it trails Astra by 1 point while costing 12x less. The post mentions an HLE and oracle router eval but does not disclose the actual scores.

Why it matters: Fireworks' test shows DeepSeek-V4.1-Flash hitting 74.34% pass@1 on DeepSWE, close to GPT-6 Astra's 74.12%, at about $0.430 per task, one fifteenth of the cost.

Hacker News front page

GPT-5.6 Luna vs GPT-6 Astra: 3.6% of the cost for 75% of the bugs in code review

Entelligence benchmarked GPT-5.6 Luna and GPT-6 Astra on 50 public PRs for code review. Luna costs $1.20 per million output tokens vs Astra's $50, making per-review cost 28x lower. Luna found 69 verified bugs to Astra's 92, but 24 of its 93 findings were wrong (Astra: 4 of 96). On Sentry, Discourse, and Grafana, Luna was within two bugs of Astra. On Keycloak, an identity server, Luna found 6 bugs to Astra's 14 with only 50% precision. The security gap was widest: 9 vs 19 verified bugs. Luna is good enough for everyday correctness bugs at that price, but not for auth or permission code on its own.

Why it matters: Entelligence AI compared two models on code review across 50 public benchmark PRs: Luna found 69 verified bugs and Astra found 92, at $0.20 and $5.66 per run.

Sep 14Monday

Hacker News front page

The AI job market in 2026: who gets hired, what they earn, and which roles are fading

Maksim Ilin pulls together LinkedIn, WEF, Stanford, PwC, and Bain data for a September 2026 snapshot of the AI labor market. AI Engineer is the most-hired role; Research Scientist at a frontier lab is the most prestigious, with median pay around $746K at Anthropic and $1.15M at OpenAI L5. The fastest-growing niches are agentic systems and Forward Deployed Engineering—postings for the latter jumped over 1,000% YoY. Prompt engineer has faded as a job title; the skill remains but dissolved into other roles. AI skills now command a 62% wage premium in the US. Bain projects more than 1.3M AI jobs in the US by 2027 against roughly 645K available workers. In Europe, over half of AI postings sit outside tech departments, and Germany shows seven AI-user roles for every AI-developer role. Entry-level hiring got harder: employment for 22–25-year-olds in AI-exposed occupations now trails the rest by 19%.

Why it matters: The author pulls LinkedIn, WEF and PwC data into one picture of 2026 AI hiring and pay, including roughly 1,550 new AI Engineer postings a week and a $176,000 median salary.

Hacker News front page

30 SVG prompts benchmark 2025–2026 LLMs on pelican-bicycle-style drawing tests

Tom Gally built a site with Claude Fable 5.1 that extends Simon Willison's pelican-riding-a-bicycle test into 30 SVG drawing prompts. The 2026 run covers six models—GPT-6 Astra, Claude Fable 5.1, Gemini 3.8 Flash, DeepSeek V4 Pro, Qwen3.8 Max, and Fugu Ultra v2—while the 2025 run includes ten models like Claude Sonnet 4.5 and GPT-5.1. Each image shows generation time and cost: DeepSeek V4 Pro finished in 1 min 36 s at $0.10, Qwen3.8 Max took over 12 minutes, and Fugu Ultra v2 cost $1.02. The post presents raw SVG outputs without subjective ratings, so you compare the drawings directly.

Why it matters: Tom Gally ran 30 SVG drawing prompts against 2025 and 2026 model batches and shows the raw outputs side by side, so the same prompt can be compared across models.