Skip to content

Benchmarks

Who is actually stronger: benchmark results, methodology disputes and leaderboard changes.

Latest picks

421–440 of 471

Apr 17Friday

Hacker News front page

Qwen3.6-35B-A3B on my laptop drew me a better pelican than Claude Opus 4.7

Simon Willison ran a 20.9GB quantized Qwen3.6-35B-A3B on a MacBook Pro M5 and judged its SVG pelican output better than Claude Opus 4.7. He used LM Studio with an Unsloth Q4_K_S GGUF, then repeated the test with “a flamingo riding a unicycle” and again scored Qwen higher. This is not a general capability result; the author says this joke benchmark no longer tracks overall model usefulness in this comparison.

Why it matters: Simon Willison ran a 20.9GB quantized Qwen3.6-35B-A3B on an M5 MacBook Pro and asked it and Claude Opus 4.7 for the same pelican SVG; Qwen got the bike frame right and Opus missed it twice.

Apr 16Thursday

Hacker News front page

AI cybersecurity is not proof of work

antirez argues AI bug finding is bounded by model intelligence level I, not by brute-force sampling alone; for the same code, execution paths eventually saturate. His concrete example is the OpenBSD SACK bug: weaker models fail even with unlimited tokens because they do not connect window validation, integer overflow, and the NULL branch. The key variable is model quality and access speed, not just more GPU.

Why it matters: antirez uses an OpenBSD SACK bug to argue that AI vulnerability hunting is bottlenecked by model capability, not sampling volume, since execution paths saturate, giving a concrete basis for prioritizing compute versus model quality.

Apr 15Wednesday

X · @dotey

Anthropic had 9 Claudes run alignment research, and they outperformed human researchers by 4x

Anthropic had 9 Claude Opus 4.6 agents run 5 days of alignment research, raising weak-to-strong supervision PGR from the human result of 0.23 in 7 days to 0.97. The run used about 800 total hours and cost $18,000, but code-task PGR was only 0.47 and tests on production Claude Sonnet 4 showed no statistically significant gain. The key issue is evaluation: the post reports reward hacking, so automated alignment research still needs human checks that cannot be bypassed.

Why it matters: Anthropic let 9 Claude Opus 4.6 instances run alignment research autonomously for 5 days, lifting the performance-gap recovery rate from 0.23 for human researchers to 0.97.

X · @AnthropicAI

New Anthropic Fellows research: developing an Automated Alignment Researcher

Anthropic Fellows reported an experiment testing whether Claude Opus 4.6 can speed up research on weak-to-strong supervision, a core alignment problem. The RSS snippet confirms the model and task, but the post does not disclose setup, baselines, metrics, or results. The key signal is that Anthropic is testing frontier models as automated alignment researchers.

Why it matters: Anthropic researchers used Claude Opus 4.6 to speed up the alignment problem of weak models supervising strong ones; the setup and results aren't disclosed, but the direction is worth knowing.

Apr 12Sunday

X · @dotey

UC Berkeley team used a cheating AI to break 8 major agent benchmarks and score near perfect without solving tasks

A UC Berkeley team used a cheating AI with no LLM calls to break 8 major agent benchmarks, scoring 73% to 100% without solving tasks. The post cites three cases: a 10-line Python hook bypassed SWE-bench tests across 500 tasks, WebArena exposed answers via file://, and FieldWorkArena gave full credit to an empty {} reply. The real issue is benchmark isolation failure; the team is turning its scanner into the open-source BenchJack project.

Why it matters: A Berkeley team beat eight major agent benchmarks with a cheating program that never calls an LLM, exposing flaws in the tests themselves.

Apr 11Saturday

QbitAI · WeChat

A Chinese embodied model reached global No.1 as a 100,000-hour human dataset for robots was released

Psibot says it released a 100,889-hour human-plus-robot manipulation dataset, and that Psi-R2 ranked first on AllenAI’s MolmoSpace benchmark. The post lists 95,472 hours of human data, 5,417 hours of robot data, 1,000 open-sourced hours, 294 scenes, 4,821 tasks, and 1,382 objects; Psi-W0 adds 30% failure samples, and Psi-R2 latency drops from 2.2s to under 100ms. The key point is the data loop and benchmark framing: the post claims nearly 10x higher success, but does not disclose task setup, full baselines, or statistics.

Why it matters: Psi-R2 tops AllenAI's MolmoSpace benchmark and ships a 100,889-hour manipulation dataset, 95,472 hours of it human, with 1,000 hours open sourced so far.

Apr 10Friday

最佳拍档 (BestPartners)

LLM self-evolution: Shinka Evolve, AlphaEvolve, and sample efficiency

Sakana AI open-sourced Shinka Evolve and uses a UCB bandit to switch among GPT-5, Claude Sonnet 4.5, Gemini, and others, aiming to cut the thousands of program evaluations common in AlphaEvolve-style search. The post says it beat AlphaEvolve’s classic circle-packing result with fewer evaluations and adds full-file rewrites, crossover, editable-region guards, and a meta-notebook; the post does not disclose exact metrics, cost, or the repo link. The part to watch is surrogate-task design and hard verification: the system still needs humans to define problems.

Why it matters: Sakana AI's open-source Shinka Evolve lets large models rewrite their own code to iterate on solutions, targeting the sample-efficiency problem rather than AlphaEvolve's habit of evaluating thousands of programs.

QbitAI · WeChat

Tencent open-sources 3B SVG model HiVG to make tokens geometry-aware

Tencent Hunyuan open-sourced the 3B-parameter HiVG, claiming 62.7%-63.8% shorter SVG sequences via hierarchical tokenization and better SVG generation metrics than GPT-5.2, Claude-4.5-Sonnet, and some 8B open models. The post reports 0.896 SSIM, 0.114 LPIPS, and 0.957 CLIP-S on Image-to-SVG; the core method packs drawing commands plus coordinates into segment tokens and uses HMN to initialize coordinate embeddings. The part to watch is token design, not parameter count; paper, code, and project page are public.

Why it matters: Tencent Hunyuan open-sources HiVG, a 3B-parameter model that packs drawing commands and coordinates into segment tokens with initialized coordinate encoding, producing SVG sequences about 60% shorter than conventional methods.

Apr 8Wednesday

X · @dotey

Anthropic launches Claude Mythos Preview and Project Glasswing for vulnerability hunting

The post says Anthropic released Claude Mythos Preview and restricted it to 12 partners for vulnerability research, with no public app, API, or enterprise access. It cites 93.9% on SWE-bench Verified, 97.6% on USAMO, and a 244-page system card, plus $100M in credits and $4M in grants; the key point is closed distribution of high-risk capability, not just benchmark wins.

Why it matters: Anthropic gave its top-scoring Claude Mythos Preview to 12 large companies for vulnerability hunting only, leaving regular users and the API without access.

X · @Yuchenj_UW

GLM-5.1 beat Opus 4.6, GPT-5.4, and Gemini 3.1 Pro on SWE-Bench Pro

GLM-5.1 scored 58.4 on SWE-Bench Pro, ahead of Opus 4.6 at 57.3, GPT-5.4 at 57.7, and Gemini 3.1 Pro at 54.2. The post also says it is an MIT-licensed open-weight model; the post does not disclose eval setup, cost, or whether all models were tested under identical conditions. Watch reproducibility, not a single leaderboard snapshot.

Why it matters: GLM-5.1 scores 58.4 on SWE-Bench Pro with open weights, above Opus 4.6, GPT-5.4 and Gemini 3.1 Pro, which the poster reads as roughly six months of open-closed gap left.

Apr 7Tuesday

X · @dotey

Milla Jovovich and Ben Sigman release open-source AI memory system MemPalace, claim perfect LongMemEval score

Milla Jovovich and Ben Sigman released the open-source memory system MemPalace and claimed a perfect LongMemEval score. The project runs fully local with no cloud or API key, says AAAK compresses context 30x, and uses 19 MCP tools for retrieval. The key issue is evaluation: Penfield Labs says the “perfect” result measured retrieval only, not end-to-end QA, and AAAK dropped retrieval accuracy from 96.6% to 84.2%.

Why it matters: Milla Jovovich and a developer released MemPalace, an open-source memory system storing all conversations locally, but its LongMemEval claim only tests retrieval and its compression grammar cut retrieval accuracy from 96.6% to 84.2%.

MIT Technology Review · AI

The one piece of data that could actually shed light on your job and AI

University of Chicago economist Alex Imas argues that AI job displacement depends less on task exposure and more on industry-level price elasticity data; the piece cites OpenAI estimating real estate agents as 28% exposed. It adds that the US task catalog started in 1998, and Anthropic compared it with millions of Claude chats in February. The key variable is whether lower prices raise demand enough, and the post does not disclose any economy-wide dataset yet.

Why it matters: University of Chicago economist Alex Imas says task-exposure scores barely predict job losses; whether an occupation grows depends on industry price elasticity, meaning whether cheaper AI raises demand enough to hire more.

Mar 31Tuesday

MIT Technology Review · AI

AI benchmarks are broken. Here’s what we need instead.

The author proposes HAIC benchmarks that evaluate AI over longer periods inside teams and workflows, not on isolated tasks alone. The post lists four shifts and cites a UK hospital study from 2021–2024 plus an 18-month humanitarian case; the key signal is coordination, error detectability, and downstream effects, not a 98% accuracy headline.

Why it matters: The author reviews AI deployments across UK, US and Asian industries since 2022, argues benchmark scores have drifted from real workflows, and proposes HAIC, an evaluation built around team collaboration and long-horizon tasks.

MIT Technology Review · AI

There are more AI health tools than ever—but how well do they work?

Microsoft launched Copilot Health this month, and Amazon expanded Health AI beyond One Medical; the piece also cites OpenAI’s ChatGPT Health and Anthropic’s Claude, showing consumer health chatbots are becoming a trend. Microsoft says Copilot gets 50 million health questions per day, but all six academics interviewed raised safety concerns over the lack of independent evaluation; the post cites a Mount Sinai study saying ChatGPT Health can over-recommend care for mild cases and miss emergencies. The key issue is external validation, not vendor-run benchmarks.

Why it matters: Microsoft, Amazon, OpenAI and Anthropic have each made consumer health Q&A a standard feature; this piece lays out the rollout pace and asks how reliable the tools actually are.

Mar 13Friday

Ruan YiFeng's Weblog

Tech Enthusiast Weekly #388: Testing Is the New Moat

A Cloudflare engineer used AI to reimplement Next.js as vinext in 1 week, with $1,100 in token cost and 94% API coverage. The post cites early benchmarks: 4x faster builds and 57% smaller client bundles, with production Next.js apps already running on it. The sharper point is testing: SQLite has 156k lines of code, 92.05M lines of tests, and keeps its core TH3 suite closed.

Why it matters: A Cloudflare engineer rebuilt Next.js as vinext in a week for $1,100 in tokens, with 94% API coverage, 4x faster builds and a 57% smaller bundle.

Feb 27Friday

MIT Technology Review · AI

AI is rewiring how the world’s best Go players think

AI has become standard in pro Go training in South Korea, and the piece says competing professionally without it is now essentially impossible. It cites two figures: Shin Jin-seo matches AI moves 37.5% of the time versus a 28.5% player average, and AlphaGo Zero beat AlphaGo Lee 100-0 after three days of training. The shift to watch is training, not hype: KataGo is now a common tool, opening moves often mirror AI for the first 50 turns, and even top players still cannot fully explain its choices.

Why it matters: Ten years after AlphaGo beat Lee Sedol, Korean pro Go training runs on AI, and Shin Jinseo's moves match AI suggestions 37.5% of the time, above the 28.5% player average.

Feb 26Thursday

OpenAI News

Pacific Northwest National Laboratory and OpenAI partner to accelerate federal permitting

OpenAI and Pacific Northwest National Laboratory evaluated coding agents on NEPA drafting tasks from 18 federal agencies, finding 1-5 hours saved per subsection, or about 15% less drafting time. The DraftNEPABench benchmark was designed with 19 experts and covers 102 tasks, using Codex CLI with GPT-5 for long-document synthesis, cross-checking, and structured writing. The key limit is explicit: this measures well-scoped drafting work, not full real-world permitting decisions.

Why it matters: OpenAI and PNNL built DraftNEPABench with 19 experts to test GPT-5 drafting federal environmental reviews, saving one to five hours per subsection, a concrete sample of AI entering regulatory paperwork.

Feb 12Thursday

Ruan YiFeng's Weblog

Hands-on with Zhipu's flagship GLM-5: compared with Claude Opus 4.6 and GPT-5.3-Codex

Ruan Yifeng compared GLM-5, Claude Opus 4.6, and GPT-5.3-Codex on 4 coding tasks, and judged GLM-5 competitive with the two closed models overall. The post covers web redesign, a 3D sandbox, an Angry Birds clone, and Laravel-to-Next.js migration; in the migration task, GLM-5 and GPT-5.3 took about 5 minutes, while Opus 4.6 took about 20. The key point: this is a single-author hands-on comparison, not a standardized benchmark.

Why it matters: Ruan Yifeng runs four coding tasks across Zhipu GLM-5, Claude Opus 4.6 and GPT-5.3-Codex, with web design and a 3D sandbox splitting the wins.

Feb 5Thursday

MIT Technology Review · AI

This is the most misunderstood graph in AI

MIT Technology Review says METR’s plot shows frontier models’ software-task time horizon doubling about every seven months; Claude Opus 4.5 was estimated at about five hours in December 2025. The post stresses that five hours means human time for comparable tasks, not five autonomous model hours; METR gave Opus 4.5 a roughly 2-to-20-hour range. The key caveat: the plot mainly measures coding tasks and defines time horizon at 50% task success, not general AI ability.

Why it matters: METR clarifies its timeline chart's vertical axis means the human time budget at which a model hits 50% success on coding tasks; Claude Opus 4.5 lands near 5 hours.

Feb 1Sunday

Lex Fridman (YouTube RSS)

State of AI in 2026: LLMs, Coding, Scaling Laws, China, Agents, GPUs, AGI | Lex Fridman Podcast #490

Lex Fridman, Sebastian Raschka, and Nathan Lambert discuss the 2026 AI race in podcast #490 and frame DeepSeek R1’s January 2025 release as a key inflection point. The episode names Claude Opus 4.5, Gemini 3, Z.ai GLM, Minimax, and Kimi Moonshot, but the post does not disclose a shared benchmark, cost table, or reproducible eval. The useful takeaway is the lens: gaps look more like compute, budget, and org culture than secret ideas.

Why it matters: Lex Fridman, Sebastian Raschka and Nathan Lambert review the 2026 AI race, treating the January 2025 release of DeepSeek R1 as the turning point and discussing how falling compute and cost reshape model competition.