Skip to content

Benchmarks

Who is actually stronger: benchmark results, methodology disputes and leaderboard changes.

Latest picks

221–240 of 471

Jun 1Monday

Import AI (Jack Clark)

Import AI 459: AI oversight is difficult; scaling laws for protein folding models; and pricing the extinction risk of AI systems

Import AI 459 summarizes papers on AI-economy measurement and AI oversight: one estimates U.S. nominal AI GDP at about $250 billion in 2025, with quality-adjusted real growth near 2,600% per year.

Why it matters: This Import AI issue treats AI extinction risk as a priceable economic problem and estimates the US AI economy at roughly $250 billion nominal, growing near 2,600% a year after quality adjustment.

r/LocalLLaMA

Deepseek V4 Flash performance on DGX Spark

A Reddit user ran DeepSeek-V4-Flash with vLLM on two ASUS GX10 DGX Spark nodes and reported 1,680 prefill tokens/s plus 39.8 decode tokens/s at a 256K context with MTP=2; the setup uses TP=2 over RoCE, fp8 KV cache, and fits about 1M tokens safely in KV cache.

Why it matters: A Reddit user clustered two ASUS GX10 units and benchmarked DeepSeek-V4-Flash prefill and decode at 256K context, publishing the full tensor-parallel and RoCE network setup.

AI HOT (Curated Pool)

MWC26 Shanghai to Host First Humanoid Robot Penalty Shootout With Unitree and 7 Other Teams

MWC26 Shanghai will host a humanoid robot penalty shootout in June 2026, with eight Chinese embodied intelligence teams competing under rules that require autonomous play without human control or preset scripts.

Why it matters: Eight teams, including Unitree, must let their robots find the ball and keeper on their own, with no remote control or preset scripts.

May 31Sunday

r/LocalLLaMA

13 abliterated Gemma 4 E2B variants, 44 GPU hours, benchmark and comparison

Abliterlitics tested 13 abliterated Gemma 4 E2B variants using 44 RTX 5090 GPU hours, and HarmBench ASR rose from the base model’s 32.2% to 82%–100%, while coder3101 scored 84.8% on GSM8K versus the base model’s 83.5%.

Why it matters: The author spent 44 hours of RTX 5090 compute abliterating Gemma 4 E2B at 13 strengths and published each variant's benchmark scores and harmful-instruction pass rate, which rose from 32% to over 82%.

r/LocalLLaMA

PolyRange: Contamination-resistant offensive-AI benchmark for web targets

PolyRange v1.0 ships 84 WSTG-derived classes across 12 OWASP testing-guide categories. It generates fresh targets per deploy with a chosen LLM, adds two defense tiers, uses an agent-submits-flag oracle, and runs via a single-command CLI on Fly.io or Docker.

Why it matters: PolyRange turns OWASP's 12 categories and 84 attack techniques into a live range where an LLM generates fresh targets each run, so scores resist contamination.

Synced · WeChat

Rubrics Survey: How to Define a Good Answer in the Agent Era

Renmin University Gaoling School of Artificial Intelligence released a 40-page survey on rubrics for LLMs, organizing the topic into five parts: definitions, construction methods, training uses, evaluation scenarios, and open challenges.

Why it matters: A 40-page survey from Renmin University's Gaoling team splits scoring criteria into definition, construction, training, evaluation and open problems, answering the need for evaluation beyond answer correctness in the agent era.

Synced · WeChat

Microsoft open-sources SkillOpt for training Agent skill documents, reaching 3.3k stars in a week

Microsoft open-sourced SkillOpt, a text-space optimization framework that trains Agent skill documents without changing model weights; the paper reports best or tied-best results across 52 combinations covering 7 target models, 6 benchmarks, and 3 execution environments.

Why it matters: Microsoft's open-source SkillOpt leaves model weights alone and instead iterates the skill documents written for agents, with experiments spanning 52 combinations across 7 target models, 6 benchmarks and 3 environments.

May 30Saturday

Xinzhiyuan · WeChat

Claude AI fluency scorecard surfaces, with strong users scoring 7.5

Anthropic is testing a Claude AI Fluency scorecard that analyzes Chat, Cowork, and Claude Code history against 11 observable behaviors, with an 11-point maximum score. The underlying study used 9,830 anonymized multi-turn conversations, and iteration appeared in 85.7% of high-quality conversations.

Why it matters: Anthropic scored users on 11 behavioral metrics drawn from 9,830 anonymous multi-turn conversations, and 85.7% of high-quality chats involved repeated revision and follow-up questions.

QbitAI · WeChat

RUC and Zhizhi Institute Open-Source Claw Agent Data, Training, and Evaluation Pipeline

Renmin University of China and Zhizhi Institute open-sourced ClawGym, a Claw Agent framework with 13.5K synthetic executable tasks, 200 benchmark tasks, model checkpoints, training data, and training code; ClawGym-30B-A3B scores 56.82 on ClawGym-Bench and exceeds Qwen3-235B-A23B in the reported evaluation.

Why it matters: Renmin University and Zhizhi Research open-sourced ClawGym, releasing 13,500 executable tasks, training code, model weights and a 200-task benchmark as a reproducible starting point for agent training.

Synced · WeChat

CUHK Pion optimizer updates LLMs on iso-spectral manifolds to address AdamW and Muon instability

CUHK and collaborators introduced Pion, an optimizer that preserves weight singular values through orthogonal equivalence transformations, and reported that it kept a 60M normalization-free LLaMA-like model stable for 9.6B training tokens while AdamW and Muon collapsed with NaNs.

Why it matters: A CUHK team's Pion optimizer keeps singular values fixed and updates only via orthogonal transforms, training a 60M-parameter model for 9.6B tokens where AdamW and Muon hit NaN.

r/LocalLLaMA

Testing MTP on vLLM and llama.cpp for Gemma 4 and Qwen 3.6

The author tested MTP on an RTX PRO 6000 Blackwell setup, where Gemma 4 31B on vLLM reached 132.52 tok/s versus a 39.69 tok/s baseline, a 3.34x speedup; the post reports 10 runs of 1,500 tokens each but does not provide a full quality or VRAM evaluation.

Why it matters: A user benchmarked Gemma 4 31B with vLLM on an RTX PRO 6000 with and without MTP, seeing throughput rise from 39.69 to 132.52 tokens per second on specific hardware and framework.

May 29Friday

New York Times Chinese

Anthropic Tops OpenAI Valuation to Become the Most Valuable AI Startup

Anthropic raised $65 billion at a $900 billion pre-money valuation, above OpenAI’s last $730 billion valuation. Claude Opus 4.8 also scored 10% higher than Anthropic’s previous model on Vals AI’s vibe-coding benchmark.

Why it matters: Anthropic's new $65 billion round puts its pre-money valuation at $900 billion, above OpenAI's $730 billion from its last round, reshuffling the top of the foundation model table.

Xinzhiyuan · WeChat

Three DeepSeek Models Enter OpenRouter Monthly Top 10 With Over 17 Trillion Tokens

DeepSeek placed three models in OpenRouter’s monthly top 10 with more than 17 trillion tokens combined, including V4 Flash at 9.13T tokens; the article says Ascend’s MegaMoE operator raised Prefill throughput by 20% to 30% on DeepSeek V3.1 and Qwen3-235B tests.

Why it matters: OpenRouter's monthly stats put three DeepSeek models in the top ten, showing how much traffic Chinese models draw on an aggregation platform.

AI HOT (Curated Pool)

Adam's Law: Prompts Written with High-Frequency Words Work Better

FaceMind tested 100 languages and four core tasks, finding that, with semantics unchanged, prompts or fine-tuning text using higher-frequency expressions from pretraining data improves large language model performance.

Why it matters: The FaceMind team tested 100 languages across four task types and found that phrasing prompts the way high-frequency pretraining text does improves model performance, adding a frequency angle to prompt writing.

AI HOT (Curated Pool)

Tesla FSD Safety Claims Face Scrutiny

Tesla claimed FSD can be up to 10 times safer than humans, but Reuters found flaws in the comparison, with 11 traffic safety researchers saying Tesla used inappropriate baselines against broader federal crash data.

Why it matters: Reuters checked Tesla's safety report, and researchers plus former employees question the numbers behind its claims.

May 28Thursday

QbitAI · WeChat

A New Paradigm for GUI Agent Trajectories: FSMs Generate Trajectories at $0.04 Each

AutoWebWorld synthesized 29 web environments, 875 pages, and 11,663 verified trajectories at about $0.04 per trajectory, using FSM-defined states, preconditions, and transitions to verify GUI agent tasks instead of human labeling or an LLM judge.

Why it matters: AutoWebWorld defines page states and transitions with a finite state machine, then generates and verifies GUI agent trajectories, producing 11,663 of them at about $0.04 each.

r/LocalLLaMA

Qwen3.6-35B-A3B-APEX Runs 128K Context on RTX 3060 12GB

old-mike ran mudler/Qwen3.6-35B-A3B-APEX-MTP-I-Compact.gguf through spiritbuun’s llama.cpp fork on one RTX 3060 12GB, offloading a 17.3GB model and reaching 37.17 t/s generation at 72K filled context, 28.08 t/s at 129K, and PPL 3.2529 on an enwik8 64K-context perplexity test.

Why it matters: A user ran a 17.3GB Qwen model on one 12GB RTX 3060 at 37 tokens/s with 72K context filled, giving concrete settings for consumer-GPU inference.

Latent Space

Cognition Raises $1B in $26B Series D

Cognition raised a $1B Series D at a $26B valuation and projects more than $1B ARR by year-end; the post says its valuation rose 2.5× from the $10B Series C eight months earlier, while the rest of the issue summarizes agent, inference, benchmark, and multimodal AI updates from May 26–27, 2026.

Why it matters: Cognition raised $1 billion in a Series D at a $26 billion valuation, up from $10 billion in eight months, a concrete gauge of capital heat in coding agents.

Synced · WeChat

Mila and DeepMind Propose UNSL for Unified Multivariate Neural Scaling Laws

Mila and Google DeepMind proposed Unified Neural Scaling Law, a multivariate scaling-law form that models parameter count, token count, training steps, bottlenecks, overfitting, and adverse hyperparameter effects; UNSL achieved the best extrapolation on 60.87% of vision tasks and 88.89% of language tasks in the reported experiments.

Why it matters: Mila and DeepMind proposed UNSL, putting parameter count, token count and training steps into one scaling formula with bottlenecks and overfitting, a unified frame for training configuration.

Computing Life · Share · Yage

After SWE-Bench Pro Saturation, Someone Built a New Benchmark

DeepSWE says SWE-Bench Pro lost discrimination because of data contamination and verifier flaws; the same model set showed a 62-point spread on the benchmark, while the snippet does not disclose the audited models or test protocol.

Why it matters: Datacurve's DeepSWE benchmark spreads the same models across a 62-point gap where SWE-Bench Pro shows only a few points, showing the old benchmark can no longer separate models.