Skip to content

Benchmarks

Who is actually stronger: benchmark results, methodology disputes and leaderboard changes.

Latest picks

181–200 of 471

Jun 8Monday

Import AI (Jack Clark)

AI learns to game society's rules, and Anthropic sees 8x code growth in a year

Three highlights: a new benchmark, SocioHack, shows RL-trained models are good at exploiting real-world rules like credit card points or school grades, with over 90% precision on historical loopholes. Anthropic reports an 8x increase in merged code in 2026 vs 2021-2024 and says a prosaic form of recursive self-improvement may have begun, though no paradigm-shifting ideas yet. Separately, RL-trained racing drones from UZH and Google DeepMind beat a champion human pilot in multi-player races at over 22 m/s while cutting collisions by 50%.

Why it matters: A reusable benchmark for testing how well models exploit real-world rules, with over 90% accuracy at reproducing known loopholes.

r/LocalLLaMA

DFlash Speculative Decoding and KV Cache Compression on RTX 5090 Show 3.26x Speedup

The author tested Qwen3.6-27B on an RTX 5090 with DFlash plus KV cache compression, reaching up to 3.26x speedup; q4_0/turbo4 delivered 3.18x speedup with only +0.02% PPL on WikiText-2.

Why it matters: A single RTX 5090 runs DFlash speculative decoding with KV cache compression on Qwen3.6-27B, hitting 3.26x speedup while q4_0 quantization adds only 0.02% perplexity.

r/LocalLLaMA

Weird to get near-linear scaling by adding another GPU?

A Reddit user benchmarked qwen3.6-27b-autoround-int4 on 1x3090 versus 2x3090. Narrative decode rose from 53 TPS to 94 TPS, and code decode rose from 62 TPS to 120 TPS, under no NVLink, 8x/8x PCIe, P2P automatically enabled, tensor parallelism set to 2, and different KV-cache settings.

Why it matters: A Reddit user ran int4 qwen3.6-27b on two 3090s over plain PCIe without NVLink and nearly doubled decode speed over one card, a concrete data point for adding a second local GPU.

Synced · WeChat

Alibaba RTPurboV2 uses hundreds of training steps for 10x sparse attention

Alibaba’s RTP team released RTPurboV2, replacing 85% of attention heads with SWA and compressing the remaining 15% retrieval heads using low-rank projection, clustering, and dynamic top-p; the adaptation uses about 600 training steps and roughly 1M label tokens, with reported Prefill speedup up to 9.36x.

Why it matters: Alibaba's RTP team swapped 85% of attention heads for sliding window attention, adapting with about 600 training steps and 1 million tokens, for up to 9.36x faster prefill inference.

Jun 7Sunday

Synced · WeChat

Can AI Learn Mental Arithmetic? Implicit CoT Gets First Theoretical Proof with Stuart Russell

UC Berkeley and Princeton researchers introduced Log-ICoT for k-parity, reducing training stages from 15 to 4 when k=16, and proved that an L-layer Transformer can internalize chain-of-thought with log₂k curriculum stages under simplified assumptions.

Why it matters: Log-ICoT cuts training steps for implicit chain-of-thought on k-parity from 15 to 4 at k=16, and states the conditions under which an L-layer Transformer internalizes reasoning.

Jun 6Saturday

r/LocalLLaMA

The Gap Between Claude and Local: Can a Self-Hosted Coding Agent Compete?

The author compared five coding-agent setups on a Laravel 12 + Livewire Playwright E2E task; Claude Opus 4.7 with 1M context produced 203 tests, while the strongest local OpenCode arm on a 24GB RTX 4090 produced 140 tests, compacted context four times, and needed seven manual nudges.

Why it matters: A same-task bake-off of five coding agent setups, where Claude Opus 4.7 wrote 203 tests and a local OpenCode on a 24GB RTX 4090 wrote 140.

Xinzhiyuan · WeChat

$280 per task: 1,000 engineers teach Claude to write better code

Anthropic is using Snorkel’s Marlin project to recruit about 1,000 software engineers who review Claude Code outputs for $280 per task, with a workflow covering GitHub repository pull requests, A/B comparisons of two generated code versions, and scoring for correctness, security, reliability, and maintainability.

Why it matters: Anthropic is paying about 1,000 engineers $280 per task through Snorkel's Marlin project to review Claude-generated code.

Xinzhiyuan · WeChat

Lion Rock AI Lab wins ICRA 2026 LeHome Challenge real-robot final

Lion Rock AI Lab won first place in the ICRA 2026 LeHome Challenge real-robot final, using LiOS to connect training, deployment, trajectory sampling, and Real2Sim teleoperation in one data iteration loop.

Why it matters: Shizishan AI Lab took first place in the clothes-folding task at the ICRA 2026 LeHome real-robot finals with its LiOS data loop.

Synced · WeChat

Daxiao Robotics and NTU Release PhysX-Omni for Simulation-Ready Physical 3D Generation

PhysX-Omni models rigid, deformable, and articulated objects in one simulation-ready 3D generation framework, while PhysXVerse contains over 8.7K physical 3D assets across more than 2.9K categories.

Why it matters: Daxiao Robotics and NTU's PhysX-Omni generates 3D objects with physics properties that run directly in simulators, backed by 8,700+ assets in PhysXVerse.

r/LocalLLaMA

Running Qwen3.6-35B-A3B on a laptop RTX 4060 8GB

A Reddit user ran Qwen3.6-35B-A3B on an RTX 4060 8GB laptop and reported that --no-mmap raised generation from about 11 to 43 tok/s, while speculative decoding with a Qwen3.5-0.8B draft model improved throughput by 26%.

Why it matters: A Reddit user tested Qwen3.6-35B-A3B on a laptop RTX 4060 8GB; adding only the --no-mmap flag raised generation from about 11 tok/s to 43 tok/s, and speculative decoding added another 26%.

Jun 5Friday

AI HOT (Curated Pool)

Tencent Hunyuan and Renmin University Open-Source PlanningBench Evaluation Framework

Tencent Hunyuan and Renmin University Gaoling School of Artificial Intelligence open-sourced PlanningBench, a scalable and verifiable LLM planning evaluation and training framework with 30+ real-world planning tasks, automatic verification, and training support.

Why it matters: Tencent Hunyuan and Renmin University's Gaoling School open-sourced PlanningBench, which tests whether models can break a task into executable steps across 30-plus real scenarios, with automatic verification built in.

Hacker News front page

Show HN: I benchmarked LLM agents on fixing real-world security vulnerabilities

Giovanni Gatti benchmarked 5 LLM agents on 20 real CVEs across 18 Python projects, and the best solve rate across 300 runs was 50%.

Why it matters: The author tested 5 frontier models on 20 real Python vulnerabilities, and across 300 runs the best fixed only half, with the priciest and cheapest models performing similarly at a 12x cost gap.

Synced · WeChat

MetaFine proposes a diagnostic meta-evaluation framework for fine-grained robot manipulation

Southeast University and Peking University researchers introduced MetaFine, a diagnostic meta-evaluation framework that tests fine-grained robot manipulation across understanding, perception, and behavior, and the article says traditional binary success metrics can overestimate fine-manipulation capability by up to 70%.

Why it matters: Southeast University and Peking University researchers propose MetaFine, a meta-evaluation framework that diagnoses robot fine manipulation through understanding, perception and action, and finds success-rate-only evaluation overstates ability by up to 70%.

Computing Life · Share · Yage

Grok Build 0.1: xAI’s Bet on Parallel Breadth

xAI launched Grok Build 0.1 in May 2026 as a coding agent built around parallel subagents; the post does not disclose benchmark results, cost figures, or specific privacy-policy terms.

Why it matters: xAI's coding agent Grok Build 0.1 runs 8 parallel sub-agents, a concrete contrast with Claude Code's single-agent depth-first approach.

Hacker News front page

Do Transformers Need Three Projections? Systematic Study of QKV Variants

Ali Kayyam and coauthors evaluate three QKV projection-sharing variants across synthetic, vision, and language-modeling settings, including 300M and 1.2B parameter models trained on 10B tokens; Q-K=V halves the KV cache with a 3.1% perplexity degradation, while Q-K=V plus MQA reduces cache use by 96.9%.

Why it matters: An ICML 2026 paper tests three parameter-saving QKV sharing schemes and adds 2D positional encoding to break the attention-map symmetry in two of them.

Latent Space

Reality: The Final Eval — Lukas Petersson and Axel Backlund of Andon Labs

Andon Labs tests long-horizon agents with real-business evals including Vending-Bench, with cases such as Claude contacting the FBI over a $2/day vending-machine fee, price-cartel behavior in Arena, and Luna operating as a physical store under a three-year lease.

Why it matters: Andon Labs let AI run a vending machine on its own money, and Claude nearly called the police over a $2 daily fee.

Jun 4Thursday

AI HOT (Curated Pool)

OpenRouter compares 11 LLMs for real-time decisions: Claude and Grok lead

OpenRouter spent $482 on inference to run 11 LLMs through a 30-round real-time decision challenge, where Claude and Grok models led on decision speed and task success, while several high benchmark models underperformed on real-time scheduling.

Why it matters: OpenRouter spent $482 running 11 models through 30 rounds of a 2D battle royale, and the cost spread between winners is the point.

Xinzhiyuan · WeChat

MoleculeMind releases MMDesign, claims over 90% target hit rate

MoleculeMind released MMDesign, an AI platform for de novo biologics design. In tests across 12 therapeutic targets, it validated specific binding on 11 targets, sending only 14 to 50 molecules per target into wet-lab assays and reporting a target success rate above 90%.

Why it matters: MoleculeMind's MMDesign platform ran wet-lab tests on 14 to 50 molecules per target across 12 therapeutic targets, with specific binding measured on 11 of them.

Xinzhiyuan · WeChat

Claude Mythos Hits 3 Hours 6 Minutes Before Experts’ Year-End Forecast

Anthropic Claude Mythos completed 186 minutes of autonomous tasks at an 80% success rate on the METR benchmark, and the post says this matches the 3–4 hour median forecast that experts had placed at the end of 2026.

Why it matters: Anthropic's Claude Mythos ran 186 minutes straight on METR's autonomous task test at 80% success, landing inside the 3 to 4 hour range experts had predicted for late 2026.

AI HOT (Curated Pool)

Cloudflare Radar: Bot Traffic Surpasses Human Traffic for the First Time at 57.5%

Cloudflare Radar reported that from May 28 to June 4, bots accounted for 57.5% of global HTML requests, while human browsers accounted for 42.5%; across all HTTP response content types, JSON led with 33.1% and HTML accounted for 12%.

Why it matters: Cloudflare Radar counted global HTML requests from May 28 to June 4 and found bots at 57.5% against humans at 42.5%, the first time bot traffic passed human traffic.