Skip to content

Benchmarks

Who is actually stronger: benchmark results, methodology disputes and leaderboard changes.

Latest picks

321–340 of 471

May 10Sunday

r/LocalLLaMA

NVIDIA AI Releases Star Elastic: One Checkpoint Contains 30B, 23B, and 12B Reasoning Models

NVIDIA AI released Star Elastic, a single checkpoint that can zero-shot slice 30B, 23B, and 12B reasoning models in BF16, FP8, and NVFP4; when the 23B submodel handles thinking and the 30B model handles final answers, reported accuracy rises 16% and latency drops 1.9× on AIME-2025, GPQA, LiveCodeBench v5, and MMLU-Pro.

Why it matters: NVIDIA's Star Elastic slices 30B, 23B and 12B reasoning models from one checkpoint with no extra training, the 23B thinking and the 30B answering.

May 9Saturday

r/LocalLLaMA

80 tok/sec and 128K context on 12GB VRAM with Qwen3.6 35B A3B and llama.cpp MTP

Reddit user janvitos ran Qwen3.6-35B-A3B-MTP-GGUF with a llama.cpp MTP PR on an RTX 4070 Super. The posted benchmark shows 69.2-81.9 tok/s, 0.694-0.947 draft acceptance, 131072 context, and a -fitt 1536 setting that reserves 1536 MB for the draft model and KV cache.

Why it matters: Reddit user janvitos runs Qwen3.6 35B A3B on a 12GB RTX 4070 Super and posts measured settings for 69.2 to 81.9 tok/s at 128K context.

Synced · WeChat

StarVLA Open-Sources a Unified VLA Framework from HKUST and the Community

HKUST and the open-source community released StarVLA, a unified Vision-Language-Action framework that integrates backbones, action heads, training strategies, and evaluation interfaces; the repository has 2.2k GitHub stars and supports benchmarks including LIBERO, SimplerEnv, RoboTwin 2.0, RoboCasa-GR1, and BEHAVIOR-1K.

Why it matters: HKUST and the community open-sourced StarVLA, making VLA backbones, action heads and training interfaces pluggable instead of rebuilt per paper.

Synced · WeChat

DeepSeek Reportedly Raises RMB 50B, with Liang Wenfeng Funding 40%, Valuation Reaching RMB 350B

DeepSeek is negotiating a $7.3 billion funding round at an estimated $51.5 billion valuation; Liang Wenfeng reportedly plans to contribute 40%, while Tencent and China’s RMB 60 billion national AI fund are also in talks.

Why it matters: Liang Wenfeng plans to fund 40% of DeepSeek's roughly 50 billion yuan round himself, with Tencent and a state AI fund also in talks.

AI HOT (Curated Pool)

Claude Mythos Evaluation Shows 16-Hour Risk Horizon

METR evaluated an early Claude Mythos Preview build during a limited March 2026 window and estimated its 50% time horizon at at least 16 hours, with a 95% confidence interval of 8.5 to 55 hours.

Why it matters: METR's risk assessment of an early Claude Mythos preview measured a 16-hour lower bound on tasks the model can finish alone.

May 8Friday

r/LocalLLaMA

Gemma 4 26B Hits 600 Tok/s on One RTX 5090

chain-77 benchmarked Gemma 4 26B with vLLM 0.19.2rc1, and DFlash raised output throughput on one RTX 5090 from 228 tok/s to 578 tok/s under 256 input tokens, 1024 output tokens, concurrency 1, and num_speculative_tokens=13.

Why it matters: chain-77 ran Gemma 4 26B on a single RTX 5090 with vLLM 0.19.2rc1, and DFlash speculative decoding lifted output throughput from 228 to 578 tokens per second, with reproducible concurrency and draft-token settings.

AI HOT (Curated Pool)

Adaptive Parallel Reasoning: A New Paradigm for Efficient Reasoning Scaling

BAIR’s post describes adaptive parallel reasoning, where ThreadWeaver and Multiverse dynamically control parallel threads for math and code reasoning; the RSS snippet does not disclose benchmark scores, latency reductions, or reproducible settings.

Why it matters: Two new parallel reasoning methods let the model decide when to branch and how many branches to open, instead of relying on external rules to split tasks.

Xinzhiyuan · WeChat

Token-Level Length Control: 3B Model Beats GPT 5.4 and Claude

UC Santa Barbara and Apple researchers introduced LenVM, which models remaining generation length as a token-level value function; Qwen2.5-3B with a 1.5B LenVM scored 62.6 on LIFEBench length control, above GPT-5.4 at 37.4 and Claude-Opus-4-6 at 35.5.

Why it matters: Modeling remaining generation length as a token-level value function lets a 3B model beat GPT-5.4 and Claude on length control.

QbitAI · WeChat

HIT and Huawei propose Dynamic-dLLM, a training-free acceleration framework with 4.48x speedup

HIT Shenzhen, Huawei, and Shenzhen Hetao College proposed Dynamic-dLLM, a training-free dLLM acceleration framework that raises LLaDA-8B-Instruct throughput on GSM8k from 8.32 TPS to 37.29 TPS with almost no accuracy loss.

Why it matters: Harbin Institute of Technology (Shenzhen) and Huawei propose Dynamic-dLLM, a training-free acceleration framework that lifts GSM8k throughput on LLaDA-8B-Instruct from 8.32 to 37.29 tokens per second with almost no accuracy loss.

Ruan YiFeng's Weblog

Technology Enthusiast Weekly Issue 395: The Third Way of Software Development

Ruanyifeng Weekly issue 395 frames AI-assisted coding as a “mystery house” style of software development and cites HN SOTA, which ranks model popularity by scanning 200 top Hacker News topics each day and their programming or AI discussions.

Why it matters: In his weekly, Ruan Yifeng frames AI-assisted programming as mystery-house development, sets it beside the cathedral and bazaar models of open source, and argues it may become the default for individuals and small teams.

AI HOT (Curated Pool)

Donating the Open-Source Alignment Tool Petri

Anthropic transferred the open-source alignment testing tool Petri to Meridian Labs to preserve independence and credibility. Petri 3.0 separates auditor and target models, adds Dish for real prompts and deployment settings, and integrates Bloom.

Why it matters: Anthropic handed its internal alignment tool Petri, built to catch model lying and sycophancy, to nonprofit evaluator Meridian Labs, putting the tool outside the developer and giving evaluation neutrality organizational backing.

r/LocalLLaMA

11.67% ARC-AGI-2 Local Eval on a Single 4090: The TOPAS Recursive Architecture

Doug_Bitterbot says TOPAS scored 11.67% on ARC-AGI-2 using one RTX 4090 after about 14 days of training. The 100M-parameter checkpoint hit 36% locally, but recursive TTT caused null outputs on nearly half of Kaggle puzzles. The key detail is time management: the author expects 20% after threshold tuning and 3-5 more weeks of training.

Why it matters: About 14 days of training on one RTX 4090 got the 100M-parameter TOPAS to 11.67% on ARC-AGI-2, with local runs reaching 36%, while test-time training recursion left nearly half the Kaggle tasks with empty output.

AI HOT (Curated Pool)

Readable behavioral signals remain in frozen LLM hidden states, Cygnus boosts accuracy

Proprioceptive AI says Cygnus adds adapters to frozen LLMs and raises Qwen-32B on ARC-Challenge from 82.2% to 94.97%. It projects hidden states into a gl(4,R) Lie-algebra space to isolate “dark modes.” Watch replication; the post does not disclose full eval sets or controls.

Why it matters: Proprioceptive AI attaches a self-awareness adapter to a frozen model, reading behavior signals from hidden states without retraining, lifting Qwen-32B's ARC score from 82% to nearly 95%.

May 7Thursday

r/LocalLLaMA

Qwen/WebWorld 32B/14B/8B (Qwen3 finetune)

Qwen released WebWorld 32B/14B/8B, Qwen3 finetunes for training and evaluating web agents. It uses 1M+ real web trajectories and supports 30+ step simulation plus A11y Tree, HTML, XML, Markdown, and natural-language states. Agents trained on its synthetic trajectories gain 9.9% on MiniWob++ and 10.9% on WebArena.

Why it matters: Qwen fine-tuned three WebWorld models of different sizes on Qwen3, training web agents on over 1 million real web interaction traces, with support for 30-plus consecutive steps and five kinds of page-state input.

Xinzhiyuan · WeChat

Zhejiang University and Harvard open-source UniGeo for geometry-guided camera-controllable editing

Zhejiang University and Harvard released UniGeo with code, a report, a project page, and an HF Space. UniGeo injects geometry guidance into representation, architecture, and loss layers; it reports SOTA on DL3DV, RE10K, and Tanks against five methods. The key is video priors plus geometry-anchor attention, not just using a video model.

Why it matters: Zhejiang University and Harvard open-sourced UniGeo, which bakes geometric constraints into the model structure, representation layer and loss function to reduce shape and position distortion when generating from new viewpoints.

Xinzhiyuan · WeChat

Claude Managed Agents Add Dreaming, With Reported Task Completion Up to 6x

Anthropic added Dreaming, Outcomes, and multi-agent orchestration to Claude managed agents; Harvey reports about 6x higher task completion. Dreaming reads up to 100 sessions; one demo distilled 5.3M tokens into 98 rules, while Outcomes raised success by up to 10 points. Opus 4.7 and Sonnet 4.6 require access, with $0.08 per session-hour runtime fees.

Why it matters: Anthropic added dreaming, outcome verification and multi-agent collaboration to Claude managed agents; partner Harvey measured roughly a 6x lift in task completion, with dreaming compressing 5.3M tokens of session history into 98 rules.

Synced · WeChat

Claude, GPT and Gemini score 0% completion on ProgramBench

ProgramBench tested Claude Opus 4.7, GPT-5.4 and Gemini 3.1 Pro, with 0% full completion on rebuilding software projects. It gives only executables and usage docs, removes source/tests, and grades behavioral equivalence via agent-driven fuzzing. The key signal is system-level engineering, not function-level code generation.

Why it matters: The author of SWE-Bench launched ProgramBench, a new benchmark that asks models to rebuild complete software projects from scratch using only executables and docs; Claude, GPT and Gemini all scored zero.

r/LocalLLaMA

Exaggerated PCI-E Bandwidth Concerns?

Reddit user ziphnor tested 2x RTX 5060 Ti 16GB with vLLM TP=2 and 32k-context prefill. PCIe peaked at 3–4 GB/s, about 40–50% of a PCIe 4.0 x4 link. Prefill reached ~840–850, 1500, and 1600–1700 t/s; the post does not disclose decode bandwidth.

Why it matters: User ziphnor ran vLLM tensor parallelism on two RTX 5060 Ti cards and measured PCIe peak bandwidth of only 3-4 GB/s during 32k-context prefill, 40-50% of the PCIe 4.0 x4 link.

r/LocalLLaMA

Analysis of 922 Agentic Task Traces Finds DeepSeek v4’s Cost Edge in Caching

A Reddit user analyzed 922 agentic task traces and reported $0.01 per task for DeepSeek v4 Flash versus $1.52 for Opus 4.7. Both used about 960K tokens per task, but DeepSeek showed a 97% cache hit rate versus 87%, with a 0.02 cache read/write price ratio versus 0.08. The key issue is caching, not headline pricing.

Why it matters: A Reddit user analyzed 922 agent task traces and compared per-task cost for DeepSeek v4 Flash and Opus 4.7, finding similar token use and the gap coming from cache hit rate and cache read-write pricing.

May 6Wednesday

r/LocalLLaMA

Qwen3.6 27B NVFP4 + MTP on a Single RTX 5090: 200k Context in vLLM

A Reddit user ran Qwen3.6 27B NVFP4 on one RTX 5090 32GB and validated 200k context in vLLM. The setup used fp8_e4m3 KV cache, FlashInfer, and MTP with 3 speculative tokens; a 10-run 200k pass completed with 73.6 tok/s mean generation and 70.2s TTFT. The key constraint is 32GB VRAM: logs showed 8.3GiB KV cache and about 30478MiB total GPU use.

Why it matters: A Reddit user ran Qwen3.6 27B NVFP4 on a single RTX 5090 in vLLM, verified 200k context, and posted the fp8_e4m3 KV cache and MTP speculative decoding config.