Benchmarks

Who is actually stronger: benchmark results, methodology disputes and leaderboard changes.

453 picksRelated topicsModel releases Research & technical reports Reasoning

Latest picks

301–320 of 453

May 10Sunday

Xinzhiyuan · WeChatMay 10, 12:49

Harsh Claim: Top Silicon Valley AI Is One Year Ahead of the World

Elad Gil claims top AI lab employees are 3-4 months ahead of Silicon Valley, while Silicon Valley is 3-6 months ahead of New York; the post cites Mythos’ 73% success rate in expert cyberattack simulations as evidence in a disputed “geographic time gap” argument.

Why it matters: HKR-H/K/R all pass: the lab-to-user lag hook is clickable, and the post cites 3–4 months, 3–6 months, and a 73% Mythos figure. It is secondhand commentary, not a model or product release, so it stays in the 72–77 threshold band.

Take

“Top labs are one year ahead of the world” reads like insider fanfic, not a testable claim. The summary gives two lag numbers: lab employees lead Silicon Valley by 3–4 months, and Silicon Valley leads New York by 3–6 months. Then it cites Mythos hitting 73% success in expert cyberattack simulations. That supports rising agentic-cyber capability; it does not validate a geographic time gradient.

I’d frame this as access asymmetry, not city physics. OpenAI and Anthropic staff see internal models, gated features, unpublished evals, and enterprise pilots before the rest of the market. New York’s gap is less about ZIP code and more about fewer training clusters, research peers, and deployment feedback loops. The WeChat body is blocked by verification, so Mythos setup, baseline, and success definition are not disclosed. Without those, 73% is a shareable number, not a proof point.

QbitAI · WeChatMay 10, 11:52

Zhejiang University introduces AdaMARP, an AI role-playing framework with scene direction

Zhejiang University and Tencent Youtu proposed AdaMARP for immersive role-playing, using a four-channel message format and a scene manager; its data pipeline includes 81 literary works, 20 synthetic themes, and AdaptiveBench with 100 evaluation seeds.

Why it matters: ACL 2026 role-play agent work brings four-channel messaging, a scene manager, and an 81-book dataset, clearing HKR-H/K. Narrow use cases and missing open-source or production evidence keep it at threshold featured.

Take

AdaMARP is useful because it turns role-play into trainable interfaces, rather than asking one prompt to carry the whole story. The four channels are Thought, Action, Environment, and Speech. The scene manager has five actions: init_scene, pick_speaker, switch_scene, add_role, and end. That is a cleaner engineering split than Character.AI-style long chat, because speaker choice and scene changes become explicit events.

I don’t buy the “immersive role-playing is solved” framing. The data pipeline uses 81 literary works and 20 synthetic themes, while AdaptiveBench has only 100 evaluation seeds and relies on model-based scoring. The hard failures in role-play are long memory, adversarial user turns, and world-state consistency across hours. The article gives no human blind test, retention data, pricing, or stress test over long sessions. ACL acceptance validates the research setup, not the product experience.

r/LocalLLaMAMay 10, 08:48

NVIDIA AI Releases Star Elastic: One Checkpoint Contains 30B, 23B, and 12B Reasoning Models

NVIDIA AI released Star Elastic, a single checkpoint that can zero-shot slice 30B, 23B, and 12B reasoning models in BF16, FP8, and NVFP4; when the 23B submodel handles thinking and the 30B model handles final answers, reported accuracy rises 16% and latency drops 1.9× on AIME-2025, GPQA, LiveCodeBench v5, and MMLU-Pro.

Why it matters: HKR-H/K/R all pass: Star Elastic has a concrete mechanism and testable numbers for inference deployment. Its reach is still narrower than a frontier-model release, so it sits in the high-quality featured band.

May 9Saturday

r/LocalLLaMAMay 9, 19:57

80 tok/sec and 128K context on 12GB VRAM with Qwen3.6 35B A3B and llama.cpp MTP

Reddit user janvitos ran Qwen3.6-35B-A3B-MTP-GGUF with a llama.cpp MTP PR on an RTX 4070 Super. The posted benchmark shows 69.2-81.9 tok/s, 0.694-0.947 draft acceptance, 131072 context, and a -fitt 1536 setting that reserves 1536 MB for the draft model and KV cache.

Why it matters: HKR-H/K/R all pass with concrete single-user benchmark data and reproducible settings. Source is one Reddit post, so verification is thin; this lands above featured threshold, not in must-write range.

Synced · WeChatMay 9, 10:59

StarVLA Open-Sources a Unified VLA Framework from HKUST and the Community

HKUST and the open-source community released StarVLA, a unified Vision-Language-Action framework that integrates backbones, action heads, training strategies, and evaluation interfaces; the repository has 2.2k GitHub stars and supports benchmarks including LIBERO, SimplerEnv, RoboTwin 2.0, RoboCasa-GR1, and BEHAVIOR-1K.

Why it matters: HKR-H/K/R all pass: StarVLA ships a concrete open-source VLA framework with unified interfaces, 2.2k stars, and named robotics benchmarks. The robotics scope keeps it in the 78–84 band, below model-release weight.

Take

StarVLA’s useful move is not another robot policy; it forces VLA chaos into one test harness. It supports LIBERO, SimplerEnv, RoboTwin 2.0, RoboCasa-GR1, and BEHAVIOR-1K, with 2.2k GitHub stars. Swapping FAST, OFT, π₀, and GR00T-style action heads under shared backbones is the kind of setup this field badly lacked.

I don’t buy the “PyTorch moment” branding yet. PyTorch won through kernels, training ergonomics, teaching, cloud support, and paper defaults moving together. StarVLA looks closer to a VLA LLaMA-Factory: very useful for reproduction and assembly, not yet the field’s substrate. The reported 30K-step 98.8% on LIBERO and RoboCasa-GR1 jump from 48.8% to 57.3% are strong hooks. Robotics benchmarks still leak confidence fast. Multi-lab reruns, real-robot transfers, and public failure cases decide whether this becomes infrastructure or just a very good repo.

Synced · WeChatMay 9, 10:59

DeepSeek Reportedly Raises RMB 50B, with Liang Wenfeng Funding 40%, Valuation Reaching RMB 350B

DeepSeek is negotiating a $7.3 billion funding round at an estimated $51.5 billion valuation; Liang Wenfeng reportedly plans to contribute 40%, while Tencent and China’s RMB 60 billion national AI fund are also in talks.

Why it matters: HKR-H/K/R all pass: the DeepSeek funding rumor has large numbers, a founder contribution ratio, and named backers. Because it is still reported as talks with no official confirmation, it stays at 84 and featured, not p1.

Take

DeepSeek’s raise exposes the capital bill behind the low-cost myth. The reported $7.3B round, $51.5B valuation, and Liang Wenfeng personally covering about 40% do not look like ordinary financing. That looks like founder credibility being posted as collateral to bring in Tencent and China’s RMB 60B national AI fund. V3 and R1 won the narrative on efficiency, but V4 did not trigger the same global market shock, and the article says third-party tests still trail top Chinese and US models on some metrics.

I don’t buy the clean “research lab staying independent” story anymore. Agent workloads raise inference spend and release cadence pressure. OpenAI and Anthropic already turned enterprise revenue into compute fuel. If DeepSeek ships V4.1 in June, that only confirms it has entered the same burn curve.

AI HOT (Curated Pool)May 9, 09:32

Claude Mythos Evaluation Shows 16-Hour Risk Horizon

METR evaluated an early Claude Mythos Preview build during a limited March 2026 window and estimated its 50% time horizon at at least 16 hours, with a 95% confidence interval of 8.5 to 55 hours.

Why it matters: HKR-H/K/R all pass: METR reports a concrete 16h risk-horizon estimate for Claude Mythos Preview. The single X-source and limited eval window keep it below P1, but it is strong featured safety signal.

May 8Friday

r/LocalLLaMAMay 8, 22:13

Gemma 4 26B Hits 600 Tok/s on One RTX 5090

chain-77 benchmarked Gemma 4 26B with vLLM 0.19.2rc1, and DFlash raised output throughput on one RTX 5090 from 228 tok/s to 578 tok/s under 256 input tokens, 1024 output tokens, concurrency 1, and num_speculative_tokens=13.

Why it matters: HKR-H/K/R all pass: the single-GPU throughput hook is strong, and the post gives reproducible settings plus before/after speed. Reddit single-post evidence and one hardware setup keep it in the featured-threshold band.

AI HOT (Curated Pool)May 8, 17:00

Adaptive Parallel Reasoning: A New Paradigm for Efficient Reasoning Scaling

BAIR’s post describes adaptive parallel reasoning, where ThreadWeaver and Multiverse dynamically control parallel threads for math and code reasoning; the RSS snippet does not disclose benchmark scores, latency reductions, or reproducible settings.

Why it matters: BAIR authority supports the 72+ band, and HKR-H/K/R all pass. The post names mechanisms and dynamic thread control, but lacks scores, latency gains, and reproducible conditions, so it stays below 78.

Take

BAIR is trying to give inference scaling a cleaner abstraction: let the model decide when to split work, how many threads to spawn, and how to merge results. That is a sharper idea than hard-coded Best-of-N, MCTS, or Tree of Thoughts. The concrete hook is solid: sequential reasoning grows linearly in exploration tokens, worsens context-rot, and can leave users waiting tens of minutes or hours on complex tasks.

I buy the direction, not the “next paradigm” framing yet. ThreadWeaver is co-led by one author, and the post labels itself part survey, part perspective. The missing pieces are the whole story: no benchmark scores, no latency reduction, no thread budget, no failure cases, no reproducible settings. Without those, ThreadWeaver and Multiverse show a research taste shift, not a changed inference cost curve.

Xinzhiyuan · WeChatMay 8, 12:12

Token-Level Length Control: 3B Model Beats GPT 5.4 and Claude

UC Santa Barbara and Apple researchers introduced LenVM, which models remaining generation length as a token-level value function; Qwen2.5-3B with a 1.5B LenVM scored 62.6 on LIFEBench length control, above GPT-5.4 at 37.4 and Claude-Opus-4-6 at 35.5.

Why it matters: HKR-H/K/R all pass: the headline has a sharp small-model-vs-frontier hook, and the post gives LenVM's mechanism plus 62.6/37.4 benchmark numbers. The topic is narrow research, not a model or major product release, so it fits the 78-84 band.

Take

LenVM’s sharp edge is that it makes “write shorter” a decoding-time control problem, not another prompt-compliance hope. Qwen2.5-3B plus a 1.5B LenVM scores 62.6 on LIFEBench length control, ahead of GPT-5.4 at 37.4 and Claude-Opus-4-6 at 35.5. On GSM8K with a 200-token budget, Pass@1 rises from about 6% under hard truncation to about 63% with LenVM-guided decoding.

I would be careful with the “10x” headline, because the baseline is hard truncation, not a tuned compression decoder or distilled short-reasoning model. The mechanism still looks clean: assign each token a fixed negative reward, then predict discounted remaining length as a value function. For agent cost control, batching, and KV-cache planning, this token-level length signal is more usable than another long-CoT leaderboard jump.

QbitAI · WeChatMay 8, 12:05

HIT and Huawei propose Dynamic-dLLM, a training-free acceleration framework with 4.48x speedup

HIT Shenzhen, Huawei, and Shenzhen Hetao College proposed Dynamic-dLLM, a training-free dLLM acceleration framework that raises LLaDA-8B-Instruct throughput on GSM8k from 8.32 TPS to 37.29 TPS with almost no accuracy loss.

Why it matters: HKR-H/K/R all pass: the 4.48x speedup is clickable, and GSM8k TPS figures add concrete substance. It is inference-optimization research, not a mainstream model launch, so it fits the 78–84 band.

Ruan YiFeng's WeblogMay 8, 07:40

Technology Enthusiast Weekly Issue 395: The Third Way of Software Development

Ruanyifeng Weekly issue 395 frames AI-assisted coding as a “mystery house” style of software development and cites HN SOTA, which ranks model popularity by scanning 200 top Hacker News topics each day and their programming or AI discussions.

Why it matters: HKR-H/K/R pass: the “third way/mystery house” framing, HN SOTA’s 200 daily HN topics, and developer workflow anxiety all land. It is commentary, not a model or product release, so it stays at 72.

AI HOT (Curated Pool)May 8, 05:29

Donating the Open-Source Alignment Tool Petri

Anthropic transferred the open-source alignment testing tool Petri to Meridian Labs to preserve independence and credibility. Petri 3.0 separates auditor and target models, adds Dish for real prompts and deployment settings, and integrates Bloom.

Why it matters: HKR-H/K/R all pass: the independent donation is a real hook, Petri 3.0 and Dish add testable mechanisms, and audit credibility resonates. Anthropic open-source safety tooling is strong, but below a model-release-level event.

Take

Anthropic is moving Petri to Meridian Labs to buy external trust for Claude evals, not just to look open. Petri has been used in every Claude alignment assessment since Claude Sonnet 4.5; Petri 3.0 now separates the auditor and target models, while Dish runs tests with the real system prompt and deployment scaffold. That is closer to agent reality than another scripted red-team suite.

I buy the direction, but not the clean independence story. MCP moved to the Linux Foundation and still spread largely through Anthropic’s ecosystem gravity. Petri has the same problem: the repo can be independent while the eval taste was set inside one frontier lab. Meridian needs public, reproducible runs on non-Claude models and government evaluations, or this becomes credible-looking infrastructure with Anthropic DNA.

r/LocalLLaMAMay 8, 04:56

11.67% ARC-AGI-2 Local Eval on a Single 4090: The TOPAS Recursive Architecture

Doug_Bitterbot says TOPAS scored 11.67% on ARC-AGI-2 using one RTX 4090 after about 14 days of training. The 100M-parameter checkpoint hit 36% locally, but recursive TTT caused null outputs on nearly half of Kaggle puzzles. The key detail is time management: the author expects 20% after threshold tuning and 3-5 more weeks of training.

Why it matters: HKR-H/K/R all pass, but this is a single Reddit post with unstable Kaggle submissions. It clears featured, not the higher research-release band.

AI HOT (Curated Pool)May 8, 03:22

Readable behavioral signals remain in frozen LLM hidden states, Cygnus boosts accuracy

Proprioceptive AI says Cygnus adds adapters to frozen LLMs and raises Qwen-32B on ARC-Challenge from 82.2% to 94.97%. It projects hidden states into a gl(4,R) Lie-algebra space to isolate “dark modes.” Watch replication; the post does not disclose full eval sets or controls.

Why it matters: HKR-H/K/R pass: the claim is novel, quantified, and practitioner-relevant. Kept at low featured because the source is an X post and full eval set, training details, and controls are not disclosed.

Take

Cygnus should not be converted into a capability story yet. A 12.77-point ARC-Challenge gain on frozen Qwen-32B is loud enough to demand replication first. The mechanism is specific: adapters project hidden states into a gl(4,R) Lie-algebra space and extract “dark modes.” If it holds, this is closer to test-time state correction than ordinary LoRA.

The eval boundary is the problem. The post gives one RTX 3090, 82.2% to 94.97%, coverage from 3B to 405B models, and 50,000 concurrent users. It does not give the split, prompt format, seed handling, or whether ARC validation was touched. ARC-style benchmarks have been over-optimized by reasoning wrappers for a year. Without an external rerun, this smells like a sharp interpretability-to-performance demo with a very fragile headline number.

May 7Thursday

r/LocalLLaMAMay 7, 22:28

Qwen/WebWorld 32B/14B/8B (Qwen3 finetune)

Qwen released WebWorld 32B/14B/8B, Qwen3 finetunes for training and evaluating web agents. It uses 1M+ real web trajectories and supports 30+ step simulation plus A11y Tree, HTML, XML, Markdown, and natural-language states. Agents trained on its synthetic trajectories gain 9.9% on MiniWob++ and 10.9% on WebArena.

Why it matters: HKR-H/K/R all pass: WebWorld has an agent hook, concrete scale, and benchmark gains. It is a useful Qwen research release for agent builders, but limited source detail keeps it below the 85 must-write band.

Xinzhiyuan · WeChatMay 7, 12:02

Zhejiang University and Harvard open-source UniGeo for geometry-guided camera-controllable editing

Zhejiang University and Harvard released UniGeo with code, a report, a project page, and an HF Space. UniGeo injects geometry guidance into representation, architecture, and loss layers; it reports SOTA on DL3DV, RE10K, and Tanks against five methods. The key is video priors plus geometry-anchor attention, not just using a video model.

Why it matters: HKR-H and HKR-K pass: open code, HF Space, and three geometry-guidance layers make it testable. HKR-R is weak because it is specialized vision-generation research, so this sits near the featured floor.

Take

UniGeo’s useful claim is geometry control, not the open-source packaging. The concrete hook is three-level geometry injection: representation, architecture, and loss, plus geometry-anchor attention. The summary says it reports SOTA on DL3DV, RE10K, and Tanks against five methods. That is the right failure surface for camera-controllable editing, because these systems usually break on cross-frame geometry drift before they break on image quality.

I would still discount the result for now. The WeChat body is CAPTCHA-blocked, so the actual metrics, ablations, failure cases, and camera-trajectory ranges are not visible here. The Zero123 / SyncDreamer lineage already showed that adding geometry language to a paper does not guarantee usable control. If the ablation shows geometry-anchor attention carries the result, UniGeo is serious. If it is mostly leaderboard gain on DL3DV, the paper is thinner than the title.

Xinzhiyuan · WeChatMay 7, 12:02

Claude Managed Agents Add Dreaming, With Reported Task Completion Up to 6x

Anthropic added Dreaming, Outcomes, and multi-agent orchestration to Claude managed agents; Harvey reports about 6x higher task completion. Dreaming reads up to 100 sessions; one demo distilled 5.3M tokens into 98 rules, while Outcomes raised success by up to 10 points. Opus 4.7 and Sonnet 4.6 require access, with $0.08 per session-hour runtime fees.

Why it matters: HKR-H/K/R all pass: Anthropic adds Dreaming, Outcomes, and multi-agent orchestration with 100-session memory, $0.08/session-hour runtime, and Harvey’s ~6x completion claim. This is a same-day Claude agent update.

Synced · WeChatMay 7, 10:05

Claude, GPT and Gemini score 0% completion on ProgramBench

ProgramBench tested Claude Opus 4.7, GPT-5.4 and Gemini 3.1 Pro, with 0% full completion on rebuilding software projects. It gives only executables and usage docs, removes source/tests, and grades behavioral equivalence via agent-driven fuzzing. The key signal is system-level engineering, not function-level code generation.

Why it matters: HKR-H/K/R all pass: the 0% result is clickable, the setup is concrete, and the coding-agent gap matters to practitioners. Still, it is a single benchmark report, below a major model or product release.

r/LocalLLaMAMay 7, 03:54

Exaggerated PCI-E Bandwidth Concerns?

Reddit user ziphnor tested 2x RTX 5060 Ti 16GB with vLLM TP=2 and 32k-context prefill. PCIe peaked at 3–4 GB/s, about 40–50% of a PCIe 4.0 x4 link. Prefill reached ~840–850, 1500, and 1600–1700 t/s; the post does not disclose decode bandwidth.

Why it matters: HKR-H/K/R all pass: a myth-busting PCIe bandwidth test with concrete vLLM conditions and numbers. Single Reddit source limits authority, but the named first-person experiment lifts it to the featured threshold.