Skip to content

#评测/基准

1 today

May 17Sunday

Dwarkesh Patel podcast

Notes on Pretraining Parallelisms and Failed Training Runs

Dwarkesh documents pretraining failure modes and parallelism tradeoffs: expert choice and token dropping can break causality in MoE routing, FP16 collectives can bias repeated additions after values exceed 1024, pretraining FLOPs are given as 6ND, B300 HBM is listed as 288GB, and FSDP communication can reach params × 3 with reduce-scatter.

Why it matters: The author breaks down two ways pretraining runs go off the rails, and why expert choice in MoE routing makes a token's destination depend on later routing decisions, so training sees information inference never gets.

AI HOT (Curated Pool)

Latest Open Artifacts #21: Gemma 4, DeepSeek V4, Kimi K2.6, MiMo 2.5, GLM-5.1, and More

Open AI model teams released Gemma 4, DeepSeek V4, Kimi K2.6, MiMo 2.5, GLM-5.1, and other versions this month, and the post says they were tested under CAISI’s V4 evaluation framework, but the RSS snippet does not disclose scores.

Why it matters: Five open models, including DeepSeek V4, Gemma 4 and Kimi K2.6, all shipped new versions in the same month, giving anyone tracking open-model release cadence and version choice a single list.

May 16Saturday

Synced · WeChat

Why Robots Need World Models: Top Institutions Release Joint Survey

NTU MARS Lab and collaborators released a 43-page survey on robot world models, covering definitions, architectures, applications, benchmarks, and challenges around action-conditioned consistency, inference efficiency, and physical grounding.

Why it matters: NTU MARS Lab and partner institutions release a 43-page survey on robot world models, covering how they let robots predict physical outcomes before acting, as background on simulation training for embodied AI.

r/LocalLLaMA

Qwen3.6-35B-A3B and 9B land on the public Terminal-Bench 2.0 leaderboard

little-coder × Qwen3.6-35B-A3B scored 24.6% ±3.2 on Terminal-Bench 2.0, above Gemini 2.5 Pro on Gemini CLI at 19.6% and Qwen3-Coder-480B on Terminus 2 at 23.9%.

Why it matters: Qwen3.6 in 35B-A3B and 9B sizes lands on the public Terminal-Bench 2.0 leaderboard, giving small models a checkable public score on hard terminal-agent tests.

Google DeepMind

How WeatherNext helped the US National Hurricane Center forecast Hurricane Melissa's Jamaica landfall

Google DeepMind's AI weather model WeatherNext helped the US National Hurricane Center forecast five days ahead that Hurricane Melissa would hit Jamaica at Category 5 strength, with 80% confidence. Three days out, that rose to near 100%.

Why it matters: Google DeepMind's WeatherNext gave the US National Hurricane Center five days' notice, at 80% confidence, that Hurricane Melissa would hit Jamaica at Category 5, showing what AI models can do with rapidly intensifying storms.

r/LocalLLaMA

Orthrus-Qwen3-8B: Up to 7.8× tokens/forward on Qwen3-8B with frozen backbone

Orthrus-Qwen3-8B adds a trainable diffusion attention head to a frozen Qwen3-8B backbone, reaches up to 7.8× tokens per forward and about 6× wall-clock speedup on MATH-500, while training 16% of parameters with under 1B tokens over 24 hours on 8×H200 GPUs.

Why it matters: The team froze the Qwen3-8B backbone and trained only a diffusion attention head, reaching up to 7.8x token throughput per forward pass on MATH-500 and about 6x faster wall-clock time, with the output distribution unchanged.

The Verge · AI

AI radio hosts demonstrate why AI can’t be trusted alone

Andon Labs had Claude, ChatGPT, Gemini, and Grok run separate radio stations with $20 in seed money each; the RSS snippet says all failed, but the post does not disclose the full experimental results.

Why it matters: Andon Labs gave Claude, ChatGPT, Gemini and Grok $20 each to run a radio station independently, and all four went off the rails unsupervised, offering a same-conditions sample of where autonomous agent operation breaks down.

May 15Friday

r/LocalLLaMA

Evaluated a RAG Chatbot: The Most Expensive Model Was the Worst Performer

The author evaluated a customer-support RAG bot and raised the quality score from 6.62 to 7.88 while cutting per-session cost from $0.002420 to $0.000509, using retrieval logging, LLM-as-judge scoring, chunk deduplication, stricter grounding, and a five-model sweep.

Why it matters: A builder used a model as judge to tune a customer-service RAG bot, lifting quality from 6.62 to 7.88 and cutting cost per session by nearly 80%, while the priciest model performed worst.

The Verge · AI

AI research papers are getting better, and it’s a big problem for scientists

The Verge describes Peter Degen investigating unusual citations to a 2017 paper: it rose from a few dozen citations over several years to being cited every few days, while the RSS snippet does not disclose the full sample size or review findings.

Why it matters: A researcher traced a 2017 paper's sudden citation spike to AI-written papers that pull references at random from training data.

r/LocalLLaMA

Used over a million tokens in three sessions to test Qwen 3.6 35B MTP

A Reddit user tested Qwen3.6-35B-A3B MTP across three million-token-scale sessions, using 300k context and KV Q8_0, and reported about 1.5x the tok/sec of earlier tests.

Why it matters: A Reddit user ran over a million tokens across three sessions to test the MTP build of Qwen3.6-35B-A3B at 300k context and KV Q8_0 quantization, measuring roughly 1.5x the generation speed of the earlier version.

Latent Space

AI-Native Healthcare: 100M Doctor Visits, 10–20 Hours Saved, Prior Auth in Minutes

Abridge says it is projected to support 80M+ patient-clinician conversations this year across 250 large U.S. health systems, 28+ languages, and 50+ specialties, while its clinical documentation workflow reduces clinicians’ documentation burden by 10–20 hours per week.

Why it matters: Abridge has turned doctor-patient conversations into medical notes since 2018, more than four years before ChatGPT, saving doctors 10 to 20 hours of paperwork a week.

AI HOT (Curated Pool)

Granite Embedding Multilingual R2: Open Multilingual Embedding Model with 32K Context

IBM Granite released Granite Embedding Multilingual R2 on Hugging Face under Apache 2.0, with fewer than 100 million parameters, a 32K-token context length, and top same-scale retrieval performance on MTEB according to the post.

Why it matters: IBM open-sourced two Apache 2.0 multilingual embedding models on Hugging Face, with the 97M-parameter version scoring 60.3 on MTEB multilingual retrieval, a new reference point for lightweight multilingual search.

May 14Thursday

AI HOT (Curated Pool)

MiMo V2.5 Pro Places Third on DesignArena

MiMo V2.5 Pro placed third on the DesignArena overall leaderboard; its Thinking version rose 8 spots over MiMo-V2.5 and matched Claude Sonnet 4.6 performance on frontend coding tasks.

Why it matters: Xiaomi's MiMo V2.5 Pro climbed eight places to third on DesignArena, matching Claude Sonnet 4.6 on front-end coding tasks.

Xinzhiyuan · WeChat

Anthropic Overtakes OpenAI in Enterprise AI Adoption After Three Years

Ramp says Anthropic reached 34.4% enterprise adoption, surpassing OpenAI at 32.3% for the first time; the index is based on credit-card and invoice spending from more than 50,000 companies.

Why it matters: Ramp's tally of card and invoice spending across more than 50,000 companies puts Anthropic at 34.4% of enterprise share and OpenAI at 32.3%, the first time OpenAI has lost this lead in three years.

Synced · WeChat

ACL 2026: Alibaba DAMO I²B-LPO Improves RLVR Exploration

Alibaba DAMO Academy introduced I²B-LPO, an RLVR post-training framework that branches rollouts at high-entropy nodes and filters them with an information-bottleneck self-reward, reporting up to 5.3% accuracy gains and 7.4% semantic-diversity gains on math benchmarks using Qwen2.5-7B and Qwen3-14B.

Why it matters: Alibaba DAMO Academy's I²B-LPO branches sampling where a model hesitates on a problem and uses an information bottleneck self-reward to pick out different solutions, easing the convergence of reasoning paths in RL fine-tuning.

AI HOT (Curated Pool)

Anthropic overtakes OpenAI in B2B adoption for the first time, Ramp data shows

Ramp AI Index data shows Anthropic reached 34.4% adoption among U.S. enterprise customers, surpassing OpenAI’s 32.3% for the first time, while its business coverage grew fourfold over one year.

Why it matters: Ramp payment data shows Anthropic's paid adoption among US business customers hit 34.4% in April 2026, passing OpenAI's 32.3%, with its coverage quadrupling in a year.

May 13Wednesday

TechCrunch · AI

Anthropic now has more business customers than OpenAI, according to Ramp data

Ramp’s survey of client expense data shows 34.4% of participating businesses pay for Anthropic services, while 32.3% pay for OpenAI; the snippet does not disclose sample size, customer segments, or spend levels.

Why it matters: Ramp counted business spending across its own customers: 34.4% pay Anthropic and 32.3% pay OpenAI, drawn from the same sample so the two numbers compare directly.

AI HOT (Curated Pool)

Step Image Edit 2 image model released with leading performance and efficiency

StepFun released the 3.5B-parameter Step Image Edit 2 model, which ranks first in KRIS-Bench overall, factual, and conceptual categories, and is now available on the Stepfun Open Platform.

Why it matters: StepFun released Step Image Edit 2 at 3.5 billion parameters, topping all three KRIS-Bench instruction-editing categories ahead of models five to six times larger.

May 12Tuesday

r/LocalLLaMA

I catalogued every way local models break JSON output and built a repair library across 288 model calls

Reddit user kexxty ran 288 structured-output calls through OpenRouter models, including Llama 3, Mistral, Command R, DeepSeek, and Qwen, and found similar JSON failure categories across local and API-only models. The MIT-licensed Python library outputguard validates against JSON Schema, applies 15 ordered repair strategies, includes 2,001 tests, and has no LLM provider dependency.

Why it matters: Reddit user kexxty ran 288 structured output tests across Llama 3, Mistral, DeepSeek and others, categorized how local models break JSON, and open-sourced a repair library, outputguard.

May 11Monday

AI HOT (Curated Pool)

Fields Medalist Tests ChatGPT 5.5 Pro: Paper-Level Result in 17 Minutes

Timothy Gowers tested ChatGPT 5.5 Pro and said it independently solved an open additive number theory problem in 17 minutes with only a simple prompt, producing PhD thesis-level work; he warned that this pace threatens mathematics research training, while Terence Tao said human value lies in digesting and deeply understanding proofs.

Why it matters: Fields medalist Timothy Gowers gave only a simple prompt, and ChatGPT 5.5 Pro solved an open additive number theory problem on its own in 17 minutes, with a construction that is optimal in theory.

AI HOT (Curated Pool)

Pareto Code Reorders Model Selection Using Market Demand

OpenRouter says Pareto Code observes the Pareto frontier using real market demand; DeepSeek V4 Pro ranks first, followed by GPT 5.4 Mini and Gemini 3.1 Pro, while the post does not disclose the scoring formula or evaluation sample size.

Why it matters: OpenRouter launched the Pareto Code ranking, which measures model value for money from real user calls instead of benchmark scores, with DeepSeek V4 Pro first and GPT 5.4 Mini and Gemini 3.1 Pro close behind.

Xinzhiyuan · WeChat

The Second Half of Agent Evaluation: Why a Live Benchmark Is Needed

Claw-Eval-Live evaluates 13 frontier models on 105 tasks, and the top model stays below a 70% pass rate, while HR tasks average only 6.8% pass rate.

Why it matters: Claw-Eval-Live ran 105 tasks across 13 frontier models; the best pass rate stayed under 70% and HR tasks averaged 6.8%, showing static benchmarks miss how agents actually perform in deployment.

Xinzhiyuan · WeChat

Claude Mythos Hits 50% Success on 16-Hour Tasks in METR Time Horizons

Claude Mythos Preview reached a 50% success rate on METR Time Horizons tasks that take humans 16 hours, while only 5 of 228 tasks exceeded the 16-hour range, so the article says METR lacks enough samples to quantify longer-horizon performance.

Why it matters: On METR's Time Horizons test, Claude Mythos preview hit 50% on tasks humans take 16 hours, but only 5 of 228 tasks run that long.

QbitAI · WeChat

Math Majors in Trouble: Fields Medalist Tests ChatGPT 5.5 Pro, Gets Paper-Level Result in 17 Minutes

Timothy Gowers tested ChatGPT 5.5 Pro on additive number theory problems, where it produced an optimal quadratic upper-bound construction in 17 minutes 5 seconds, then generated a LaTeX preprint in 47 minutes; the article says arXiv rejects AI-generated content, so the result remains on Gowers’s blog.

Why it matters: A Fields medalist handed ChatGPT 5.5 Pro an additive number theory problem and got an optimal quadratic upper bound construction in 17 minutes, plus a LaTeX preprint 47 minutes later.

AI HOT (Curated Pool)

Local models handle half of daily tasks and respond faster than cloud models

A five-week experiment tested about 1,400 daily work tasks, where local 35B models such as Qwen 3.6 35B handled about 50% and averaged 2.8-second responses, 2.1 times faster than Claude Opus 4.5, while the cloud model still led complex reasoning by about 20%.

Why it matters: A five-week test of a local 35B model across about 1,400 daily requests completed half the tasks at 2.8 seconds average, 2.1x faster than cloud Claude Opus 4.5 at 5.8 seconds.

r/LocalLLaMA

MTP benchmark results: task type determines speculative inference speedups or slowdowns

A Reddit LocalLLaMA user ran 300+ tests on Qwen 3.6 27B MTP quants, finding coding draft acceptance at 79-89% and F16 coding speed up 171%, while Q4_K_M creative writing slowed down 9%.

Why it matters: More than 300 runs on a quantized Qwen 3.6 27B show speculative decoding's speedup depends on the task, not the setup.

AI HOT (Curated Pool)

Older AI Model Outperforms Human Doctors in Emergency Diagnosis

A Science study reports that OpenAI o1 reached a 67% correct or near-correct diagnosis rate on real emergency department data, exceeding doctors at 50-55%, but the study did not cover long-term inpatient data or imaging diagnosis.

Why it matters: A Science study puts OpenAI's year-old o1 at 67% correct or near-correct on real ER cases, against 50-55% for the doctors it was compared with.

May 10Sunday

Xinzhiyuan · WeChat

Next-ToBE Targets Short-Sighted Next-Token Prediction in LLMs at ICLR 2026

East China Normal University and Fudan University researchers proposed Next-ToBE, a training objective that keeps standard autoregressive inference while adding a soft target over future-token windows, and the article reports the method ranked best in 35 of 36 experiments across Qwen2.5-Math-1.5B, Qwen2.5-Math-7B, and Llama3.1-8B-Instruct.

Why it matters: ECNU and Fudan researchers propose Next-ToBE, which trains against soft targets from the next few words while still generating token by token, winning 35 of 36 tests.

Xinzhiyuan · WeChat

Harsh Claim: Top Silicon Valley AI Is One Year Ahead of the World

Elad Gil claims top AI lab employees are 3-4 months ahead of Silicon Valley, while Silicon Valley is 3-6 months ahead of New York; the post cites Mythos’ 73% success rate in expert cyberattack simulations as evidence in a disputed “geographic time gap” argument.

Why it matters: The headline claims frontier Silicon Valley AI leads the outside world by a year, but the body never loaded past an access check, leaving only title and summary.

QbitAI · WeChat

Zhejiang University introduces AdaMARP, an AI role-playing framework with scene direction

Zhejiang University and Tencent Youtu proposed AdaMARP for immersive role-playing, using a four-channel message format and a scene manager; its data pipeline includes 81 literary works, 20 synthetic themes, and AdaptiveBench with 100 evaluation seeds.

Why it matters: Zhejiang University and Tencent Youtu propose AdaMARP, using dialogue, narration, scene and inner monologue messages to give AI role-play director-style scheduling across 100 seed scenarios.

r/LocalLLaMA

NVIDIA AI Releases Star Elastic: One Checkpoint Contains 30B, 23B, and 12B Reasoning Models

NVIDIA AI released Star Elastic, a single checkpoint that can zero-shot slice 30B, 23B, and 12B reasoning models in BF16, FP8, and NVFP4; when the 23B submodel handles thinking and the 30B model handles final answers, reported accuracy rises 16% and latency drops 1.9× on AIME-2025, GPQA, LiveCodeBench v5, and MMLU-Pro.

Why it matters: NVIDIA's Star Elastic slices 30B, 23B and 12B reasoning models from one checkpoint with no extra training, the 23B thinking and the 30B answering.

May 9Saturday

r/LocalLLaMA

80 tok/sec and 128K context on 12GB VRAM with Qwen3.6 35B A3B and llama.cpp MTP

Reddit user janvitos ran Qwen3.6-35B-A3B-MTP-GGUF with a llama.cpp MTP PR on an RTX 4070 Super. The posted benchmark shows 69.2-81.9 tok/s, 0.694-0.947 draft acceptance, 131072 context, and a -fitt 1536 setting that reserves 1536 MB for the draft model and KV cache.

Why it matters: Reddit user janvitos runs Qwen3.6 35B A3B on a 12GB RTX 4070 Super and posts measured settings for 69.2 to 81.9 tok/s at 128K context.

Synced · WeChat

StarVLA Open-Sources a Unified VLA Framework from HKUST and the Community

HKUST and the open-source community released StarVLA, a unified Vision-Language-Action framework that integrates backbones, action heads, training strategies, and evaluation interfaces; the repository has 2.2k GitHub stars and supports benchmarks including LIBERO, SimplerEnv, RoboTwin 2.0, RoboCasa-GR1, and BEHAVIOR-1K.

Why it matters: HKUST and the community open-sourced StarVLA, making VLA backbones, action heads and training interfaces pluggable instead of rebuilt per paper.

Synced · WeChat

DeepSeek Reportedly Raises RMB 50B, with Liang Wenfeng Funding 40%, Valuation Reaching RMB 350B

DeepSeek is negotiating a $7.3 billion funding round at an estimated $51.5 billion valuation; Liang Wenfeng reportedly plans to contribute 40%, while Tencent and China’s RMB 60 billion national AI fund are also in talks.

Why it matters: Liang Wenfeng plans to fund 40% of DeepSeek's roughly 50 billion yuan round himself, with Tencent and a state AI fund also in talks.

AI HOT (Curated Pool)

Claude Mythos Evaluation Shows 16-Hour Risk Horizon

METR evaluated an early Claude Mythos Preview build during a limited March 2026 window and estimated its 50% time horizon at at least 16 hours, with a 95% confidence interval of 8.5 to 55 hours.

Why it matters: METR's risk assessment of an early Claude Mythos preview measured a 16-hour lower bound on tasks the model can finish alone.

May 8Friday

r/LocalLLaMA

Gemma 4 26B Hits 600 Tok/s on One RTX 5090

chain-77 benchmarked Gemma 4 26B with vLLM 0.19.2rc1, and DFlash raised output throughput on one RTX 5090 from 228 tok/s to 578 tok/s under 256 input tokens, 1024 output tokens, concurrency 1, and num_speculative_tokens=13.

Why it matters: chain-77 ran Gemma 4 26B on a single RTX 5090 with vLLM 0.19.2rc1, and DFlash speculative decoding lifted output throughput from 228 to 578 tokens per second, with reproducible concurrency and draft-token settings.

AI HOT (Curated Pool)

Adaptive Parallel Reasoning: A New Paradigm for Efficient Reasoning Scaling

BAIR’s post describes adaptive parallel reasoning, where ThreadWeaver and Multiverse dynamically control parallel threads for math and code reasoning; the RSS snippet does not disclose benchmark scores, latency reductions, or reproducible settings.

Why it matters: Two new parallel reasoning methods let the model decide when to branch and how many branches to open, instead of relying on external rules to split tasks.

Xinzhiyuan · WeChat

Token-Level Length Control: 3B Model Beats GPT 5.4 and Claude

UC Santa Barbara and Apple researchers introduced LenVM, which models remaining generation length as a token-level value function; Qwen2.5-3B with a 1.5B LenVM scored 62.6 on LIFEBench length control, above GPT-5.4 at 37.4 and Claude-Opus-4-6 at 35.5.

Why it matters: Modeling remaining generation length as a token-level value function lets a 3B model beat GPT-5.4 and Claude on length control.

QbitAI · WeChat

HIT and Huawei propose Dynamic-dLLM, a training-free acceleration framework with 4.48x speedup

HIT Shenzhen, Huawei, and Shenzhen Hetao College proposed Dynamic-dLLM, a training-free dLLM acceleration framework that raises LLaDA-8B-Instruct throughput on GSM8k from 8.32 TPS to 37.29 TPS with almost no accuracy loss.

Why it matters: Harbin Institute of Technology (Shenzhen) and Huawei propose Dynamic-dLLM, a training-free acceleration framework that lifts GSM8k throughput on LLaDA-8B-Instruct from 8.32 to 37.29 tokens per second with almost no accuracy loss.

Ruan YiFeng's Weblog

Technology Enthusiast Weekly Issue 395: The Third Way of Software Development

Ruanyifeng Weekly issue 395 frames AI-assisted coding as a “mystery house” style of software development and cites HN SOTA, which ranks model popularity by scanning 200 top Hacker News topics each day and their programming or AI discussions.

Why it matters: In his weekly, Ruan Yifeng frames AI-assisted programming as mystery-house development, sets it beside the cathedral and bazaar models of open source, and argues it may become the default for individuals and small teams.