Skip to content

Benchmarks

Who is actually stronger: benchmark results, methodology disputes and leaderboard changes.

Latest picks

301–320 of 471

May 15Friday

Latent Space

AI-Native Healthcare: 100M Doctor Visits, 10–20 Hours Saved, Prior Auth in Minutes

Abridge says it is projected to support 80M+ patient-clinician conversations this year across 250 large U.S. health systems, 28+ languages, and 50+ specialties, while its clinical documentation workflow reduces clinicians’ documentation burden by 10–20 hours per week.

Why it matters: Abridge has turned doctor-patient conversations into medical notes since 2018, more than four years before ChatGPT, saving doctors 10 to 20 hours of paperwork a week.

AI HOT (Curated Pool)

Granite Embedding Multilingual R2: Open Multilingual Embedding Model with 32K Context

IBM Granite released Granite Embedding Multilingual R2 on Hugging Face under Apache 2.0, with fewer than 100 million parameters, a 32K-token context length, and top same-scale retrieval performance on MTEB according to the post.

Why it matters: IBM open-sourced two Apache 2.0 multilingual embedding models on Hugging Face, with the 97M-parameter version scoring 60.3 on MTEB multilingual retrieval, a new reference point for lightweight multilingual search.

May 14Thursday

AI HOT (Curated Pool)

MiMo V2.5 Pro Places Third on DesignArena

MiMo V2.5 Pro placed third on the DesignArena overall leaderboard; its Thinking version rose 8 spots over MiMo-V2.5 and matched Claude Sonnet 4.6 performance on frontend coding tasks.

Why it matters: Xiaomi's MiMo V2.5 Pro climbed eight places to third on DesignArena, matching Claude Sonnet 4.6 on front-end coding tasks.

Xinzhiyuan · WeChat

Anthropic Overtakes OpenAI in Enterprise AI Adoption After Three Years

Ramp says Anthropic reached 34.4% enterprise adoption, surpassing OpenAI at 32.3% for the first time; the index is based on credit-card and invoice spending from more than 50,000 companies.

Why it matters: Ramp's tally of card and invoice spending across more than 50,000 companies puts Anthropic at 34.4% of enterprise share and OpenAI at 32.3%, the first time OpenAI has lost this lead in three years.

Synced · WeChat

ACL 2026: Alibaba DAMO I²B-LPO Improves RLVR Exploration

Alibaba DAMO Academy introduced I²B-LPO, an RLVR post-training framework that branches rollouts at high-entropy nodes and filters them with an information-bottleneck self-reward, reporting up to 5.3% accuracy gains and 7.4% semantic-diversity gains on math benchmarks using Qwen2.5-7B and Qwen3-14B.

Why it matters: Alibaba DAMO Academy's I²B-LPO branches sampling where a model hesitates on a problem and uses an information bottleneck self-reward to pick out different solutions, easing the convergence of reasoning paths in RL fine-tuning.

AI HOT (Curated Pool)

Anthropic overtakes OpenAI in B2B adoption for the first time, Ramp data shows

Ramp AI Index data shows Anthropic reached 34.4% adoption among U.S. enterprise customers, surpassing OpenAI’s 32.3% for the first time, while its business coverage grew fourfold over one year.

Why it matters: Ramp payment data shows Anthropic's paid adoption among US business customers hit 34.4% in April 2026, passing OpenAI's 32.3%, with its coverage quadrupling in a year.

May 13Wednesday

TechCrunch · AI

Anthropic now has more business customers than OpenAI, according to Ramp data

Ramp’s survey of client expense data shows 34.4% of participating businesses pay for Anthropic services, while 32.3% pay for OpenAI; the snippet does not disclose sample size, customer segments, or spend levels.

Why it matters: Ramp counted business spending across its own customers: 34.4% pay Anthropic and 32.3% pay OpenAI, drawn from the same sample so the two numbers compare directly.

AI HOT (Curated Pool)

Step Image Edit 2 image model released with leading performance and efficiency

StepFun released the 3.5B-parameter Step Image Edit 2 model, which ranks first in KRIS-Bench overall, factual, and conceptual categories, and is now available on the Stepfun Open Platform.

Why it matters: StepFun released Step Image Edit 2 at 3.5 billion parameters, topping all three KRIS-Bench instruction-editing categories ahead of models five to six times larger.

May 12Tuesday

r/LocalLLaMA

I catalogued every way local models break JSON output and built a repair library across 288 model calls

Reddit user kexxty ran 288 structured-output calls through OpenRouter models, including Llama 3, Mistral, Command R, DeepSeek, and Qwen, and found similar JSON failure categories across local and API-only models. The MIT-licensed Python library outputguard validates against JSON Schema, applies 15 ordered repair strategies, includes 2,001 tests, and has no LLM provider dependency.

Why it matters: Reddit user kexxty ran 288 structured output tests across Llama 3, Mistral, DeepSeek and others, categorized how local models break JSON, and open-sourced a repair library, outputguard.

May 11Monday

AI HOT (Curated Pool)

Fields Medalist Tests ChatGPT 5.5 Pro: Paper-Level Result in 17 Minutes

Timothy Gowers tested ChatGPT 5.5 Pro and said it independently solved an open additive number theory problem in 17 minutes with only a simple prompt, producing PhD thesis-level work; he warned that this pace threatens mathematics research training, while Terence Tao said human value lies in digesting and deeply understanding proofs.

Why it matters: Fields medalist Timothy Gowers gave only a simple prompt, and ChatGPT 5.5 Pro solved an open additive number theory problem on its own in 17 minutes, with a construction that is optimal in theory.

AI HOT (Curated Pool)

Pareto Code Reorders Model Selection Using Market Demand

OpenRouter says Pareto Code observes the Pareto frontier using real market demand; DeepSeek V4 Pro ranks first, followed by GPT 5.4 Mini and Gemini 3.1 Pro, while the post does not disclose the scoring formula or evaluation sample size.

Why it matters: OpenRouter launched the Pareto Code ranking, which measures model value for money from real user calls instead of benchmark scores, with DeepSeek V4 Pro first and GPT 5.4 Mini and Gemini 3.1 Pro close behind.

Xinzhiyuan · WeChat

The Second Half of Agent Evaluation: Why a Live Benchmark Is Needed

Claw-Eval-Live evaluates 13 frontier models on 105 tasks, and the top model stays below a 70% pass rate, while HR tasks average only 6.8% pass rate.

Why it matters: Claw-Eval-Live ran 105 tasks across 13 frontier models; the best pass rate stayed under 70% and HR tasks averaged 6.8%, showing static benchmarks miss how agents actually perform in deployment.

Xinzhiyuan · WeChat

Claude Mythos Hits 50% Success on 16-Hour Tasks in METR Time Horizons

Claude Mythos Preview reached a 50% success rate on METR Time Horizons tasks that take humans 16 hours, while only 5 of 228 tasks exceeded the 16-hour range, so the article says METR lacks enough samples to quantify longer-horizon performance.

Why it matters: On METR's Time Horizons test, Claude Mythos preview hit 50% on tasks humans take 16 hours, but only 5 of 228 tasks run that long.

QbitAI · WeChat

Math Majors in Trouble: Fields Medalist Tests ChatGPT 5.5 Pro, Gets Paper-Level Result in 17 Minutes

Timothy Gowers tested ChatGPT 5.5 Pro on additive number theory problems, where it produced an optimal quadratic upper-bound construction in 17 minutes 5 seconds, then generated a LaTeX preprint in 47 minutes; the article says arXiv rejects AI-generated content, so the result remains on Gowers’s blog.

Why it matters: A Fields medalist handed ChatGPT 5.5 Pro an additive number theory problem and got an optimal quadratic upper bound construction in 17 minutes, plus a LaTeX preprint 47 minutes later.

AI HOT (Curated Pool)

Local models handle half of daily tasks and respond faster than cloud models

A five-week experiment tested about 1,400 daily work tasks, where local 35B models such as Qwen 3.6 35B handled about 50% and averaged 2.8-second responses, 2.1 times faster than Claude Opus 4.5, while the cloud model still led complex reasoning by about 20%.

Why it matters: A five-week test of a local 35B model across about 1,400 daily requests completed half the tasks at 2.8 seconds average, 2.1x faster than cloud Claude Opus 4.5 at 5.8 seconds.

r/LocalLLaMA

MTP benchmark results: task type determines speculative inference speedups or slowdowns

A Reddit LocalLLaMA user ran 300+ tests on Qwen 3.6 27B MTP quants, finding coding draft acceptance at 79-89% and F16 coding speed up 171%, while Q4_K_M creative writing slowed down 9%.

Why it matters: More than 300 runs on a quantized Qwen 3.6 27B show speculative decoding's speedup depends on the task, not the setup.

AI HOT (Curated Pool)

Older AI Model Outperforms Human Doctors in Emergency Diagnosis

A Science study reports that OpenAI o1 reached a 67% correct or near-correct diagnosis rate on real emergency department data, exceeding doctors at 50-55%, but the study did not cover long-term inpatient data or imaging diagnosis.

Why it matters: A Science study puts OpenAI's year-old o1 at 67% correct or near-correct on real ER cases, against 50-55% for the doctors it was compared with.

May 10Sunday

Xinzhiyuan · WeChat

Next-ToBE Targets Short-Sighted Next-Token Prediction in LLMs at ICLR 2026

East China Normal University and Fudan University researchers proposed Next-ToBE, a training objective that keeps standard autoregressive inference while adding a soft target over future-token windows, and the article reports the method ranked best in 35 of 36 experiments across Qwen2.5-Math-1.5B, Qwen2.5-Math-7B, and Llama3.1-8B-Instruct.

Why it matters: ECNU and Fudan researchers propose Next-ToBE, which trains against soft targets from the next few words while still generating token by token, winning 35 of 36 tests.

Xinzhiyuan · WeChat

Harsh Claim: Top Silicon Valley AI Is One Year Ahead of the World

Elad Gil claims top AI lab employees are 3-4 months ahead of Silicon Valley, while Silicon Valley is 3-6 months ahead of New York; the post cites Mythos’ 73% success rate in expert cyberattack simulations as evidence in a disputed “geographic time gap” argument.

Why it matters: The headline claims frontier Silicon Valley AI leads the outside world by a year, but the body never loaded past an access check, leaving only title and summary.

QbitAI · WeChat

Zhejiang University introduces AdaMARP, an AI role-playing framework with scene direction

Zhejiang University and Tencent Youtu proposed AdaMARP for immersive role-playing, using a four-channel message format and a scene manager; its data pipeline includes 81 literary works, 20 synthetic themes, and AdaptiveBench with 100 evaluation seeds.

Why it matters: Zhejiang University and Tencent Youtu propose AdaMARP, using dialogue, narration, scene and inner monologue messages to give AI role-play director-style scheduling across 100 seed scenarios.