Benchmarks

Who is actually stronger: benchmark results, methodology disputes and leaderboard changes.

453 picksRelated topicsModel releases Research & technical reports Reasoning

Latest picks

181–200 of 453

Jun 4Thursday

Xinzhiyuan · WeChatJun 4, 19:17

Claude Mythos Hits 3 Hours 6 Minutes Before Experts’ Year-End Forecast

Anthropic Claude Mythos completed 186 minutes of autonomous tasks at an 80% success rate on the METR benchmark, and the post says this matches the 3–4 hour median forecast that experts had placed at the end of 2026.

Why it matters: HKR-H/K/R all pass: the 3h06m autonomy result is a strong hook, METR 80%/186 minutes gives concrete signal, and agent safety lands with practitioners. Single-source coverage without release details or reproducible setup keeps it below p1.

AI HOT (Curated Pool)Jun 4, 10:54

Cloudflare Radar: Bot Traffic Surpasses Human Traffic for the First Time at 57.5%

Cloudflare Radar reported that from May 28 to June 4, bots accounted for 57.5% of global HTML requests, while human browsers accounted for 42.5%; across all HTTP response content types, JSON led with 33.1% and HTML accounted for 12%.

Why it matters: Cloudflare Radar supplies a concrete window and ratios, clearing HKR-H/K/R. The post does not separate AI crawlers, search bots, and malicious automation, so it sits just above the featured threshold.

Latent SpaceJun 4, 03:27

Scaling Past Informal AI - Carina Hong, Axiom Math

Axiom solved all 12 Putnam problems in 2025 and scored 8/12 within the time limit; Carina Hong says its Verina ProofGen result reached 187/189, while the last disclosed OpenAI o3 result on that benchmark was 4.9%.

Why it matters: HKR-H/K/R all pass: Putnam results, the o3 comparison, and 187/189 give it a real hook. It stays at 80 because this is a Latent Space interview/research story, not a broad model release.

Latent SpaceJun 4, 01:13

Satya Nadella: No Priors x Latent Space Crossover Special at Microsoft Build

Satya Nadella said in a Build interview that Microsoft frames AI as a multi-model enterprise platform spanning MAI, OpenClaw, Scout, and Work IQ; the transcript cites a 5B reasoning model that can hill climb from collected traces and private evals.

Why it matters: HKR-H/K/R all pass: Satya is a strong hook, and the post adds Microsoft’s multi-model enterprise stack plus a 5B reasoning-trace mechanism. It is still a Build interview, not a standalone model launch, so 78 fits.

Jun 3Wednesday

QbitAI · WeChatJun 3, 13:10

Daxiao Robot and NTU Release PhysX-Omni for Unified Physical 3D Generation

Daxiao Robot and NTU introduced PhysX-Omni, a unified simulation-ready physical 3D generation framework for rigid, deformable, and articulated objects, with PhysXVerse covering 8.7K assets across 2.9K categories and PhysX-Bench evaluating six dimensions including geometry, scale, material, affordance, kinematics, and description.

Why it matters: HKR-H/K/R all pass: unified physical 3D generation is a clear hook, the dataset and benchmark numbers add substance, and robotics simulation data is a real practitioner pain. No open-source or product adoption is disclosed, so it stays at 78.

QbitAI · WeChatJun 3, 13:10

Papers with Code returns with CVPR coverage and Hugging Face-led rebuild

Hugging Face’s open-source team launched paperswithcode.co in May 2026, using AI agents to parse papers and restore SOTA leaderboards tied to the original platform’s 9,300-plus benchmarks.

Why it matters: HKR-H/K/R all pass: a beloved research portal returns, with 9,300 restored leaderboards and agent-based paper parsing. The impact is strong for research workflows, not model-release scale.

Computing Life · Share · YageJun 3, 08:00

After vibe coding: the industrialization of AI programming

MAI filtered 265,000 trainable tasks from 4.87 million open-source PRs and built a three-layer judging system. The key change after vibe coding is the industrialization of training infrastructure.

Why it matters: HKR-H/K/R all pass via the post-vibe-coding angle, 4.87M PR corpus, 265K tasks, and code-agent infra stakes. No model scores, open-source scope, or product access are disclosed, so it stays below P1.

Take

MAI is betting on the data factory, not the IDE surface. It filtered 265,000 trainable tasks from 4.87 million open-source PRs, roughly 5.4% retention. That rejection rate says the coding-agent bottleneck has moved from autocomplete quality to task manufacturing, judging, and feedback loops.

The three-layer judging setup is the tell, though the snippet does not disclose the layers. SWE-bench pushed the field toward real issues. Cursor pulled users into natural-language code editing. MAI is describing the missing middle: infrastructure that converts repository history into verifiable training work. I’m cautious about the “industrialization” label, but this smells less like a benchmark launch and more like a claim that coding agents will be won by whoever can mass-produce reliable evaluation fuel.

AI HOT (Curated Pool)Jun 3, 08:00

Intelligence Cost-Performance

Microsoft added average token usage to its model release card; the model scored 71.6 on SWE-Bench Verified while using about one-third of Claude Haiku 4.5’s tokens.

Why it matters: HKR-H/K/R all pass: the score-per-token angle is clickable, with concrete 71.6 and one-third-token claims. The article is thin on full test setup and pricing, so it lands at 78.

AI HOT (Curated Pool)Jun 3, 04:51

Microsoft releases MAI-Thinking-1 model

Microsoft released MAI-Thinking-1, an MoE model with 35B active parameters and 1T total parameters, pretrained from scratch on 30T tokens without third-party model distillation.

Why it matters: HKR-H/K/R all pass: Microsoft released MAI-Thinking-1 with concrete MoE scale and training-token figures. Benchmarks, access, and pricing are not disclosed, so it stays in the 78–84 band rather than P1.

TechCrunch · AIJun 3, 03:02

New Microsoft Tool Lets Devs Spin Up AI Behavior Tests Using Text Descriptions

Microsoft released Adaptive Spec-driven Scoring for Evaluation and Regression Testing, an open source framework that creates AI evaluations and regression tests from text descriptions; the post does not disclose supported models, scoring metrics, or usage conditions.

Why it matters: HKR-H/K/R pass: text-described behavior tests are a clear dev hook, with a concrete open-source Microsoft framework. Missing supported models, metrics, and run conditions keeps it in the mid-weight product-update band.

Take

Microsoft should not get credit for “automated evals” yet. The disclosed fact is narrower: Adaptive Spec-driven Scoring for Evaluation and Regression Testing creates AI evaluations and regression tests from text descriptions. The snippet gives no supported models, scoring metrics, thresholds, or reproduction rules.

The useful part is obvious: teams hate translating product specs into regression cases. ASSER T attacks that workflow pain. The trap is also obvious. LLM-as-judge, promptfoo, and OpenAI Evals already showed that generating cases is easy; stable scoring is the hard part. Without metric contracts, calibration, and version pinning, text-authored tests become another pile of prompt debt. Open source helps adoption, but the trust boundary sits in the default scoring contract, not the repo badge.

Hacker News front pageJun 3, 02:47

Microsoft's MAI-Code-1-Flash Scores 51% SWE-Bench Pro with Just 5B Active Params

The title says Microsoft's MAI-Code-1-Flash scores 51% on SWE-Bench Pro with 5B active parameters; the post does not disclose the evaluation setup, training data, release date, or deployment conditions.

Why it matters: HKR-H/K/R pass on the 51% SWE-Bench Pro with 5B active params claim from Microsoft. Missing eval setup, training data, and release timing keep it in the 72–77 band.

Take

MAI-Code-1-Flash is sharp because 5B active parameters hit 51% on SWE-Bench Pro, not because Microsoft published another coding model. Coding agents have moved from “can it patch?” to “how much does each attempted patch cost?” If that 5B-active number holds in reproducible runs, Copilot can run issue triage, patch drafting, and test repair at a very different margin profile.

I’d still haircut the claim. The post does not disclose eval setup, training data, tool-use policy, pass@, or failure distribution. SWE-Bench-style scores have become easy to bend with retrieval, repeated test runs, and scaffolding. The “Flash” name smells like a deployment model, probably a small MoE, not a lab trophy. Without latency, token pricing, and Azure/Copilot availability, 51% is a sign on the door, not proof of production economics.

AI HOT (Curated Pool)Jun 3, 02:12

Microsoft releases its first advanced reasoning AI model, MAI-Thinking-1

Microsoft released MAI-Thinking-1 at Build 2026, describing it as a medium-sized reasoning model that matches leading models on key software engineering benchmarks.

Why it matters: HKR-H/K/R all pass: Microsoft released its first advanced reasoning model with a mid-sized design and SWE benchmark claim. Exact scores, access, and pricing are not disclosed, so it stays below 85.

Take

Microsoft’s sharp move is not MAI-Thinking-1 itself; it is the public no-distillation claim. That matters because Microsoft has spent years leaning on OpenAI for frontier capability, then renegotiated the partnership as the relationship loosened. Now it needs an asset developers can view as Microsoft-native, not Azure packaging around another lab’s work.

The evidence is thin. The body says “medium-sized” and claims parity with leading models on “key” software-engineering benchmarks, but gives no parameter count, SWE-bench score, pricing, or context window. For practitioners, that is not yet a capability signal you can route workloads around. It smells more like leverage in the OpenAI negotiation. OpenAI has GPT-5-class branding, Anthropic owns a lot of coding mindshare with Claude Sonnet 4.5, and Microsoft still has to turn this Build demo into public evals people trust.

r/LocalLLaMAJun 3, 01:46

Using Gemma 4 E4B with LiteRT: about 2.4× faster text generation than Q4 GGUF

The author tested Gemma 4 E4B on an RTX 4060 Ti 16GB, where LiteRT averaged 157.2 tok/s for text generation versus 66.3 tok/s for llama.cpp Q4 GGUF; image captioning on 111 full-resolution images improved only 1.1×, at about 72 seconds versus 80 seconds.

Why it matters: HKR-H/K/R all pass, with a first-person benchmark including hardware, throughput, and sample count. Source authority is limited to one Reddit test, so it sits at the featured threshold rather than the 78+ band.

r/LocalLLaMAJun 3, 00:16

Benchmarks of 20 Small LLMs on a 6GB RTX 4050

The author benchmarked 20 small LLMs on a 6GB RTX 4050 using LM Studio’s OpenAI-compatible API, with N=5 speed runs at 1k, 8k, and 32k context; unsloth/lfm2.5-vl-1.6b led throughput at 207 tok/s on 1k context while using 3.0GB VRAM.

Why it matters: HKR-H/K/R all pass: the low-VRAM GPU hook is concrete, the post gives speed/context/VRAM numbers, and it speaks to local-inference cost pressure. Source authority is a Reddit post, so it stays in the lower featured band.

Jun 2Tuesday

Ben's BitesJun 2, 21:29

Opus 4.8

Ben’s Bites says Claude Opus 4.8 is out, and Claude Code can write an orchestration script before launching subagents in parallel to work through complex tasks.

Why it matters: HKR-H/K/R all pass for a substantive Anthropic/Claude release and Claude Code agent update. The post is thin on benchmarks, pricing, and context window, so it stays low in the 85–94 band.

Take

Opus 4.8’s useful move is Claude Code writing an orchestration script before launching parallel subagents. That order matters. Anthropic is not proving free-form multi-agent swarms work; it is turning task decomposition, dependencies, and checks into a deterministic wrapper around smaller agent loops.

The evidence is messy in a familiar way. Simon Willison calls 4.8 modest but useful, mainly because it admits uncertainty and catches more flaws in its own code. Every says it jumps from 4.7 and competes with GPT-5.5 on an internal senior-engineer benchmark. Datacurve puts it below GPT-5.5, barely above 5.4, while using far more tokens. The ARC-AGI-3 claim says it triples 5.5’s score, but the harness is doing too much work here. I’d trust the Claude Code workflow change before I trust the leaderboard flex.

r/LocalLLaMAJun 2, 19:05

Replaced Claude with local Qwen3.6-27B in my multi-agent orchestrator for 2 weeks

The author ran Qwen3.6-27B on one RTX 3090 across 47 multi-step coding workflows. Plan generation reached about 95% schema validity, but tool-call formatting errors were about 12%, and practical long-context use degraded past about 12k tokens.

Why it matters: HKR-H/K/R all pass: a named first-person local-vs-Claude experiment with concrete numbers. The single Reddit source and 47-workflow scope keep it below the 78–84 band.

Take

This reads like the split point for local coding agents: planning is usable, interface reliability is still the tax. The body is blocked by Reddit 403, so the usable evidence is the title and summary: one RTX 3090, Qwen3.6-27B, 47 coding workflows, about 95% plan-schema validity, about 12% tool-call formatting errors, and long-context degradation past roughly 12k tokens.

I don’t buy the “replaced Claude” framing yet. Claude is expensive in multi-step coding loops, but its value often sits in boring failure reduction: cleaner tool calls, fewer retries, and better context endurance. Qwen3.6-27B getting plans right says local orchestration costs are collapsing. A 12% tool-call format error rate says you pay back part of that saving with parsers, validators, retries, and dead-loop handling.

Synced · WeChatJun 2, 13:30

DataMaster: When AI Becomes Its Own Data Engineer

DataMaster searches, cleans, and combines data while keeping the model and training algorithm fixed; on MLE-Bench Lite, it raised the medal rate from 35.91% to 68.18%.

Why it matters: HKR-H/K/R all pass: DataMaster changes the data pipeline under fixed model and training code, lifting MLE-Bench Lite medal rate from 35.91% to 68.18%. This is still a single research release without production validation, so it lands at 78 featured.

Xinzhiyuan · WeChatJun 2, 12:00

Chinese AI chip firm raises nearly 1B yuan as next-generation card is due this year

Motern AI completed a nearly 1 billion yuan Series C round and plans to release its SparsePrime inference card this year; the article says its S30 and S40 cards achieved three consecutive wins in MLPerf Inference.

Why it matters: HKR-H/K/R all pass, but this is still a funding and roadmap item; SparsePrime specs, production timing, and customers are not disclosed. Featured threshold, not P1.

Take

Motern is selling the right cost narrative, but not yet a proven replacement path for dense GPU serving. The article gives hard hooks: nearly ¥1B Series C, SparsePrime launching this year, and S30/S40 claiming three straight MLPerf Inference wins. The missing numbers matter more: card price, memory, tokens per second, vLLM throughput, LLM accuracy loss, and real utilization in thousand-card clusters are not disclosed.

I buy the direction. Agent loops made inference cost the bill that actually hurts, and sparse compute attacks that directly. But MLPerf is a controlled track; production LLM serving is CUDA compatibility, scheduler behavior, custom kernels, and model churn. “Near-zero migration” from PyTorch, TensorFlow, and vLLM is the claim I’d test first, not the funding headline.

Xinzhiyuan · WeChatJun 2, 12:00

CAS Opens MobileGym, a Browser-Based Agent Training Environment for Mobile Apps

CASIA released MobileGym, a browser-based Android simulation environment covering 28 apps, with about 400MB per instance, 3-second cold start, JSON state snapshots, and programmatic task verification for mobile-agent training and evaluation.

Why it matters: MobileGym is practical open-source infrastructure for agent training and evaluation, with enough concrete numbers and mechanisms to pass HKR-H/K/R. It fits the 78–84 quality band, below major lab model-release weight.

Take

MobileGym is useful because it attacks the dirty-environment problem before chasing leaderboard theater. The concrete hooks are strong: 28 simulated apps, about 400MB per instance, 3-second cold starts, JSON state snapshots, and a 256-task evaluation finished in 6 minutes. With GRPO, Qwen3-VL-4B moved from 9.4% to 22.2% test success.

The part I buy is programmatic verification, not the phone-in-browser demo. In 118 real-phone trajectories, a VLM judge missed 12 cases; GPT-5.4 still had a 10.2% error rate, just on different tasks. AndroidWorld already had verifiable tasks, but it did not reach WeChat or Alipay-style daily apps. MobileGym dodges accounts, risk controls, reset pain, and parallel rollout limits through simulation. The bill comes later: app realism has to be earned task by task.

New York Times ChineseJun 2, 11:37

China Is Trying to Use AI to Predict Dissent

Geedge is developing an AI system to predict dissent using telecom, social media, and location data, according to 100,000 leaked documents reviewed by Vanderbilt researchers; U.S. officials say there is no evidence that the predictive technology has been finalized or deployed.

Why it matters: HKR-H/K/R all pass: the NYT report adds leaked-file evidence, data-source detail, and a clear surveillance-governance nerve. Deployment is unconfirmed, so this stays in the 78–84 band rather than P1.

Take

Geedge’s threat is not sci-fi prediction; it is old surveillance infrastructure wired to scalable classifiers. The evidence is concrete: 100,000 leaked documents show MESA Lab working in early 2024 on profiles built from telecom, social media, and location data, with meeting notes about “identifying intent.” U.S. officials also say there is no evidence the predictive system has been finalized or deployed.

I don’t buy the headline version that AI can already forecast dissent. The practical version is harsher: police systems use models like DeepSeek to triage huge alert streams and move human review earlier. Geedge hitting GPU limits and older models says export controls bite less on chatbots than on scaled multimodal surveillance. The missing number is false-positive rate, and that is the number a system like this would hide first.