Skip to content

Benchmarks

Who is actually stronger: benchmark results, methodology disputes and leaderboard changes.

Latest picks

201–220 of 471

Jun 4Thursday

Latent Space

Scaling Past Informal AI - Carina Hong, Axiom Math

Axiom solved all 12 Putnam problems in 2025 and scored 8/12 within the time limit; Carina Hong says its Verina ProofGen result reached 187/189, while the last disclosed OpenAI o3 result on that benchmark was 4.9%.

Why it matters: Axiom solved all 12 problems of the 2025 Putnam in seven months, scoring 8/12 timed and 120 untimed, above DeepSeek's known 103.

Latent Space

Satya Nadella: No Priors x Latent Space Crossover Special at Microsoft Build

Satya Nadella said in a Build interview that Microsoft frames AI as a multi-model enterprise platform spanning MAI, OpenClaw, Scout, and Work IQ; the transcript cites a 5B reasoning model that can hill climb from collected traces and private evals.

Why it matters: Nadella explains why Microsoft treats AI as an ecosystem platform rather than one model, and why MAI demands clean, traceable data from pretraining.

Jun 3Wednesday

QbitAI · WeChat

Daxiao Robot and NTU Release PhysX-Omni for Unified Physical 3D Generation

Daxiao Robot and NTU introduced PhysX-Omni, a unified simulation-ready physical 3D generation framework for rigid, deformable, and articulated objects, with PhysXVerse covering 8.7K assets across 2.9K categories and PhysX-Bench evaluating six dimensions including geometry, scale, material, affordance, kinematics, and description.

Why it matters: One pipeline generates rigid, soft and articulated 3D assets, so teams skip separate processing and manual mesh fixes.

QbitAI · WeChat

Papers with Code returns with CVPR coverage and Hugging Face-led rebuild

Hugging Face’s open-source team launched paperswithcode.co in May 2026, using AI agents to parse papers and restore SOTA leaderboards tied to the original platform’s 9,300-plus benchmarks.

Why it matters: Hugging Face's open source team launched paperswithcode.co, using AI agents to parse papers automatically, restoring SOTA leaderboards for over 9,300 benchmarks from the original platform and organizing CVPR papers with their code.

Computing Life · Share · Yage

After vibe coding: the industrialization of AI programming

MAI filtered 265,000 trainable tasks from 4.87 million open-source PRs and built a three-layer judging system. The key change after vibe coding is the industrialization of training infrastructure.

Why it matters: MAI filtered 487 million open-source PRs down to 265,000 trainable problems and built a three-layer scoring system, extending code grading from unit-test passes to tool-use efficiency and front-end visual quality.

AI HOT (Curated Pool)

Intelligence Cost-Performance

Microsoft added average token usage to its model release card; the model scored 71.6 on SWE-Bench Verified while using about one-third of Claude Haiku 4.5’s tokens.

Why it matters: Microsoft adds average token use to its model card, giving a 71.6 SWE-Bench Verified score a cost dimension next to Claude Haiku 4.5.

AI HOT (Curated Pool)

Microsoft releases MAI-Thinking-1 model

Microsoft released MAI-Thinking-1, an MoE model with 35B active parameters and 1T total parameters, pretrained from scratch on 30T tokens without third-party model distillation.

Why it matters: Microsoft's MAI-Thinking-1 carries 1 trillion total parameters and 35 billion active per pass, pretrained from scratch on 30T tokens with no third-party distillation, making its MoE sparsity and in-house training path the draw.

TechCrunch · AI

New Microsoft Tool Lets Devs Spin Up AI Behavior Tests Using Text Descriptions

Microsoft released Adaptive Spec-driven Scoring for Evaluation and Regression Testing, an open source framework that creates AI evaluations and regression tests from text descriptions; the post does not disclose supported models, scoring metrics, or usage conditions.

Why it matters: Microsoft's open-source ASSERT framework turns plain-language test requirements into AI behavior checks and regression tests, so developers skip writing scripts by hand.

Hacker News front page

Microsoft's MAI-Code-1-Flash Scores 51% SWE-Bench Pro with Just 5B Active Params

The title says Microsoft's MAI-Code-1-Flash scores 51% on SWE-Bench Pro with 5B active parameters; the post does not disclose the evaluation setup, training data, release date, or deployment conditions.

Why it matters: Microsoft's MAI-Code-1-Flash hits 51% on SWE-Bench Pro with only 5B active parameters, a useful reference point for how far small code models go.

AI HOT (Curated Pool)

Microsoft releases its first advanced reasoning AI model, MAI-Thinking-1

Microsoft released MAI-Thinking-1 at Build 2026, describing it as a medium-sized reasoning model that matches leading models on key software engineering benchmarks.

Why it matters: Microsoft's first in-house reasoning model matches top-tier software engineering benchmarks, built on self-generated training data with no distillation.

r/LocalLLaMA

Using Gemma 4 E4B with LiteRT: about 2.4× faster text generation than Q4 GGUF

The author tested Gemma 4 E4B on an RTX 4060 Ti 16GB, where LiteRT averaged 157.2 tok/s for text generation versus 66.3 tok/s for llama.cpp Q4 GGUF; image captioning on 111 full-resolution images improved only 1.1×, at about 72 seconds versus 80 seconds.

Why it matters: A hands-on RTX 4060 Ti test puts LiteRT text generation at roughly 2.4x llama.cpp Q4 GGUF, with image processing about even.

r/LocalLLaMA

Benchmarks of 20 Small LLMs on a 6GB RTX 4050

The author benchmarked 20 small LLMs on a 6GB RTX 4050 using LM Studio’s OpenAI-compatible API, with N=5 speed runs at 1k, 8k, and 32k context; unsloth/lfm2.5-vl-1.6b led throughput at 207 tok/s on 1k context while using 3.0GB VRAM.

Why it matters: A rare speed reference for running 20 small models on a 6GB laptop GPU, tested at 1k, 8k and 32k context.

Jun 2Tuesday

Ben's Bites

Opus 4.8

Ben’s Bites says Claude Opus 4.8 is out, and Claude Code can write an orchestration script before launching subagents in parallel to work through complex tasks.

Why it matters: Anthropic releases Claude Opus 4.8, letting Claude Code write a scheduling script and spin up parallel subagents for complex tasks, while developers say deterministic workflows around small agent loops hold up better.

r/LocalLLaMA

Replaced Claude with local Qwen3.6-27B in my multi-agent orchestrator for 2 weeks

The author ran Qwen3.6-27B on one RTX 3090 across 47 multi-step coding workflows. Plan generation reached about 95% schema validity, but tool-call formatting errors were about 12%, and practical long-context use degraded past about 12k tokens.

Why it matters: Two weeks of real multi-step coding workflows on one RTX 3090, with concrete numbers on where a local 27B model holds up and where it breaks.

Synced · WeChat

DataMaster: When AI Becomes Its Own Data Engineer

DataMaster searches, cleans, and combines data while keeping the model and training algorithm fixed; on MLE-Bench Lite, it raised the medal rate from 35.91% to 68.18%.

Why it matters: DataMaster has the model handle external data search, cleaning and multi-table joins itself without changing model architecture or training algorithms, lifting medal rate on MLE-Bench Lite from 35.91% to 68.18%.

Xinzhiyuan · WeChat

Chinese AI chip firm raises nearly 1B yuan as next-generation card is due this year

Motern AI completed a nearly 1 billion yuan Series C round and plans to release its SparsePrime inference card this year; the article says its S30 and S40 cards achieved three consecutive wins in MLPerf Inference.

Why it matters: Moffett AI closed a nearly 1 billion yuan Series C, following three straight MLPerf wins for its S30 and S40 sparse-computing inference cards, with the next-generation SparsePrime due this year.

Xinzhiyuan · WeChat

CAS Opens MobileGym, a Browser-Based Agent Training Environment for Mobile Apps

CASIA released MobileGym, a browser-based Android simulation environment covering 28 apps, with about 400MB per instance, 3-second cold start, JSON state snapshots, and programmatic task verification for mobile-agent training and evaluation.

Why it matters: A browser-based Android simulator with 28 apps and JSON snapshots gives agent builders a cheap test bed.

New York Times Chinese

China Is Trying to Use AI to Predict Dissent

Geedge is developing an AI system to predict dissent using telecom, social media, and location data, according to 100,000 leaked documents reviewed by Vanderbilt researchers; U.S. officials say there is no evidence that the predictive technology has been finalized or deployed.

Why it matters: Leaked files show Jizhi built AI meant to predict dissent from telecom, social and location data, though the documents put it at the research stage.

AI HOT (Curated Pool)

NVIDIA Cosmos 3 Tops Open-Weight Image and Video Generation Rankings

NVIDIA Cosmos 3 ranked first in Artificial Analysis’s open-weight text-to-image and image-to-video categories, with 16B Nano and 64B Super variants, and the release includes weights, code, curated datasets, and fine-tuning recipes under the OpenMDW 1.1 license.

Why it matters: NVIDIA's Cosmos 3 took first on Artificial Analysis's open-weights board for both text-to-image and image-to-video, with a Mixture-of-Transformers design pairing an autoregressive reasoner with a diffusion generator.

Jun 1Monday

AI HOT (Curated Pool)

MiniMax Releases Open-Source M3 with Coding, Long-Context, and Multimodal Capabilities

MiniMax released the open-source M3 model with coding, a 1M-token context window, and native multimodal support; M3 scores 59.0% on SWE-Bench Pro, 83.5% on BrowseComp, and costs about one-twelfth per token versus GPT-5.5.

Why it matters: MiniMax has open-sourced M3, putting code, million-token long context and native multimodality in one model, scoring 59.0% on SWE-Bench Pro against GPT-5.5's 58.6%.