Skip to content

Benchmarks

Who is actually stronger: benchmark results, methodology disputes and leaderboard changes.

Latest picks

261–280 of 471

May 22Friday

Bloomberg Technology

Pentagon Tests Rival AI Models in Race to Replace Anthropic

The Pentagon is testing rival AI models with 25 departmental “power users” as it seeks alternatives to Anthropic’s Claude, according to a senior defense official; the RSS snippet does not disclose the candidate model list, evaluation criteria, or deployment timeline.

Why it matters: The Pentagon is testing OpenAI and Google models with senior users across 25 departments as it looks for alternatives to its current Anthropic Claude.

May 21Thursday

r/LocalLLaMA

Agent Execution Tax: New Procurement Metric for Browser Agent Benchmarks?

Fireworks ran 720 browser-agent tasks on WebVoyager and reported a 22.9% Agent Execution Tax, defined as wasted over productive inference; MiniMax M2.5 cost 2.3x less per successful task than Gemini, while GLM-5 reached 57.1% accuracy and Kimi K2.5 had 0% parse retries across 852 calls.

Why it matters: Fireworks ran 720 browser agent tasks on WebVoyager and found 22.9% of inference calls were wasted, giving buyers a way to price that hidden cost.

r/LocalLLaMA

LLM planner: pick a rig by use case, model, or budget, or pick models for your rig

totosse17 published the LLMRequirements hardware planner with 60+ build configs, 50+ models, 130 cited tokens-per-second sources, 150+ reviewer videos, multi-region prices, idle and active watts, and a public GitHub data repo.

Why it matters: A public dataset compares 60+ local model rigs and 50+ models on measured generation speed, idle and load power, and prices across regions.

r/LocalLLaMA

Tencent Hy-MT2 30B/7B/1.8B

Tencent released Hy-MT2 translation models in 1.8B, 7B, and 30B-A3B sizes, supporting translation across 33 languages; AngelSlim 1.25-bit quantization reduces the 1.8B model’s storage requirement to 440 MB and raises inference speed by 1.5x.

Why it matters: Tencent's Hy-MT2 line covers 1.8B, 7B and 30B-A3B sizes with 33-language translation, and the quantized 1.8B build drops to 440MB of storage.

Xinzhiyuan · WeChat

USTC Papers Study Lifelong Learning for LMMs via Multimodal Knowledge Injection

USTC researchers released MMEVOKE and KORE: MMEVOKE contains 9,422 samples across 159 subcategories, while KORE uses knowledge-tree augmentation and null-space constrained fine-tuning to reduce catastrophic forgetting during multimodal knowledge injection.

Why it matters: A USTC team's MMEVOKE benchmark and KORE fine-tuning method offer reusable evaluation and training ideas for forgetting in continual learning of multimodal models.

Latent Space

OpenAI GPT-next Disproves 80-Year-Old Erdős Planar Unit Distance Problem for Under $1000

OpenAI said an internal general-purpose reasoning model disproved the 1946 Erdős planar unit distance problem by finding a new family of constructions; the reasoning summary reportedly spans about 125 pages, while outside observers speculate the run used under 32 hours or under $1,000.

Why it matters: A general OpenAI reasoning model disproved Erdős's 1946 planar unit distance conjecture, producing a roughly 125-page summary; outside observers estimate under 32 hours of runtime and under $1,000 in cost.

Synced · WeChat

Xie Saining’s Team Releases Second-Generation Representation Autoencoder RAEv2

Xie Saining’s team, Adobe Research, and the Australian National University released RAEv2, which reaches gFID 1.06 after 80 epochs on ImageNet-256 and reduces EPFID@2 from 177 epochs to 35 epochs while keeping compute at 189 GFLOPs.

Why it matters: Xie Saining's team with Adobe Research and ANU released RAEv2, cutting ImageNet-256 convergence from 177 epochs to 35 while holding compute at 189 GFLOPs.

r/LocalLLaMA

HalBench: Custom sycophancy and hallucination benchmark tests 4 frontier models

HalBench tested 4 frontier models on 3,200 false-premise prompts, with Sonnet 4.6 ranking first at a 0.565 mean score and Gemini 3.1 Pro last at 0.339; higher scores mean the model more often named the false premise and pushed back instead of complying.

Why it matters: A developer built HalBench, testing four frontier models with 3,200 false-premise prompts to see whether they go along or push back, and ranked each model's score.

May 20Wednesday

AI HOT (Curated Pool)

Gemini 3.5 Flash launches with stronger performance and speed

Google opened Gemini 3.5 Flash after Google I/O across its products and API; the post says it outperforms Gemini 3.1 Pro on most benchmarks and generates tokens 4x faster than other frontier models.

Why it matters: Gemini 3.5 Flash is now open to product teams and API users, with Google claiming it beats 3.1 Pro on most benchmarks.

AI HOT (Curated Pool)

Google releases Gemini 3.5 Flash with 55 intelligence score

Google released Gemini 3.5 Flash, raising its intelligence score by 9 points to 55, exceeding 280 output tokens per second, and increasing operating cost by 5.5 times versus the previous generation.

Why it matters: Gemini 3.5 Flash trades a higher intelligence score and faster output for running costs 5.5 times the previous generation.

AI HOT (Curated Pool)

Google releases Gemini 3.5 Flash for complex agent workflows

Google introduced Gemini 3.5 Flash at Google I/O for long-running agent workflows; it outscored 3.1 Pro on Terminal-Bench and MCP Atlas, runs up to 4x faster than other frontier models, and reaches up to 12x speed gains in Google Antigravity.

Why it matters: Google's first Gemini 3.5 model, Flash, targets long multi-step agent workflows and outscores the previous 3.1 Pro on Terminal-Bench and MCP Atlas.

AI HOT (Curated Pool)

Google releases Gemini 3.5 Flash with output speed about 4x GPT-5.5

Google introduced Gemini 3.5 Flash at I/O 2026, with output speed reaching 289 tokens per second, about 4x faster than Claude Opus 4.7 and GPT-5.5 xhigh under the cited comparison.

Why it matters: Google's Gemini 3.5 Flash hits 289 output tokens per second, about four times faster than GPT-5.5 xhigh and Claude Opus 4.7.

r/LocalLLaMA

KV cache quantization benchmarks: TurboQuant is overrated, q5 deserves attention, q8 may waste VRAM

Anbeeld benchmarked KV cache quantization for Qwen 3.6 27B on one RTX 3090 at 64k and 128k context, reporting q4_0 tail KLD 32% worse than q5_0 and turbo4 running 17% slower than q4_0 with little memory saving.

Why it matters: Anbeeld benchmarks Qwen 3.6 27B KV cache quantization on an RTX 3090, with numbers like q4_0's tail KLD 32% worse than q5_0.

May 19Tuesday

r/LocalLLaMA

Sapient Intelligence releases HRM-Text 1B: 40B tokens, ~$1k pretrain

Sapient Intelligence released HRM-Text 1B, a 1B-parameter model trained from scratch on 16 GPUs for 1.9 days with 40B tokens and a reported ~$1,000 budget; its self-reported chart shows MATH 56.2 and DROP 82.2, while independent evaluation remains pending.

Why it matters: A 1B model pretrained on 16 GPUs in 1.9 days for about $1,000, beating Llama3.2 3B on MATH and DROP, shows one low-cost path for small models.

AI Chat-Group Daily (群聊日报)

May 18, 2026 Chat Group Daily

The chat group daily says AI21 Labs cut 60% of staff and stopped selling model access, and cites a University of Waterloo paper where GPT-5.4 accuracy dropped from 100% to 23% after false peer-consensus injection; the snippet also mentions Meta layoff talk at 10%, but does not disclose source details or confirmation conditions.

Why it matters: AI21 Labs cut 60% of staff and stopped selling its models on their own, as token prices fell 100x in two years and six independent model firms were absorbed.

QbitAI · WeChat

JD and CAS IIE Publish Three Papers Defining Self-Taught RLVR

JD and CAS IIE released three Self-Taught RLVR papers covering RLSD, NPO, and CoPD; RLSD reports that 200 training steps on Qwen3-VL-8B-Instruct exceed GRPO at 400 steps across 8 benchmarks.

Why it matters: JD and the Institute of Information Engineering at CAS published three papers on Self-Taught RLVR, a route where the model generates its own training signal during reinforcement learning, skipping human labels and external reward models.

AI HOT (Curated Pool)

Qwen3.7 Preview lands on Arena; Alibaba rises to fifth in vision ranking

Alibaba says Qwen3.7-Plus-Preview has landed on Arena and that Alibaba now ranks fifth in vision; the post does not disclose benchmark scores, the number of competing models, or a release timeline for the Qwen3.7 series.

Why it matters: Alibaba put Qwen3.7-Plus-Preview on an arena and claims fifth place on a vision leaderboard, with no scores or launch date.

r/LocalLLaMA

21 GPUs benchmarked running a small TTS model, with 5GB peak VRAM

A Reddit user rented 21 GPUs on vast.ai to benchmark OmniVoice, a small TTS model with about 5GB peak VRAM, using xRT as the audio generation speed metric and averaging 3 voice-cloning runs with reference audio.

Why it matters: A Reddit user benchmarked 21 rented GPUs on a small voice-cloning TTS model, averaging three runs per card, as a buying reference for anyone building speech synthesis.

r/LocalLLaMA

llama.cpp MTP support landed: Qwen3.6 27B reaches 2.44× on Strix Halo

llama.cpp merged MTP speculative decoding in PR #22673; Qwen3.6 27B Q8_0 rose from 7.4 to 18.1 tok/s on Strix Halo, while a dual RTX 3090 Q8_0 setup rose from 25.7 to 55.9 tok/s.

Why it matters: After llama.cpp merged MTP speculative decoding, Qwen3.6 27B measured 7.4 tok/s to 18.1 tok/s on Strix Halo, giving local inference a concrete speedup reference.

May 18Monday

AI HOT (Curated Pool)

The Open Agent Leaderboard

IBM Research published the Open Agent Leaderboard on Hugging Face to evaluate agents across language understanding, tool use, and multi-step reasoning tasks; the post does not disclose dataset size, model scores, or the evaluation date.

Why it matters: IBM Research put an open agent leaderboard on Hugging Face that scores whole systems across six scenario benchmarks and reports quality alongside running cost.