Skip to content

Benchmarks

Who is actually stronger: benchmark results, methodology disputes and leaderboard changes.

Latest picks

341–360 of 471

May 6Wednesday

r/LocalLLaMA

An Open Benchmark for Testing RAG on Realistic Company-Internal Data

EnterpriseRAG-Bench released a 500k-document corpus for testing RAG on company-internal data. It simulates Redwood Inference across 9 sources and includes 500 questions over 10 retrieval failure modes. Baselines show BM25 beats vector search overall, while agentic/bash retrieval has the best completeness at higher cost and latency.

Why it matters: EnterpriseRAG-Bench tests internal-company RAG with 500k documents and 500 questions across 9 data sources and 10 retrieval failure modes; BM25 beat vector retrieval overall.

Xinzhiyuan · WeChat

Salesforce plans to hire 1,000 graduates as agent roles expand

Salesforce CEO Marc Benioff said the company will hire 1,000 graduates or interns for Agentforce growth. The post cites Agentforce ARR up 169% to $800 million, with roles covering prompts, evals, agent supervision, and delivery. The key shift is entry roles moving from execution to agent orchestration and output checks.

Why it matters: Salesforce plans to hire 1,000 new graduates and interns as Agentforce annual recurring revenue reaches $800 million, with roles covering prompt writing, evals, agent supervision and delivery.

NVIDIA Blog

NVIDIA and ServiceNow Partner on Autonomous AI Agents for Enterprises

NVIDIA and ServiceNow expanded their partnership with Project Arc, an enterprise desktop agent. It connects via Action Fabric and uses OpenShell for sandboxed, policy-governed execution. Blackwell delivers over 50x Hopper’s token output per watt and nearly 35x lower cost per million tokens.

Why it matters: NVIDIA and ServiceNow built Project Arc, a desktop AI agent that reaches enterprise software through Action Fabric and runs permissioned actions inside an OpenShell sandbox.

The Verge · AI

OpenAI claims ChatGPT’s new default model hallucinates way less

OpenAI says ChatGPT’s default GPT-5.5 Instant reduced hallucinations in internal evaluations. Versus GPT-5.3 Instant, hallucinated claims fell 52.5% on high-stakes prompts. Inaccurate claims fell 37.3% on flagged hard chats; the post does not disclose full eval size.

Why it matters: OpenAI made GPT-5.5 Instant the ChatGPT default and put a number on it: 52.5% fewer fabricated answers on high-risk hallucination prompts than the previous model.

May 5Tuesday

r/LocalLLaMA

ProgramBench: Can We Really Rebuild Huge Binaries from Scratch?

ProgramBench released 200 tasks for agents rebuilding programs from target executables and usage files. The team spent about $50k generating 6M lines of black-box behavioral tests, with no internet or decompilation. GitHub, Hugging Face, and Docker images are open-sourced, with pip-based evaluation available.

Why it matters: ProgramBench gives agents 200 tasks to rebuild programs from nothing but a binary and its docs, with no network or decompiling, and the results show agents mostly cannot do it.

r/LocalLLaMA

Heretic 1.3 Released: Reproducible Models, Integrated Benchmarks, Lower Peak VRAM

Heretic 1.3 adds reproducible runs, integrated benchmarks, lower peak VRAM, and broader model support. The project claims 20,000 GitHub stars and 13 million model downloads. Reproduce directories capture PyTorch, GPU, driver, and accelerator details; benchmarks use lm-evaluation-harness for MMLU, EQ-Bench, GSM8K, and HellaSwag. The post names Qwen3.5 and Gemma 4 support, but does not disclose VRAM reduction figures.

Why it matters: Heretic 1.3 records the PyTorch, GPU and driver environment so others can reproduce results, and adds built-in benchmarking with lower peak VRAM use.

r/LocalLLaMA

Interactive Guide from Hugging Face Comparing RL Environments Across Frameworks

Hugging Face’s post-training team published an interactive guide comparing RL environment frameworks. The team spent one month building environments in verifiers, OpenEnv, Nemo-Gym, OpenRewards, and others, then trained models to study scaling. The post does not disclose benchmark scores, model sizes, or training costs.

Why it matters: Hugging Face's post-training team spent a month building the same RL environment in verifiers, OpenEnv, Nemo-Gym and other frameworks, then trained models to compare how each scales.

The Verge · AI

Google, Microsoft, and xAI Will Let the US Government Review New AI Models

Google DeepMind, Microsoft, and xAI agreed to let CAISI review new AI models before public release. CAISI says it will run pre-deployment evaluations and targeted research, after 40 reviews since 2024; the post does not disclose model names. The key issue is review scope and release timing, not the announcement alone.

Why it matters: Google, Microsoft and xAI will hand new models to CAISI for evaluation before public release; the agency says it has reviewed 40 models since 2024, but scope and release impact are unclear.

OpenAI News

GPT-5.5 Instant System Card

OpenAI published a GPT-5.5 Instant system card; the title confirms one model version. The post body is empty and does not disclose eval scores, safety limits, context window, or release date.

Why it matters: OpenAI published the GPT-5.5 Instant system card, confirming the model codename gpt-5.5-instant, with the series raised to a high capability level on cybersecurity and bioweapon preparation risks, plus matching safeguards.

r/LocalLLaMA

Prompt injection benchmark: delimiter and strict prompt took Gemma 4 from 21% to 100% defense rate

A Reddit user posted a prompt-injection benchmark covering 15 models, 7 attack types, and 6,100+ cases. The setup wraps untrusted documents in long random delimiters; Gemma 4 E4B rose from 21.6% to 100% defense. The key detail is the reproducible metric: blocked/(blocked+failed).

Why it matters: A Reddit user ran over 6,100 prompt injection tests across 15 models and 7 attack styles, and a long random delimiter plus strict instructions lifted Gemma 4 E4B's defense rate from 21.6% to 100%.

r/LocalLLaMA

DeepSeek V4 Pro matches GPT-5.2 on FoodTruck Bench, 10 weeks later and about 17x cheaper

DeepSeek V4 Pro ranked No. 4 on FoodTruck Bench. The 30-day agentic benchmark uses 34 tools, persistent memory, and daily reflection; its median is within 3% of GPT-5.2 at about 17x lower workload cost. Xiaomi MiMo v2.5 Pro also ranked No. 6, with 5/5 survival, 1,019% median ROI, and $2.41 per run.

Why it matters: DeepSeek V4 Pro placed fourth in the 30-day FoodTruck Bench agent evaluation, with a median score within 3% of GPT-5.2 and a per-turn cost about one seventeenth of it.

Xinzhiyuan · WeChat

$1 for 10 Stars: ICSE Paper Exposes Fake GitHub Star Market

CMU researchers scanned GitHub events from July 2019 to Dec. 2024, flagging 6 million suspected fake stars. StarScout ran on about 20 TiB and found 18,617 repositories and 301,000 accounts. The supply-chain risk is concrete: GitHub deleted 90.42% of flagged repos, and about 30% of live samples were spam, phishing, or malware.

Why it matters: A CMU team scanned five years of GitHub logs and found roughly 6 million suspected fake stars, showing how cheap repo credibility has become.

Synced · WeChat

Anthropic cofounder says AI self-improvement has a 60% chance by 2028

Anthropic cofounder Jack Clark says human-free AI R&D has over a 60% chance by end-2028. He cites SWE-Bench, CORE-Bench, MLE-Bench, and PostTrainBench: Claude Mythos Preview reaches 93.9% on SWE-Bench, and Opus 4.5 reaches 95.5% on CORE-Bench. The key signal is longer task horizons and post-training capability, not the “singularity” framing.

Why it matters: A named Anthropic co-founder puts a number on AI doing its own R&D, with two benchmark results attached.

r/LocalLLaMA

Benching Local Qwen as a Codex Validator, Co-agent, and Challenger

robert896r1 tested Qwen3.6 27B GGUF beside Codex as a coding validator and released a reproducible eval suite. The runs covered Bartowski, Unsloth, 65k/128k context, and q8/f16 KV cache; three 128k profiles tied for best, with no measured q8 KV accuracy loss in this suite. The useful signal is the sidecar eval: missed directives, overbuilding, UI judgment, and long-context misses, not a universal leaderboard.

Why it matters: The author runs a GGUF-quantized Qwen3.6 27B beside Codex as a code validator and published a reproducible benchmark covering two quantization sources, 65k and 128k context, and q8 and f16 KV cache precision.

May 4Monday

r/LocalLLaMA

M3 Ultra + DGX Spark = M5 Ultra-lite?

A Reddit user benchmarked DGX Spark against M3 Ultra in llama.cpp at pp16384, with Spark 1.4× to 3.4× faster across 4 models. Qwen 27B hit 778 t/s vs 340 t/s, while Mistral 128B hit 241 t/s vs 72 t/s. The concrete tuning note is mmap=0: loading fell from minutes to about 20 seconds.

Why it matters: A Reddit user benchmarked DGX Spark against M3 Ultra at pp16384 context in llama.cpp, with Spark 1.4x to 3.4x faster on Qwen 27B and Mistral 128B.

Import AI (Jack Clark)

Import AI 455: Automating AI Research

Jack Clark argues that no-human-involved AI R&D has a 60%+ chance of arriving by the end of 2028, citing SWE-Bench gains from Claude 2 at about 2% to Claude Mythos Preview at 93.9%, plus METR task horizons rising from 30 seconds in 2022 to 12 hours in 2026.

Why it matters: Jack Clark uses SWE-Bench and METR to put the odds of AI research running without humans above 60% by end-2028, giving a trackable timeline.

r/LocalLLaMA

Mistral Medium 3.5 128B and Qwen 3.5 122B A10B on 4x RTX 3080 20GB

A Reddit user benchmarked Mistral Medium 3.5 128B and Qwen 3.5 122B A10B on 4x RTX 3080 20GB. llama.cpp tensor split raised Mistral tg128 from 10.37 to 21.59 t/s, but Qwen MoE fell from 60.08 to 53.49 t/s. vLLM served Qwen GPTQ-Int4 at 187.04 tok/s; the key signal is MoE sensitivity to parallel strategy.

Why it matters: A Reddit user ran two large models on 4 RTX 3080 20GB cards, and Mistral Medium 3.5 128B hit 21.59 tokens per second with tensor split, up from 10.37.

Xinzhiyuan · WeChat

Top AI wrote dozens of pages of derivation before reviewers found the problem was wrong

Xinzhiyuan says Google DeepMind used Aletheia on 700 Erdős problems and got 13 original answers. The pipeline had Gemini Deep Think produce 200 candidates, then a verifier reduced them to 63. The post says Erdős-75 had a wrong premise, yet Aletheia wrote dozens of proof pages.

Why it matters: DeepMind's Aletheia pipeline ran Gemini Deep Think on 700 Erdős problems, and Erdős-75, whose statement is itself wrong, still produced dozens of pages, showing where automated verification breaks down.

TechCrunch · AI

In Harvard Study, AI Gave More Accurate ER Diagnoses Than Two Doctors

A Harvard study compared LLMs with two doctors on ER diagnoses; at least one model was more accurate. The post does not disclose model names, sample size, or accuracy rates.

Why it matters: A Harvard Medical School team compared large language models with human doctors on real emergency cases, at least one model beat two physicians, and the study is published in Science.

May 3Sunday

r/LocalLLaMA

Local LLM Benchmark for Backend Generation via Function Calling: GLM vs Qwen vs DeepSeek

AutoBe posted a controlled backend-generation benchmark and says qwen3.5-35b-a3b matches gpt-5.4 on DB/API design. One shopping-mall run uses 200–300M tokens, costing $1,000–$1,500 per model at GPT 5.5 pricing. The key caveat is n=4 projects and self-scoring harness bias.

Why it matters: AutoBe tests database and API design through function calling, and qwen3.5-35b-a3b matches gpt-5.4 on 4 projects, with one mall project burning 200 to 300 million tokens.