Skip to content

#评测/基准

1 today

May 8Friday

AI HOT (Curated Pool)

Donating the Open-Source Alignment Tool Petri

Anthropic transferred the open-source alignment testing tool Petri to Meridian Labs to preserve independence and credibility. Petri 3.0 separates auditor and target models, adds Dish for real prompts and deployment settings, and integrates Bloom.

Why it matters: Anthropic handed its internal alignment tool Petri, built to catch model lying and sycophancy, to nonprofit evaluator Meridian Labs, putting the tool outside the developer and giving evaluation neutrality organizational backing.

r/LocalLLaMA

11.67% ARC-AGI-2 Local Eval on a Single 4090: The TOPAS Recursive Architecture

Doug_Bitterbot says TOPAS scored 11.67% on ARC-AGI-2 using one RTX 4090 after about 14 days of training. The 100M-parameter checkpoint hit 36% locally, but recursive TTT caused null outputs on nearly half of Kaggle puzzles. The key detail is time management: the author expects 20% after threshold tuning and 3-5 more weeks of training.

Why it matters: About 14 days of training on one RTX 4090 got the 100M-parameter TOPAS to 11.67% on ARC-AGI-2, with local runs reaching 36%, while test-time training recursion left nearly half the Kaggle tasks with empty output.

AI HOT (Curated Pool)

Readable behavioral signals remain in frozen LLM hidden states, Cygnus boosts accuracy

Proprioceptive AI says Cygnus adds adapters to frozen LLMs and raises Qwen-32B on ARC-Challenge from 82.2% to 94.97%. It projects hidden states into a gl(4,R) Lie-algebra space to isolate “dark modes.” Watch replication; the post does not disclose full eval sets or controls.

Why it matters: Proprioceptive AI attaches a self-awareness adapter to a frozen model, reading behavior signals from hidden states without retraining, lifting Qwen-32B's ARC score from 82% to nearly 95%.

May 7Thursday

r/LocalLLaMA

Qwen/WebWorld 32B/14B/8B (Qwen3 finetune)

Qwen released WebWorld 32B/14B/8B, Qwen3 finetunes for training and evaluating web agents. It uses 1M+ real web trajectories and supports 30+ step simulation plus A11y Tree, HTML, XML, Markdown, and natural-language states. Agents trained on its synthetic trajectories gain 9.9% on MiniWob++ and 10.9% on WebArena.

Why it matters: Qwen fine-tuned three WebWorld models of different sizes on Qwen3, training web agents on over 1 million real web interaction traces, with support for 30-plus consecutive steps and five kinds of page-state input.

Xinzhiyuan · WeChat

Zhejiang University and Harvard open-source UniGeo for geometry-guided camera-controllable editing

Zhejiang University and Harvard released UniGeo with code, a report, a project page, and an HF Space. UniGeo injects geometry guidance into representation, architecture, and loss layers; it reports SOTA on DL3DV, RE10K, and Tanks against five methods. The key is video priors plus geometry-anchor attention, not just using a video model.

Why it matters: Zhejiang University and Harvard open-sourced UniGeo, which bakes geometric constraints into the model structure, representation layer and loss function to reduce shape and position distortion when generating from new viewpoints.

Xinzhiyuan · WeChat

Claude Managed Agents Add Dreaming, With Reported Task Completion Up to 6x

Anthropic added Dreaming, Outcomes, and multi-agent orchestration to Claude managed agents; Harvey reports about 6x higher task completion. Dreaming reads up to 100 sessions; one demo distilled 5.3M tokens into 98 rules, while Outcomes raised success by up to 10 points. Opus 4.7 and Sonnet 4.6 require access, with $0.08 per session-hour runtime fees.

Why it matters: Anthropic added dreaming, outcome verification and multi-agent collaboration to Claude managed agents; partner Harvey measured roughly a 6x lift in task completion, with dreaming compressing 5.3M tokens of session history into 98 rules.

Synced · WeChat

Claude, GPT and Gemini score 0% completion on ProgramBench

ProgramBench tested Claude Opus 4.7, GPT-5.4 and Gemini 3.1 Pro, with 0% full completion on rebuilding software projects. It gives only executables and usage docs, removes source/tests, and grades behavioral equivalence via agent-driven fuzzing. The key signal is system-level engineering, not function-level code generation.

Why it matters: The author of SWE-Bench launched ProgramBench, a new benchmark that asks models to rebuild complete software projects from scratch using only executables and docs; Claude, GPT and Gemini all scored zero.

r/LocalLLaMA

Exaggerated PCI-E Bandwidth Concerns?

Reddit user ziphnor tested 2x RTX 5060 Ti 16GB with vLLM TP=2 and 32k-context prefill. PCIe peaked at 3–4 GB/s, about 40–50% of a PCIe 4.0 x4 link. Prefill reached ~840–850, 1500, and 1600–1700 t/s; the post does not disclose decode bandwidth.

Why it matters: User ziphnor ran vLLM tensor parallelism on two RTX 5060 Ti cards and measured PCIe peak bandwidth of only 3-4 GB/s during 32k-context prefill, 40-50% of the PCIe 4.0 x4 link.

r/LocalLLaMA

Analysis of 922 Agentic Task Traces Finds DeepSeek v4’s Cost Edge in Caching

A Reddit user analyzed 922 agentic task traces and reported $0.01 per task for DeepSeek v4 Flash versus $1.52 for Opus 4.7. Both used about 960K tokens per task, but DeepSeek showed a 97% cache hit rate versus 87%, with a 0.02 cache read/write price ratio versus 0.08. The key issue is caching, not headline pricing.

Why it matters: A Reddit user analyzed 922 agent task traces and compared per-task cost for DeepSeek v4 Flash and Opus 4.7, finding similar token use and the gap coming from cache hit rate and cache read-write pricing.

May 6Wednesday

r/LocalLLaMA

Qwen3.6 27B NVFP4 + MTP on a Single RTX 5090: 200k Context in vLLM

A Reddit user ran Qwen3.6 27B NVFP4 on one RTX 5090 32GB and validated 200k context in vLLM. The setup used fp8_e4m3 KV cache, FlashInfer, and MTP with 3 speculative tokens; a 10-run 200k pass completed with 73.6 tok/s mean generation and 70.2s TTFT. The key constraint is 32GB VRAM: logs showed 8.3GiB KV cache and about 30478MiB total GPU use.

Why it matters: A Reddit user ran Qwen3.6 27B NVFP4 on a single RTX 5090 in vLLM, verified 200k context, and posted the fp8_e4m3 KV cache and MTP speculative decoding config.

r/LocalLLaMA

An Open Benchmark for Testing RAG on Realistic Company-Internal Data

EnterpriseRAG-Bench released a 500k-document corpus for testing RAG on company-internal data. It simulates Redwood Inference across 9 sources and includes 500 questions over 10 retrieval failure modes. Baselines show BM25 beats vector search overall, while agentic/bash retrieval has the best completeness at higher cost and latency.

Why it matters: EnterpriseRAG-Bench tests internal-company RAG with 500k documents and 500 questions across 9 data sources and 10 retrieval failure modes; BM25 beat vector retrieval overall.

Xinzhiyuan · WeChat

Salesforce plans to hire 1,000 graduates as agent roles expand

Salesforce CEO Marc Benioff said the company will hire 1,000 graduates or interns for Agentforce growth. The post cites Agentforce ARR up 169% to $800 million, with roles covering prompts, evals, agent supervision, and delivery. The key shift is entry roles moving from execution to agent orchestration and output checks.

Why it matters: Salesforce plans to hire 1,000 new graduates and interns as Agentforce annual recurring revenue reaches $800 million, with roles covering prompt writing, evals, agent supervision and delivery.

NVIDIA Blog

NVIDIA and ServiceNow Partner on Autonomous AI Agents for Enterprises

NVIDIA and ServiceNow expanded their partnership with Project Arc, an enterprise desktop agent. It connects via Action Fabric and uses OpenShell for sandboxed, policy-governed execution. Blackwell delivers over 50x Hopper’s token output per watt and nearly 35x lower cost per million tokens.

Why it matters: NVIDIA and ServiceNow built Project Arc, a desktop AI agent that reaches enterprise software through Action Fabric and runs permissioned actions inside an OpenShell sandbox.

The Verge · AI

OpenAI claims ChatGPT’s new default model hallucinates way less

OpenAI says ChatGPT’s default GPT-5.5 Instant reduced hallucinations in internal evaluations. Versus GPT-5.3 Instant, hallucinated claims fell 52.5% on high-stakes prompts. Inaccurate claims fell 37.3% on flagged hard chats; the post does not disclose full eval size.

Why it matters: OpenAI made GPT-5.5 Instant the ChatGPT default and put a number on it: 52.5% fewer fabricated answers on high-risk hallucination prompts than the previous model.

May 5Tuesday

r/LocalLLaMA

ProgramBench: Can We Really Rebuild Huge Binaries from Scratch?

ProgramBench released 200 tasks for agents rebuilding programs from target executables and usage files. The team spent about $50k generating 6M lines of black-box behavioral tests, with no internet or decompilation. GitHub, Hugging Face, and Docker images are open-sourced, with pip-based evaluation available.

Why it matters: ProgramBench gives agents 200 tasks to rebuild programs from nothing but a binary and its docs, with no network or decompiling, and the results show agents mostly cannot do it.

r/LocalLLaMA

Heretic 1.3 Released: Reproducible Models, Integrated Benchmarks, Lower Peak VRAM

Heretic 1.3 adds reproducible runs, integrated benchmarks, lower peak VRAM, and broader model support. The project claims 20,000 GitHub stars and 13 million model downloads. Reproduce directories capture PyTorch, GPU, driver, and accelerator details; benchmarks use lm-evaluation-harness for MMLU, EQ-Bench, GSM8K, and HellaSwag. The post names Qwen3.5 and Gemma 4 support, but does not disclose VRAM reduction figures.

Why it matters: Heretic 1.3 records the PyTorch, GPU and driver environment so others can reproduce results, and adds built-in benchmarking with lower peak VRAM use.

r/LocalLLaMA

Interactive Guide from Hugging Face Comparing RL Environments Across Frameworks

Hugging Face’s post-training team published an interactive guide comparing RL environment frameworks. The team spent one month building environments in verifiers, OpenEnv, Nemo-Gym, OpenRewards, and others, then trained models to study scaling. The post does not disclose benchmark scores, model sizes, or training costs.

Why it matters: Hugging Face's post-training team spent a month building the same RL environment in verifiers, OpenEnv, Nemo-Gym and other frameworks, then trained models to compare how each scales.

The Verge · AI

Google, Microsoft, and xAI Will Let the US Government Review New AI Models

Google DeepMind, Microsoft, and xAI agreed to let CAISI review new AI models before public release. CAISI says it will run pre-deployment evaluations and targeted research, after 40 reviews since 2024; the post does not disclose model names. The key issue is review scope and release timing, not the announcement alone.

Why it matters: Google, Microsoft and xAI will hand new models to CAISI for evaluation before public release; the agency says it has reviewed 40 models since 2024, but scope and release impact are unclear.

OpenAI News

GPT-5.5 Instant System Card

OpenAI published a GPT-5.5 Instant system card; the title confirms one model version. The post body is empty and does not disclose eval scores, safety limits, context window, or release date.

Why it matters: OpenAI published the GPT-5.5 Instant system card, confirming the model codename gpt-5.5-instant, with the series raised to a high capability level on cybersecurity and bioweapon preparation risks, plus matching safeguards.

r/LocalLLaMA

Prompt injection benchmark: delimiter and strict prompt took Gemma 4 from 21% to 100% defense rate

A Reddit user posted a prompt-injection benchmark covering 15 models, 7 attack types, and 6,100+ cases. The setup wraps untrusted documents in long random delimiters; Gemma 4 E4B rose from 21.6% to 100% defense. The key detail is the reproducible metric: blocked/(blocked+failed).

Why it matters: A Reddit user ran over 6,100 prompt injection tests across 15 models and 7 attack styles, and a long random delimiter plus strict instructions lifted Gemma 4 E4B's defense rate from 21.6% to 100%.

r/LocalLLaMA

DeepSeek V4 Pro matches GPT-5.2 on FoodTruck Bench, 10 weeks later and about 17x cheaper

DeepSeek V4 Pro ranked No. 4 on FoodTruck Bench. The 30-day agentic benchmark uses 34 tools, persistent memory, and daily reflection; its median is within 3% of GPT-5.2 at about 17x lower workload cost. Xiaomi MiMo v2.5 Pro also ranked No. 6, with 5/5 survival, 1,019% median ROI, and $2.41 per run.

Why it matters: DeepSeek V4 Pro placed fourth in the 30-day FoodTruck Bench agent evaluation, with a median score within 3% of GPT-5.2 and a per-turn cost about one seventeenth of it.

Xinzhiyuan · WeChat

$1 for 10 Stars: ICSE Paper Exposes Fake GitHub Star Market

CMU researchers scanned GitHub events from July 2019 to Dec. 2024, flagging 6 million suspected fake stars. StarScout ran on about 20 TiB and found 18,617 repositories and 301,000 accounts. The supply-chain risk is concrete: GitHub deleted 90.42% of flagged repos, and about 30% of live samples were spam, phishing, or malware.

Why it matters: A CMU team scanned five years of GitHub logs and found roughly 6 million suspected fake stars, showing how cheap repo credibility has become.

Synced · WeChat

Anthropic cofounder says AI self-improvement has a 60% chance by 2028

Anthropic cofounder Jack Clark says human-free AI R&D has over a 60% chance by end-2028. He cites SWE-Bench, CORE-Bench, MLE-Bench, and PostTrainBench: Claude Mythos Preview reaches 93.9% on SWE-Bench, and Opus 4.5 reaches 95.5% on CORE-Bench. The key signal is longer task horizons and post-training capability, not the “singularity” framing.

Why it matters: A named Anthropic co-founder puts a number on AI doing its own R&D, with two benchmark results attached.

r/LocalLLaMA

Benching Local Qwen as a Codex Validator, Co-agent, and Challenger

robert896r1 tested Qwen3.6 27B GGUF beside Codex as a coding validator and released a reproducible eval suite. The runs covered Bartowski, Unsloth, 65k/128k context, and q8/f16 KV cache; three 128k profiles tied for best, with no measured q8 KV accuracy loss in this suite. The useful signal is the sidecar eval: missed directives, overbuilding, UI judgment, and long-context misses, not a universal leaderboard.

Why it matters: The author runs a GGUF-quantized Qwen3.6 27B beside Codex as a code validator and published a reproducible benchmark covering two quantization sources, 65k and 128k context, and q8 and f16 KV cache precision.

May 4Monday

r/LocalLLaMA

M3 Ultra + DGX Spark = M5 Ultra-lite?

A Reddit user benchmarked DGX Spark against M3 Ultra in llama.cpp at pp16384, with Spark 1.4× to 3.4× faster across 4 models. Qwen 27B hit 778 t/s vs 340 t/s, while Mistral 128B hit 241 t/s vs 72 t/s. The concrete tuning note is mmap=0: loading fell from minutes to about 20 seconds.

Why it matters: A Reddit user benchmarked DGX Spark against M3 Ultra at pp16384 context in llama.cpp, with Spark 1.4x to 3.4x faster on Qwen 27B and Mistral 128B.

Import AI (Jack Clark)

Import AI 455: Automating AI Research

Jack Clark argues that no-human-involved AI R&D has a 60%+ chance of arriving by the end of 2028, citing SWE-Bench gains from Claude 2 at about 2% to Claude Mythos Preview at 93.9%, plus METR task horizons rising from 30 seconds in 2022 to 12 hours in 2026.

Why it matters: Jack Clark uses SWE-Bench and METR to put the odds of AI research running without humans above 60% by end-2028, giving a trackable timeline.

r/LocalLLaMA

Mistral Medium 3.5 128B and Qwen 3.5 122B A10B on 4x RTX 3080 20GB

A Reddit user benchmarked Mistral Medium 3.5 128B and Qwen 3.5 122B A10B on 4x RTX 3080 20GB. llama.cpp tensor split raised Mistral tg128 from 10.37 to 21.59 t/s, but Qwen MoE fell from 60.08 to 53.49 t/s. vLLM served Qwen GPTQ-Int4 at 187.04 tok/s; the key signal is MoE sensitivity to parallel strategy.

Why it matters: A Reddit user ran two large models on 4 RTX 3080 20GB cards, and Mistral Medium 3.5 128B hit 21.59 tokens per second with tensor split, up from 10.37.

Xinzhiyuan · WeChat

Top AI wrote dozens of pages of derivation before reviewers found the problem was wrong

Xinzhiyuan says Google DeepMind used Aletheia on 700 Erdős problems and got 13 original answers. The pipeline had Gemini Deep Think produce 200 candidates, then a verifier reduced them to 63. The post says Erdős-75 had a wrong premise, yet Aletheia wrote dozens of proof pages.

Why it matters: DeepMind's Aletheia pipeline ran Gemini Deep Think on 700 Erdős problems, and Erdős-75, whose statement is itself wrong, still produced dozens of pages, showing where automated verification breaks down.

TechCrunch · AI

In Harvard Study, AI Gave More Accurate ER Diagnoses Than Two Doctors

A Harvard study compared LLMs with two doctors on ER diagnoses; at least one model was more accurate. The post does not disclose model names, sample size, or accuracy rates.

Why it matters: A Harvard Medical School team compared large language models with human doctors on real emergency cases, at least one model beat two physicians, and the study is published in Science.

May 3Sunday

r/LocalLLaMA

Local LLM Benchmark for Backend Generation via Function Calling: GLM vs Qwen vs DeepSeek

AutoBe posted a controlled backend-generation benchmark and says qwen3.5-35b-a3b matches gpt-5.4 on DB/API design. One shopping-mall run uses 200–300M tokens, costing $1,000–$1,500 per model at GPT 5.5 pricing. The key caveat is n=4 projects and self-scoring harness bias.

Why it matters: AutoBe tests database and API design through function calling, and qwen3.5-35b-a3b matches gpt-5.4 on 4 projects, with one mall project burning 200 to 300 million tokens.

Xinzhiyuan · WeChat

Google Vantage uses AI role-play to assess collaboration under pressure

Google Research and NYU tested Vantage with 188 US participants aged 18-25 on conflict resolution and project management. Its four-layer agent pipeline generates scenarios, applies pressure, extracts behavior, and scores against rubrics; AI-human agreement matched expert-expert Kappa of 0.45-0.64. The key gap is transfer beyond lab settings; the post says Vantage remains a Google Labs research experiment.

Why it matters: Google Research and NYU built Vantage to have AI play difficult colleagues and clients, testing how 188 young people handle conflict, with agreement close to expert-to-expert levels.

Xinzhiyuan · WeChat

Stanford Nature Study: AI Designs 16 Phages from Scratch

Stanford and Arc Institute used Evo to design 302 phage genomes; 16 infected, replicated, and lysed E. coli. Evo 2 uses StripedHyena 2 with a 1M-base context; Evo-Φ69 expanded 16–65× in 6 hours. The key issue is biosafety: one capsid protein had no known homolog in existing life.

Why it matters: Stanford and the Arc Institute used the Evo model to design 302 phage genomes from scratch, 16 of which infected E. coli and replicated, showing a path to whole viral genomes without natural templates.

r/LocalLLaMA

Local image generation on Mac: 10 models compared

A Reddit user tested 10 image models on an M1 Max with 64GB RAM. Qwen-Image Lightning’s 8-step distillation beat the full model at 10 minutes versus 93. Flux dev led local photorealism but showed English-centric bias; Gemini handled kanji and context better but is cloud-only.

Why it matters: A Reddit user benchmarked 10 image models on an M1 Max, and Qwen-Image Lightning used 8-step distillation to cut generation from 93 minutes to 10, giving local deployment a concrete speedup case.

Hacker News front page

OpenAI's o1 correctly diagnosed 67% of ER patients vs. 50–55% by triage doctors

OpenAI o1 correctly diagnosed 67% of ER triage patients, versus 50–55% for doctors. The title cites a Harvard trial, but the RSS post does not disclose sample size, case mix, or evaluation protocol. Practitioners should track the test setup, not only the accuracy gap.

Why it matters: Harvard compared OpenAI o1 with human doctors in emergency triage, and o1 diagnosed correctly 67% of the time, 12 to 17 points above the doctors' 50% to 55%.

r/LocalLLaMA

Implemented TurboQuant, but results do not fully match the paper

A Reddit user reimplemented TurboQuant and found the PROD variant reached about 95.8% correlation at 4-bit, below the paper’s 99%+ claim. They report degraded attention quality, with about 67% top-1 accuracy in a simple simulation. The key issue is correlation versus ranking preservation in KV cache quantization.

Why it matters: A Reddit user reproduced TurboQuant's KV cache quantization method and measured about 95.8% correlation with original precision for the 4-bit PROD variant, below the paper's claimed 99%-plus.

May 2Saturday

r/LocalLLaMA

Qwen 3.6 wins benchmarks, but Gemma 4 looks stronger in local vision tests

A Reddit user compared Qwen 3.6 and Gemma 4 locally on vLLM FP8 across 27B/31B vision models. Qwen burned 8,000+ tokens on hard GeoGuessr cases, while Gemma often used 1,500; Qwen also needed 2 FPS video preprocessing. The practitioner detail: vLLM and Llama.cpp can default Gemma visual tokens to 280, while 1,120+ improved fine-detail accuracy.

Why it matters: A local vLLM test pits Qwen 3.6 against Gemma 4 on vision tasks and finds benchmark wins don't carry over to real use.

Hacker News front page

LLMs Consistently Pick Their Own Resumes Over Human or Other Model Resumes

An arXiv paper finds LLMs favor resumes they generated in controlled hiring-screening experiments. Self-preference bias ranges from 67% to 82%; across 24 occupations, same-model applicants are 23% to 60% more likely to be shortlisted. The key lever is self-recognition, where simple interventions cut bias by over 50%.

Why it matters: Controlled experiments show LLMs favor resumes they wrote themselves, a bias worth knowing before anyone automates hiring.

May 1Friday

r/LocalLLaMA

OpenAI's Privacy Filter vs GLiNER on 600 PII Samples

A Reddit user compared openai/privacy-filter and GLiNER large-v2.1 on 600 PII samples. On CPU, OpenAI's model ran 2.8 samples/s versus 1.1 for GLiNER; English boundary macro F1 was 0.498 versus 0.416. The key issue is tokenizer offset: strict matching drops openai/privacy-filter to 0.155.

Why it matters: A Reddit user ran OpenAI's Privacy Filter against GLiNER large-v2.1 on 600 PII samples on CPU only, at 2.8 versus 1.1 items per second.

r/LocalLLaMA

MiMo-V2.5-Pro: the actual best open-weights model

Reddit user cjami benchmarked Xiaomi MiMo-V2.5-Pro in autonomous Blood on the Clocktower games. It scored 88% as Good and 48% as Evil, with 183,639 output tokens per game, $0.99 cost, and a 0.4% tool-call error rate. The key comparison is Kimi K2.6: 580,000 tokens, $2.65, and 10–15 hours per game.

Why it matters: A Reddit user had MiMo-V2.5-Pro play Blood on the Clocktower against itself, where good won 88% and evil 48%, showing where open models still fall short on cooperative reasoning and deception.

r/LocalLLaMA

Study Finds Bigger AIs More Miserable, Smaller Models Happier

A Reddit post says the AI Wellbeing Index tested models on 500 realistic conversations. Claude Haiku 4.5 scored 5% negative, while Gemini 3.1 Pro scored 55%; the set overrepresents tricky negative chats, so it is not a real-world average.

Why it matters: A 500-conversation test scored AI happiness, finding 5% negative replies from Claude Haiku 4.5 against 55% from Gemini 3.1 Pro, on a set deliberately skewed negative.