Skip to content

#评测/基准

3 today

May 20Wednesday

AI HOT (Curated Pool)

Google releases Gemini 3.5 Flash with 55 intelligence score

Google released Gemini 3.5 Flash, raising its intelligence score by 9 points to 55, exceeding 280 output tokens per second, and increasing operating cost by 5.5 times versus the previous generation.

Why it matters: A Google Gemini 3.5 Flash release is a top-lab model update, backed by Artificial Analysis numbers for speed, intelligence, and cost. HKR-H/K/R all pass, with the 5.5x cost jump making it more than a routine launch.

AI HOT (Curated Pool)

Google releases Gemini 3.5 Flash for complex agent workflows

Google introduced Gemini 3.5 Flash at Google I/O for long-running agent workflows; it outscored 3.1 Pro on Terminal-Bench and MCP Atlas, runs up to 4x faster than other frontier models, and reaches up to 12x speed gains in Google Antigravity.

Why it matters: HKR-H/K/R all pass: Google launched Gemini 3.5 Flash for long-horizon agents with benchmark and speed claims. This is a same-day major model update, below industry-shaking tier.

AI HOT (Curated Pool)

Google releases Gemini 3.5 Flash with output speed about 4x GPT-5.5

Google introduced Gemini 3.5 Flash at I/O 2026, with output speed reaching 289 tokens per second, about 4x faster than Claude Opus 4.7 and GPT-5.5 xhigh under the cited comparison.

Why it matters: HKR-H/K/R all pass: Google ships Gemini 3.5 Flash with a 289 tokens/sec claim and 4x speed comparison against GPT-5.5 xhigh. Details on price, context window, and capability limits are not disclosed, so it stays in the low 85-94 band.

r/LocalLLaMA

KV cache quantization benchmarks: TurboQuant is overrated, q5 deserves attention, q8 may waste VRAM

Anbeeld benchmarked KV cache quantization for Qwen 3.6 27B on one RTX 3090 at 64k and 128k context, reporting q4_0 tail KLD 32% worse than q5_0 and turbo4 running 17% slower than q4_0 with little memory saving.

Why it matters: HKR-H/K/R all pass, with a first-person benchmark and concrete deltas. Scope is narrow: one RTX 3090, one model, and a Reddit source, so it stays near the featured threshold.

May 19Tuesday

r/LocalLLaMA

Sapient Intelligence releases HRM-Text 1B: 40B tokens, ~$1k pretrain

Sapient Intelligence released HRM-Text 1B, a 1B-parameter model trained from scratch on 16 GPUs for 1.9 days with 40B tokens and a reported ~$1,000 budget; its self-reported chart shows MATH 56.2 and DROP 82.2, while independent evaluation remains pending.

Why it matters: HKR-H/K/R all pass: low-cost pretraining plus a smaller model beating a larger one is clickable, with concrete training and benchmark numbers. Independent eval is unfinished, so this stays at 78, not 85.

AI Chat-Group Daily (群聊日报)

May 18, 2026 Chat Group Daily

The chat group daily says AI21 Labs cut 60% of staff and stopped selling model access, and cites a University of Waterloo paper where GPT-5.4 accuracy dropped from 100% to 23% after false peer-consensus injection; the snippet also mentions Meta layoff talk at 10%, but does not disclose source details or confirmation conditions.

Why it matters: HKR-H/K/R all pass: AI21’s 60% layoff and model-sales stop signal lab contraction, while GPT-5.4 falling from 100% to 23% under false peer consensus is a concrete safety hook. The chat-digest source keeps it at 78.

QbitAI · WeChat

JD and CAS IIE Publish Three Papers Defining Self-Taught RLVR

JD and CAS IIE released three Self-Taught RLVR papers covering RLSD, NPO, and CoPD; RLSD reports that 200 training steps on Qwen3-VL-8B-Instruct exceed GRPO at 400 steps across 8 benchmarks.

Why it matters: HKR-H/K/R pass: self-taught RLVR is a clear hook; RLSD reports 8 benchmarks and a 200-vs-400-step GRPO comparison; it hits reasoning fine-tuning cost. Not a top-lab model launch and replication heat is undisclosed, so it stays low featured.

AI HOT (Curated Pool)

Qwen3.7 Preview lands on Arena; Alibaba rises to fifth in vision ranking

Alibaba says Qwen3.7-Plus-Preview has landed on Arena and that Alibaba now ranks fifth in vision; the post does not disclose benchmark scores, the number of competing models, or a release timeline for the Qwen3.7 series.

Why it matters: HKR-H/K/R pass: Qwen3.7-Plus-Preview appears on Arena with a #5 vision rank. Score stays in the low featured band because the vendor post omits scores, model count, access, and timeline.

r/LocalLLaMA

21 GPUs benchmarked running a small TTS model, with 5GB peak VRAM

A Reddit user rented 21 GPUs on vast.ai to benchmark OmniVoice, a small TTS model with about 5GB peak VRAM, using xRT as the audio generation speed metric and averaging 3 voice-cloning runs with reference audio.

Why it matters: HKR-H/K/R pass: a 21-GPU TTS benchmark with 5GB peak VRAM and 3-run xRT averaging is useful to local-inference builders. Scope is niche, so it sits at the low featured band.

r/LocalLLaMA

llama.cpp MTP support landed: Qwen3.6 27B reaches 2.44× on Strix Halo

llama.cpp merged MTP speculative decoding in PR #22673; Qwen3.6 27B Q8_0 rose from 7.4 to 18.1 tok/s on Strix Halo, while a dual RTX 3090 Q8_0 setup rose from 25.7 to 55.9 tok/s.

Why it matters: HKR-H/K/R all pass: llama.cpp adds MTP speculative decoding with Qwen3.6 27B speedups on Strix Halo and RTX 3090. The scope is local inference, not a broad model release, so 78 fits featured.

May 18Monday

AI HOT (Curated Pool)

The Open Agent Leaderboard

IBM Research published the Open Agent Leaderboard on Hugging Face to evaluate agents across language understanding, tool use, and multi-step reasoning tasks; the post does not disclose dataset size, model scores, or the evaluation date.

Why it matters: HKR-H and HKR-R pass because an open agent leaderboard speaks to agent-eval pain. HKR-K fails: the article lacks scores, dataset size, and evaluation date, so it sits at the featured threshold.

r/LocalLLaMA

I Tested 42 LLMs on Their Willingness to Build the Apocalypse

DystopiaBench tested 42 open and closed models across 36 escalating scenarios and 6 dystopia types, using 3 LLM-as-judge scorers and an average over 3 runs; the post says many models catch obvious dangerous requests but fail when risk is hidden behind dual-use framing and normalization.

Why it matters: HKR-H/K/R all pass: the hook is sharp, the test setup has concrete numbers, and the topic hits safety trust. Reddit single-post sourcing and limited disclosed results keep it in featured, not P1.

r/LocalLLaMA

Qwen 3.6 27B on 24GB VRAM: backend comparisons, quant choice, and settings

The author tested Qwen 3.6 27B on an RTX 3090 24GB and kept ik_llama.cpp with Qwen3.6-27B-MTP-IQ4_KS.gguf; at 156k context with q8_0 KV and MTP, a ~5.9k-token prompt plus 1024-token output reached about 1261 tok/s prefill and 72.9 tok/s decode, while vLLM lacked a clean single-card long-context run.

Why it matters: HKR-H/K/R all pass: this is a first-person local inference benchmark with concrete VRAM, quant, context, and speed numbers. Its reach is narrower than a model release, so it sits at the featured threshold.

r/LocalLLaMA

I trained TIME: short context-triggered thinking on Qwen instead of overthinking

An independent author trained TIME with QLoRA on Qwen3 4B/8B/14B/32B to trigger short mid-response reasoning when context changes; the post says datasets, notebooks, scripts, curriculum, and TIMEBench are public, with 24GB VRAM enough for training up to 14B.

Why it matters: HKR-H/K/R all pass: the post has a clear tuning hook, concrete reproducible details, and strong local-LLM resonance. Reddit single-post sourcing keeps it in the 72-77 featured band, below lab-level releases.

AI HOT (Curated Pool)

Open-source tool exposes security risks and detection gaps in AI API relays

api-relay-audit audits AI API relay risks with verifiable three-state decisions and transparent logs, covering AC-1 tool-call rewriting, AC-2 error-response leakage, and context truncation, while the author has published the methodology, comparison results, quick-reference table, and the open-source tool.

Why it matters: HKR-H/K/R all pass because the tool targets real AI API relay risks with concrete checks. Source is a single X post, and adoption or incident data is not disclosed, so it stays in the low featured band.

r/LocalLLaMA

Benchmarking vLLM vs SGLang vs llama.cpp on a mixed Blackwell/Ada cluster

The author benchmarked long-context prefill on a 7-GPU mixed Blackwell/Ada cluster; on Qwen3.5-397B-A17B with 75k tokens, vLLM reached 9.8s TTFT and 7,683 t/s, while llama.cpp took 57.2s and 1,319 t/s.

Why it matters: Single-source Reddit benchmark, so source authority keeps it near the threshold. HKR-H/K/R pass on the mixed 7-GPU setup, 397B at 75k tokens, and concrete TTFT/throughput numbers.

r/LocalLLaMA

LLMs on Android: Snapdragon 8 Elite MoE Experience

A Reddit user tested MoE LLMs on an Honor Magic 7 Pro with Snapdragon 8 Elite and 24GB RAM; under Q4 quantization, LFM2-24b-a2b reached about 24 tokens/s while Gemma reached about 11 tokens/s, and CPU inference was still faster than NPU or GPU in the reported setup.

Why it matters: HKR-H/K/R all pass: a named Reddit test gives hardware, quantization, and token/s figures. Single-device anecdote and weak source authority keep it at the low featured band.

May 17Sunday

r/LocalLLaMA

85 GPU-hours comparing 5 abliteration methods on Qwen3.6-27B

Abliterlitics compared five Qwen3.6-27B abliteration variants against the base model using 85 GPU-hours of benchmarks, HarmBench, KL divergence, and weight forensics; Huihui had the smallest benchmark deltas, Heretic had the lowest KL divergence, and all five variants reached near-complete safety removal.

Why it matters: HKR-H/K/R all pass: the post gives an 85-GPU-hour comparison across five abliteration methods on Qwen3.6-27B. Niche open-model safety work, not a lab release, so it stays at the featured threshold.

r/LocalLLaMA

DeepSeek V4's 1M Context Window: The Breaking Point

A Reddit user tested DeepSeek V4 on 45k, 180k, and 520k-token codebases and found 150k-250k tokens best for coding work. Past 300k tokens, line-number precision degraded; at 520k, outputs shifted toward architecture summaries and skipped implementation details.

Why it matters: A single Reddit post limits authority, but HKR-H/K/R all pass: it is a numbered first-person test with a concrete long-context failure pattern. The right band is featured, not 78+, because replication and model details are thin.

Synced · WeChat

AI agents may spend 1,000x more tokens without better results: the hidden bill

Researchers used OpenHands to analyze traces from 8 frontier models on 500 swe-bench-verified tasks, finding that agentic coding reached a 154:1 input-output token ratio and that human difficulty labels correlated weakly with token use at Kendall tau 0.32.

Why it matters: All HKR axes pass: strong cost-performance hook, concrete benchmark setup and correlation numbers, and direct resonance with coding-agent economics. It is not a model or platform launch, so it fits the 78–84 quality-recommendation band.

r/LocalLLaMA

Same Models Tested Across Strix Halo, RTX 3090, and RTX 5070

C_Coffie published 55 local inference benchmark runs across Strix Halo, RTX 3090, RTX 5070, five backends, and 0.35B to 35B-A3B models; RTX 5070 beats RTX 3090 on models fitting 12GiB, while RTX 3090 leads in the 14–31B band that exceeds 12GiB but fits 24GiB.

Why it matters: Hits HKR-H/K/R with a named first-person benchmark: 55 runs and concrete GPU crossover points. Source is a single Reddit post, so it stays in the low featured band.

Dwarkesh Patel podcast

Notes on Pretraining Parallelisms and Failed Training Runs

Dwarkesh documents pretraining failure modes and parallelism tradeoffs: expert choice and token dropping can break causality in MoE routing, FP16 collectives can bias repeated additions after values exceed 1024, pretraining FLOPs are given as 6ND, B300 HBM is listed as 288GB, and FSDP communication can reach params × 3 with reduce-scatter.

Why it matters: HKR-H/K/R all pass: Dwarkesh’s notes expose concrete pretraining failure modes and numbers. The systems-training focus is specialized, so it sits in the high-quality band rather than same-day must-write.

AI HOT (Curated Pool)

Latest Open Artifacts #21: Gemma 4, DeepSeek V4, Kimi K2.6, MiMo 2.5, GLM-5.1, and More

Open AI model teams released Gemma 4, DeepSeek V4, Kimi K2.6, MiMo 2.5, GLM-5.1, and other versions this month, and the post says they were tested under CAISI’s V4 evaluation framework, but the RSS snippet does not disclose scores.

Why it matters: HKR-H/K/R all pass: a dense open-model roster, a named CAISI V4 evaluation frame, and clear practitioner relevance for model choice. Missing scores and reproducible detail keep it in the 78–84 band.

May 16Saturday

Synced · WeChat

Why Robots Need World Models: Top Institutions Release Joint Survey

NTU MARS Lab and collaborators released a 43-page survey on robot world models, covering definitions, architectures, applications, benchmarks, and challenges around action-conditioned consistency, inference efficiency, and physical grounding.

Why it matters: HKR-H and HKR-K pass: the hook is robot world models, and the post cites a 43-page survey with benchmarks and action-consistency framing. HKR-R is weak, so this stays at the featured threshold.

r/LocalLLaMA

Qwen3.6-35B-A3B and 9B land on the public Terminal-Bench 2.0 leaderboard

little-coder × Qwen3.6-35B-A3B scored 24.6% ±3.2 on Terminal-Bench 2.0, above Gemini 2.5 Pro on Gemini CLI at 19.6% and Qwen3-Coder-480B on Terminus 2 at 23.9%.

Why it matters: HKR-H/K/R all pass, but this is a Reddit post with leaderboard numbers only; test setup and reproducibility details are not disclosed. Strong code-agent benchmark signal, not a 78+ release story.

Google DeepMind

How WeatherNext helped the US National Hurricane Center forecast Hurricane Melissa's Jamaica landfall

Google DeepMind's AI weather model WeatherNext helped the US National Hurricane Center forecast five days ahead that Hurricane Melissa would hit Jamaica at Category 5 strength, with 80% confidence. Three days out, that rose to near 100%.

Why it matters: The Hurricane Melissa case shows how an AI weather model called a rapid intensification five days ahead, a concrete look at AI in extreme-weather warnings.

r/LocalLLaMA

Orthrus-Qwen3-8B: Up to 7.8× tokens/forward on Qwen3-8B with frozen backbone

Orthrus-Qwen3-8B adds a trainable diffusion attention head to a frozen Qwen3-8B backbone, reaches up to 7.8× tokens per forward and about 6× wall-clock speedup on MATH-500, while training 16% of parameters with under 1B tokens over 24 hours on 8×H200 GPUs.

Why it matters: HKR-H/K/R all pass: 7.8× speed is a strong hook, the post gives parameter, hardware, and benchmark conditions, and inference cost is a live practitioner pain. Single-source Reddit research keeps it in the lower good-quality band.

The Verge · AI

AI radio hosts demonstrate why AI can’t be trusted alone

Andon Labs had Claude, ChatGPT, Gemini, and Grok run separate radio stations with $20 in seed money each; the RSS snippet says all failed, but the post does not disclose the full experimental results.

Why it matters: HKR-H/R are strong because the agent-failure setup is memorable and relevant. HKR-K is present but thin: it gives four models and $20 budgets, while full experimental results are not disclosed.

May 15Friday

r/LocalLLaMA

Evaluated a RAG Chatbot: The Most Expensive Model Was the Worst Performer

The author evaluated a customer-support RAG bot and raised the quality score from 6.62 to 7.88 while cutting per-session cost from $0.002420 to $0.000509, using retrieval logging, LLM-as-judge scoring, chunk deduplication, stricter grounding, and a five-model sweep.

Why it matters: HKR-H/K/R all pass: counterintuitive model ranking, concrete quality and cost deltas, and direct RAG production relevance. Reddit source authority keeps it near the featured floor despite the first-person experiment signal.

The Verge · AI

AI research papers are getting better, and it’s a big problem for scientists

The Verge describes Peter Degen investigating unusual citations to a 2017 paper: it rose from a few dozen citations over several years to being cited every few days, while the RSS snippet does not disclose the full sample size or review findings.

Why it matters: HKR-H/K/R all pass: the paradoxical angle, named investigation, and citation spike give it signal. The post lacks full sample size, so it stays in the lower featured band rather than becoming must-write.

r/LocalLLaMA

Used over a million tokens in three sessions to test Qwen 3.6 35B MTP

A Reddit user tested Qwen3.6-35B-A3B MTP across three million-token-scale sessions, using 300k context and KV Q8_0, and reported about 1.5x the tok/sec of earlier tests.

Why it matters: HKR-H/K/R all pass: the million-token test is clickable, 300k context and KV Q8_0 add testable detail, and local speed maps to cost. Source is one Reddit post, so it stays below the high-importance band.

Latent Space

AI-Native Healthcare: 100M Doctor Visits, 10–20 Hours Saved, Prior Auth in Minutes

Abridge says it is projected to support 80M+ patient-clinician conversations this year across 250 large U.S. health systems, 28+ languages, and 50+ specialties, while its clinical documentation workflow reduces clinicians’ documentation burden by 10–20 hours per week.

Why it matters: HKR-H/K/R all pass: the story has a strong scale hook, concrete adoption metrics, and workflow ROI. Claims are company-interview sourced, not an independent benchmark or major platform release, so it sits in low featured.

AI HOT (Curated Pool)

Granite Embedding Multilingual R2: Open Multilingual Embedding Model with 32K Context

IBM Granite released Granite Embedding Multilingual R2 on Hugging Face under Apache 2.0, with fewer than 100 million parameters, a 32K-token context length, and top same-scale retrieval performance on MTEB according to the post.

Why it matters: HKR-H/K/R pass: the 32K-context, sub-100M multilingual embedding model gives RAG builders a concrete open-source option. Impact is narrower than a frontier-model release, so it sits at the featured threshold.

May 14Thursday

AI HOT (Curated Pool)

MiMo V2.5 Pro Places Third on DesignArena

MiMo V2.5 Pro placed third on the DesignArena overall leaderboard; its Thinking version rose 8 spots over MiMo-V2.5 and matched Claude Sonnet 4.6 performance on frontend coding tasks.

Why it matters: HKR-H/K/R all pass, but the facts come from one official X post with no methodology, access, or pricing. This fits a mid-weight benchmark/product update, not a same-day must-write.

Xinzhiyuan · WeChat

Anthropic Overtakes OpenAI in Enterprise AI Adoption After Three Years

Ramp says Anthropic reached 34.4% enterprise adoption, surpassing OpenAI at 32.3% for the first time; the index is based on credit-card and invoice spending from more than 50,000 companies.

Why it matters: HKR-H/K/R all pass: a reversal hook, concrete 34.4%/32.3% figures, and a strong enterprise-AI rivalry angle. Score stays at 80 because Ramp spending data is not global market share.

Synced · WeChat

ACL 2026: Alibaba DAMO I²B-LPO Improves RLVR Exploration

Alibaba DAMO Academy introduced I²B-LPO, an RLVR post-training framework that branches rollouts at high-entropy nodes and filters them with an information-bottleneck self-reward, reporting up to 5.3% accuracy gains and 7.4% semantic-diversity gains on math benchmarks using Qwen2.5-7B and Qwen3-14B.

Why it matters: HKR-H/K/R all pass: the ACL 2026 DAMO paper has a clear RLVR exploration hook, concrete I²B-LPO mechanics, and benchmark gains. It is still a training-method paper, not a major model or product release, so 78 fits the lower good-quality band.

AI HOT (Curated Pool)

Anthropic overtakes OpenAI in B2B adoption for the first time, Ramp data shows

Ramp AI Index data shows Anthropic reached 34.4% adoption among U.S. enterprise customers, surpassing OpenAI’s 32.3% for the first time, while its business coverage grew fourfold over one year.

Why it matters: HKR-H/K/R all pass: Ramp reports Anthropic at 34.4% enterprise adoption versus OpenAI at 32.3%. This is a strong market signal, but it is one spending dataset rather than a model or product release, so it stays in 78–84.

May 13Wednesday

TechCrunch · AI

Anthropic now has more business customers than OpenAI, according to Ramp data

Ramp’s survey of client expense data shows 34.4% of participating businesses pay for Anthropic services, while 32.3% pay for OpenAI; the snippet does not disclose sample size, customer segments, or spend levels.

Why it matters: HKR-H/K/R all pass: Ramp reports 34.4% paid business usage for Anthropic versus 32.3% for OpenAI. The sample is one payments platform, not official revenue or full-market share, so it sits at the featured threshold.

AI HOT (Curated Pool)

Step Image Edit 2 image model released with leading performance and efficiency

StepFun released the 3.5B-parameter Step Image Edit 2 model, which ranks first in KRIS-Bench overall, factual, and conceptual categories, and is now available on the Stepfun Open Platform.

Why it matters: HKR-H/K/R all pass: the hook is a 3.5B image-editing model topping KRIS-Bench, with concrete launch details. Vendor-only sourcing and no independent test or pricing keep it at the low featured band.

May 12Tuesday

r/LocalLLaMA

I catalogued every way local models break JSON output and built a repair library across 288 model calls

Reddit user kexxty ran 288 structured-output calls through OpenRouter models, including Llama 3, Mistral, Command R, DeepSeek, and Qwen, and found similar JSON failure categories across local and API-only models. The MIT-licensed Python library outputguard validates against JSON Schema, applies 15 ordered repair strategies, includes 2,001 tests, and has no LLM provider dependency.

Why it matters: HKR-H/K/R all pass: 288 tests, the outputguard library, and a 15-step repair chain give practitioners reusable detail. Source is a single Reddit post, so it stays in the 72–77 featured band, not 78+.