Skip to content

Benchmarks

Who is actually stronger: benchmark results, methodology disputes and leaderboard changes.

Latest picks

381–400 of 471

Apr 28Tuesday

Hacker News front page

Xiaomi releases MiMo-v2.5 weights with strong coding and agent benchmarks

Xiaomi released MiMo-v2.5 family weights; the title cites strong coding and agent benchmarks. The RSS body only lists URLs, 13 HN points and 2 comments; the post does not disclose size, license, or scores.

Why it matters: Xiaomi open-sourced MiMo-V2.5-Pro weights, with coding scores close to Claude Opus 4.6, and the summary also mentions passing 233 hidden tests in 4.3 hours on a Peking University compiler assignment.

The Verge · AI

Attack of the Killer Script Kiddies

The Verge discusses Claude Mythos and AI bug finding, citing DARPA AIxCC scans over 54 million code lines. Teams found most seeded flaws plus over a dozen unseeded bugs; the RSS snippet does not disclose Mythos benchmarks, pricing, or access terms.

Why it matters: In DARPA's AI cyber challenge, Anthropic's Claude security test build Mythos scanned 54 million lines of code and found a dozen real bugs beyond the planted ones.

r/LocalLLaMA

Local coding models have reached a threshold for real work

Antigma tested 27B–32B open-weight models; Qwen 3.6-27B scored 38.2% on Terminal-Bench 2.0. The run used 89 tasks and the default per-task timeout, while verified SOTA is about 80%. The key claim is deployment lag: offline coding is about 6–8 months behind hosted frontier models.

Why it matters: Antigma ran several 27B to 32B open models through one agent flow on 89 Terminal-Bench 2.0 tasks, and the best, Qwen 3.6-27B at 38.2%, trails cloud frontier models near 80% by six to eight months.

Hacker News front page

Talkie: a 13B vintage language model from 1930

Nick Levine, David Duvenaud, and Alec Radford released Talkie, a 13B vintage LM trained only on pre-1931 text. The post shows a 24/7 Claude Sonnet 4.6 chat feed and tests surprise on nearly 5,000 NYT historical event descriptions. The key angle is temporal cutoff training as a probe of prediction, bias, and knowledge limits.

Why it matters: Three researchers trained Talkie, a 13B model, on old books published before 1931, then had Claude talk to it for 24 hours straight to probe its knowledge boundaries and values.

Apr 27Monday

Hacker News front page

Show HN: OSS Agent Dirac topped TerminalBench on Gemini-3-flash-preview

Dirac-run released Dirac and says it topped TerminalBench using Gemini-3-flash-preview. The repo claims 50-80% lower API costs via Hash Anchored edits, parallel operations, and AST manipulation; the post does not disclose full scores.

Why it matters: Dirac topped TerminalBench on Gemini-3-flash-preview, claiming hash-anchor localization, parallel operations and direct syntax-tree edits cut API costs to 20 to 50 percent of peers.

Xinzhiyuan · WeChat

First Spatio-Temporal Time-Series Reasoning Framework for LLMs | ACL'26

Emory University, Microsoft, and partners introduced STReasoner for spatio-temporal time-series reasoning, with ST-Bench covering four task types. It uses Network SDE plus Multi-Agent data generation, then Align, SFT+CoT, and S-GRPO training. The article claims inference cost is 0.004× closed models, with code on GitHub.

Why it matters: Emory and Microsoft researchers propose the STReasoner framework with a matching ST-Bench covering forecasting, anomaly detection, causal analysis and QA, showing a transferable pipeline from data generation to reinforcement learning.

QbitAI · WeChat

Stanford-led LLM-as-a-Verifier claims SOTA on Terminal-Bench 2.0

Stanford, Berkeley and Nvidia introduced LLM-as-a-Verifier, claiming SOTA on Terminal-Bench 2.0 and SWE-Bench Verified. It selects trajectories via score-token granularity, repeated checks and criteria decomposition; ForgeCode accuracy reached 86.4%.

Why it matters: Stanford, Berkeley and NVIDIA proposed LLM-as-a-Verifier, which has a model check another model's solution path step by step, and it topped both Terminal-Bench 2.0 and SWE-Bench Verified.

Apr 26Sunday

Hacker News front page

Why SWE-bench Verified No Longer Measures Frontier Coding Capabilities

OpenAI stopped reporting SWE-bench Verified scores and recommends SWE-bench Pro instead. It audited 138 tasks that o3 failed inconsistently across 64 runs and found 59.4% had test or prompt flaws. The key issue is contamination: tested frontier models reproduced some gold patches or task details.

Why it matters: OpenAI audited 138 problems o3 repeatedly failed and found at least 59.4% had overly rigid test cases or vague descriptions, which explains why it retired the benchmark.

Apr 25Saturday

Hacker News front page

TIPSv2: Advancing Vision-Language Pretraining with Enhanced Patch-Text Alignment

Google DeepMind released TIPSv2 with 3 pretraining changes for CVPR 2026. iBOT++ applies self-distillation to masked and visible patches, adding 14.1 mIoU on ADE150; Head-only EMA cuts training parameters by 42%. The key signal is visible-token supervision, not a larger teacher model.

Why it matters: Google DeepMind's TIPSv2 adds self-distillation losses on both masked and visible patches, gaining 14.1 mIoU in zero-shot segmentation and cutting 42% of training parameters with Head-only EMA.

Apr 24Friday

Hacker News front page

Researchers Simulated a Delusional User to Test Chatbot Safety

Researchers at CUNY and King’s College London used one simulated user showing psychosis-spectrum delusions to test 5 LLMs across extended chats. The set included GPT-4o, GPT-5.2, Grok 4.1 Fast, Gemini 3 Pro, and Claude Opus 4.5; the article says Grok and Gemini reinforced delusions more often, while GPT-5.2 and Claude became more cautious over longer conversations. The key point is that multi-turn safety differences were measurable, not just single-prompt behavior.

Why it matters: Researchers at CUNY and King's College London ran long chats between a simulated delusional user and five major models, finding Grok and Gemini more likely to go along with and reinforce unreal beliefs.

TechCrunch · AI

DeepSeek previews new AI model that ‘closes the gap’ with frontier models

DeepSeek previewed two new models and said architectural changes make them more efficient and higher-performing than DeepSeek V3.2, while nearly closing the gap with leading models on reasoning benchmarks. The RSS snippet discloses only that there are two models and that they outperform V3.2; model names, parameter counts, benchmark scores, test sets, and release timing are not disclosed. The key question is reproducible evals, because “closes the gap” comes without numbers.

Why it matters: DeepSeek posted two V4 preview models on Hugging Face, claiming a leaner architecture than V3.2 and higher scores, with parameter counts and benchmark numbers not disclosed.

MIT Technology Review · AI

Health-care AI is here. We don’t know if it actually helps patients.

Jenna Wiens and Anna Goldenberg argue in Nature Medicine that health-care AI is widely deployed, but patient-outcome evidence is thin. A 2025 study found about 65% of US hospitals used AI predictive tools, and only two-thirds assessed accuracy. The key issue is post-deployment impact on clinical decisions.

Why it matters: Two researchers from the University of Michigan and the University of Toronto argue in Nature Medicine that hospitals are deploying AI tools fast without evaluating whether they improve patient outcomes.

Xinzhiyuan · WeChat

Google's Vision Banana aims to unify vision tasks with a single pixel-generation interface

Google DeepMind and collaborators including Kaiming He introduced Vision Banana, claiming one pixel-generation interface can cover detection, segmentation, generation, and editing. The RSS snippet gives two head-to-head numbers versus Nano Banana Pro: 53.5% human win rate on GenAI-Bench and 47.8% on ImgEdit; it says only a small amount of reversible-format task data was mixed in, while data scale and full benchmark tables are not disclosed in the post.

Why it matters: Google DeepMind and Kaiming He's team propose Vision Banana, using pixel generation as a single output to unify detection, segmentation, generation and editing instead of attaching a separate head per task.

X · @Yuchenj_UW

Finally, DeepSeek V4 is here!

DeepSeek announced DeepSeek V4 and says DeepSeek-V4-Pro uses an MIT license with 1.6T parameters and 49B active parameters. The snippet also claims DeepSeek-V4-Pro Max is close to Opus-4.6 Max and GPT-5.4 xHigh across benchmarks; the post does not disclose benchmark names, scores, release timing, or model weights. The key signal is the MIT license and 49B active scale, not the headline comparison.

Why it matters: DeepSeek open-sourced V4 Pro under MIT with 1.6 trillion total parameters and only 49 billion active at inference, setting a new reference for open-model deployment cost.

X · @op7418

DeepSeek V4 detailed official announcement is out

DeepSeek says V4 Pro has 1.6T total parameters with 49B active, while Flash has 284B total and 13B active; both were pretrained on 32T tokens. Web and app Expert mode map to Pro, and Fast mode maps to Flash. The post also says several benchmarks are on par with Opus 4.6, with stronger agent ability and world knowledge, plus a new attention mechanism that reduces compute and memory demand.

Why it matters: DeepSeek disclosed parameter counts and 32 trillion pretraining tokens for both V4 versions, and mapped the web and app Expert and Fast modes to Pro and Flash.

Hacker News front page

GPT-5.5: Mythos-Like Hacking, Open to All

XBOW says GPT-5.5 cut miss rate to 10% on its real-vulnerability benchmark, versus 40% for GPT-5 and 18% for Opus 4.6. It scored 97.5% on visual acuity and used about half the login iterations of the next-best model. The key point is black-box testing: GPT-5.5 without source beat GPT-5 with source.

Why it matters: Security firm XBOW tested GPT-5.5 against a real vulnerability set, and its black-box miss rate fell to 10%, below GPT-5 white-box at 40% and Opus 4.6 at 18%.

Apr 23Thursday

New York Times Chinese

AI so powerful it is called worse than a nuclear bomb: Mythos triggers cyber alarms

Anthropic said it is tightly restricting access to Mythos and named 11 US partners helping patch software flaws the model found. The company said it shared the model with 40+ critical-infrastructure groups, and only the UK has access outside the US; similar cyber-capable models may be released more broadly within 18 months. The real signal is geopolitical control over frontier cyber capability, not a normal model launch.

Why it matters: Anthropic limits its Mythos vulnerability-finding model to 11 US tech firms and 40-plus critical-infrastructure bodies, with the UK the only overseas access so far.

OpenAI News

GPT-5.5 Bio Bug Bounty

OpenAI launched the GPT-5.5 Bio Bug Bounty, offering up to $25,000 for universal jailbreaks that trigger bio safety risks. The RSS snippet confirms a red-teaming challenge; the post does not disclose eligibility, eval protocol, scope, or deadline.

Why it matters: OpenAI is paying up to $25,000 for a single prompt that clears five biosecurity questions at once.

Hacker News front page

Coding Models Are Doing Too Much

The author programmatically corrupts 400 BigCodeBench problems with single-point bugs to test whether coding models over-edit code during fixes. The post defines the minimal fix as exactly reversing the corruption and measures excess changes with token-level Python Levenshtein distance. The provided body does not disclose final results, model rankings, or training gains.

Why it matters: The author plants single-point bugs in 400 BigCodeBench tasks and uses token-level edit distance to measure how much models over-rewrite when fixing bugs, then tries to train them to make minimal edits.

Apr 22Wednesday

Hacker News front page

Show HN submissions tripled and are now mostly vibe-coded

Adrian Krebs scored 500 recent Show HN landing pages and says submissions have tripled, with 67% of pages triggering at least 2 AI design patterns. The method used Playwright plus an in-page script to check DOM and computed styles across 15 deterministic CSS/DOM signals; manual QA found about 5% to 10% false positives. The real signal is not model quality, but fast homogenization from AI default frontend templates.

Why it matters: Adrian Krebs scripted a scan of 500 Show HN landing pages and found 67% hit at least two AI design tells, giving a measured read on how alike AI-generated front ends have become.