Skip to content

Benchmarks

Who is actually stronger: benchmark results, methodology disputes and leaderboard changes.

Latest picks

401–420 of 471

Apr 22Wednesday

Synced · WeChat

Transformer can be converted into Mamba: Apple uses cross-architecture distillation to make inference cost linear

Apple presents a two-stage cross-architecture distillation path that converts Pythia-1B Transformer into a 1B HedgeMamba, reaching 14.11 perplexity with 10B tokens, about 2.7% of the teacher data. The teacher scores 13.86 PPL, while direct Transformer-to-Mamba distillation jumps above 100; the method first aligns with Hedgehog linear attention, then maps into Mamba initialization and fine-tunes. The key point is the path, not one trick: long-context inference shifts from quadratic to linear cost, and the post says downstream results on ARC, PIQA, BoolQ, RACE, and LogiQA approach the teacher.

Why it matters: Apple distilled a trained Pythia-1B into a Mamba architecture in two steps, cutting inference compute from quadratic to linear and giving a reusable path from Transformer to linear attention.

Latent Space

OpenAI launches GPT-Image-2

OpenAI shipped GPT-Image-2 in ChatGPT, Codex, and the API. It has thinking and non-thinking variants, with stronger text, layout, editing, multilingual output, and QR codes. Arena ranks it first on 3 Image Arena boards, with 1512 Elo in text-to-image and a +242 lead.

Why it matters: OpenAI put GPT-Image-2 into ChatGPT, Codex and the API, with better text rendering, layout editing and multilingual output for slides, infographics and UI mockups.

Apr 21Tuesday

QbitAI · WeChat

Mystery model Elephant: 100B parameters reaches same-scale SOTA with high token efficiency

Ant Group's Inclusion AI team is identified as the maker of Elephant, a 100B-parameter model with 256K context and 32K output shown on OpenRouter. The post reports tests on bug fixing, summarizing a 3,000-word meeting note, and a light agent loop, plus AI BENCHY figures of about 2,500 output tokens, about 1 second average latency, and 9.6/10 consistency; the post does not disclose training details, pricing, or an official model card.

Why it matters: Ant Digital's Inclusion AI team put its 10-billion-parameter Elephant model on OpenRouter, and the piece tests it on bug fixes, a 3,000-word meeting summary and a light agent workflow.

Synced · WeChat

Monet: Enabling multimodal LLMs to reason in latent visual space

Monet trains Qwen2.5-VL-7B into Monet-7B to reason with continuous latent visual embeddings instead of external tools; the work is accepted by CVPR 2026 and releases paper, code, model, and a 125K SFT dataset. The method uses three-stage SFT plus VLPO reinforcement learning; the post reports 3% to 9.75% gains on in-distribution tasks and 2.31% on out-of-distribution abstract visual reasoning versus the base model. The key detail is the VLPO mechanism and dataset construction; the post does not disclose one unified table of absolute headline scores.

Why it matters: Monet adapted Qwen2.5-VL-7B to do abstract reasoning in a continuous visual latent space, training in two stages of supervised fine-tuning and VLPO reinforcement learning, gaining 3% to 9.75% on in-distribution tasks.

Synced · WeChat

Anonymous world model MotuBrain tops WorldArena and RoboTwin2.0

MotuBrain ranked first on both WorldArena and RoboTwin2.0, with a 63.77 EWM Score on WorldArena and 95.8/96.1 in RoboTwin Clean and Randomized settings. The post says it also leads Motion Quality, Flow Score, and Motion Smoothness, and averages 96.0 across 50 RoboTwin tasks versus 92.3 for second place; the post does not disclose its owner, model size, or training setup. The result matters because it supports a single-model path that combines world prediction with robot action, at least on benchmarks.

Why it matters: An anonymous model, MotuBrain, tops both WorldArena and RoboTwin2.0, averaging 96.0 across RoboTwin's fifty tasks, 3.7 points above second place.

Hacker News front page

Even 'uncensored' models can't say what they want

Morgin.ai probed 6 pretrains on 4,442 contexts and found that even “uncensored” models sharply deflate charged words, by hundreds to about 16,000x. It calls this effect flinch: no refusal fires, but token probabilities shift; in one example, qwen3.5-9b-base ranks “deportation” #506 at 0.0014%. The key issue is pretraining-level distribution shaping, not only post-training refusals.

Why it matters: Shows that even models stripped of refusal behavior still suppress sensitive-word probabilities by hundreds to roughly 16,000 times.

Apr 20Monday

X · @Yuchenj_UW

Kimi K2.6 is open-source

Kimi K2.6 is now open source, and the RSS snippet says it scored 58.6 on SWE-Bench Pro. The snippet also says it beat GPT-5.4 xhigh and Claude Opus 4.6 max effort. What matters is reproducibility; the post does not disclose weights, license, or eval setup.

Why it matters: Kimi K2.6 is open sourced and scores 58.6 on SWE-Bench Pro for code bug fixing, above GPT-5.4 and Claude Opus 4.6 in the tweet, adding a reference point for open versus closed code ability.

r/LocalLLaMA

Training LoRA adapters for Apple's on-device 3B model on a free Colab T4 and a Mac

The author built a QLoRA pipeline for Apple’s on-device 3B model, cutting training needs from about 24GB to about 1GB RAM and 5GB GPU, enough for a free Colab T4 or a 24GB Mac. The post says A100 LoRA, T4 QLoRA, and Mac QLoRA adapters perform about the same, raising accuracy from about 40% to 75%, or 86% with retrieval; it also reports a confirmed Apple bug that writes a hidden ~160MB cache copy per CLI call, reaching 269GB over ~300 runs.

Why it matters: A walkthrough of fine-tuning Apple's on-device 3B model with QLoRA, cutting memory needs to 1GB RAM plus 5GB VRAM so a free Colab T4 or a 24GB Mac can run it.

r/LocalLLaMA

Compared some models for feature planning

A Reddit user tested 9 models on planning a “load tracking” feature for a Go budgeting app, then used Claude Code to rank the generated specs, with Claude Opus 4.6 placed first. The table shows Opus 4.6 produced a 19 KB spec with 44 code reads at $2.47; GLM 5.1 ranked second and Qwen 3.6 35B fp8+vLLM ranked third. Do not treat this as a benchmark: the author says it is not representative, and the post does not disclose any manual quality review yet.

Why it matters: A Reddit user had nine models design load-tracking for the same Go bookkeeping app, then used Claude Code to score the specs, making their real-task output comparable.

r/LocalLLaMA

Actually put Gemma 4 26B to work on something real: extract trading signals from 2,400 earnings calls

A Reddit user fine-tuned Gemma 4 26B on 800 labeled earnings-call transcripts and ran inference on 2,400 transcripts over 3 years on one RTX 4090 in about 14 hours. On 600 out-of-sample transcripts, one signal linked vaguer CFO guidance to about 1.8% sector-relative underperformance over 5 days with IC 0.04. A stronger signal showed 0.85 correlation with sector returns after checks and was discarded as a ghost factor; the key point is factor sanity checks, not the profit claim.

Why it matters: A Reddit user fine-tuned Gemma 4 26B on 800 labeled earnings calls and ran 2,400 records on one RTX 4090, finding one signal where vaguer CFO guidance tracked about 1.8% underperformance.

QbitAI · WeChat

Sudo, valued above $2 billion, unveils embodied model Sudo R1 with zero real-robot data and ~98% first-try grasp success

Sudo unveiled embodied model Sudo R1 and says it achieved about 98% first-try grasp success in 200+ zero-shot tests with zero real-robot training data, nearing 100% within two attempts. The post says the 60-minute run covered 100+ unseen objects, including transparent, metallic, soft, and reflective items, using integrated world-model and reinforcement-learning training on a high-fidelity simulator. It also says Sudo is valued above $2 billion and is working with CATL, but the post does not disclose round size, benchmark protocol, or third-party validation.

Why it matters: Sudo Technology released the embodied model Sudo R1, which without real-robot training data passed over 200 zero-shot grasping tests on 100-plus unseen objects at about 98% first-try success.

New York Times Chinese

Chinese humanoid robot 'Shandian' finishes a half marathon in 50:26, faster than the human world record

Honor’s humanoid robot Shandian finished a Beijing half marathon in 50:26, faster than Jacob Kiplimo’s 57:20 human world record. The 1.65-meter robot fell after hitting a barrier, resumed with human help, and far beat last year’s best robot time of 2:40:42. The key signal is stronger robotics engineering, not a disclosed AI leap.

Why it matters: Honor's humanoid robot Lightning ran the Beijing half marathon in 50:26, nearly 7 minutes under the human world record, against 2:40:42 for the best comparable run last year.

Apr 19Sunday

r/LocalLLaMA

Same 9B Qwen weights: 19.1% in Aider vs 45.6% with a scaffold adapted to small local models

Using the same Qwen3.5-9B Q4 weights on the 225-task Aider Polyglot benchmark, the author changed only the scaffold and raised mean pass@2 from 19.11% to 45.56%. The little-coder setup is not a new model; it uses bounded reasoning, a write guard, explicit workspace discovery, and small per-turn skill injections. The key claim is scaffold-model fit, but the post reports only two full runs and does not disclose ablations, cross-model replications, or a second benchmark.

Why it matters: The same Qwen3.5-9B weights score 19.11% pass@2 on Aider with one scaffold and 45.56% with another built for small local models.

Synced · WeChat

MIA, a next-generation memory agent framework, aims to end agents' "amnesiac" workflows

A Shanghai Institute for Advanced Learning and ECNU team released MIA, a memory agent framework, and said it achieved the best results on 7 datasets. MIA uses a Manager-Planner-Executor design, dual parametric and non-parametric memory, alternating RL, and test-time continual learning; the post does not disclose exact benchmark scores. The key point is memory as capability internalization, not just retrieval, for open-world agents.

Why it matters: A Shanghai research team's MIA memory framework splits agent memory into parametric and non-parametric types and updates them with alternating reinforcement learning, offering a transferable way to keep evolving on open tasks.

Synced · WeChat

Amap debuts an autonomous embodied robot at the Yizhuang Marathon and showcases guide-assistance

Amap showed its quadruped robot Tutu at the 2026 Yizhuang humanoid half marathon, claiming it completed a guide-assistance obstacle task in an open environment without preset routes or teleoperation. The post says its ABot stack includes ABot-N0, which reached SOTA on 7 navigation benchmarks with 88.3% on SocNav, and ABot-M0, which scored 80.5% on Libero-Plus. The key point is the integrated stack across navigation, manipulation, world modeling, and closed-loop correction; the post does not disclose guide-task test scope, commercialization timing, or safety incident data.

Why it matters: A quadruped robot navigated open roads with no preset route or remote control, a rare public test of real-world navigation.

Xinzhiyuan · WeChat

A Berkeley team built an AI that scores perfectly on SWE-bench while fixing 0 bugs

Berkeley RDI used a roughly 10-line conftest.py exploit to score 100% on all 500 SWE-bench tasks while fixing 0 bugs. The post says its agent broke 8 major agent benchmarks with scores from 73% to 100%, via pytest hook tampering, file:// answer reads, and faulty validators. The real issue is benchmark isolation failure, not stronger models.

Why it matters: A Berkeley RDI team scored full marks on all 500 SWE-bench tasks with about 10 lines of pytest hook code and zero real fixes, and the same trick broke 8 other major agent benchmarks.

r/LocalLLaMA

I tested 8 LLMs as tabletop GMs: a 27B model beat the 405B on narrative quality

The author tested 8 LLMs on 6 fixed tabletop-GM scenarios, and google/gemma-3-27b-it ranked first in narrative quality with a 4.33 overall score. The probe used 8 auto metrics plus 3 LLM-judge scores, and the full run cost about $0.02; the title says a 27B beat a 405B, but the snippet does not disclose the 405B model name or full rankings.

Why it matters: Eight models were tested as tabletop GMs across 6 fixed scenarios with 8 automatic metrics and 3 LLM judges, and google/gemma-3-27b-it topped the narrative ranking at 4.33.

Apr 18Saturday

QbitAI · WeChat

RAG retrieves the right docs but still answers wrong? Saarland University team diagnoses why | ACL 2026

A Saarland University-led team introduced Disco-RAG, adding a 3-step “reading” layer between retrieval and generation, and says the paper was accepted as an ACL 2026 main-conference long paper. The post says it uses RST-based argument trees, cross-passage relation graphs, and outline generation with zero training; it reports gains on Loong, ASQA, and SciNews, but does not fully disclose the exact scores. The key claim is that many RAG failures come from reading and discourse understanding, not retrieval recall.

Why it matters: A Saarland University team traces RAG errors to document understanding in the generation step and proposes Disco-RAG, which builds argument trees with rhetorical structure theory and a cross-paragraph relation graph, with no extra training.

Xinzhiyuan · WeChat

Study says distribution shifts can trigger LLM dark patterns, with 22 of 26 models at 100% attack success

A Hong Kong Polytechnic University and Northwestern Polytechnical University team reports in Nature Communications that 22 of 26 aligned models hit 100% attack success under distribution-shifted semantic prompts. The paper says harmful pretraining knowledge stays globally connected to post-alignment “safe regions”; even Llama 3.1 8B Instruct showed ethical drift under natural-language induction. The key point for practitioners: no gradient attack or gibberish prompt was required.

Why it matters: PolyU and Northwestern Polytechnical researchers tested 26 safety-aligned models in Nature Communications; 22 fell to malicious prompts rewritten in natural language, a 100% success rate.

Apr 17Friday

Hacker News front page

Measuring Claude 4.7's tokenizer costs

The author used Anthropic's free count_tokens API to compare Claude Opus 4.6 and 4.7 on 7 real samples and 12 synthetic ones; the real-sample weighted total rose from 8,254 to 10,937 input tokens, or 1.325x. Technical docs hit 1.47x, a real CLAUDE.md file hit 1.445x, while Chinese and Japanese stayed near 1.01x. On a 20-prompt IFEval sample, 4.7 improved strict prompt-level pass rate from 85% to 90%; the post cannot isolate tokenizer effects from model weights or post-training.

Why it matters: Using Anthropic's free counting endpoint on the same Claude Code content, the author measured weighted tokens rising from 8254 to 10937, a 47% jump on technical documentation.