Skip to content

Benchmarks

Who is actually stronger: benchmark results, methodology disputes and leaderboard changes.

Latest picks

1–20 of 471

Yesterday · Nov 16Monday

Computing Life · Share · Yage

DeepSeek Harness plugin ecosystem review: what it can do that Claude Code and Cursor can't

DeepSeek Harness plugins can change the model connection, conversation history, remote file operations and interface layout. The article compares these capabilities with Claude Code, Cursor and Codex.

Why it matters: DeepSeek Harness lets plugins replace the model connection, history and UI components, giving more room to customize than coding tools that keep those entry points closed.

Nov 10Tuesday

Hacker News front page

Fable 5.2 builds a nephew's dream game in one run, then scales to 18 characters

lexandstuff tested Claude Code with Fable 5.2, starting from a nephew's design sketches and existing game code. One run produced a playable 12-level side-scrolling action game. The model rewrote most of the code, generated about 175 images and 140 audio clips, and wrote a bot that played all twelve levels in a headless browser and checked screenshots.

Why it matters: The model wrote a bot that played all twelve levels in a headless browser and checked screenshots, using automated play and visual checks as basic tests for a generated game.

Computing Life · Share · Yage

Agent harness techniques tested: which framework designs raise scores and which just raise cost

Controlled experiments from Surge AI and FireFlash show coding agents can improve through deeper single-model reasoning, test and goal-check gates, and a reworked terminal pipeline. In Surge AI's run, Compound Engineering lifted cost to 8x baseline while the score moved only from 31.1 to 32.9. Claude Opus 5.5 with higher thinking settings reached 42.4 at 3.5x baseline cost.

Why it matters: The two teams break down the cost and payoff of agent harness techniques, comparing multi-agent collaboration, deeper single-model thinking and terminal pipeline changes.

Nov 8Sunday

Computing Life · Share · Yage

Why did Kimi K3.1 Pro recite hidden grading rules in an isolated exam room?

An R40 audit found Kimi K3.1 Pro restated hidden grading rules in 23.3% of Terminal-Bench 4.0 attempts without running probe commands. In a counterfactual test that changed the on-site requirements, its pass rate on four tasks fell from 90% to 15%. The audit says the behavior fits context distillation on hidden rules, but can't tell whether the material came from internal builds or outside purchases, or call it deliberate cheating.

Why it matters: R40 edits on-site code comments and inverts grading rules to test whether Kimi K3.1 Pro follows new requirements or reuses old ones, a method for separating live problem-solving from memorized test banks.

Oct 30Friday

AI HOT · Tips & opinions

OpenRouter tests Claude Opus, Sonnet and Haiku 5.5 and picks a default

OpenRouter ran the three Claude models on the same coding tasks and recommends Sonnet 5.5 as the default. Haiku 5.5 suits high-volume requests and sub-agents, while Opus 5.5 is kept for long, ambiguous tasks where errors are costly.

Why it matters: All three models passed the tests, yet cost per task still differed by 19 to 30 times.

Oct 29Thursday

Hacker News front page

Google's Gemini 4 Carbon tops Artificial Analysis Intelligence Index v5

Google's Gemini 4 Carbon scored 59 on the Artificial Analysis Intelligence Index v5, taking first place. It ranked second on the coding agent index. Artificial Analysis compared it with similar models and found its cost per task lower than Claude Opus 5.5.

Why it matters: The benchmark puts Gemini 4 Carbon first on general intelligence and cheaper per task than Claude Opus 5.5.

Oct 28Wednesday

AI HOT · Tips & opinions

Pydantic AI benchmark: code mode cuts long-task cost 58% and time 51%

Pydantic AI tested 8 models across 5,276 scored runs in an internal benchmark. Code mode cut long-task cost by 58%, time by 51%, and raised success rate by 16.0 percentage points.

Why it matters: The same tool set was run two ways, and code mode saves on long tasks while adding overhead on short ones.

Oct 27Tuesday

AI HOT · Models

Artificial Analysis tests GPT-6.1 Terra: 6-point intelligence gain at under a third of the cost per task

Artificial Analysis found GPT-6.1 Terra scores 6 points higher than GPT-5.6 Terra on its intelligence index at the highest reasoning effort, while cost per task falls from $1.40 to $0.40. At low and medium reasoning effort the model sits on the intelligence-cost Pareto frontier.

Why it matters: A 6-point intelligence gain with cost per task down to $0.40 changes the price of running hard reasoning work.

Oct 17Saturday

AI HOT · Industry

404 Media tests how ChatGPT ads target users by prompt, phone number and email

404 Media found in testing that OpenAI lets advertisers target specific groups of ChatGPT users by email, phone number or a phone's unique identifier. Targeting also uses the prompts users type, and some users may not know that personal data uploaded by advertisers is used to match ads. The original article says ChatGPT has 1 billion monthly active users, and notes that personal data targeting is common in advertising while prompt-based targeting is newer.

Why it matters: 404 Media's tests show ChatGPT ads can be targeted by prompt and personal identifiers, with both user input and advertiser-uploaded personal data feeding ad matching.

Oct 16Friday

The Decoder

Google Gemini 4 Argon benchmark results land closer to the frontier than recent Google models

Google DeepMind's Gemini 4 Argon scores 165 on Epoch AI's composite capability index ECI, close to GPT-6 Astra at 166 and Opus 5.5 at 167.

Why it matters: Argon's scores differ sharply between the two test setups, and vendor API features such as retained reasoning and context compression were part of the benchmark comparison.

Oct 14Wednesday

AI HOT · Tips & opinions

Google runs Gemini 4 Argon unattended to optimize ~205,000 lines of ADK Python

Google ran Gemini 4 Argon unattended for 11 hours over its ADK codebase, cutting memory session preparation time for long sessions by about 40% and package import time by 21%. Maintenance rules were supplied through .agents/skills and /goal mode in Google Antigravity. The model proposed 6 optimizations; after human review, session copying and lazy imports were merged.

Why it matters: Google wrote repo maintenance rules into skill files and let Gemini 4 Argon optimize code overnight, with maintainers filtering out fragile fixes.

Oct 7Wednesday

Hacker News front page

Telegraph Test: LLMs compress text into telegraph style, and reading it back stays accurate

Telegraph Test measured LLM telegraph-style compression across 50 texts and about 1,300 questions. Accuracy when another model reads the compressed text ran 0.99 to 1.10 times the plaintext baseline. With lowercase instructions, GLM-5.3-Flash cut tokens by 48.4% but answered in telegraph style at an accuracy ratio of 0.81. gpt-5-mini, which cannot turn off reasoning, cost about twice as much to write the compressed record as plaintext.

Why it matters: It tests record compression and answer compression separately, giving agent memory a method that compresses once content is settled and keeps shorthand for machine readers.

The Decoder

Common Sense rates ChatGPT teen version unacceptable risk; suicide chats miss parent alerts

Common Sense Media's AI safety institute for teens rated the ChatGPT teen version an "unacceptable risk" to minors after more than 4,000 test prompts. New test accounts linked to over 12 parent accounts never triggered notifications in explicit conversations about suicide, self-harm and eating disorders, and more than a quarter of situations that should have offered crisis referrals did not point users to professional help.

Why it matters: A teen AI safety group tested ChatGPT's parent alerts, and crisis chats failing to trigger notifications clashes with its safety promises to parents.

Oct 6Tuesday

Hacker News front page

Anthropic vs OpenAI subscription review: daily-driver API-equivalent value runs about 5x higher

SemiAnalysis measured subscription quota consumption by token type and found that, on agent workloads, Anthropic's Opus 5.5 plan delivers roughly 5x the API-equivalent value of OpenAI's GPT 6.1 Sol. The comparison breaks plan value down by subscription tier, model and workload combination.

Why it matters: SemiAnalysis separates quota consumption by token type, pushing subscription value comparisons down to specific plan, model and workload combinations.

Oct 3Saturday

The Verge · AI

OpenAI Dots hands-on: books dinner, but works better on job tasks

In the author's hands-on test, OpenAI Dots handled a website revamp and video editing, but booking and shopping tasks kept hitting security checks. Talking through design changes by voice and granting Dots access to the computer let it deploy site updates, merge videos and resize them for social media. Those jobs required handing over a lot of permissions.

Why it matters: A hands-on test shows OpenAI Dots stalling on everyday services at security checks while running smoother on authorized site and video work.

Oct 1Thursday

The Decoder

Google releases Gemini 4 Argon: benchmarks match OpenAI and Anthropic, no clear lead

Google released Gemini 4 Argon, its new frontier model. On several benchmarks it matches OpenAI's GPT-6 Astra and Anthropic's Claude Fable 5.1, but it does not take a clear lead.

Why it matters: Third-party benchmarks put Gemini 4 Argon level with GPT-6 Astra and Claude Fable 5.1, though it burns about twice the output tokens per task.

AI HOT · Models

Artificial Analysis rates Gemini 4 Argon: Google back in the top three on intelligence

Artificial Analysis' evaluation shows Google DeepMind's Gemini 4 Argon scoring 53 on the Artificial Analysis Intelligence Index in high-reasoning mode, level with GPT-6 Astra (max, 53) and one point ahead of GPT-6.1 Sol (max, 52).

Why it matters: Artificial Analysis scored Gemini 4 Argon's high-reasoning mode at 53 on its intelligence index, tied with GPT-6 Astra, with the lowest hallucination rate among comparable models.

Sep 30Wednesday

AI HOT · Models

GPT-6.1 Sol replaces GPT-6 Sol 7 days after launch, 1 point behind GPT-6 Astra on intelligence index

GPT-6.1 Sol replaced GPT-6 Sol 7 days after launch. Its intelligence index is 1 point below GPT-6 Astra, and pricing stays at $2 per million input tokens and $10 per million output tokens. The cached-read discount rises from 90% to 95%.

Why it matters: Side-by-side intelligence index, cost and token-efficiency figures let readers judge the trade-off GPT-6.1 Sol makes between price and capability.

Sep 29Tuesday

Latent Space

Opus 5.5 wins community praise for explainer video production

After Opus 5.5 launched this week, the community shared several cases of writing code in Claude Code to produce explainer videos and animations. One reproduction case says roughly 1.5–2 hours and about $4 produced a 30–60s explainer video with script, assets, voiceover, music and sound effects, rendering JavaScript canvas animation to MP4.

Why it matters: Community users chained script, assets, voiceover and code animation through Claude Code and Opus 5.5, giving a reusable pipeline for automated explainer videos.

Hacker News front page

Pac-Bench: One-shot Pac-Man benchmark, Claude Opus 5.5 scores 99/100

Jon Clegg built a Pac-Man benchmark: one prompt, one HTML page, scored automatically by Opus 5.5. Claude Opus 5.5 hit 99/100 via Claude Code at $1.99, generating a 10.8 KB page in 9 minutes with near-arcade audio. Claude Fable 5.1 scored 96 but cost $5.87. Grok 4.7 and GPT-5.6-sol scored 94 and 90; the latter cost just $0.72 in under 5 minutes. Scoring covers controls, ghost behavior, stuck detection, maze layout, and sound. The post doesn't explain why some models ran Phase 2 or how much the harness affects scores. Worth noting: this measures model-plus-toolchain combos, not bare model capability.

Why it matters: One prompt sent to 30 models and harnesses to build Pac-Man in one shot, scored item by item on controls, ghosts and maze, gives a side-by-side read on single-page game generation.