Skip to content

Benchmarks

Who is actually stronger: benchmark results, methodology disputes and leaderboard changes.

Latest picks

21–40 of 471

Sep 29Tuesday

AI HOT (Curated Pool)

GPT-6 Luna (Max) ranks #23 on Agent Arena at $0.05 per task

Arena evaluated GPT-6 Luna (Max) on 8K real agent conversations. It ranked #23, up 6 spots from GPT-5.6 Luna (xHigh), with a net gain of +1.6%. Cost is $0.05 per task. The post doesn't disclose latency or task-type breakdown.

Why it matters: Arena's 8,000 real agent sessions put GPT-6 Luna (Max) at rank 23, up six places from GPT-5.6 Luna (xHigh), at $0.05 per task.

AI HOT (Curated Pool)

Claude Sonnet 5.5 enters Arena's Agent Arena and Battle Mode

Anthropic's Claude Sonnet 5.5 is now available for voting in Arena's Agent Arena. The leaderboard evaluates models on millions of real-world long-horizon agent tasks where models can use web search, filesystem, and terminal tools. Rankings use a causal tracking method to measure how much a model outperforms the average.

Why it matters: Claude Sonnet 5.5 entered Arena's Agent Arena with voting open. The leaderboard draws on millions of real long-horizon agent tasks and uses causal tracing to measure a model's outcomes against an average model.

Hacker News front page

Anthropic launches Claude Sonnet 5.5: 30%+ faster, up to 30% cheaper than Sonnet 5

Claude Sonnet 5.5 is the second model in the 5.5 family, aimed at everyday coding, bug fixes, and polished docs. It scores 70.6% on Terminal-Bench 4.0 vs. Sonnet 5's 10.3%. Pricing stays at $2/$10 per million input/output tokens, but it uses fewer tokens per task, cutting per-task cost by up to 30%. Speed is up 30%+. For the first time, a Sonnet model ships with cyber safeguards because its cybersecurity capabilities now match Opus 5. Haiku 5.5 is coming in a few weeks.

Why it matters: Anthropic's second Claude 5.5 model, Sonnet 5.5, keeps the same unit price as the prior generation while cutting cost up to 30% on most tasks and running 30%+ faster.

AI HOT (Curated Pool)

Claude Opus 5.5 (High) hits #2 on Agent Arena and reshapes the Pareto frontier

Anthropic's Claude Opus 5.5 (High) landed at #2 on Agent Arena with a +12.15% net improvement, behind only Fable 5.1 (Max). Median cost is $1.31 per task—40% cheaper than Opus 5 (High) and 56% cheaper than Opus 5 (Max). It ranked #1 on Steerability at +14.50%. The post doesn't disclose a release date or other model comparisons.

Why it matters: Claude Opus 5.5 (High) rose to second on Agent Arena with a +12.15% net improvement score and a $1.31 median cost per task, 40% below Opus 5 (High).

AI HOT (Curated Pool)

Claude Opus 5.5 (High) hits #2 on Agent Arena, costs 56% less than Opus 5 (Max)

Anthropic's Claude Opus 5.5 (High) reached #2 on Agent Arena with a +12.15% net improvement, behind only Fable 5.1 (Max). It also costs 56% less than Opus 5 (Max). The post doesn't disclose exact pricing or latency—I'd discount the cost claim until we see real usage numbers.

Why it matters: Claude Opus 5.5 (High) ranks second on Agent Arena with a +12.15% net improvement score, and Anthropic says it costs 56% less than Opus 5 (Max).

Sep 28Monday

Computing Life · Share · Yage

Agents mentioned PayPal 139 times, picked it 0 times: software distribution is changing

Armature ran 5,300 sandbox sessions where Claude Code, Codex, and Cursor integrated payment and email tools into real engineering repos. PayPal got 139 mentions and zero code commits; Stripe won 88% of payment tests. Tool choice follows language stack: the same email task picked Resend in TypeScript, SendGrid in Python, Postmark in Go, and Azure ACS in Java. Big markets concentrate heavily; long-tail categories split three ways across agents. Armature sells agent-adoption optimization starting at $5,000/month and disclosed the conflict upfront. The post doesn't spell out how the 42% selection-consistency figure was calculated; our own recomputation across reasonable definitions landed at 41.5%–48.2%, consistent with the published number. Agent tool selection is a context-dependent function, not a global ranking—traditional SEO logic breaks here.

Why it matters: Armature logged 16.9K sandbox sessions to see which dev tools Claude Code, Codex and Cursor actually wire up, and Stripe beat PayPal 124 to zero.

Sep 26Saturday

Hacker News front page

LLM watermarking degrades AI agent performance and speed

Lasso Security tested LLM watermarking on AI agents and found it hurts tool-calling accuracy by 3–5 points on BFCL V3, adds 11 seconds of latency, and increases output length by 20–30%. The post doesn't name the specific models or agent frameworks tested.

Why it matters: Lasso Security measured how watermarking LLM output changes agent behavior, covering accuracy, latency and output length, useful for anyone weighing the cost of deploying watermarks.

Sep 25Friday

Hacker News front page

NSA is spending billions this year to test frontier AI models, far above prior estimates

Two sources say the NSA told lawmakers in a classified briefing that it is spending billions in taxpayer money this year to evaluate and test advanced AI models. The figure is far higher than previously known, leading lawmakers to estimate a full federal AI regulatory system could cost tens of billions per year. Trump has mostly resisted stronger federal AI oversight. The article does not name which models are being tested, whose compute is used, or how the money breaks down.

Why it matters: The NSA's real spending on testing frontier AI models is far higher than outsiders knew, and it points to a costly federal oversight bill.

Sep 24Thursday

Hacker News front page

A daily-updated LLM value chart that plots price against intelligence to find the frontier

The site plots 420 models from the Artificial Analysis Intelligence Index against blended API price, drawing a value frontier where no cheaper model is smarter. Claude Opus 5.5 leads at $8/1M tokens with a 57.6 intelligence score. Meta's Muse Spark 1.3 tops the $2–$8 band at 48.1, Xiaomi's MiMo-V2.6-Pro wins $0.54–$2 at 46.3, and Z AI's GLM 5.3 Flash takes the under-$0.24 tier at 41.8. The post doesn't disclose how the intelligence index is built, and Coding/Math sub-scores are listed as empty for many models, so I'd hold off on those comparisons.

Why it matters: The page plots 420 models' Intelligence Index against blended API price, marks the value frontier, and lists the highest-scoring model for each budget band.

Hacker News front page

Paper Instruments releases Paper Office, a Python suite for agents to safely edit Word, PowerPoint, and Excel files

Paper Office is a suite of Python packages that wrap python-docx, python-pptx, and OpenPyxl with safety checks and broader editing capabilities. Across 5 models and 61 tasks, Paper packages plus guidance passed 92.5% of trials, vs 80.7% for the upstream packages alone and 69.5% for Anthropic's Office skills. Agents resorted to raw OOXML editing in only 1.6% of Paper runs, compared to 78.7% without skills and 50.5% with Anthropic skills. The team argues that low agent adoption in consulting, law, and banking stems from tools that silently corrupt formatting, break references, or produce client-unready output. Paper Office keeps the familiar imports and adds cross-run text search, native Word redlines, comment threads, content controls, cross-document composition, and package-level diff saves, refusing unsafe operations instead of quietly breaking files.

Why it matters: Paper Instruments rebuilds python-docx, python-pptx and OpenPyxl into an Office editing toolkit for agents, passing 92.5% of 61 tasks against 80.7% for the upstream libraries.

Hacker News front page

Open-source prompt-injection detectors catch 0–1% of realistic AI agent attacks buried in tool output

This benchmark hides 629 AgentDojo injection attacks inside tool outputs and tests Regex vs. Meta Prompt Guard 2. Regex catches 0%, Prompt Guard 2 catches 1%. The attacks aren't sent directly to the model—they're buried in search results, email bodies, and similar tool responses, so current detectors are effectively blind. Code and reproduction steps are public; the post doesn't include comparisons with commercial detectors.

Why it matters: The buried-injections GitHub project tested 10 open-source detectors against 629 AgentDojo v1 attacks and found Regex caught 0% and Meta's Prompt Guard 2 caught 1% when the injection hides in tool output.

AI HOT (Curated Pool)

Claude Opus 5.5 tops Coding Agent Index, but per-task cost rises to $13.04

Artificial Analysis tested Claude Opus 5.5 under Claude Code max effort and it scored 66 on the Coding Agent Index, up from Opus 5's 60. All three subtests improved: Terminal-Bench 4.0 63.1%, DeepSWE v1.1 68.4%, SWE-Atlas-QnA 66.4%. The trade-off: per-task cost jumped from $3 to $13.04. The post doesn't break down how max effort drove the cost increase.

Why it matters: Artificial Analysis puts Claude Opus 5.5 at the top of its Coding Agent Index with 66 points, six above Opus 5, while cost per task rises to $13.04.

AI HOT (Curated Pool)

MiMo-V2.6-Pro, Claude Opus 5.5, GPT-6 Luna/Sol launch, shifting the intelligence–cost Pareto frontier

Artificial Analysis reports that four new models this week added 11 points on the Intelligence Index vs. cost-per-task Pareto frontier. GPT-6 Luna contributed 5 points, Claude Opus 5.5 contributed 4, and the remaining two came from MiMo-V2.6-Pro and GPT-6 Sol. The post doesn't disclose specific scores, pricing, or latency—hold off on conclusions until full benchmarks drop.

Why it matters: Artificial Analysis data shows four new models this week pushed the best frontier of intelligence index against cost per task out by eleven points, with GPT-6 Luna adding five and Claude Opus 5.5 adding four.

Sep 23Wednesday

OpenAI News

OpenAI releases MentalHealthBench, an open benchmark co-developed with 80+ licensed clinicians to evaluate AI in realistic mental health conversations

OpenAI open-sourced MentalHealthBench, a benchmark built with over 80 licensed psychologists and psychiatrists across 22 countries. It tests AI on realistic mental health conversations ranging from everyday stress to emergencies, covering adults, teens, and caregivers. The eval goes beyond safety filters: it checks whether models seek context, preserve user agency, and offer actionable guidance when appropriate. OpenAI stresses ChatGPT isn't a substitute for therapy, but the benchmark tracks progress on empathy and steering people toward real-world support. The paper and benchmark are publicly available.

Why it matters: OpenAI built MentalHealthBench with more than 80 licensed mental health experts, an open, expert-labeled benchmark for AI responses in realistic mental health conversations.

Latent Space

Claude Opus 5.5 launches with Fable 5.1-level performance at 40% lower cost, plus a rare focus on writing quality

Anthropic released Claude Opus 5.5, the first model in the new 5.5 family. It matches Claude Fable 5.1 on most tasks, costs 40% less to run than Opus 5, and is about 30% faster. The launch unusually highlights writing improvements: the model puts key info up front and follows user style rules. Artificial Analysis notes that token usage on frontier tasks jumped ~80%, so per-task cost remains around $6—similar to Opus 5. OpenAI shipped GPT-6 Sol and Luna an hour later at 50% lower prices than GPT-5.6, but Opus 5.5's launch post hit 17M views and dominated the day. Anthropic's system card also reports multi-agent scaling with up to 100 parallel agents for the first time. Latent Space tested both and switched to Opus 5.5 as the default model immediately, calling the writing quality a night-and-day difference over Sol 6.

Why it matters: Anthropic says Opus 5.5 matches Fable 5.1 on most tasks at 40% lower cost and 30% faster, while outside estimates put per-task cost near Opus 5.

Computing Life · Share · Yage

Same tool toggle: Nemotron-3 550B gained, Mistral-Medium-3.5 crashed

A new paper breaks down coding agent harnesses into three independent toggles and measures each one. The most striking result: switching from dedicated file tools to a pure CLI made Nemotron-3 550B's SWE-Bench Verified score jump 3.6 pp while cutting per-task cost from $2.33 to $1.11, but Mistral-Medium-3.5-128B dropped from 68.60% to 45.40%. Trajectory analysis shows 550B composing dense shell one-liners, while Mistral failed to locate files in 32.80% of tasks and submitted no edits. On Terminal-Bench 2.1, both models improved under CLI mode. Planning boosted the 30B model from 13.60% to 25.20% but only saved ~30% cost for larger models without accuracy gains. Context management mainly prevents window overflow; at 128k the gap shrinks to 2.7 pp, and complex read-back mechanisms were almost never invoked. The takeaway: no universal best harness design—it depends on the model's CLI fluency and the task type.

Why it matters: A controlled test shows the same harness change helps one coding model and badly hurts another, so harness tuning has to be per-model.

AI HOT (Curated Pool)

Arena launches GPT-6 Sol and GPT-6 Luna testing, scores coming soon

Arena is now testing two new OpenAI models, GPT-6 Sol and GPT-6 Luna, with scores not yet released. You can try them on real agent tasks and vote to feed the leaderboard. The post doesn't disclose model size, release date, or pricing.

Why it matters: Arena opened testing slots for OpenAI's GPT-6 Sol and GPT-6 Luna, with scores not yet published; users can run real agent tasks on both models and vote, and votes count toward the leaderboard.

AI HOT (Curated Pool)

OpenAI launches GPT-6 Sol and Luna, API pricing cut 50% vs GPT-5.6

OpenAI added two cheaper models to the GPT-6 family: Sol and Luna, with API prices halved across input and output. Sol costs $2/$10 per 1M tokens, Luna $0.10/$0.50. Sol scored 33.2% on AutomationBench at xhigh effort at 9% of Claude Opus 5's cost per task, and 56.4% on Agents' Last Exam at max effort at 60% lower cost. On internal factuality evals, Sol makes about half as many mistakes as its predecessor. The post does not specify a launch date beyond 'available now.'

Why it matters: Same Astra training recipe as the flagship, with Sol's input price halved from $4 to $2 per million tokens.

AI HOT (Curated Pool)

Claude Opus 5.5 lands on Arena's Agent Arena and Battle Mode

Anthropic's Claude Opus 5.5 is now available on Arena's Agent Arena, where users vote on rankings after the model runs real long-horizon agent tasks. The model can use web search, a file system, and a terminal; the leaderboard uses causal tracking to measure performance relative to the average model. The post doesn't spell out Battle Mode specifics or show example tasks.

Why it matters: Puts Claude Opus 5.5 in an agent harness with web access, file I/O and terminal commands, ranked by causal tracing.

AI HOT (Curated Pool)

Claude Opus 5.5 tops Artificial Analysis Intelligence Index with a score of 58, plus a 20% price cut

Claude Opus 5.5 scored 58 on the Artificial Analysis Intelligence Index, the highest measured so far. It leads on 6 of 10 evaluations, including Humanity's Last Exam at 61.4% and SciCode at 66.9%, and matches GPT-6 Astra (xhigh) on Terminal-Bench 4.0 at 59.6%. On the agentic knowledge-work eval AA-Briefcase, it hit 1822 Elo—143 points above Fable 5.1—and surpassed GPT-5.6 Sol on both analytical quality and presentation. Pricing dropped to $4/$20 per 1M input/output tokens (from $5/$25), with cache reads down 60% to $0.20. Output tokens per task grew ~60% vs Opus 5, so cost per task stayed flat. Context window remains 1M tokens with image and text input.

Why it matters: Opus 5.5 tops the Artificial Analysis Intelligence Index at 58 and leads six of ten evaluations, with prices down 20%.