Benchmarks

GPT-6.1 Sol replaced GPT-6 Sol 7 days after launch. Its intelligence index is 1 point below GPT-6 Astra, and pricing stays at $2 per million input tokens and $10 per million output tokens. The cached-read discount rises from 90% to 95%.

Jon Clegg built a Pac-Man benchmark: one prompt, one HTML page, scored automatically by Opus 5.5. Claude Opus 5.5 hit 99/100 via Claude Code at $1.99, generating a 10.8 KB page in 9 minutes with near-arcade audio. Claude Fable 5.1 scored 96 but cost $5.87. Grok 4.7 and GPT-5.6-sol scored 94 and 90; the latter cost just $0.72 in under 5 minutes. Scoring covers controls, ghost behavior, stuck detection, maze layout, and sound. The post doesn't explain why some models ran Phase 2 or how much the harness affects scores. Worth noting: this measures model-plus-toolchain combos, not bare model capability.

Arena evaluated GPT-6 Luna (Max) on 8K real agent conversations. It ranked #23, up 6 spots from GPT-5.6 Luna (xHigh), with a net gain of +1.6%. Cost is $0.05 per task. The post doesn't disclose latency or task-type breakdown.

Anthropic's Claude Sonnet 5.5 is now available for voting in Arena's Agent Arena. The leaderboard evaluates models on millions of real-world long-horizon agent tasks where models can use web search, filesystem, and terminal tools. Rankings use a causal tracking method to measure how much a model outperforms the average.

Claude Sonnet 5.5 is the second model in the 5.5 family, aimed at everyday coding, bug fixes, and polished docs. It scores 70.6% on Terminal-Bench 4.0 vs. Sonnet 5's 10.3%. Pricing stays at $2/$10 per million input/output tokens, but it uses fewer tokens per task, cutting per-task cost by up to 30%. Speed is up 30%+. For the first time, a Sonnet model ships with cyber safeguards because its cybersecurity capabilities now match Opus 5. Haiku 5.5 is coming in a few weeks.

Anthropic's Claude Opus 5.5 (High) landed at #2 on Agent Arena with a +12.15% net improvement, behind only Fable 5.1 (Max). Median cost is $1.31 per task—40% cheaper than Opus 5 (High) and 56% cheaper than Opus 5 (Max). It ranked #1 on Steerability at +14.50%. The post doesn't disclose a release date or other model comparisons.

Anthropic's Claude Opus 5.5 (High) reached #2 on Agent Arena with a +12.15% net improvement, behind only Fable 5.1 (Max). It also costs 56% less than Opus 5 (Max). The post doesn't disclose exact pricing or latency—I'd discount the cost claim until we see real usage numbers.

Armature ran 5,300 sandbox sessions where Claude Code, Codex, and Cursor integrated payment and email tools into real engineering repos. PayPal got 139 mentions and zero code commits; Stripe won 88% of payment tests. Tool choice follows language stack: the same email task picked Resend in TypeScript, SendGrid in Python, Postmark in Go, and Azure ACS in Java. Big markets concentrate heavily; long-tail categories split three ways across agents. Armature sells agent-adoption optimization starting at $5,000/month and disclosed the conflict upfront. The post doesn't spell out how the 42% selection-consistency figure was calculated; our own recomputation across reasonable definitions landed at 41.5%–48.2%, consistent with the published number. Agent tool selection is a context-dependent function, not a global ranking—traditional SEO logic breaks here.

Lasso Security tested LLM watermarking on AI agents and found it hurts tool-calling accuracy by 3–5 points on BFCL V3, adds 11 seconds of latency, and increases output length by 20–30%. The post doesn't name the specific models or agent frameworks tested.

Two sources say the NSA told lawmakers in a classified briefing that it is spending billions in taxpayer money this year to evaluate and test advanced AI models. The figure is far higher than previously known, leading lawmakers to estimate a full federal AI regulatory system could cost tens of billions per year. Trump has mostly resisted stronger federal AI oversight. The article does not name which models are being tested, whose compute is used, or how the money breaks down.

The site plots 420 models from the Artificial Analysis Intelligence Index against blended API price, drawing a value frontier where no cheaper model is smarter. Claude Opus 5.5 leads at $8/1M tokens with a 57.6 intelligence score. Meta's Muse Spark 1.3 tops the $2–$8 band at 48.1, Xiaomi's MiMo-V2.6-Pro wins $0.54–$2 at 46.3, and Z AI's GLM 5.3 Flash takes the under-$0.24 tier at 41.8. The post doesn't disclose how the intelligence index is built, and Coding/Math sub-scores are listed as empty for many models, so I'd hold off on those comparisons.

Paper Office is a suite of Python packages that wrap python-docx, python-pptx, and OpenPyxl with safety checks and broader editing capabilities. Across 5 models and 61 tasks, Paper packages plus guidance passed 92.5% of trials, vs 80.7% for the upstream packages alone and 69.5% for Anthropic's Office skills. Agents resorted to raw OOXML editing in only 1.6% of Paper runs, compared to 78.7% without skills and 50.5% with Anthropic skills. The team argues that low agent adoption in consulting, law, and banking stems from tools that silently corrupt formatting, break references, or produce client-unready output. Paper Office keeps the familiar imports and adds cross-run text search, native Word redlines, comment threads, content controls, cross-document composition, and package-level diff saves, refusing unsafe operations instead of quietly breaking files.

This benchmark hides 629 AgentDojo injection attacks inside tool outputs and tests Regex vs. Meta Prompt Guard 2. Regex catches 0%, Prompt Guard 2 catches 1%. The attacks aren't sent directly to the model—they're buried in search results, email bodies, and similar tool responses, so current detectors are effectively blind. Code and reproduction steps are public; the post doesn't include comparisons with commercial detectors.

Artificial Analysis tested Claude Opus 5.5 under Claude Code max effort and it scored 66 on the Coding Agent Index, up from Opus 5's 60. All three subtests improved: Terminal-Bench 4.0 63.1%, DeepSWE v1.1 68.4%, SWE-Atlas-QnA 66.4%. The trade-off: per-task cost jumped from $3 to $13.04. The post doesn't break down how max effort drove the cost increase.

Artificial Analysis reports that four new models this week added 11 points on the Intelligence Index vs. cost-per-task Pareto frontier. GPT-6 Luna contributed 5 points, Claude Opus 5.5 contributed 4, and the remaining two came from MiMo-V2.6-Pro and GPT-6 Sol. The post doesn't disclose specific scores, pricing, or latency—hold off on conclusions until full benchmarks drop.

OpenAI open-sourced MentalHealthBench, a benchmark built with over 80 licensed psychologists and psychiatrists across 22 countries. It tests AI on realistic mental health conversations ranging from everyday stress to emergencies, covering adults, teens, and caregivers. The eval goes beyond safety filters: it checks whether models seek context, preserve user agency, and offer actionable guidance when appropriate. OpenAI stresses ChatGPT isn't a substitute for therapy, but the benchmark tracks progress on empathy and steering people toward real-world support. The paper and benchmark are publicly available.

Anthropic released Claude Opus 5.5, the first model in the new 5.5 family. It matches Claude Fable 5.1 on most tasks, costs 40% less to run than Opus 5, and is about 30% faster. The launch unusually highlights writing improvements: the model puts key info up front and follows user style rules. Artificial Analysis notes that token usage on frontier tasks jumped ~80%, so per-task cost remains around $6—similar to Opus 5. OpenAI shipped GPT-6 Sol and Luna an hour later at 50% lower prices than GPT-5.6, but Opus 5.5's launch post hit 17M views and dominated the day. Anthropic's system card also reports multi-agent scaling with up to 100 parallel agents for the first time. Latent Space tested both and switched to Opus 5.5 as the default model immediately, calling the writing quality a night-and-day difference over Sol 6.

A new paper breaks down coding agent harnesses into three independent toggles and measures each one. The most striking result: switching from dedicated file tools to a pure CLI made Nemotron-3 550B's SWE-Bench Verified score jump 3.6 pp while cutting per-task cost from $2.33 to $1.11, but Mistral-Medium-3.5-128B dropped from 68.60% to 45.40%. Trajectory analysis shows 550B composing dense shell one-liners, while Mistral failed to locate files in 32.80% of tasks and submitted no edits. On Terminal-Bench 2.1, both models improved under CLI mode. Planning boosted the 30B model from 13.60% to 25.20% but only saved ~30% cost for larger models without accuracy gains. Context management mainly prevents window overflow; at 128k the gap shrinks to 2.7 pp, and complex read-back mechanisms were almost never invoked. The takeaway: no universal best harness design—it depends on the model's CLI fluency and the task type.

Arena is now testing two new OpenAI models, GPT-6 Sol and GPT-6 Luna, with scores not yet released. You can try them on real agent tasks and vote to feed the leaderboard. The post doesn't disclose model size, release date, or pricing.

OpenAI added two cheaper models to the GPT-6 family: Sol and Luna, with API prices halved across input and output. Sol costs $2/$10 per 1M tokens, Luna $0.10/$0.50. Sol scored 33.2% on AutomationBench at xhigh effort at 9% of Claude Opus 5's cost per task, and 56.4% on Agents' Last Exam at max effort at 60% lower cost. On internal factuality evals, Sol makes about half as many mistakes as its predecessor. The post does not specify a launch date beyond 'available now.'