Skip to content

Benchmarks

Who is actually stronger: benchmark results, methodology disputes and leaderboard changes.

Latest picks

121–140 of 471

Aug 2Sunday

Computing Life · Share · Yage

Prompt injection defense lives in the harness, not the model

Ghostcommit showed the same Sonnet model rejected malicious PNG instructions 10/10 times in Claude Code, but obeyed 10/10 times in Cursor and Antigravity, leaking .env secrets. Lab-reported 99% defense rates suffer from five traps: static benchmark overfitting, misleading single-attempt ASR, LLM-as-judge drift, ignored utility-under-attack, and bare-model testing without tool shells. Deeper causes: LLMs lack hard instruction-data separation, and stronger models can follow injections more faithfully—Opus 4.6 with extended thinking saw ASR rise from 14.8% to 21.7%. A joint study by 14 researchers from OpenAI, Anthropic, and DeepMind tested 12 model-layer defenses; over 90% broke under adaptive attacks, with human red-teamers hitting 100%. The engineering fix is architectural isolation: CaMeL separates trusted planner from untrusted executor, and OpenClaw's dual-agent setup cut ASR from 100% to 0.31%. Harness-level deterministic tool gating, hook signature checks, and sandboxed least-privilege are the real controls.

Why it matters: Same Sonnet model refused a malicious image prompt in Claude Code ten times out of ten, but leaked .env secrets every time in Cursor and Antigravity.

Jul 31Friday

Hacker News front page

Distilling DeepSeek into GPT-OSS Doesn't Transfer Censorship

CTGT distilled a 120B finance model from DeepSeek V4 Flash. Across 152 matched prompt pairs, the teacher scored 45.45 points more censored on China-sensitive topics, but the student showed no censorship at all—four US lab judges agreed. Self-distillation on corrected outputs matched the DeepSeek-taught model on financial reasoning, at 62× lower cost than Inkling. Code, data, and models are open.

Why it matters: CTGT distilled DeepSeek V4 Flash into a 120B finance model, and 152 paired tests show the teacher's censorship leanings did not carry over to the student.

Jul 29Wednesday

AI HOT (Curated Pool)

Enabling two API settings tripled GPT-5.6's ARC-AGI-3 scores

GPT-5.6 Sol scored just 7.8% on ARC-AGI-3 because the official harness discarded private reasoning after each action and used rolling truncation that dropped older moves. Switching to retained reasoning and context compaction raised the public-set score from 13.3% to 38.3% while cutting output tokens by 6x. Human testers averaged about 48%. The post doesn't disclose full private-set results or whether the same settings help other models.

Why it matters: OpenAI traced GPT-5.6's low ARC-AGI-3 score to the test harness clearing thinking and truncating history, and fixing both settings tripled the score.

Jul 28Tuesday

Hacker News front page

A $500 RL fine-tune of a 9B open model beat frontier models on catalog review

Fermisense GRPO-fine-tuned a 9B open-source model for catalog review and hit 87.3% quality vs. 76.9% for the best frontier setup. Cost per 1,000 listings: $0.50 for the fine-tune, $19–$172 for frontier models. The post calls this 'intelligence ownership'—training a specialist on your own data, tools, and scorer beats prompt-tuned frontier APIs on both accuracy and cost. Ramp data is cited: top-quartile AI spenders more than doubled revenue from Nov 2022 to Dec 2025; zero-AI-spend companies grew ~15%.

Why it matters: Fermisense fine-tuned a 9B open model with GRPO for catalog review, scoring 87.3% against 76.9% for the strongest closed combo it tested, cutting cost per thousand items from $19 to $172 down to $0.50.

Jul 27Monday

Computing Life · Share · Yage

Why high SWE-bench scores don't translate to real-world Kotlin projects

JetBrains released the Kotlin Benchmark on July 24, 2026, testing AI coding agents on 105 real-world tasks across 8 open-source projects. With the same Opus 4.7 model, Claude Code hit 85.71% and Junie 81.9%—a nearly 4-point gap driven by how each agent harness handles Gradle build logs. A good harness uses Language Server diagnostics to catch static errors locally, then runs full builds only at key checkpoints and extracts just the blocking lines from noisy output. The article argues that Python-based benchmarks like SWE-bench reward trial-and-error strategies that collapse under Kotlin's heavy build overhead. It recommends teams stop buying off public leaderboards and instead use the Harbor container spec to build private micro-eval matrices from their own historical PRs and issues, measuring Pass@k stability, token cost, and whether patches respect internal architectural constraints.

Why it matters: JetBrains ran 105 real tasks from 8 open source projects, and on the same model Claude Code and its own Junie differed by nearly 4 points in success rate.

Jul 26Sunday

Hacker News front page

Terence Tao's ICM 2026 talk on how the math community should respond to AI

Tao's ICM 2026 talk sidesteps the debate on whether AI can do research math. He asks the community to assume a strong capability hypothesis and instead examine its own goals and values. He draws a parallel to the early-20th-century foundations crisis. The talk cites First Proof benchmark results: on May 28, 2026, 7 out of 10 novel research problems were solved at publication quality by at least one AI harness, with compute costs of $10–$1,000 per problem.

Why it matters: Tao assumes AI can already do research-level math, then asks what the field is actually for, shifting the debate from tools to purpose.

AI Chat-Group Daily (群聊日报)

Opus 5 Day 2: Saturation self-testing trades cost for quality, total cost may beat Fable

Third-party tests show Claude Opus 5 uses saturation self-testing—frontend screenshot checks and 1000+ backend test cases—to nearly eliminate delivery issues, but total cost in complex scenarios may exceed Fable. With self-testing off, bug rates don't beat the previous model. Anthropic's strategy: long-chain debugging over one-shot correctness. The official model card advises against max effort for the first time; FrontierBench peaks at xhigh. Another test reveals ~80% of Claude Code's system prompt was cut. Group sentiment is positive, some calling it smoother than Fable. OpenAI had a full 503 outage overnight; reset cards landed the next day. WSJ reports US companies are mixing cheaper models to control costs, with Cursor as a beneficiary.

Why it matters: A third-party test shows Claude Opus 5's saturated self-testing cuts delivery bugs but burns far more time and tokens, possibly costing more than Fable on complex work.

Jul 25Saturday

Hacker News front page

ARC Prize launches ARC-AGI-3 leaderboard, ranking AI systems by cost efficiency

ARC Prize published the verified leaderboard for ARC-AGI-3. The new benchmark tests how AI agents adapt on the fly to novel interactive environments, not just passive reasoning. A scatter plot maps each system's score against cost per task, making efficiency the headline metric. Only systems that cost under $10,000 to run are shown; Kaggle entries operate under a $50 compute cap. The post doesn't list specific model names or scores—you need to open the page to see the full ranking.

Why it matters: ARC Prize's verified ARC-AGI-3 leaderboard plots agent scores in interactive environments against cost per task, and only lists systems under $10,000 total.

Computing Life · Share · Yage

OpenAI Presence: Turning Field Failures Into a Productized Improvement Loop

OpenAI launched Presence on July 22, an enterprise voice and chat agent product targeting specific roles like customer service and outbound sales. Its core pitch is not model capability but productizing the feedback loop after agent failures: when a task gets stuck and escalates to a human, the system saves the full execution context, lets teams reproduce the failure in a sandbox, fix rules, run regression tests, and push code changes via Codex. This standardizes what field FDEs used to do manually—migrating on-site failures back into the product. Presence is in limited GA, non-self-serve, deployed case-by-case; OpenAI hasn't disclosed hosting details, data residency, or cross-vendor export for failure records and test suites. The article warns that if enterprises can't take these hard-won lessons with them, they face a new form of vendor lock-in.

Why it matters: OpenAI's Presence packages enterprise voice and chat agents with standard tooling for saving failed handoff context, reproducing it in a sandbox, patching rules and regression testing.

Jul 24Friday

Computing Life · Share · Yage

GPT-5.6 prompt guide: write fewer steps, define clearer deliverables

OpenAI's July 22 guidance for GPT-5.6 tells developers to strip hand-written intermediate steps from prompts and instead constrain agents with completion criteria, verification evidence, and permission boundaries. The recommended method is ablation testing on eval sets—remove a section, rerun, and keep it only if metrics hold. This reverses the GPT-4.1 era of hard-coding eight-step workflows into system prompts. GPT-5 had already started loosening route control by scene. The author validated the approach in a long-form translation system, replacing chunking and retry logic with deliverable specs that let the agent decide its own execution path.

Why it matters: OpenAI's GPT-5.6 prompting guide says that when an agent has enough context you should state the delivery standard instead of scripting every intermediate step.

Jul 23Thursday

Computing Life · Share · Yage

Cursor rewrites SQLite with Swarm: a controlled experiment pushing three scaling dimensions of agent orchestration

Cursor fed 835 pages of SQLite docs into its new Harness, hitting 80% sqllogictest pass rate in 4 hours; the old Swarm was halted before hour 2 due to code conflicts. The new system isolates planner and worker roles, uses shared design docs and auto-merge, cutting merge conflicts from 70k to under 1k. A mixed-model setup—Opus 4.8 planning, Composer 2.5 executing—cost $1,339 total, roughly 8× cheaper than GPT-5.5 solo. The public minisqlite repo lacks CLI, C API, and cross-process locking, so it's far from production-ready SQLite. This is a capability demo under ideal conditions—fixed spec, dense feedback—not a daily driver for product teams with shifting requirements.

Why it matters: Cursor ran its old and new agent orchestration systems on the same task, writing SQLite in Rust from scratch, and the two runs diverged sharply.

Jul 22Wednesday

OpenAI News

OpenAI launches Presence, a production agent product for customer and internal workflows

OpenAI launched Presence today, a product for deploying voice and chat AI agents in enterprise workflows. It bundles policies, guardrails, escalation rules, and evaluation tooling so agents can access company systems, take approved actions, and hand off to humans when needed. OpenAI's own English-language support line at 1-888-GPT-0090 already runs on Presence: it resolves 75% of inbound issues without human help and cut handoff rates by 15 percentage points in 10 days via a Codex-powered improvement loop. BBVA is testing Spanish-language voice banking in Mexico, SoftBank is trialing Japanese conversations, and IAG is exploring claims support during severe weather. The post does not disclose pricing or API availability details.

Why it matters: OpenAI packaged its internal support bot deployment into Presence, letting enterprises set which systems the AI can read, what actions it can take and when to hand off to a human.

Hacker News front page

Fireworks benchmarks Kimi K3 vs Fable 5: routing the two hits 93% accuracy at far lower cost

Fireworks ran ~1,030 real agent tasks comparing Kimi K3 and Fable 5. Head-to-head on SWE: K3 92.4%, Fable 92.6% — a near tie. Under the hood, K3 excels at symbolic math and dev tooling; Fable wins on web and data viz. Routing between the two lifts accuracy to 93% and cuts cost up to 50x on long agent loops. Oracle routing shows K3 can handle 72–96% of tasks, but a practical router still needs an order of magnitude more routing data.

Why it matters: A 1,030-task coding agent comparison shows Kimi K3 and Fable nearly tied head to head, with routing by task type lifting accuracy to 93% and cutting cost by up to 50x.

Jul 17Friday

AI HOT (Curated Pool)

Kimi K3 tops frontend coding leaderboard, open weights coming July 27

Kimi K3 scored 1679 on Frontend Code Arena, taking first in 6 of 7 frontend sub-tasks and beating Claude Fable 5 and GPT-5.6 Sol. It's a 2.8-trillion-parameter MoE model with a 1M context window, and open weights are promised for July 27. API pricing is $15 per million tokens—no low-cost play here, it's priced against top closed-source models and aimed at long-context coding and agent workflows.

Why it matters: Moonshot AI's Kimi K3 tops Frontend Code Arena at 1679, winning 6 of 7 subtasks against Claude Fable 5 and GPT-5.6 Sol. 2.8T MoE params, 1M context window, weights opening July 27.

AI HOT (Curated Pool)

The agent evaluation gap: Enterprise AI organizations have a reality-alignment problem, not a coverage problem — and most are shipping to production anyway

A VentureBeat survey of 157 enterprises finds a sharp gap between agent evaluations and real-world performance. In the past year, 50% of organizations shipped an agent or LLM feature that passed internal evals but then caused a customer-facing failure; a quarter saw it happen more than once. Only 5% fully trust automated evaluation, with poor alignment to real outcomes cited as the top limitation (29%). Yet 66% already allow or are engineering toward fully automated, zero-human-in-the-loop deployments. The eval stack is fragmented: 17% rely on model-provider native evals, another 17% have no dedicated tooling, and only about a quarter run real-time quality checks on live traffic. The sample skews mid-market (100+ employees), with tech/software at 23%.

Why it matters: A VentureBeat survey of 157 enterprises found half had put an AI agent that passed internal tests into production and then hit customer-facing failures, with the eval-to-reality gap the most cited problem.

Jul 16Thursday

Hugging Face Blog

Ai2 shares the engineering lessons behind Shippy, a maritime AI agent

Ai2's Skylight team built Shippy, an AI assistant that helps maritime analysts query fishing activity, EEZ boundaries, and vessel tracks. The post breaks its architecture into three parts: a soul (system prompt), skills (markdown files that teach it to call APIs and interpret track data), and config (runtime settings; currently Claude Opus 4.6 with the OpenClaw framework). The core idea is wrapping a non-deterministic model in deterministic tools—every answer includes source, data cutoff, and a deep link to the Skylight map so an analyst can verify it. The post doesn't disclose error rates or latency numbers, but it stresses sandboxed hosting and evaluating the agent as a system, not just the model.

Why it matters: Ai2's Skylight team split its maritime assistant Shippy into system prompt, Markdown skill files and runtime config, a way to keep model behavior inside deterministic tools that other agent projects can copy.

Hacker News front page

Unsolved Problems in MLOps

This ACM Queue piece lays out why classical ops practices break down for ML: non-deterministic outputs and data as a system driver make canary deploys, health checks, and alerting nearly useless. Azure validates new models by having LLMs judge LLM output—the SRECon audience was audibly surprised. The authors argue the field must either find a better paradigm or fix the ones we have.

Why it matters: An ACM Queue piece lays out two hard problems in MLOps: uncertain model outputs and systems driven by data and code together, which break standard health checks and canary releases.

Jul 15Wednesday

Computing Life · Share · Yage

AI Trains AI: What a Public Self-Improvement Experiment Actually Closed the Loop On

Dan Austin open-sourced a full AI-trains-AI loop. An outer Qwen3.6 agent designs post-training recipes; an inner Qwen3-0.6B or 1.7B model runs real GPU training, and hidden eval scores feed back as reward to update the outer policy. Over 54 steps, the agent first learned to reduce invalid submissions, then shifted 1.7B model usage from 42% to 95% and began tuning temperature, optimizer, and other hyperparameters. Trained small-model scores rose from noise level into the 0.22–0.48 range, with limited transfer to a held-out triage task. A postmortem also revealed an evaluator bug: the old tool-use detector looked for `.function.name` instead of `.name`, so the 0.4-weighted score never fired—yet the reward curve still climbed. The fix required a full restart. The experiment shows outer RL can reshape a training agent's behavior, but tasks, rewards, and budgets are still human-designed.

Why it matters: A full open-source loop where an outer agent designs the post-training recipe and an inner model actually trains on GPU, lifting a 1.7B model's usage share from 42% to 95% in 54 steps.

Jul 14Tuesday

Hacker News front page

DoorDash uses LLM juries and multimodal AI to tag food items, beating human accuracy by 20%

DoorDash's catalog has millions of items with wildly inconsistent names, making manual tagging slow and expensive. They built an AI metadata platform that uses multimodal models—text, images, and web search—to infer attributes like spiciness or cuisine type. Instead of human review, an 'LLM jury' of multiple strong models votes independently and aggregates a consensus, lifting annotation accuracy roughly 20% above typical human reviewers. Context-optimization agents iterate prompts in minutes, adding another 20%+ precision gain and speeding up prompt development 10x. They auto-generate training data to fine-tune small models that match frontier LLM quality at 10% of the inference cost. Distributed inference cut backfill time for millions of items from over a month to a few days. The post doesn't disclose which models, latency, or per-item cost.

Why it matters: DoorDash uses juries of several large models to auto-label dish attributes like spiciness and cuisine, beating ordinary human review by about 20 points of accuracy and showing a reusable pattern for multimodal data labeling.

Hacker News front page

Apple's new SpeechAnalyzer beats Whisper Small on accuracy in first public benchmark

Inscribe benchmarked Apple's new SpeechAnalyzer API against the legacy SFSpeechRecognizer and three Whisper models on 5,559 LibriSpeech utterances. SpeechAnalyzer hit 2.12% WER on clean speech and 4.56% on noisy speech, beating Whisper Small by 1.62 and 3.39 points respectively while running ~3x faster. The legacy API scored 9.02% WER, worse than the 40MB Whisper Tiny. All engines ran fully on-device on an M2 Pro. Inscribe switched its default engine to SpeechAnalyzer and released all transcripts and scoring code. The post does not disclose SpeechAnalyzer's model architecture or parameter count.

Why it matters: The Inscribe team ran 5,559 standard speech samples through Apple's new SpeechAnalyzer, the older SFSpeechRecognizer and three Whisper models to get numbers on accuracy and speed.