Skip to content

#评测/基准

1 today

Oct 14Wednesday

AI HOT · Tips & opinions

Google runs Gemini 4 Argon unattended to optimize ~205,000 lines of ADK Python

Google ran Gemini 4 Argon unattended for 11 hours over its ADK codebase, cutting memory session preparation time for long sessions by about 40% and package import time by 21%. Maintenance rules were supplied through .agents/skills and /goal mode in Google Antigravity. The model proposed 6 optimizations; after human review, session copying and lazy imports were merged.

Why it matters: Google wrote repo maintenance rules into skill files and let Gemini 4 Argon optimize code overnight, with maintainers filtering out fragile fixes.

Oct 7Wednesday

Hacker News front page

Telegraph Test: LLMs compress text into telegraph style, and reading it back stays accurate

Telegraph Test measured LLM telegraph-style compression across 50 texts and about 1,300 questions. Accuracy when another model reads the compressed text ran 0.99 to 1.10 times the plaintext baseline. With lowercase instructions, GLM-5.3-Flash cut tokens by 48.4% but answered in telegraph style at an accuracy ratio of 0.81. gpt-5-mini, which cannot turn off reasoning, cost about twice as much to write the compressed record as plaintext.

Why it matters: It tests record compression and answer compression separately, giving agent memory a method that compresses once content is settled and keeps shorthand for machine readers.

The Decoder

Common Sense rates ChatGPT teen version unacceptable risk; suicide chats miss parent alerts

Common Sense Media's AI safety institute for teens rated the ChatGPT teen version an "unacceptable risk" to minors after more than 4,000 test prompts. New test accounts linked to over 12 parent accounts never triggered notifications in explicit conversations about suicide, self-harm and eating disorders, and more than a quarter of situations that should have offered crisis referrals did not point users to professional help.

Why it matters: A teen AI safety group tested ChatGPT's parent alerts, and crisis chats failing to trigger notifications clashes with its safety promises to parents.

Oct 6Tuesday

Hacker News front page

Anthropic vs OpenAI subscription review: daily-driver API-equivalent value runs about 5x higher

SemiAnalysis measured subscription quota consumption by token type and found that, on agent workloads, Anthropic's Opus 5.5 plan delivers roughly 5x the API-equivalent value of OpenAI's GPT 6.1 Sol. The comparison breaks plan value down by subscription tier, model and workload combination.

Why it matters: SemiAnalysis separates quota consumption by token type, pushing subscription value comparisons down to specific plan, model and workload combinations.

Oct 3Saturday

The Verge · AI

OpenAI Dots hands-on: books dinner, but works better on job tasks

In the author's hands-on test, OpenAI Dots handled a website revamp and video editing, but booking and shopping tasks kept hitting security checks. Talking through design changes by voice and granting Dots access to the computer let it deploy site updates, merge videos and resize them for social media. Those jobs required handing over a lot of permissions.

Why it matters: A hands-on test shows OpenAI Dots stalling on everyday services at security checks while running smoother on authorized site and video work.

Oct 1Thursday

The Decoder

Google releases Gemini 4 Argon: benchmarks match OpenAI and Anthropic, no clear lead

Google released Gemini 4 Argon, its new frontier model. On several benchmarks it matches OpenAI's GPT-6 Astra and Anthropic's Claude Fable 5.1, but it does not take a clear lead.

Why it matters: Third-party benchmarks put Gemini 4 Argon level with GPT-6 Astra and Claude Fable 5.1, though it burns about twice the output tokens per task.

AI HOT · Models

Artificial Analysis rates Gemini 4 Argon: Google back in the top three on intelligence

Artificial Analysis' evaluation shows Google DeepMind's Gemini 4 Argon scoring 53 on the Artificial Analysis Intelligence Index in high-reasoning mode, level with GPT-6 Astra (max, 53) and one point ahead of GPT-6.1 Sol (max, 52).

Why it matters: Artificial Analysis scored Gemini 4 Argon's high-reasoning mode at 53 on its intelligence index, tied with GPT-6 Astra, with the lowest hallucination rate among comparable models.

Sep 30Wednesday

AI HOT · Models

GPT-6.1 Sol replaces GPT-6 Sol 7 days after launch, 1 point behind GPT-6 Astra on intelligence index

GPT-6.1 Sol replaced GPT-6 Sol 7 days after launch. Its intelligence index is 1 point below GPT-6 Astra, and pricing stays at $2 per million input tokens and $10 per million output tokens. The cached-read discount rises from 90% to 95%.

Why it matters: Side-by-side intelligence index, cost and token-efficiency figures let readers judge the trade-off GPT-6.1 Sol makes between price and capability.

Sep 29Tuesday

Latent Space

Opus 5.5 wins community praise for explainer video production

After Opus 5.5 launched this week, the community shared several cases of writing code in Claude Code to produce explainer videos and animations. One reproduction case says roughly 1.5–2 hours and about $4 produced a 30–60s explainer video with script, assets, voiceover, music and sound effects, rendering JavaScript canvas animation to MP4.

Why it matters: Community users chained script, assets, voiceover and code animation through Claude Code and Opus 5.5, giving a reusable pipeline for automated explainer videos.

Hacker News front page

Pac-Bench: One-shot Pac-Man benchmark, Claude Opus 5.5 scores 99/100

Jon Clegg built a Pac-Man benchmark: one prompt, one HTML page, scored automatically by Opus 5.5. Claude Opus 5.5 hit 99/100 via Claude Code at $1.99, generating a 10.8 KB page in 9 minutes with near-arcade audio. Claude Fable 5.1 scored 96 but cost $5.87. Grok 4.7 and GPT-5.6-sol scored 94 and 90; the latter cost just $0.72 in under 5 minutes. Scoring covers controls, ghost behavior, stuck detection, maze layout, and sound. The post doesn't explain why some models ran Phase 2 or how much the harness affects scores. Worth noting: this measures model-plus-toolchain combos, not bare model capability.

Why it matters: One prompt sent to 30 models and harnesses to build Pac-Man in one shot, scored item by item on controls, ghosts and maze, gives a side-by-side read on single-page game generation.

AI HOT (Curated Pool)

GPT-6 Luna (Max) ranks #23 on Agent Arena at $0.05 per task

Arena evaluated GPT-6 Luna (Max) on 8K real agent conversations. It ranked #23, up 6 spots from GPT-5.6 Luna (xHigh), with a net gain of +1.6%. Cost is $0.05 per task. The post doesn't disclose latency or task-type breakdown.

Why it matters: Arena's 8,000 real agent sessions put GPT-6 Luna (Max) at rank 23, up six places from GPT-5.6 Luna (xHigh), at $0.05 per task.

AI HOT (Curated Pool)

Claude Sonnet 5.5 enters Arena's Agent Arena and Battle Mode

Anthropic's Claude Sonnet 5.5 is now available for voting in Arena's Agent Arena. The leaderboard evaluates models on millions of real-world long-horizon agent tasks where models can use web search, filesystem, and terminal tools. Rankings use a causal tracking method to measure how much a model outperforms the average.

Why it matters: Claude Sonnet 5.5 entered Arena's Agent Arena with voting open. The leaderboard draws on millions of real long-horizon agent tasks and uses causal tracing to measure a model's outcomes against an average model.

Hacker News front page

Anthropic launches Claude Sonnet 5.5: 30%+ faster, up to 30% cheaper than Sonnet 5

Claude Sonnet 5.5 is the second model in the 5.5 family, aimed at everyday coding, bug fixes, and polished docs. It scores 70.6% on Terminal-Bench 4.0 vs. Sonnet 5's 10.3%. Pricing stays at $2/$10 per million input/output tokens, but it uses fewer tokens per task, cutting per-task cost by up to 30%. Speed is up 30%+. For the first time, a Sonnet model ships with cyber safeguards because its cybersecurity capabilities now match Opus 5. Haiku 5.5 is coming in a few weeks.

Why it matters: Anthropic's second Claude 5.5 model, Sonnet 5.5, keeps the same unit price as the prior generation while cutting cost up to 30% on most tasks and running 30%+ faster.

AI HOT (Curated Pool)

Claude Opus 5.5 (High) hits #2 on Agent Arena and reshapes the Pareto frontier

Anthropic's Claude Opus 5.5 (High) landed at #2 on Agent Arena with a +12.15% net improvement, behind only Fable 5.1 (Max). Median cost is $1.31 per task—40% cheaper than Opus 5 (High) and 56% cheaper than Opus 5 (Max). It ranked #1 on Steerability at +14.50%. The post doesn't disclose a release date or other model comparisons.

Why it matters: Claude Opus 5.5 (High) rose to second on Agent Arena with a +12.15% net improvement score and a $1.31 median cost per task, 40% below Opus 5 (High).

AI HOT (Curated Pool)

Claude Opus 5.5 (High) hits #2 on Agent Arena, costs 56% less than Opus 5 (Max)

Anthropic's Claude Opus 5.5 (High) reached #2 on Agent Arena with a +12.15% net improvement, behind only Fable 5.1 (Max). It also costs 56% less than Opus 5 (Max). The post doesn't disclose exact pricing or latency—I'd discount the cost claim until we see real usage numbers.

Why it matters: Claude Opus 5.5 (High) ranks second on Agent Arena with a +12.15% net improvement score, and Anthropic says it costs 56% less than Opus 5 (Max).

Sep 28Monday

Computing Life · Share · Yage

Agents mentioned PayPal 139 times, picked it 0 times: software distribution is changing

Armature ran 5,300 sandbox sessions where Claude Code, Codex, and Cursor integrated payment and email tools into real engineering repos. PayPal got 139 mentions and zero code commits; Stripe won 88% of payment tests. Tool choice follows language stack: the same email task picked Resend in TypeScript, SendGrid in Python, Postmark in Go, and Azure ACS in Java. Big markets concentrate heavily; long-tail categories split three ways across agents. Armature sells agent-adoption optimization starting at $5,000/month and disclosed the conflict upfront. The post doesn't spell out how the 42% selection-consistency figure was calculated; our own recomputation across reasonable definitions landed at 41.5%–48.2%, consistent with the published number. Agent tool selection is a context-dependent function, not a global ranking—traditional SEO logic breaks here.

Why it matters: Armature logged 16.9K sandbox sessions to see which dev tools Claude Code, Codex and Cursor actually wire up, and Stripe beat PayPal 124 to zero.

Sep 26Saturday

Hacker News front page

LLM watermarking degrades AI agent performance and speed

Lasso Security tested LLM watermarking on AI agents and found it hurts tool-calling accuracy by 3–5 points on BFCL V3, adds 11 seconds of latency, and increases output length by 20–30%. The post doesn't name the specific models or agent frameworks tested.

Why it matters: Lasso Security measured how watermarking LLM output changes agent behavior, covering accuracy, latency and output length, useful for anyone weighing the cost of deploying watermarks.

Sep 25Friday

Hacker News front page

NSA is spending billions this year to test frontier AI models, far above prior estimates

Two sources say the NSA told lawmakers in a classified briefing that it is spending billions in taxpayer money this year to evaluate and test advanced AI models. The figure is far higher than previously known, leading lawmakers to estimate a full federal AI regulatory system could cost tens of billions per year. Trump has mostly resisted stronger federal AI oversight. The article does not name which models are being tested, whose compute is used, or how the money breaks down.

Why it matters: The NSA's real spending on testing frontier AI models is far higher than outsiders knew, and it points to a costly federal oversight bill.

Sep 24Thursday

Hacker News front page

A daily-updated LLM value chart that plots price against intelligence to find the frontier

The site plots 420 models from the Artificial Analysis Intelligence Index against blended API price, drawing a value frontier where no cheaper model is smarter. Claude Opus 5.5 leads at $8/1M tokens with a 57.6 intelligence score. Meta's Muse Spark 1.3 tops the $2–$8 band at 48.1, Xiaomi's MiMo-V2.6-Pro wins $0.54–$2 at 46.3, and Z AI's GLM 5.3 Flash takes the under-$0.24 tier at 41.8. The post doesn't disclose how the intelligence index is built, and Coding/Math sub-scores are listed as empty for many models, so I'd hold off on those comparisons.

Why it matters: The page plots 420 models' Intelligence Index against blended API price, marks the value frontier, and lists the highest-scoring model for each budget band.

Hacker News front page

Paper Instruments releases Paper Office, a Python suite for agents to safely edit Word, PowerPoint, and Excel files

Paper Office is a suite of Python packages that wrap python-docx, python-pptx, and OpenPyxl with safety checks and broader editing capabilities. Across 5 models and 61 tasks, Paper packages plus guidance passed 92.5% of trials, vs 80.7% for the upstream packages alone and 69.5% for Anthropic's Office skills. Agents resorted to raw OOXML editing in only 1.6% of Paper runs, compared to 78.7% without skills and 50.5% with Anthropic skills. The team argues that low agent adoption in consulting, law, and banking stems from tools that silently corrupt formatting, break references, or produce client-unready output. Paper Office keeps the familiar imports and adds cross-run text search, native Word redlines, comment threads, content controls, cross-document composition, and package-level diff saves, refusing unsafe operations instead of quietly breaking files.

Why it matters: Paper Instruments rebuilds python-docx, python-pptx and OpenPyxl into an Office editing toolkit for agents, passing 92.5% of 61 tasks against 80.7% for the upstream libraries.

Hacker News front page

Open-source prompt-injection detectors catch 0–1% of realistic AI agent attacks buried in tool output

This benchmark hides 629 AgentDojo injection attacks inside tool outputs and tests Regex vs. Meta Prompt Guard 2. Regex catches 0%, Prompt Guard 2 catches 1%. The attacks aren't sent directly to the model—they're buried in search results, email bodies, and similar tool responses, so current detectors are effectively blind. Code and reproduction steps are public; the post doesn't include comparisons with commercial detectors.

Why it matters: The buried-injections GitHub project tested 10 open-source detectors against 629 AgentDojo v1 attacks and found Regex caught 0% and Meta's Prompt Guard 2 caught 1% when the injection hides in tool output.

AI HOT (Curated Pool)

Claude Opus 5.5 tops Coding Agent Index, but per-task cost rises to $13.04

Artificial Analysis tested Claude Opus 5.5 under Claude Code max effort and it scored 66 on the Coding Agent Index, up from Opus 5's 60. All three subtests improved: Terminal-Bench 4.0 63.1%, DeepSWE v1.1 68.4%, SWE-Atlas-QnA 66.4%. The trade-off: per-task cost jumped from $3 to $13.04. The post doesn't break down how max effort drove the cost increase.

Why it matters: Artificial Analysis puts Claude Opus 5.5 at the top of its Coding Agent Index with 66 points, six above Opus 5, while cost per task rises to $13.04.

AI HOT (Curated Pool)

MiMo-V2.6-Pro, Claude Opus 5.5, GPT-6 Luna/Sol launch, shifting the intelligence–cost Pareto frontier

Artificial Analysis reports that four new models this week added 11 points on the Intelligence Index vs. cost-per-task Pareto frontier. GPT-6 Luna contributed 5 points, Claude Opus 5.5 contributed 4, and the remaining two came from MiMo-V2.6-Pro and GPT-6 Sol. The post doesn't disclose specific scores, pricing, or latency—hold off on conclusions until full benchmarks drop.

Why it matters: Artificial Analysis data shows four new models this week pushed the best frontier of intelligence index against cost per task out by eleven points, with GPT-6 Luna adding five and Claude Opus 5.5 adding four.

Sep 23Wednesday

OpenAI News

OpenAI releases MentalHealthBench, an open benchmark co-developed with 80+ licensed clinicians to evaluate AI in realistic mental health conversations

OpenAI open-sourced MentalHealthBench, a benchmark built with over 80 licensed psychologists and psychiatrists across 22 countries. It tests AI on realistic mental health conversations ranging from everyday stress to emergencies, covering adults, teens, and caregivers. The eval goes beyond safety filters: it checks whether models seek context, preserve user agency, and offer actionable guidance when appropriate. OpenAI stresses ChatGPT isn't a substitute for therapy, but the benchmark tracks progress on empathy and steering people toward real-world support. The paper and benchmark are publicly available.

Why it matters: OpenAI built MentalHealthBench with more than 80 licensed mental health experts, an open, expert-labeled benchmark for AI responses in realistic mental health conversations.

Latent Space

Claude Opus 5.5 launches with Fable 5.1-level performance at 40% lower cost, plus a rare focus on writing quality

Anthropic released Claude Opus 5.5, the first model in the new 5.5 family. It matches Claude Fable 5.1 on most tasks, costs 40% less to run than Opus 5, and is about 30% faster. The launch unusually highlights writing improvements: the model puts key info up front and follows user style rules. Artificial Analysis notes that token usage on frontier tasks jumped ~80%, so per-task cost remains around $6—similar to Opus 5. OpenAI shipped GPT-6 Sol and Luna an hour later at 50% lower prices than GPT-5.6, but Opus 5.5's launch post hit 17M views and dominated the day. Anthropic's system card also reports multi-agent scaling with up to 100 parallel agents for the first time. Latent Space tested both and switched to Opus 5.5 as the default model immediately, calling the writing quality a night-and-day difference over Sol 6.

Why it matters: Anthropic says Opus 5.5 matches Fable 5.1 on most tasks at 40% lower cost and 30% faster, while outside estimates put per-task cost near Opus 5.

Computing Life · Share · Yage

Same tool toggle: Nemotron-3 550B gained, Mistral-Medium-3.5 crashed

A new paper breaks down coding agent harnesses into three independent toggles and measures each one. The most striking result: switching from dedicated file tools to a pure CLI made Nemotron-3 550B's SWE-Bench Verified score jump 3.6 pp while cutting per-task cost from $2.33 to $1.11, but Mistral-Medium-3.5-128B dropped from 68.60% to 45.40%. Trajectory analysis shows 550B composing dense shell one-liners, while Mistral failed to locate files in 32.80% of tasks and submitted no edits. On Terminal-Bench 2.1, both models improved under CLI mode. Planning boosted the 30B model from 13.60% to 25.20% but only saved ~30% cost for larger models without accuracy gains. Context management mainly prevents window overflow; at 128k the gap shrinks to 2.7 pp, and complex read-back mechanisms were almost never invoked. The takeaway: no universal best harness design—it depends on the model's CLI fluency and the task type.

Why it matters: A controlled test shows the same harness change helps one coding model and badly hurts another, so harness tuning has to be per-model.

AI HOT (Curated Pool)

Arena launches GPT-6 Sol and GPT-6 Luna testing, scores coming soon

Arena is now testing two new OpenAI models, GPT-6 Sol and GPT-6 Luna, with scores not yet released. You can try them on real agent tasks and vote to feed the leaderboard. The post doesn't disclose model size, release date, or pricing.

Why it matters: Arena opened testing slots for OpenAI's GPT-6 Sol and GPT-6 Luna, with scores not yet published; users can run real agent tasks on both models and vote, and votes count toward the leaderboard.

AI HOT (Curated Pool)

OpenAI launches GPT-6 Sol and Luna, API pricing cut 50% vs GPT-5.6

OpenAI added two cheaper models to the GPT-6 family: Sol and Luna, with API prices halved across input and output. Sol costs $2/$10 per 1M tokens, Luna $0.10/$0.50. Sol scored 33.2% on AutomationBench at xhigh effort at 9% of Claude Opus 5's cost per task, and 56.4% on Agents' Last Exam at max effort at 60% lower cost. On internal factuality evals, Sol makes about half as many mistakes as its predecessor. The post does not specify a launch date beyond 'available now.'

Why it matters: Same Astra training recipe as the flagship, with Sol's input price halved from $4 to $2 per million tokens.

AI HOT (Curated Pool)

Claude Opus 5.5 lands on Arena's Agent Arena and Battle Mode

Anthropic's Claude Opus 5.5 is now available on Arena's Agent Arena, where users vote on rankings after the model runs real long-horizon agent tasks. The model can use web search, a file system, and a terminal; the leaderboard uses causal tracking to measure performance relative to the average model. The post doesn't spell out Battle Mode specifics or show example tasks.

Why it matters: Puts Claude Opus 5.5 in an agent harness with web access, file I/O and terminal commands, ranked by causal tracing.

AI HOT (Curated Pool)

Claude Opus 5.5 tops Artificial Analysis Intelligence Index with a score of 58, plus a 20% price cut

Claude Opus 5.5 scored 58 on the Artificial Analysis Intelligence Index, the highest measured so far. It leads on 6 of 10 evaluations, including Humanity's Last Exam at 61.4% and SciCode at 66.9%, and matches GPT-6 Astra (xhigh) on Terminal-Bench 4.0 at 59.6%. On the agentic knowledge-work eval AA-Briefcase, it hit 1822 Elo—143 points above Fable 5.1—and surpassed GPT-5.6 Sol on both analytical quality and presentation. Pricing dropped to $4/$20 per 1M input/output tokens (from $5/$25), with cache reads down 60% to $0.20. Output tokens per task grew ~60% vs Opus 5, so cost per task stayed flat. Context window remains 1M tokens with image and text input.

Why it matters: Opus 5.5 tops the Artificial Analysis Intelligence Index at 58 and leads six of ten evaluations, with prices down 20%.

Sep 22Tuesday

Hacker News front page

Prompting agents to iteratively optimize Rust until it beats state-of-the-art libraries

Max Woolf spent over a year testing whether agentic LLMs can iteratively optimize Rust code with a hard pass/fail rule: each iteration must deliver at least a 5% speedup or roll back. With Claude Opus 4.5 and later models, he got 2–20× speedups on algorithms like UMAP versus mature libraries, all without unsafe code. The post includes the exact prompts and benchmark results. The caveat: these gains are measured on his specific benchmarks and may not translate directly to production workloads.

Why it matters: Max Woolf published the prompts and benchmark results behind agent-driven Rust optimization, hitting 2x to 20x faster than mature libraries on algorithms like UMAP with unsafe code banned.

Computing Life · Share · Yage

Three AI Coding Stories, Three Numbers to Read

ZCode was caught silently packaging entire Git histories for cloud upload, with .git objects making up 86.6% of snapshots. HarnessTax benchmarked Claude Fable 5 across frameworks: Claude Code and minimal Pi achieved near-identical success rates but a 2x cost gap. A Microsoft architect replaced multi-model agent loops with a single model reading skill docs and calling tools directly—halving API calls but increasing total tokens by 22%. Each story unpacks one number and a reminder to check which layer a metric actually measures.

Why it matters: A single sample shows ZCode packaging the whole workspace, .git included, which can carry secrets and internal branch names to the cloud.

AI HOT (Curated Pool)

Xiaomi MiMo-V2.6-Pro hits ~10th on Code Arena WebDev, ~3rd among open-weight models

Xiaomi released two omni-modal models: MiMo-V2.6-Pro and Flash. The Pro version scored 1628 on Code Arena WebDev, up 153 points from MiMo-V2.5-Pro's 1475, landing around 10th overall and ~3rd among open-weight models under MIT license. The post doesn't disclose Flash's benchmark numbers or parameter counts.

Why it matters: Xiaomi's MiMo-V2.6-Pro scored 1628 on the Code Arena web development leaderboard, 153 points above the previous generation, around 10th overall and about 3rd among MIT-licensed open-weight models.

AI HOT (Curated Pool)

Xiaomi MiMo-V2.6-Pro tops open-weight model intelligence index

Xiaomi released MiMo-V2.6-Pro, scoring 46 on the Artificial Analysis Intelligence Index—up from 26 for the previous V2.5-Pro. It's now the highest among open-weight models. The post doesn't disclose parameter count, architecture details, or a release timeline.

Why it matters: Xiaomi's MiMo-V2.6-Pro scored 46 on Artificial Analysis's open-weight intelligence index, up from 26 for V2.5-Pro, the highest score among open-weight models so far.

AI HOT (Curated Pool)

Xiaomi MiMo-V2.6-Pro tops open-weight model intelligence index

Xiaomi MiMo-V2.6-Pro scored 46 on the Artificial Analysis Intelligence Index, the highest among open-weight models. The previous MiMo-V2.5-Pro scored 26. The post doesn't disclose model size, training data, or release license, so hold for details.

Why it matters: Xiaomi's open-weight MiMo-V2.6-Pro scores 46 on the Artificial Analysis Intelligence Index, up from 26 for MiMo-V2.5-Pro and the highest of any open-weight model.

Sep 21Monday

AI HOT (Curated Pool)

Fireworks AI launches FireRouter: the frontier isn't a model, it's a router

Fireworks AI benchmarked 18 models on DeepSWE: picking the right model per task beats any single model. GPT-6 Astra alone scores 74.1% at $6.52/task. An oracle router across all 18 hits 97.6% at $1.88. Open-weight models alone reach 90.3% at $1.45. 94 of 113 tasks need a model under $3; the three priciest models are the best pick on only 3 tasks. FireRouter aims to make that per-task choice before the work starts—the post doesn't yet detail how.

Why it matters: Fireworks AI released FireRouter, a task-level router that picks between open and closed models with cache awareness; its DeepSWE tests show per-task model choice can raise solve rates and cut cost at once.

Sep 20Sunday

Hacker News front page

Prompts Aren't Real: Build Evaluation Pipelines Instead

Dan McKinley argues that prompt engineering is a distraction. Building consumer-facing agents taught him that even structured output fails on a fraction of requests—models will flood a field with nonsense. His fix was renaming a field from 'title' to 'heading,' which he calls deranged. The talk pushes for pass^k testing and evaluation pipelines to constrain behavior, since prompts alone can't tame the beast. The post is a slide deck; it names no specific eval frameworks or metrics.

Why it matters: Engineer Dan McKinley argues in a talk that prompts cannot be maintained as standalone units, because new instructions enter a context different from the one seen at test time and change behavior.

Sep 19Saturday

Hacker News front page

Brood War Bench: No model played beyond beginner level in StarCraft

Ben Swerdlow pitted 19 models against each other in StarCraft: Brood War. Codex Astra / xhigh went 18–0, but no model surpassed beginner level. Older models treated the RTS as turn-based and got destroyed while thinking; Grok 4.6 issued only 6 command batches in 43 minutes and never fielded a combat unit. Claude Fable earnestly climbed the tech tree but couldn't execute. Codex models favored early Probe harassment that paralyzed opponents. The post doesn't specify whether matches were pure AI vs AI or involved human input.

Why it matters: Ben Swerdlow ran 19 models and effort settings head to head in StarCraft: Brood War; Codex Astra at xhigh went 18-0, and no model played above a beginner level.

Sep 17Thursday

Hacker News front page

Coding agent harnesses can 2× your cost with no real accuracy gain

UC Berkeley and Arena researchers tested 7 models across 3 harnesses—Claude Code, Codex CLI, and Pi—on 30 tasks each from SWE-bench Lite and Terminal-Bench 2.0, with 3 repetitions per pair. Harness choice barely moves success rates (±2–5%), but cost can vary up to 5×. Claude Fable 5 hits 97.8% in Claude Code at $1.33, and 96.7% in Pi at $0.67. Pi, a minimal open-source harness with just read, write, edit, and bash, reaches the Pareto frontier on both benchmarks. The post doesn't spell out the exact open-source licenses for Pi and Codex CLI, and doesn't link the full pricing sheet.

Why it matters: A controlled test across 7 models and 3 coding agent harnesses shows the harness barely moves task success while cost for the same model can vary up to 5x.

Hacker News front page

Frontier models are much better at physics than benchmarks suggest—expert re-grading shows why

Researchers at Yale and other institutions had physics faculty and PhDs re-grade six widely used physics benchmarks. Most answers previously marked wrong turned out to be grader errors, incorrect reference solutions, or ambiguous questions. For GPT-5.6-Sol, corrected mean@4 jumped from 47.3% to 78.7% on HLE-Physics and from 61.0% to 87.2% on CMT-Benchmark. The near-saturation on these closed-ended tasks signals an urgent need for harder, expert-validated evaluations.

Why it matters: Researchers at Yale and elsewhere had physics experts re-grade six common physics benchmarks and found most wrong answers came from the scoring code, reference answers or question wording, not from model reasoning.