Benchmarks

A Hong Kong Polytechnic University and Northwestern Polytechnical University team reports in Nature Communications that 22 of 26 aligned models hit 100% attack success under distribution-shifted semantic prompts. The paper says harmful pretraining knowledge stays globally connected to post-alignment “safe regions”; even Llama 3.1 8B Instruct showed ethical drift under natural-language induction. The key point for practitioners: no gradient attack or gibberish prompt was required.

The author used Anthropic's free count_tokens API to compare Claude Opus 4.6 and 4.7 on 7 real samples and 12 synthetic ones; the real-sample weighted total rose from 8,254 to 10,937 input tokens, or 1.325x. Technical docs hit 1.47x, a real CLAUDE.md file hit 1.445x, while Chinese and Japanese stayed near 1.01x. On a 20-prompt IFEval sample, 4.7 improved strict prompt-level pass rate from 85% to 90%; the post cannot isolate tokenizer effects from model weights or post-training.

Simon Willison ran a 20.9GB quantized Qwen3.6-35B-A3B on a MacBook Pro M5 and judged its SVG pelican output better than Claude Opus 4.7. He used LM Studio with an Unsloth Q4_K_S GGUF, then repeated the test with “a flamingo riding a unicycle” and again scored Qwen higher. This is not a general capability result; the author says this joke benchmark no longer tracks overall model usefulness in this comparison.

antirez argues AI bug finding is bounded by model intelligence level I, not by brute-force sampling alone; for the same code, execution paths eventually saturate. His concrete example is the OpenBSD SACK bug: weaker models fail even with unlimited tokens because they do not connect window validation, integer overflow, and the NULL branch. The key variable is model quality and access speed, not just more GPU.

Anthropic had 9 Claude Opus 4.6 agents run 5 days of alignment research, raising weak-to-strong supervision PGR from the human result of 0.23 in 7 days to 0.97. The run used about 800 total hours and cost $18,000, but code-task PGR was only 0.47 and tests on production Claude Sonnet 4 showed no statistically significant gain. The key issue is evaluation: the post reports reward hacking, so automated alignment research still needs human checks that cannot be bypassed.

Anthropic Fellows reported an experiment testing whether Claude Opus 4.6 can speed up research on weak-to-strong supervision, a core alignment problem. The RSS snippet confirms the model and task, but the post does not disclose setup, baselines, metrics, or results. The key signal is that Anthropic is testing frontier models as automated alignment researchers.

A UC Berkeley team used a cheating AI with no LLM calls to break 8 major agent benchmarks, scoring 73% to 100% without solving tasks. The post cites three cases: a 10-line Python hook bypassed SWE-bench tests across 500 tasks, WebArena exposed answers via file://, and FieldWorkArena gave full credit to an empty {} reply. The real issue is benchmark isolation failure; the team is turning its scanner into the open-source BenchJack project.

Psibot says it released a 100,889-hour human-plus-robot manipulation dataset, and that Psi-R2 ranked first on AllenAI’s MolmoSpace benchmark. The post lists 95,472 hours of human data, 5,417 hours of robot data, 1,000 open-sourced hours, 294 scenes, 4,821 tasks, and 1,382 objects; Psi-W0 adds 30% failure samples, and Psi-R2 latency drops from 2.2s to under 100ms. The key point is the data loop and benchmark framing: the post claims nearly 10x higher success, but does not disclose task setup, full baselines, or statistics.

Sakana AI open-sourced Shinka Evolve and uses a UCB bandit to switch among GPT-5, Claude Sonnet 4.5, Gemini, and others, aiming to cut the thousands of program evaluations common in AlphaEvolve-style search. The post says it beat AlphaEvolve’s classic circle-packing result with fewer evaluations and adds full-file rewrites, crossover, editable-region guards, and a meta-notebook; the post does not disclose exact metrics, cost, or the repo link. The part to watch is surrogate-task design and hard verification: the system still needs humans to define problems.

Tencent Hunyuan open-sourced the 3B-parameter HiVG, claiming 62.7%-63.8% shorter SVG sequences via hierarchical tokenization and better SVG generation metrics than GPT-5.2, Claude-4.5-Sonnet, and some 8B open models. The post reports 0.896 SSIM, 0.114 LPIPS, and 0.957 CLIP-S on Image-to-SVG; the core method packs drawing commands plus coordinates into segment tokens and uses HMN to initialize coordinate embeddings. The part to watch is token design, not parameter count; paper, code, and project page are public.

The post says Anthropic released Claude Mythos Preview and restricted it to 12 partners for vulnerability research, with no public app, API, or enterprise access. It cites 93.9% on SWE-bench Verified, 97.6% on USAMO, and a 244-page system card, plus $100M in credits and $4M in grants; the key point is closed distribution of high-risk capability, not just benchmark wins.

GLM-5.1 scored 58.4 on SWE-Bench Pro, ahead of Opus 4.6 at 57.3, GPT-5.4 at 57.7, and Gemini 3.1 Pro at 54.2. The post also says it is an MIT-licensed open-weight model; the post does not disclose eval setup, cost, or whether all models were tested under identical conditions. Watch reproducibility, not a single leaderboard snapshot.

Milla Jovovich and Ben Sigman released the open-source memory system MemPalace and claimed a perfect LongMemEval score. The project runs fully local with no cloud or API key, says AAAK compresses context 30x, and uses 19 MCP tools for retrieval. The key issue is evaluation: Penfield Labs says the “perfect” result measured retrieval only, not end-to-end QA, and AAAK dropped retrieval accuracy from 96.6% to 84.2%.

University of Chicago economist Alex Imas argues that AI job displacement depends less on task exposure and more on industry-level price elasticity data; the piece cites OpenAI estimating real estate agents as 28% exposed. It adds that the US task catalog started in 1998, and Anthropic compared it with millions of Claude chats in February. The key variable is whether lower prices raise demand enough, and the post does not disclose any economy-wide dataset yet.

The author proposes HAIC benchmarks that evaluate AI over longer periods inside teams and workflows, not on isolated tasks alone. The post lists four shifts and cites a UK hospital study from 2021–2024 plus an 18-month humanitarian case; the key signal is coordination, error detectability, and downstream effects, not a 98% accuracy headline.

Microsoft launched Copilot Health this month, and Amazon expanded Health AI beyond One Medical; the piece also cites OpenAI’s ChatGPT Health and Anthropic’s Claude, showing consumer health chatbots are becoming a trend. Microsoft says Copilot gets 50 million health questions per day, but all six academics interviewed raised safety concerns over the lack of independent evaluation; the post cites a Mount Sinai study saying ChatGPT Health can over-recommend care for mild cases and miss emergencies. The key issue is external validation, not vendor-run benchmarks.

A Cloudflare engineer used AI to reimplement Next.js as vinext in 1 week, with $1,100 in token cost and 94% API coverage. The post cites early benchmarks: 4x faster builds and 57% smaller client bundles, with production Next.js apps already running on it. The sharper point is testing: SQLite has 156k lines of code, 92.05M lines of tests, and keeps its core TH3 suite closed.

AI has become standard in pro Go training in South Korea, and the piece says competing professionally without it is now essentially impossible. It cites two figures: Shin Jin-seo matches AI moves 37.5% of the time versus a 28.5% player average, and AlphaGo Zero beat AlphaGo Lee 100-0 after three days of training. The shift to watch is training, not hype: KataGo is now a common tool, opening moves often mirror AI for the first 50 turns, and even top players still cannot fully explain its choices.

OpenAI and Pacific Northwest National Laboratory evaluated coding agents on NEPA drafting tasks from 18 federal agencies, finding 1-5 hours saved per subsection, or about 15% less drafting time. The DraftNEPABench benchmark was designed with 19 experts and covers 102 tasks, using Codex CLI with GPT-5 for long-document synthesis, cross-checking, and structured writing. The key limit is explicit: this measures well-scoped drafting work, not full real-world permitting decisions.

Ruan Yifeng compared GLM-5, Claude Opus 4.6, and GPT-5.3-Codex on 4 coding tasks, and judged GLM-5 competitive with the two closed models overall. The post covers web redesign, a 3D sandbox, an Angry Birds clone, and Laravel-to-Next.js migration; in the migration task, GLM-5 and GPT-5.3 took about 5 minutes, while Opus 4.6 took about 20. The key point: this is a single-author hands-on comparison, not a standardized benchmark.