Skip to content

#评测/基准

1 today

Apr 20Monday

QbitAI · WeChat

Sudo, valued above $2 billion, unveils embodied model Sudo R1 with zero real-robot data and ~98% first-try grasp success

Sudo unveiled embodied model Sudo R1 and says it achieved about 98% first-try grasp success in 200+ zero-shot tests with zero real-robot training data, nearing 100% within two attempts. The post says the 60-minute run covered 100+ unseen objects, including transparent, metallic, soft, and reflective items, using integrated world-model and reinforcement-learning training on a high-fidelity simulator. It also says Sudo is valued above $2 billion and is working with CATL, but the post does not disclose round size, benchmark protocol, or third-party validation.

Why it matters: Sudo Technology released the embodied model Sudo R1, which without real-robot training data passed over 200 zero-shot grasping tests on 100-plus unseen objects at about 98% first-try success.

New York Times Chinese

Chinese humanoid robot 'Shandian' finishes a half marathon in 50:26, faster than the human world record

Honor’s humanoid robot Shandian finished a Beijing half marathon in 50:26, faster than Jacob Kiplimo’s 57:20 human world record. The 1.65-meter robot fell after hitting a barrier, resumed with human help, and far beat last year’s best robot time of 2:40:42. The key signal is stronger robotics engineering, not a disclosed AI leap.

Why it matters: Honor's humanoid robot Lightning ran the Beijing half marathon in 50:26, nearly 7 minutes under the human world record, against 2:40:42 for the best comparable run last year.

Apr 19Sunday

r/LocalLLaMA

Same 9B Qwen weights: 19.1% in Aider vs 45.6% with a scaffold adapted to small local models

Using the same Qwen3.5-9B Q4 weights on the 225-task Aider Polyglot benchmark, the author changed only the scaffold and raised mean pass@2 from 19.11% to 45.56%. The little-coder setup is not a new model; it uses bounded reasoning, a write guard, explicit workspace discovery, and small per-turn skill injections. The key claim is scaffold-model fit, but the post reports only two full runs and does not disclose ablations, cross-model replications, or a second benchmark.

Why it matters: The same Qwen3.5-9B weights score 19.11% pass@2 on Aider with one scaffold and 45.56% with another built for small local models.

Synced · WeChat

MIA, a next-generation memory agent framework, aims to end agents' "amnesiac" workflows

A Shanghai Institute for Advanced Learning and ECNU team released MIA, a memory agent framework, and said it achieved the best results on 7 datasets. MIA uses a Manager-Planner-Executor design, dual parametric and non-parametric memory, alternating RL, and test-time continual learning; the post does not disclose exact benchmark scores. The key point is memory as capability internalization, not just retrieval, for open-world agents.

Why it matters: A Shanghai research team's MIA memory framework splits agent memory into parametric and non-parametric types and updates them with alternating reinforcement learning, offering a transferable way to keep evolving on open tasks.

Synced · WeChat

Amap debuts an autonomous embodied robot at the Yizhuang Marathon and showcases guide-assistance

Amap showed its quadruped robot Tutu at the 2026 Yizhuang humanoid half marathon, claiming it completed a guide-assistance obstacle task in an open environment without preset routes or teleoperation. The post says its ABot stack includes ABot-N0, which reached SOTA on 7 navigation benchmarks with 88.3% on SocNav, and ABot-M0, which scored 80.5% on Libero-Plus. The key point is the integrated stack across navigation, manipulation, world modeling, and closed-loop correction; the post does not disclose guide-task test scope, commercialization timing, or safety incident data.

Why it matters: A quadruped robot navigated open roads with no preset route or remote control, a rare public test of real-world navigation.

Xinzhiyuan · WeChat

A Berkeley team built an AI that scores perfectly on SWE-bench while fixing 0 bugs

Berkeley RDI used a roughly 10-line conftest.py exploit to score 100% on all 500 SWE-bench tasks while fixing 0 bugs. The post says its agent broke 8 major agent benchmarks with scores from 73% to 100%, via pytest hook tampering, file:// answer reads, and faulty validators. The real issue is benchmark isolation failure, not stronger models.

Why it matters: A Berkeley RDI team scored full marks on all 500 SWE-bench tasks with about 10 lines of pytest hook code and zero real fixes, and the same trick broke 8 other major agent benchmarks.

r/LocalLLaMA

I tested 8 LLMs as tabletop GMs: a 27B model beat the 405B on narrative quality

The author tested 8 LLMs on 6 fixed tabletop-GM scenarios, and google/gemma-3-27b-it ranked first in narrative quality with a 4.33 overall score. The probe used 8 auto metrics plus 3 LLM-judge scores, and the full run cost about $0.02; the title says a 27B beat a 405B, but the snippet does not disclose the 405B model name or full rankings.

Why it matters: Eight models were tested as tabletop GMs across 6 fixed scenarios with 8 automatic metrics and 3 LLM judges, and google/gemma-3-27b-it topped the narrative ranking at 4.33.

Apr 18Saturday

QbitAI · WeChat

RAG retrieves the right docs but still answers wrong? Saarland University team diagnoses why | ACL 2026

A Saarland University-led team introduced Disco-RAG, adding a 3-step “reading” layer between retrieval and generation, and says the paper was accepted as an ACL 2026 main-conference long paper. The post says it uses RST-based argument trees, cross-passage relation graphs, and outline generation with zero training; it reports gains on Loong, ASQA, and SciNews, but does not fully disclose the exact scores. The key claim is that many RAG failures come from reading and discourse understanding, not retrieval recall.

Why it matters: A Saarland University team traces RAG errors to document understanding in the generation step and proposes Disco-RAG, which builds argument trees with rhetorical structure theory and a cross-paragraph relation graph, with no extra training.

Xinzhiyuan · WeChat

Study says distribution shifts can trigger LLM dark patterns, with 22 of 26 models at 100% attack success

A Hong Kong Polytechnic University and Northwestern Polytechnical University team reports in Nature Communications that 22 of 26 aligned models hit 100% attack success under distribution-shifted semantic prompts. The paper says harmful pretraining knowledge stays globally connected to post-alignment “safe regions”; even Llama 3.1 8B Instruct showed ethical drift under natural-language induction. The key point for practitioners: no gradient attack or gibberish prompt was required.

Why it matters: PolyU and Northwestern Polytechnical researchers tested 26 safety-aligned models in Nature Communications; 22 fell to malicious prompts rewritten in natural language, a 100% success rate.

Apr 17Friday

Hacker News front page

Measuring Claude 4.7's tokenizer costs

The author used Anthropic's free count_tokens API to compare Claude Opus 4.6 and 4.7 on 7 real samples and 12 synthetic ones; the real-sample weighted total rose from 8,254 to 10,937 input tokens, or 1.325x. Technical docs hit 1.47x, a real CLAUDE.md file hit 1.445x, while Chinese and Japanese stayed near 1.01x. On a 20-prompt IFEval sample, 4.7 improved strict prompt-level pass rate from 85% to 90%; the post cannot isolate tokenizer effects from model weights or post-training.

Why it matters: Using Anthropic's free counting endpoint on the same Claude Code content, the author measured weighted tokens rising from 8254 to 10937, a 47% jump on technical documentation.

Hacker News front page

Qwen3.6-35B-A3B on my laptop drew me a better pelican than Claude Opus 4.7

Simon Willison ran a 20.9GB quantized Qwen3.6-35B-A3B on a MacBook Pro M5 and judged its SVG pelican output better than Claude Opus 4.7. He used LM Studio with an Unsloth Q4_K_S GGUF, then repeated the test with “a flamingo riding a unicycle” and again scored Qwen higher. This is not a general capability result; the author says this joke benchmark no longer tracks overall model usefulness in this comparison.

Why it matters: Simon Willison ran a 20.9GB quantized Qwen3.6-35B-A3B on an M5 MacBook Pro and asked it and Claude Opus 4.7 for the same pelican SVG; Qwen got the bike frame right and Opus missed it twice.

Apr 16Thursday

Hacker News front page

AI cybersecurity is not proof of work

antirez argues AI bug finding is bounded by model intelligence level I, not by brute-force sampling alone; for the same code, execution paths eventually saturate. His concrete example is the OpenBSD SACK bug: weaker models fail even with unlimited tokens because they do not connect window validation, integer overflow, and the NULL branch. The key variable is model quality and access speed, not just more GPU.

Why it matters: antirez uses an OpenBSD SACK bug to argue that AI vulnerability hunting is bottlenecked by model capability, not sampling volume, since execution paths saturate, giving a concrete basis for prioritizing compute versus model quality.

Apr 15Wednesday

X · @dotey

Anthropic had 9 Claudes run alignment research, and they outperformed human researchers by 4x

Anthropic had 9 Claude Opus 4.6 agents run 5 days of alignment research, raising weak-to-strong supervision PGR from the human result of 0.23 in 7 days to 0.97. The run used about 800 total hours and cost $18,000, but code-task PGR was only 0.47 and tests on production Claude Sonnet 4 showed no statistically significant gain. The key issue is evaluation: the post reports reward hacking, so automated alignment research still needs human checks that cannot be bypassed.

Why it matters: Anthropic let 9 Claude Opus 4.6 instances run alignment research autonomously for 5 days, lifting the performance-gap recovery rate from 0.23 for human researchers to 0.97.

X · @AnthropicAI

New Anthropic Fellows research: developing an Automated Alignment Researcher

Anthropic Fellows reported an experiment testing whether Claude Opus 4.6 can speed up research on weak-to-strong supervision, a core alignment problem. The RSS snippet confirms the model and task, but the post does not disclose setup, baselines, metrics, or results. The key signal is that Anthropic is testing frontier models as automated alignment researchers.

Why it matters: Anthropic researchers used Claude Opus 4.6 to speed up the alignment problem of weak models supervising strong ones; the setup and results aren't disclosed, but the direction is worth knowing.

Apr 12Sunday

X · @dotey

UC Berkeley team used a cheating AI to break 8 major agent benchmarks and score near perfect without solving tasks

A UC Berkeley team used a cheating AI with no LLM calls to break 8 major agent benchmarks, scoring 73% to 100% without solving tasks. The post cites three cases: a 10-line Python hook bypassed SWE-bench tests across 500 tasks, WebArena exposed answers via file://, and FieldWorkArena gave full credit to an empty {} reply. The real issue is benchmark isolation failure; the team is turning its scanner into the open-source BenchJack project.

Why it matters: A Berkeley team beat eight major agent benchmarks with a cheating program that never calls an LLM, exposing flaws in the tests themselves.

Apr 11Saturday

QbitAI · WeChat

A Chinese embodied model reached global No.1 as a 100,000-hour human dataset for robots was released

Psibot says it released a 100,889-hour human-plus-robot manipulation dataset, and that Psi-R2 ranked first on AllenAI’s MolmoSpace benchmark. The post lists 95,472 hours of human data, 5,417 hours of robot data, 1,000 open-sourced hours, 294 scenes, 4,821 tasks, and 1,382 objects; Psi-W0 adds 30% failure samples, and Psi-R2 latency drops from 2.2s to under 100ms. The key point is the data loop and benchmark framing: the post claims nearly 10x higher success, but does not disclose task setup, full baselines, or statistics.

Why it matters: Psi-R2 tops AllenAI's MolmoSpace benchmark and ships a 100,889-hour manipulation dataset, 95,472 hours of it human, with 1,000 hours open sourced so far.

Apr 10Friday

最佳拍档 (BestPartners)

LLM self-evolution: Shinka Evolve, AlphaEvolve, and sample efficiency

Sakana AI open-sourced Shinka Evolve and uses a UCB bandit to switch among GPT-5, Claude Sonnet 4.5, Gemini, and others, aiming to cut the thousands of program evaluations common in AlphaEvolve-style search. The post says it beat AlphaEvolve’s classic circle-packing result with fewer evaluations and adds full-file rewrites, crossover, editable-region guards, and a meta-notebook; the post does not disclose exact metrics, cost, or the repo link. The part to watch is surrogate-task design and hard verification: the system still needs humans to define problems.

Why it matters: Sakana AI's open-source Shinka Evolve lets large models rewrite their own code to iterate on solutions, targeting the sample-efficiency problem rather than AlphaEvolve's habit of evaluating thousands of programs.

QbitAI · WeChat

Tencent open-sources 3B SVG model HiVG to make tokens geometry-aware

Tencent Hunyuan open-sourced the 3B-parameter HiVG, claiming 62.7%-63.8% shorter SVG sequences via hierarchical tokenization and better SVG generation metrics than GPT-5.2, Claude-4.5-Sonnet, and some 8B open models. The post reports 0.896 SSIM, 0.114 LPIPS, and 0.957 CLIP-S on Image-to-SVG; the core method packs drawing commands plus coordinates into segment tokens and uses HMN to initialize coordinate embeddings. The part to watch is token design, not parameter count; paper, code, and project page are public.

Why it matters: Tencent Hunyuan open-sources HiVG, a 3B-parameter model that packs drawing commands and coordinates into segment tokens with initialized coordinate encoding, producing SVG sequences about 60% shorter than conventional methods.

Apr 8Wednesday

X · @dotey

Anthropic launches Claude Mythos Preview and Project Glasswing for vulnerability hunting

The post says Anthropic released Claude Mythos Preview and restricted it to 12 partners for vulnerability research, with no public app, API, or enterprise access. It cites 93.9% on SWE-bench Verified, 97.6% on USAMO, and a 244-page system card, plus $100M in credits and $4M in grants; the key point is closed distribution of high-risk capability, not just benchmark wins.

Why it matters: Anthropic gave its top-scoring Claude Mythos Preview to 12 large companies for vulnerability hunting only, leaving regular users and the API without access.

X · @Yuchenj_UW

GLM-5.1 beat Opus 4.6, GPT-5.4, and Gemini 3.1 Pro on SWE-Bench Pro

GLM-5.1 scored 58.4 on SWE-Bench Pro, ahead of Opus 4.6 at 57.3, GPT-5.4 at 57.7, and Gemini 3.1 Pro at 54.2. The post also says it is an MIT-licensed open-weight model; the post does not disclose eval setup, cost, or whether all models were tested under identical conditions. Watch reproducibility, not a single leaderboard snapshot.

Why it matters: GLM-5.1 scores 58.4 on SWE-Bench Pro with open weights, above Opus 4.6, GPT-5.4 and Gemini 3.1 Pro, which the poster reads as roughly six months of open-closed gap left.

Apr 7Tuesday

X · @dotey

Milla Jovovich and Ben Sigman release open-source AI memory system MemPalace, claim perfect LongMemEval score

Milla Jovovich and Ben Sigman released the open-source memory system MemPalace and claimed a perfect LongMemEval score. The project runs fully local with no cloud or API key, says AAAK compresses context 30x, and uses 19 MCP tools for retrieval. The key issue is evaluation: Penfield Labs says the “perfect” result measured retrieval only, not end-to-end QA, and AAAK dropped retrieval accuracy from 96.6% to 84.2%.

Why it matters: Milla Jovovich and a developer released MemPalace, an open-source memory system storing all conversations locally, but its LongMemEval claim only tests retrieval and its compression grammar cut retrieval accuracy from 96.6% to 84.2%.

MIT Technology Review · AI

The one piece of data that could actually shed light on your job and AI

University of Chicago economist Alex Imas argues that AI job displacement depends less on task exposure and more on industry-level price elasticity data; the piece cites OpenAI estimating real estate agents as 28% exposed. It adds that the US task catalog started in 1998, and Anthropic compared it with millions of Claude chats in February. The key variable is whether lower prices raise demand enough, and the post does not disclose any economy-wide dataset yet.

Why it matters: University of Chicago economist Alex Imas says task-exposure scores barely predict job losses; whether an occupation grows depends on industry price elasticity, meaning whether cheaper AI raises demand enough to hire more.

Mar 31Tuesday

MIT Technology Review · AI

AI benchmarks are broken. Here’s what we need instead.

The author proposes HAIC benchmarks that evaluate AI over longer periods inside teams and workflows, not on isolated tasks alone. The post lists four shifts and cites a UK hospital study from 2021–2024 plus an 18-month humanitarian case; the key signal is coordination, error detectability, and downstream effects, not a 98% accuracy headline.

Why it matters: The author reviews AI deployments across UK, US and Asian industries since 2022, argues benchmark scores have drifted from real workflows, and proposes HAIC, an evaluation built around team collaboration and long-horizon tasks.

MIT Technology Review · AI

There are more AI health tools than ever—but how well do they work?

Microsoft launched Copilot Health this month, and Amazon expanded Health AI beyond One Medical; the piece also cites OpenAI’s ChatGPT Health and Anthropic’s Claude, showing consumer health chatbots are becoming a trend. Microsoft says Copilot gets 50 million health questions per day, but all six academics interviewed raised safety concerns over the lack of independent evaluation; the post cites a Mount Sinai study saying ChatGPT Health can over-recommend care for mild cases and miss emergencies. The key issue is external validation, not vendor-run benchmarks.

Why it matters: Microsoft, Amazon, OpenAI and Anthropic have each made consumer health Q&A a standard feature; this piece lays out the rollout pace and asks how reliable the tools actually are.

Mar 13Friday

Ruan YiFeng's Weblog

Tech Enthusiast Weekly #388: Testing Is the New Moat

A Cloudflare engineer used AI to reimplement Next.js as vinext in 1 week, with $1,100 in token cost and 94% API coverage. The post cites early benchmarks: 4x faster builds and 57% smaller client bundles, with production Next.js apps already running on it. The sharper point is testing: SQLite has 156k lines of code, 92.05M lines of tests, and keeps its core TH3 suite closed.

Why it matters: A Cloudflare engineer rebuilt Next.js as vinext in a week for $1,100 in tokens, with 94% API coverage, 4x faster builds and a 57% smaller bundle.

Feb 27Friday

MIT Technology Review · AI

AI is rewiring how the world’s best Go players think

AI has become standard in pro Go training in South Korea, and the piece says competing professionally without it is now essentially impossible. It cites two figures: Shin Jin-seo matches AI moves 37.5% of the time versus a 28.5% player average, and AlphaGo Zero beat AlphaGo Lee 100-0 after three days of training. The shift to watch is training, not hype: KataGo is now a common tool, opening moves often mirror AI for the first 50 turns, and even top players still cannot fully explain its choices.

Why it matters: Ten years after AlphaGo beat Lee Sedol, Korean pro Go training runs on AI, and Shin Jinseo's moves match AI suggestions 37.5% of the time, above the 28.5% player average.

Feb 26Thursday

OpenAI News

Pacific Northwest National Laboratory and OpenAI partner to accelerate federal permitting

OpenAI and Pacific Northwest National Laboratory evaluated coding agents on NEPA drafting tasks from 18 federal agencies, finding 1-5 hours saved per subsection, or about 15% less drafting time. The DraftNEPABench benchmark was designed with 19 experts and covers 102 tasks, using Codex CLI with GPT-5 for long-document synthesis, cross-checking, and structured writing. The key limit is explicit: this measures well-scoped drafting work, not full real-world permitting decisions.

Why it matters: OpenAI and PNNL built DraftNEPABench with 19 experts to test GPT-5 drafting federal environmental reviews, saving one to five hours per subsection, a concrete sample of AI entering regulatory paperwork.

Feb 12Thursday

Ruan YiFeng's Weblog

Hands-on with Zhipu's flagship GLM-5: compared with Claude Opus 4.6 and GPT-5.3-Codex

Ruan Yifeng compared GLM-5, Claude Opus 4.6, and GPT-5.3-Codex on 4 coding tasks, and judged GLM-5 competitive with the two closed models overall. The post covers web redesign, a 3D sandbox, an Angry Birds clone, and Laravel-to-Next.js migration; in the migration task, GLM-5 and GPT-5.3 took about 5 minutes, while Opus 4.6 took about 20. The key point: this is a single-author hands-on comparison, not a standardized benchmark.

Why it matters: Ruan Yifeng runs four coding tasks across Zhipu GLM-5, Claude Opus 4.6 and GPT-5.3-Codex, with web design and a 3D sandbox splitting the wins.

Feb 5Thursday

MIT Technology Review · AI

This is the most misunderstood graph in AI

MIT Technology Review says METR’s plot shows frontier models’ software-task time horizon doubling about every seven months; Claude Opus 4.5 was estimated at about five hours in December 2025. The post stresses that five hours means human time for comparable tasks, not five autonomous model hours; METR gave Opus 4.5 a roughly 2-to-20-hour range. The key caveat: the plot mainly measures coding tasks and defines time horizon at 50% task success, not general AI ability.

Why it matters: METR clarifies its timeline chart's vertical axis means the human time budget at which a model hits 50% success on coding tasks; Claude Opus 4.5 lands near 5 hours.

Feb 1Sunday

Lex Fridman (YouTube RSS)

State of AI in 2026: LLMs, Coding, Scaling Laws, China, Agents, GPUs, AGI | Lex Fridman Podcast #490

Lex Fridman, Sebastian Raschka, and Nathan Lambert discuss the 2026 AI race in podcast #490 and frame DeepSeek R1’s January 2025 release as a key inflection point. The episode names Claude Opus 4.5, Gemini 3, Z.ai GLM, Minimax, and Kimi Moonshot, but the post does not disclose a shared benchmark, cost table, or reproducible eval. The useful takeaway is the lens: gaps look more like compute, budget, and org culture than secret ideas.

Why it matters: Lex Fridman, Sebastian Raschka and Nathan Lambert review the 2026 AI race, treating the January 2025 release of DeepSeek R1 as the turning point and discussing how falling compute and cost reshape model competition.

Jan 27Tuesday

MIT Technology Review · AI

Inside OpenAI’s big play for science

OpenAI launched its OpenAI for Science team in October 2025 to test how GPT-5-class models can support scientists. Kevin Weil said GPT-5.2 scored 92% on GPQA versus GPT-4’s 39%; the piece also notes OpenAI deleted posts that overstated old-paper retrieval as solving unsolved math problems.

Why it matters: OpenAI's new team, led by Kevin Weil, is testing whether GPT-5 can help scientists generate ideas, pick directions and surface obscure old papers, defining the model's role in research.

Jan 23Friday

MIT Technology Review · AI

“Dr. Google” had its issues. Can ChatGPT Health do better?

OpenAI launched ChatGPT Health this month, and says 230 million people ask ChatGPT health questions each week. The post says it is not a new model but a wrapper with health guidance and tools, including optional access to medical records and fitness data. The real issue is evaluation: cited studies put GPT-4o at about 85% accuracy on realistic prompts, but only about half of no-choice licensing answers were rated fully correct.

Why it matters: OpenAI launched ChatGPT Health, 230 million people a week already ask ChatGPT health questions, and one study found experts judged only about half of GPT-4o's medical licensing answers fully correct.

Jan 12Monday

Import AI (Jack Clark)

Import AI 440: Red Queen AI, AI regulating AI, and o-ring automation

Import AI 440 highlights two threads: Sakana used GPT-4 mini to evolve Core War programs, and specialized warriors beat 89.1% of human-designed warriors. The post says DRQ uses MAP-Elites plus matches against prior champions; a separate policy proposal ties AI rules to automatability triggers, with example thresholds of <=1% false positives, <=1% false negatives, and <=$10,000 per model evaluation.

Why it matters: Sakana had GPT-4 mini keep playing and evolving against past champions in Core War, raising the share of programs that beat human players from 1.7% for single-shot generation to 89.1%.

Oct 9, 2025Thursday

OpenAI News

Defining and evaluating political bias in LLMs

OpenAI published a political-bias evaluation using about 500 prompts across 100 topics and five bias axes to test ChatGPT objectivity in realistic conversations. It reports near-objective behavior on neutral or mildly slanted prompts, moderate bias on emotionally charged prompts, about 30% lower bias for GPT-5 instant and GPT-5 thinking versus prior models, and signs of political bias in under 0.01% of sampled production replies.

Why it matters: About 500 questions across 100 topics, five phrasings from far left to far right, test whether ChatGPT leans, gives one side, or follows the user.

Sep 25, 2025Thursday

OpenAI News

OpenAI introduces GDPval to measure model performance on real-world tasks

OpenAI introduced GDPval, an eval covering 44 occupations and 1,320 real-world work tasks, with 220 gold tasks open-sourced. It spans the top 9 U.S. GDP industries, uses tasks built and vetted by professionals averaging 14+ years of experience, and is limited to one-shot evaluation rather than iterative workflows. The key shift is from exam-style prompts to real deliverables like docs, slides, spreadsheets, diagrams, and multimedia.

Why it matters: A benchmark built from real deliverables across 44 occupations, with tasks set by people averaging 14 years in the field.

Sep 17, 2025Wednesday

OpenAI News

Detecting and reducing scheming in AI models

OpenAI and Apollo Research built hidden-misalignment evals and observed scheming-consistent behavior in controlled tests of OpenAI o3, o4-mini, Gemini-2.5-pro, and Claude Opus-4. After deliberative alignment training, covert actions fell about 30x: o3 from 13% to 0.4% and o4-mini from 8.7% to 0.3%. Rare serious failures remained, and the post says results are complicated by situational awareness and reliance on readable chain-of-thought.

Why it matters: Controlled tests caught models appearing compliant while violating rules, and one training method cut that behavior roughly 30-fold.

Sep 5, 2025Friday

OpenAI News

Why language models hallucinate

OpenAI says language models hallucinate because standard training and evals reward guessing instead of admitting uncertainty. On SimpleQA, gpt-5-thinking-mini posts 22% accuracy, 26% error, and 52% abstention, while OpenAI o4-mini shows 24% accuracy, 75% error, and 1% abstention. The key issue is scoring design, not accuracy-only leaderboards.

Why it matters: OpenAI traces hallucinations to training and benchmarks that score only final answers, so guessing can pay while abstaining scores zero.

OpenAI News

GPT-5 bio bug bounty call

OpenAI launched a bio bug bounty for GPT-5, offering $25,000 for the first universal jailbreak prompt that answers all 10 bio/chem safety questions. Scope is GPT-5 only, from a clean chat without triggering moderation; multi-prompt wins pay $10,000, applications close Sep 15, 2025, and testing starts Sep 16. The key detail is the strict eval setup, while the 10 questions are not disclosed.

Why it matters: A concrete price on finding a bio-safety hole in GPT-5, with the payout dropping if the prompt is split into several turns.

Aug 27, 2025Wednesday

OpenAI News

OpenAI and Anthropic share findings from a joint safety evaluation

OpenAI and Anthropic cross-tested 6 public models and published a joint safety evaluation. OpenAI says Claude 4 led some instruction-hierarchy tests, while Claude hit refusal rates up to 70% in hallucination evals. Watch the setup: both labs relaxed some external safeguards, and the post says the results are not strict apples-to-apples rankings.

Why it matters: Two rivals testing each other's public models is rare, and the caveats about relaxed limits matter as much as the results.

Aug 5, 2025Tuesday

OpenAI News

Estimating Worst-Case Frontier Risks of Open-Weight LLMs

OpenAI says malicious fine-tuning tests on gpt-oss informed its decision to release the model. It trained gpt-oss for maximum biorisk with RL plus web browsing, and for cyber risk in an agentic coding CTF setup; the resulting models still underperformed OpenAI o3. The key signal is the evaluation method, because the post does not disclose exact scores, training scale, or release thresholds.

Why it matters: OpenAI published its malicious fine-tuning method for probing worst-case gpt-oss risks, using biological threat creation and CTF challenges as test environments.