#安全/对齐

9 today

Aug 13, 2024Tuesday

OpenAI NewsAug 13, 18:00

Introducing SWE-bench Verified

OpenAI released SWE-bench Verified, a human-validated subset built with the benchmark’s authors to assess real software issue resolution more reliably. The post names 3 failure modes in SWE-bench: overly narrow tests, underspecified issue statements, and unreliable environment setup; as of Aug. 5, 2024, top agents scored about 20% on SWE-bench and 43% on SWE-bench Lite. The key point is that the original benchmark can systematically underestimate coding-agent ability.

Why it matters: This is a strong benchmark release, not a routine post: OpenAI re-audited SWE-bench with the original authors, named 3 defect classes, and reported new score ceilings of 20% and 43%. HKR-H/K/R all pass because it changes how builders read code-agent leaderboards.

Take

OpenAI is pulling SWE-bench back from leaderboard theater into risk evaluation. The original set spans 12 Python repos, and the top agents sat around 20% on SWE-bench and 43% on Lite as of Aug. 5, 2024. That reads like weak coding agents. The post names three benchmark defects: narrow tests, underspecified issues, and unreliable environments. Those defects reject valid patches and mislabel harness failure as model failure.

I buy this move more than another model claiming a few extra points. SWE-bench Verified is a human-validated subset built with the benchmark authors, and it fits OpenAI’s Preparedness Framework for Model Autonomy. Cursor, Devin, and Claude Code live on long-loop debugging and environment recovery, not clean single-shot pass rates. The old SWE-bench was partly blind to that.

Aug 8, 2024Thursday

OpenAI NewsAug 8, 20:00

Zico Kolter Joins OpenAI's Board of Directors

OpenAI appointed Carnegie Mellon professor Zico Kolter to its board on August 8, 2024, and added him to the Safety and Security Committee. The post says he will advise on critical safety and security decisions across all OpenAI projects alongside Bret Taylor, Sam Altman, and other members. The signal here is governance adding AI safety and robustness expertise, not a product launch.

Why it matters: The real signal is governance: OpenAI added a director with AI safety and robustness credentials and placed him on the Safety & Security Committee. HKR-K and HKR-R pass, but HKR-H is limited because this is a straightforward appointment notice, so it lands in low featured.

Take

OpenAI is buying board-level credibility here, not proving a stronger safety mechanism. Zico Kolter is a serious choice: CMU’s machine learning chair, work on classifier robustness, constrained neural layers, and 2023 research on automated ways to bypass LLM safeguards. He understands attack surfaces, not just governance theater.

The constraint is in the structure. Kolter joins the Safety and Security Committee with Bret Taylor, Paul Nakasone, and Sam Altman, and the article says the committee makes recommendations across all OpenAI projects. It does not disclose veto power, release thresholds, audit cadence, or red-team publication rules. After the 2023 board blowup, OpenAI knows “governance” needs technical cover. Kolter supplies that cover; the article does not show the brake pedal.

OpenAI NewsAug 8, 08:00

GPT-4o System Card

OpenAI published the GPT-4o System Card on August 8, 2024, reporting 3 of 4 Preparedness categories as low risk and persuasion as borderline medium. The post says GPT-4o accepts text, audio, image, and video inputs, responds to audio in as little as 232 ms with a 320 ms average, and is 50% cheaper than GPT-4 Turbo in the API. The key issue for practitioners is voice safety: the card names unauthorized voice generation, speaker identification, and sensitive trait attribution, and says only models with post-mitigation scores at medium or below can be deployed.

Why it matters: This is not a routine post: it adds concrete preparedness ratings, 232ms voice latency, and a clear deployment threshold. HKR-H/K/R all pass, but it is a safety disclosure rather than a new model or major launch, so it lands as featured, not p1.

Take

GPT-4o’s sharp edge is not multimodality; it is real-time voice becoming the product surface before safety has a good language for it. The card gives 232 ms best-case audio response and 320 ms average, close to human turn-taking. It also says the API is 50% cheaper than GPT-4 Turbo, so this is not a lab demo parked behind a waitlist.

The risk list is concrete: unauthorized voice generation, speaker identification, sensitive trait attribution, and disallowed audio. The Preparedness result still lands at three low categories and persuasion at borderline medium. I don’t buy the comfort implied by that bucket. Once voice agents enter support, tutoring, sales, and companionship, persuasion is not a text benchmark problem. Tone, latency, interruption, and simulated intimacy become part of the model behavior. OpenAI says only post-mitigation medium-or-below models can ship, but the card does not give practitioners a reproducible test for that deployment line.

Jul 24, 2024Wednesday

OpenAI NewsJul 24, 17:00

Improving Model Safety Behavior with Rule-Based Rewards

OpenAI said on July 24, 2024 it uses Rule-Based Rewards in the RLHF pipeline to reduce repeated human feedback for safety alignment. The post defines three response types—hard refusal, soft refusal, and comply—and says the method has been part of OpenAI’s safety stack since GPT-4, including GPT-4o mini. The key point is maintainability when policies change; the post excerpt does not disclose quantitative gains.

Why it matters: HKR-H/K/R all pass: explicit rules inside RLHF is a strong hook, and the post adds three response modes plus paper/code. I keep it in the 78–84 band because the excerpt does not disclose effect sizes, baselines, or failure-case detail.

Jul 18, 2024Thursday

OpenAI NewsJul 18, 18:00

GPT-4o mini: advancing cost-efficient intelligence

OpenAI released GPT-4o mini on July 18, 2024 at $0.15 per 1M input tokens and $0.60 per 1M output tokens, replacing GPT-3.5 in ChatGPT. It supports text and vision, offers a 128K context window and 16K max output, scores 82.0% on MMLU and 87.2% on HumanEval. The key detail for builders is that its API version is the first to use instruction hierarchy against jailbreaks and prompt injection.

Why it matters: This is a substantive OpenAI model launch, not a minor refresh: GPT-4o mini adds $0.15/$0.60 pricing, 128K context, 16K max output, benchmark details, and instruction hierarchy, then replaces GPT-3.5 in ChatGPT. HKR-H/K/R all pass, so it lands in P1.

Jul 17, 2024Wednesday

OpenAI NewsJul 17, 18:00

Prover-Verifier Games improve legibility of language model outputs

OpenAI trained GPT-4-family prover-verifier games so stronger models write solutions weaker models can verify; under time-limited human review, correctness-only optimization led to nearly 2x more evaluation errors. The post says the large and small models differ by about 3 orders of magnitude in pretraining compute, and checkability training recovers about half the performance gain of correctness-only optimization; the full experimental numbers are not fully disclosed in the provided text.

Why it matters: This is a substantive OpenAI research release with HKR-H/K/R all present: novel setup, clear mechanism, and strong relevance to scalable oversight. The excerpt confirms the method and the human-evaluation effect, but not the full experimental tables, so it fits the 78–84 band, نه

Take

OpenAI’s useful admission here is that optimizing for correct answers can make reasoning hostile to review. In their GPT-4-family prover setup, correctness-only optimization made time-limited human evaluators commit nearly twice as many errors. Training the prover to write chains a weaker verifier can check recovered about half the performance gain from accuracy-only tuning.

I buy the direction because legibility becomes a training target, not a post-hoc “please explain” wrapper. The setup also has the right asymmetry: the large and small models differ by roughly three orders of magnitude in pretraining compute, which resembles strong systems writing for weaker auditors. The caveat is ugly: the provided article cuts off before the full experimental tables, so that “half the gain” claim is not yet enough for product decisions.

May 28, 2024Tuesday

OpenAI NewsMay 28, 11:00

OpenAI Board Forms Safety and Security Committee

OpenAI's board formed a Safety and Security Committee; that action is the only confirmed fact so far. The source provides only a title, and the post does not disclose members, authority, reporting lines, or timing. Watch governance power, not the committee name.

Why it matters: This is an official board-level OpenAI governance move with HKR-H and HKR-R. It stays in the low featured band because HKR-K is weak: the post confirms the committee exists, but gives no members, remit, reporting line, or effective date.

Take

OpenAI is moving safety into a board process, but the independence story is thin. The committee is led by Bret Taylor, Adam D’Angelo, Nicole Seligman, and Sam Altman, and its first job is to deliver recommendations in 90 days. The same post says OpenAI has begun training its next frontier model, so the timing looks like pre-release governance scaffolding.

The technical bench is real: Jakub Pachocki, Lilian Weng, John Schulman, Aleksander Madry, and Matt Knight are named. The weak word is “recommendations.” The post does not disclose veto power, authority to pause training, external audit rights, or a reporting path outside OpenAI management. After the 2023 board fight, a committee title does not answer the only hard governance question: who can tell the product and compute machine to stop.