OpenAI published deliberative alignment on Dec 20, 2024, training o-series models to reason over written safety specs before answering. The post says o1 uses this method and needs no human-labeled CoT or answers; it says o1 beats GPT-4o on internal and external safety benchmarks, but the post does not disclose exact scores.

OpenAI published the system card for o1 and o1-mini, with a deployment gate that requires post-mitigation risk scores of medium or lower. The listed Preparedness results are low for cybersecurity, medium for CBRN and persuasion, and low for model autonomy; testing covered o1-near-final-checkpoint and o1-dec5-release. The key point for practitioners is that OpenAI confirms large-scale RL for chain-of-thought reasoning, while the post does not disclose dataset mix or full benchmark scores.

OpenAI published 2 papers on Nov 21, 2024, outlining its external human red teaming process and a new automated red teaming method. The post discloses 3 concrete design choices for external testing—threat-model-based team selection, versioned model access, and structured feedback via API or ChatGPT interfaces—but this excerpt does not fully disclose the automated method's metrics or results.

OpenAI open-sourced SimpleQA, a 4,326-question benchmark for factual short-answer QA and model calibration. Two independent AI trainers verified each item; a 1,000-question audit showed 94.4% agreement and an estimated inherent error rate near 3%. The key signal: it is built to challenge frontier models, and the post says GPT-4o scores below 40%.

OpenAI introduced sCM and scaled continuous-time consistency models to 1.5B parameters on ImageNet at 512×512. The post says sCM reaches sample quality comparable to leading diffusion models in 2 sampling steps, with about 50x wall-clock speedup. Its largest model generates one sample in 0.11s on a single A100 at batch size 1 without inference optimization.

OpenAI analyzed millions of ChatGPT requests to test whether user names trigger harmful stereotypes, finding an overall rate of about 0.1%. The study used GPT-4o as a privacy-preserving evaluator; its gender-related judgments matched human raters over 90% of the time, while race and ethnicity agreement was lower. The key signal is model drift across versions: GPT-3.5 Turbo showed the highest task-level bias.

OpenAI released MLE-bench, a benchmark built from 75 Kaggle competitions to measure ML engineering ability in AI agents. The best setup, o1-preview with AIDE scaffolding, reached at least Kaggle bronze-medal level on 16.9% of tasks; the benchmark code is open-source.

OpenAI released o1-preview and reported 74% single-sample accuracy on AIME 2024, versus 12% for GPT-4o. The post says o1 reached the 89th percentile on Codeforces and exceeded human PhD experts on GPQA Diamond; it attributes this to large-scale RL and gains from both train-time and test-time compute. The key signal is scaling reasoning with compute, not just pretraining a larger base model.

OpenAI released SWE-bench Verified, a human-validated subset built with the benchmark’s authors to assess real software issue resolution more reliably. The post names 3 failure modes in SWE-bench: overly narrow tests, underspecified issue statements, and unreliable environment setup; as of Aug. 5, 2024, top agents scored about 20% on SWE-bench and 43% on SWE-bench Lite. The key point is that the original benchmark can systematically underestimate coding-agent ability.

OpenAI published the GPT-4o System Card on August 8, 2024, reporting 3 of 4 Preparedness categories as low risk and persuasion as borderline medium. The post says GPT-4o accepts text, audio, image, and video inputs, responds to audio in as little as 232 ms with a 320 ms average, and is 50% cheaper than GPT-4 Turbo in the API. The key issue for practitioners is voice safety: the card names unauthorized voice generation, speaker identification, and sensitive trait attribution, and says only models with post-mitigation scores at medium or below can be deployed.

OpenAI said on July 24, 2024 it uses Rule-Based Rewards in the RLHF pipeline to reduce repeated human feedback for safety alignment. The post defines three response types—hard refusal, soft refusal, and comply—and says the method has been part of OpenAI’s safety stack since GPT-4, including GPT-4o mini. The key point is maintainability when policies change; the post excerpt does not disclose quantitative gains.

OpenAI trained GPT-4-family prover-verifier games so stronger models write solutions weaker models can verify; under time-limited human review, correctness-only optimization led to nearly 2x more evaluation errors. The post says the large and small models differ by about 3 orders of magnitude in pretraining compute, and checkability training recovers about half the performance gain of correctness-only optimization; the full experimental numbers are not fully disclosed in the provided text.