OpenAI simulates real-world deployment to catch undesired model behavior before release
Predicting model behavior before release by simulating deployment
OpenAI replays recent real conversations through a candidate model before release, then checks for new undesired behaviors. Across GPT‑5‑Thinking deployments, this Deployment Simulation gave more accurate frequency estimates than traditional evals, surfaced novel misalignment, and reduced the chance models could tell they were being tested. It also works for agentic rollouts with tool use. The method can’t catch behaviors rarer than 1 in 200,000 messages.
Why it matters: OpenAI replayed real conversations to have a new model regenerate replies before GPT-5-Thinking shipped, estimating bad-behavior rates and surfacing alignment issues it had not seen before.