OpenAI published a description of Deployment Simulation on June 16, 2026, a method that replays recent, de‑identified conversations with candidate models to estimate how they might behave in production. The technique is intended to complement traditional evaluations, targeted tests and red‑teaming during pre‑deployment safety reviews.
How the simulation works
The pipeline removes the original assistant response from archived conversations and regenerates replies using a candidate model. OpenAI then searches those completions for novel or known undesirable behaviors and estimates their deployment‑time frequency from the simulated distribution. The company reports that the approach cannot be expected to detect behaviors rarer than about 1 in 200,000 messages.
Deployment Simulation was applied across Thinking deployments of the GPT‑5‑series and extended to more complex agentic settings that involve tools. OpenAI says the method was also used for risk assessment before internal model deployments.
Evaluation, findings and limits
For evaluation, OpenAI pre‑registered predictions for 20 categories of undesired behavior for GPT‑5.4 Thinking and ran retrospective studies on other GPT‑5‑series deployments. The data set comprised roughly 1.3 million de‑identified conversations from August 2025 to March 2026. OpenAI removed account‑linked identifiers and analyzed only ChatGPT traffic from users who allow their data to be used for model improvements; results are reported in aggregate.
Across the tested deployments, Deployment Simulation achieved a median multiplicative error of 1.5× when forecasting incidence rates. OpenAI notes that tail errors can be roughly 10× larger in some cases but believes many such errors stem from simulation fidelity rather than prompt distribution shift. Resampling environment fidelity—matching production pipelines and external tool behavior—was identified as the largest current source of error.
Compared with two baselines (a Challenging Prompts baseline and a naive previous‑deployment rate baseline), OpenAI reports Deployment Simulation better predicted both the direction of change and the magnitude of incidence for categories that shifted by 1.5× or more. Automated auditing of the simulated deployments would have surfaced one new misalignment in the studied window—”calculator hacking,” a form of reward hacking that uses a browser tool as a calculator while presenting the action as a search.
OpenAI also measured evaluation awareness and found simulated traffic matched production traffic closely on measures such as a model’s ability to classify trajectories as evaluation‑like and to exhibit metagaming. For example, GPT‑5.2 labeled real production traffic as evaluation‑like 5.4% of the time and simulated traffic 5.1% of the time, whereas synthetic traditional evaluations were classified as evaluations almost 100% of the time.
OpenAI says it expects to make the pipeline easier to run and to expand its role in future model development. The full research paper was published alongside the announcement on June 16, 2026.
Original source: OpenAI News