Why it matters
Deployment Simulation replays privacy-filtered prefixes from prior conversations and substitutes a candidate model to estimate behavior before launch. OpenAI reports a 1.5× median multiplicative error across 20 behavior categories on 1.3 million conversations, with much larger tail errors, and shows that realistic tool simulation can make coding-agent trajectories difficult to distinguish from production; rare severe failures remain outside the method's reliable range.
My takeaway: Use representative replay to estimate common behavior, then add targeted adversarial and rare-event evaluations for the tail. Pre-register metrics, simulate tool state without live side effects, protect replayed user data, and compare predictions with post-deployment observations to expose distribution shift.