RL in Production

Off-policy evaluation, reward delay, sim-to-real, safe RL, when NOT to use RL

You have trained a stellar RL agent in simulation — 95th percentile performance on your evaluation benchmark. You deploy it to production. Within 2 days, performance drops 40%. The agent has found a way to collect rewards that technically satisfies your reward function but does not serve users. This is not a simulation failure — it is the fundamental difficulty of RL in production: specification, distribution shift, and safety.

Four production failure modes, and their fixes. First: reward hacking. The agent optimizes the letter, not the spirit, of the reward. "Maximize clicks" becomes clickbait that never delivers value. Fix: before training, spend 2 hours listing every way an agent could maximize the reward without satisfying the underlying objective. For each exploit, add a penalty term or constraint. Second: distribution shift. The simulator does not match production — agents learn to exploit simulator artifacts. Fix: domain randomization, sim-to-real transfer, real-data offline RL. Third: catastrophic forgetting. Online updates overwrite good behavior. Fix: experience replay, behavioral cloning on a buffer of historically good trajectories. Fourth: exploding Q-values. RL can be unstable without careful configuration. Fix: gradient clipping, target networks, reward normalization.

Safe exploration matters in production because exploratory actions can cause real harm. A bad recommendation alienates a user. An incorrect drug dose harms a patient. Constrained RL formulations add a safety metric as a constraint: optimize reward subject to the constraint being satisfied. Conservative policies require human approval for actions below a confidence threshold.

Offline RL addresses sample efficiency and safety together by training entirely on logged historical data without new environment interaction. Conservative Q-Learning (CQL), Implicit Q-Learning (IQL), and Decision Transformer learn from fixed datasets — critical when real environment interaction is expensive or risky.

NOT this: if the agent performs well in simulation, it will perform well in production. Simulation fidelity is never perfect. RL agents are brittle to distribution shift in ways that supervised learning models are not — the policy was trained to optimize in the simulated world, and optimization finds every crack in the simulation. Always A/B test with limited traffic before full deployment, and monitor for the specific reward hacking patterns your reward function makes possible.

Key points

• Red-team the reward function before training — list every way an agent could maximize the reward without satisfying the underlying objective. This adversarial analysis takes 2 hours and prevents the most common production failures. For each exploit, add a penalty term or constraint before training begins. Reward hacking patterns are almost always predictable in advance if you think adversarially.
• Always maintain a hard fallback policy that activates if the RL agent's action confidence drops below a threshold. RL agents degrade unpredictably. A simple rule-based or supervised learning fallback prevents a partial RL failure from becoming a total outage. Never deploy an RL agent without it. This is non-optional.
• In production, compare the RL agent's average reward per episode against a simple heuristic policy — if the RL agent underperforms the heuristic after deployment, you have distribution shift. Training distribution and deployment distribution diverged. The agent learned to optimize for the training world, which differs from the real one. First diagnostic step: compare on the heuristic baseline, then investigate what changed in the deployment distribution.
• Off-policy evaluation (OPE) estimates a new policy π_e's performance from data already logged under the old policy π_b, without deploying it live. The importance-sampling (IS) estimator reweights each logged outcome by how much more or less likely π_e was to take that action: V̂^IS = (1/N) Σ_n w_n·r_n where w_n = π_e(a_n|s_n)/π_b(a_n|s_n). For a single-step bandit this is manageable, but for a T-step sequential MDP the trajectory-level weight is a product Π_t w_t — variance grows exponentially in T, and delayed rewards make this worse by stretching T further before a reward lands. The doubly robust (DR) estimator fixes this by combining a fitted reward model R̂(s,a) with an IS correction on the residual; it stays consistent if EITHER the reward model OR the importance weights are accurate — hence "doubly robust" — which is why it dominates plain IS in practice.

Recap

• Sim 95th percentile -> 40% drop in 2 days: specification, distribution shift, and safety are the real difficulty.
• Four failure modes: reward hacking, distribution shift, catastrophic forgetting, exploding $Q$-values — each with a fix.
• Red-team the reward before training: 2 hours listing every exploit, add a penalty per exploit. #1 skill.
• Always keep a hard fallback policy that activates below a confidence threshold — non-optional.
• Safe/constrained RL: optimize reward subject to a safety constraint; conservative policies need human approval.
• Offline RL (CQL, IQL, Decision Transformer) trains on logged data — critical when real interaction is risky.
• RL agents are brittle to distribution shift in ways supervised models aren't; A/B test on limited traffic first.

Check your understanding