PPO and TRPO
Trust region, KL constraint, clipped surrogate, entropy bonus, implementation details
You are training a quadruped robot to walk. With vanilla policy gradient, you take a gradient step and the policy changes. If the step is too large, the robot attempts movements far outside the distribution of the collected data — the advantage estimates, computed under the old policy, are completely wrong for the new one. The robot was walking; after one bad update it is lying on the ground producing no useful gradient signal. Recovery is impossible because the next gradient step is also based on wrong estimates. This catastrophic policy collapse is the problem PPO and TRPO solve.
TRPO formalizes the constraint: maximize the expected advantage subject to KL(π_old || π_new) ≤ δ. The KL constraint defines a trust region — updates inside it are theoretically safe, with a guaranteed lower bound on policy improvement. The cost is second-order optimization: computing the natural gradient requires Fisher information matrix-vector products, conjugate gradient, and a line search. Correct, but expensive and complex.
PPO approximates TRPO with a clipped objective: L_CLIP = E[min(r_t A_t, clip(r_t, 1-ε, 1+ε) A_t)] where r_t = π_new/π_old. When A_t > 0 and r_t > 1+ε — the policy is already much more likely to take this good action — the gradient is killed. When A_t < 0 and r_t < 1-ε — the policy has already moved away from this bad action — the gradient is killed again. The clip enforces a soft trust region using only first-order optimization and vanilla Adam.
PPO's clip parameter ε = 0.2 allows up to 20% policy ratio change per step. This is not overly conservative — it prevents catastrophic collapse while still allowing rapid learning across multiple mini-batch epochs per collected batch. The fraction of clipped updates should be 10–30% during stable training. Below 1% means the clip is never activating and providing no stability benefit. Above 90% means the policy is drifting too far and the trust region is already broken.
NOT this: PPO is conservative and learns slowly. PPO with ε = 0.2 is not slow — the alternative (unconstrained large updates) produces policy collapse that wastes entire training runs. PPO's constraint is exactly why it achieves consistent results across random seeds when unconstrained policy gradients do not.
Key points
- Use PPO as your default policy gradient algorithm — it outperforms TRPO with comparable stability, requires no second-order optimization, and is robust to hyperparameters. TRPO's theoretical monotonic improvement guarantee is valuable only when you need formal guarantees. For most tasks, PPO's clipped objective achieves the same protective effect at a fraction of the compute cost.
- Too many mini-batch epochs K defeats PPO's trust region. After K gradient steps, the policy has drifted from π_old even though the advantages were computed under π_old. The policy ratio r_t becomes large, the clip triggers constantly, and gradient signal becomes noise. Limit K to 3–10 epochs per collected batch. If > 20% of samples have r_t outside [0.8, 1.2], reduce K.
- Track the policy ratio r_t distribution during training as the primary diagnostic. If less than 1% of samples are clipped, ε is too large — you are not getting the stability benefit. If more than 20% are outside [0.8, 1.2], the step size is too large and the trust region is being violated. Adjust K or ε accordingly.
PPO prevents catastrophic policy collapse by killing gradients when the policy ratio r_t moves outside [1-ε, 1+ε] — ensuring each update stays within a soft trust region where the advantage estimates are still valid.
Recap
- Problem: too-large a PG step -> policy leaves the data distribution, advantages go wrong, catastrophic collapse.
- TRPO: maximize advantage s.t. $KL(\pi_{old}\|\pi_{new}) \le \delta$ — a trust region, but 2nd-order and expensive.
- PPO clip: $L = E[\min(r_t A_t, \text{clip}(r_t, 1-\epsilon, 1+\epsilon)A_t)]$, $r_t = \pi_{new}/\pi_{old}$ — 1st-order, soft trust region.
- Clip kills the gradient once the policy already moved far on a good action ($r_t > 1+\epsilon$) or away from a bad one.
- $\epsilon = 0.2$ allows 20% ratio change; target 10–30% clipped updates (< 1% = no benefit, > 90% = trust region broken).
- Too many minibatch epochs $K$ defeats the trust region — limit to 3–10; PPO is the default over TRPO.
- In RLHF, the KL penalty to SFT is non-optional — remove it and the LLM reward-hacks.
Check your understanding
Q1. Which two of the following statements correctly describe when the PPO clip objective kills the gradient?
- A) When A_t > 0 (good action) and r_t has already risen past 1+ε, min(r_t·A_t, (1+ε)·A_t) = (1+ε)·A_t, a constant w.r.t. θ at the clip boundary — the gradient is killed so the policy stops reinforcing an already-overshot good action
- B) When A_t < 0 (bad action) and r_t has already fallen past 1-ε, the clipped term likewise becomes constant w.r.t. θ, killing the gradient so the policy stops further penalising an action it has already moved away from
- C) When A_t > 0 and r_t is still below 1-ε — meaning the policy hasn't yet reinforced the good action at all — the clip objective also flattens and kills the gradient at that point
- D) The clip only ever activates once the KL divergence between old and new policy is already exactly zero, since PPO is defined to update solely after a trust-region violation has occurred
Q2. You are training PPO on a continuous control task and observe that training is stable for 100 updates, then the policy collapses — mean episode reward drops from +500 to near 0 and never recovers. What happened and how do you diagnose and fix it?
- A) Policy collapse after a long stretch of stable training is caused by the value function diverging, a well-known PPO failure mode that occurs specifically when the critic's learning rate is set too high relative to the actor's; simply reduce the critic learning rate by a full order of magnitude and the policy will recover on its own
- B) The collapse indicates the environment underwent a sudden non-stationary distribution shift right around update 100; the agent's policy had become optimal for the earlier distribution but the environment changed, so monitor environment statistics continuously and retrain when distribution shift is detected
- C) Policy collapse is typically caused by too many mini-batch epochs K (policy drifts far from π_old, invalidating advantages) or too-high a learning rate; diagnose via r_t distribution and clip fraction (should be 10-30%, not 90%+); fix by reducing K, early-stopping on KL, or lowering the learning rate
- D) The collapse is caused by policy entropy collapsing all the way to zero; once the policy becomes fully deterministic it can never recover because the policy gradient is mathematically zero for a deterministic policy, so the fix is an unusually large entropy bonus β=1.0 to force the policy back into a stochastic regime
Q3. In RLHF with PPO for an LLM, why is the KL penalty to the SFT model necessary? What happens if you remove it?
- A) The KL penalty is only ever necessary during the very earliest stages of RLHF training to stabilise the raw reward model scores as they are first produced; once training has run for approximately 1000 steps, it can be safely removed entirely without any measurable degradation in output quality
- B) Without the KL penalty the LLM exploits the reward model via Goodhart's Law — generating repetitive or incoherent text that scores high but is low quality, because the policy drifts where the reward model (trained on SFT-like outputs) is uncalibrated; β trades hacking risk against improvement
- C) The KL penalty exists purely to prevent the LLM from generating toxic content by forcibly keeping every response close to the safe SFT baseline distribution; removing it would specifically cause harmful outputs even in cases where the reward model explicitly and heavily penalises them
- D) The KL penalty functions as nothing more than a computational-efficiency trick that reduces the numerical size of each policy gradient update; removing it would cause instability purely because of the resulting large gradient magnitudes coming directly from the reward model's raw output scale
Try it interactively
ML Systems Lab is a free interview-prep platform for ML engineers — work through the full interactive module, quizzes, and drills.
Open ML Systems Lab →