Policy Gradients

REINFORCE, log-derivative trick, high variance, baselines, why PG beats value-based

Consider a robotic arm reaching for a target. The state is joint angles and velocities — continuous. The action is torques applied to each joint — also continuous, varying smoothly across a large range. Q-learning requires taking the argmax over all actions to compute the optimal next step. Over a continuous torque space, this argmax is an optimization problem that must be solved at every step, for every transition in the replay buffer. It is computationally infeasible. Policy gradient methods sidestep this entirely: instead of learning Q values and deriving a policy from them, parameterize the policy directly as π_θ(a | s) = N(μ_θ(s), σ²_θ(s)). The neural network outputs a mean and variance, and actions are sampled from that Gaussian. Update θ to increase the probability of actions that led to high returns.

The Policy Gradient Theorem gives the gradient: ∇_θ J(θ) = E_π[∇_θ log π_θ(a|s) · Q^π(s, a)]. Increase the log-probability of action a in state s proportionally to how good that action was. The log-derivative trick makes this computable: ∇_θ π_θ(a|s) = π_θ(a|s) · ∇_θ log π_θ(a|s), which converts the gradient of an expectation into an expectation of a gradient — sampleable from trajectories. The environment's transition model never appears. This is model-free.

REINFORCE is the direct implementation: sample a full episode, compute G_t at each timestep, update θ ← θ + α Σ_t G_t ∇_θ log π_θ(a_t | s_t). The problem is that G_t includes all future rewards — noise unrelated to a_t's actual contribution. A good action followed by bad luck is indistinguishable from a genuinely bad action. Gradient estimates have enormous variance.

Baseline subtraction solves this. Replace G_t with (G_t - b(s_t)) where b depends only on the state, not the action. The expected gradient is unchanged — any state-dependent term subtracts to zero because the policy log-gradient sums to zero over actions. But variance drops by centering returns around the state's average value. The standard practical baseline is V^π(s_t) itself, giving the advantage A(s_t, a_t) = G_t - V^π(s_t) — how much better this action was than average. (The true variance-minimizing baseline is technically a score-weighted average of returns, not V^π(s) exactly — but V^π(s) captures almost all the benefit and is far simpler to estimate, which is why it's the one actually used in practice.)

NOT this: policy gradients are unbiased because they use sampled returns. Unbiased in expectation does not mean useful in practice. REINFORCE has extremely high variance for long-horizon tasks, and the gradient estimate from a single trajectory is dominated by random noise. This is why actor-critic methods — which replace G_t with a learned critic estimate — dominate in practice.

Key points

• Always subtract a baseline from returns in policy gradient updates. Using the mean return or a learned value function as baseline reduces variance by 10–100× with zero bias cost — the baseline integrates to zero over the policy distribution. Skipping the baseline is skipping the most important and cheapest variance reduction available.
• If σ_θ(s) → 0 early in training, the policy has collapsed to deterministic and exploration has stopped — and the gradient does not go quietly: the Gaussian score function (a-μ)/σ² blows up as σ→0, so the collapse shows up as gradient-variance explosion and numerical instability, not a clean zero signal. Add an entropy bonus H[π_θ] to the loss to maintain policy spread throughout training. Without it, the network converges to the first good-looking action it found and stops exploring whether there is something better.
• If policy gradient training is noisy with high variance in returns across episodes, you need more environment samples per update — not a learning rate change. Gradient signal-to-noise ratio improves as √(num_samples). Quadrupling the number of parallel environments halves gradient noise. Collect longer rollouts or more parallel workers before tuning any other hyperparameter.
• In a two-player zero-sum game (poker, adversarial self-play), a deterministic policy is always exploitable — the opponent observes it and plays the exact counter. The game-theoretic optimum is a Nash equilibrium, which in general requires a mixed (stochastic) strategy over actions — something argmax Q*(s,a) cannot represent at all, but a stochastic policy π_θ(a|s) can. This is a second, independent reason (beyond continuous actions) that policy gradients generalize where value-based methods break down.

Recap

• PG parameterizes the policy directly $\pi_\theta(a|s)$ — sidesteps the infeasible argmax over continuous actions.
• Policy Gradient Theorem: $\nabla_\theta J = E_\pi[\nabla_\theta \log \pi_\theta(a|s) \cdot Q^\pi(s,a)]$; log-derivative trick makes it sampleable, model-free.
• REINFORCE uses full-episode $G_t$ — enormous variance because good action + bad luck looks like a bad action.
• Baseline subtraction is mandatory: $A = G_t - V^\pi(s_t)$ cuts variance 10–100× at zero bias.
• Any state-only baseline is unbiased — policy log-gradient sums to zero over actions.
• $\sigma_\theta \to 0$ early = policy collapsed to deterministic; add an entropy bonus to keep exploring.
• Noisy returns? Add samples, not learning rate — signal-to-noise scales as $\sqrt{N}$.

Check your understanding