Actor-Critic Methods

A2C, A3C, advantage function, GAE, async vs sync, bias-variance in advantage estimation

Return to the robotic arm. With REINFORCE, you collect a full episode before updating — the arm attempts the reach, you compute G_t at every step, and you update the policy. Two problems. First, you need complete episodes. Second, G_t at step t includes rewards from steps t+1 through the end of the episode — all caused by different actions, not the one at step t. The credit assignment is noisy. Variance is high.

Actor-critic solves both. Maintain two networks simultaneously. The actor π_θ(a|s) selects actions — the policy. The critic V_φ(s) estimates the state value — how much total reward to expect from here under the current policy. After each step, update the critic using TD: the critic learns V(s_t) ≈ R + γV(s_{t+1}). Then compute the advantage A(s_t, a_t) = R + γV(s_{t+1}) - V(s_t) — how much better than expected was this particular step? Update the actor proportionally. You get updates every step, not every episode.

The advantage has a key property: E_{a~π}[A(s, a)] = 0. It is zero-mean across actions. This means it carries only relative information — this action was above average, that one was below. Unlike raw Q(s, a), which can be large and positive for all actions in a highly valuable state, the advantage removes the state's baseline value and isolates the signal about action quality. This is what makes actor-critic gradient estimates so much lower variance than REINFORCE.

Generalized Advantage Estimation (GAE) extends this. Instead of the one-step advantage R + γV(s') - V(s), GAE accumulates a weighted average of n-step advantages: Â^GAE = δ_t + γλδ_{t+1} + (γλ)²δ_{t+2} + ... where δ_t = R_{t+1} + γV(s_{t+1}) - V(s_t). λ = 0 gives the one-step TD error — low variance, high bias. λ = 1 gives the full Monte Carlo advantage — no bias, high variance. λ = 0.95 is the standard for most tasks. PPO, TRPO, and most modern on-policy actor-critics use GAE.

NOT this: the actor and critic have separate learning problems that can interfere with each other. The two networks are cooperative, not adversarial — the critic provides variance-reducing signal to the actor, and the actor's improving policy makes the critic's targets more stable. The instability risk is that a slow or inaccurate critic injects biased gradient into the actor. Mitigate by setting critic learning rate 3–10× higher than actor learning rate, so the critic leads.

Key points

• Use actor-critic over pure policy gradients for any task with episodes longer than about 50 steps. Per-step TD updates in actor-critic dramatically reduce gradient variance compared to full-trajectory REINFORCE. The actor-critic wall-clock speedup is typically 10–100× on continuous control tasks because you do not wait for episode completion.
• Set critic learning rate 3–10× higher than actor learning rate. The critic must converge to a stable estimate before the actor can use it meaningfully. If critic and actor learn at the same speed, the actor is chasing a moving value target — equivalent to applying noisy baselines that can increase gradient variance rather than reduce it.
• If actor loss improves but critic loss plateaus at a high value, the reward magnitude is too large for the critic to track. Normalize rewards to approximately [-1, 1] or clip them, then recheck critic convergence. A critic that cannot model the value function correctly injects biased advantage estimates into the actor gradient, which explains why actor performance degrades even as actor loss decreases.

Recap

• Two networks: actor $\pi_\theta(a|s)$ picks actions, critic $V_\phi(s)$ estimates state value.
• Per-step TD updates replace REINFORCE's noisy full-episode return.
• Advantage $A(s_t,a_t) = R + \gamma V(s_{t+1}) - V(s_t)$ — how much better than expected this step was.
• $E_{a\sim\pi}[A]=0$: zero-mean signal removes the state baseline, isolating action quality — lower variance than raw $Q$.
• GAE blends n-step advantages: λ=0 one-step (low var/high bias), λ=1 MC, 0.95 is the default.
• Set critic LR 3–10× the actor LR so the critic leads and the actor isn't chasing a moving target.
• Actor down, critic plateaus high = reward magnitude too large; normalize rewards to ~$[-1,1]$.

Check your understanding