Deep Q-Networks
Experience replay, target network, Double DQN, Dueling, Prioritized Replay, failure modes
Consider Atari Breakout. The state is 4 consecutive 84×84 game frames — stacked to capture motion. The action space has 3 choices: left, right, no-op. The reward is +1 per brick broken. The state space is effectively infinite: no two sequences of frames are likely to be identical. A tabular Q-table for every possible pixel configuration is physically impossible. DQN replaces the Q-table with a neural network Q(s, a; θ) that takes raw pixels as input and outputs Q-values for all three actions simultaneously.
The problem is that applying Q-learning naively to a neural network is deeply unstable. Consecutive game frames are highly correlated — if the agent is in the top-left of the screen, the next 100 transitions are all from the top-left, and gradient updates overfit to that region while forgetting everything else. This violates the IID assumption that SGD requires: gradient estimates should be drawn from the full training distribution, not a narrow slice of recent experience. DQN's first fix is experience replay: store every transition (s_t, a_t, r_t, s_{t+1}) in a replay buffer of up to 1M entries, then sample random mini-batches. Temporal correlation breaks; transitions are reused multiple times.
The second instability is that the bootstrap target R + γ max_{a'} Q_θ(s', a') depends on the same θ being updated. As θ shifts, the target shifts — you are chasing a moving target, and the feedback loop amplifies Q-values until they diverge. DQN's second fix is the target network: maintain a separate θ^- that is copied from θ only every 10,000 steps. The target is computed using θ^-, which is frozen between updates. The feedback loop breaks.
Double DQN further improves on this. The DQN target uses the same θ^- to both select the best action and evaluate it, which produces a systematic upward bias — the max over noisy estimates is always higher than the estimate of the true max. Double DQN decouples these: use θ to select the action (argmax_a Q_θ(s', a')), then use θ^- to evaluate it. Dueling DQN goes further and decomposes Q(s, a) = V(s) + A(s, a), learning state value and action advantage separately.
NOT this: DQN is the standard deep RL algorithm. DQN only works for discrete action spaces. For continuous control — robot joint torques, motor commands — DQN's argmax over actions is infeasible. Use actor-critic methods (SAC, TD3, PPO) when actions are real-valued.
Key points
- Always use experience replay and target networks together — removing either one causes DQN to diverge. Replay breaks the temporal correlation that violates SGD's IID assumption. The target network freezes the bootstrap target to prevent the moving-target feedback loop. These two problems are independent and require independent fixes; a target network alone does not solve the correlation problem, and replay alone does not solve the moving-target problem.
- A replay buffer smaller than 100K transitions memorizes recent experience and discards rare but important transitions. With a 10K buffer, the agent has effectively seen only the last few minutes of gameplay. Rare high-reward transitions — the first time the agent breaks a row of bricks — cycle out before they can be learned from. Use at least 100K for Atari-scale problems; 1M is standard for long training runs.
- If training Q-values explode or oscillate, increase the target network update interval first, then reduce the learning rate. Q-value explosion is almost always the moving-target problem. Increase the target update interval from 1K to 10K steps to slow down the feedback loop. If that does not stabilize training, halve the learning rate. If Q-values collapse to zero, check that terminal states are handled correctly and that rewards are not always negative.
DQN makes Q-learning stable for neural networks with two fixes: experience replay breaks the temporal correlation that causes gradient overfitting, and a target network freezes the bootstrap target to prevent the moving-target feedback loop that amplifies Q-values into divergence.
Recap
- DQN replaces the Q-table with a network $Q(s,a;\theta)$ — tabular is impossible for pixel states.
- Naive Q-learning on a net is unstable: consecutive frames are correlated, violating SGD's IID assumption.
- Fix 1 — experience replay: buffer up to 1M transitions, sample random minibatches to decorrelate.
- Fix 2 — target network: frozen $\theta^-$ copied every 10K steps stops the moving-target feedback loop.
- Use both together — removing either diverges; they solve independent problems.
- Double DQN decouples action selection ($\theta$) from evaluation ($\theta^-$) to kill max-operator overestimation; Dueling splits $Q = V + A$.
- Buffer < 100K memorizes recent play; DQN is discrete-action only — use actor-critic for continuous control.
Check your understanding
Q1. Vanilla DQN adds two specific mechanisms on top of neural-network Q-learning to make training stable. Which two are they?
- A) Experience replay — sample random mini-batches from a large buffer of past transitions so gradient estimates are decorrelated and match the full training distribution instead of the recent trajectory's local statistics
- B) A frozen target network θ^-, copied from θ only every K steps, so the TD bootstrap target stays stationary between updates and the moving-target feedback loop is broken
- C) A recurrent policy architecture that conditions on the full episode history, which is what actually removes the correlation between consecutive gradient updates
- D) Second-order natural-gradient updates computed via Fisher information matrix-vector products, which is what stops the network from catastrophically forgetting older regions of state space
Q2. What is the difference between Dueling DQN and standard DQN architecturally, and in what types of states does Dueling provide the largest benefit?
- A) Dueling DQN uses two entirely separate Q-networks — one trained only on even-numbered timesteps and one only on odd-numbered timesteps — specifically to reduce gradient correlation between consecutive updates; it benefits most in environments with sparse terminal rewards
- B) Dueling DQN adds an auxiliary loss term on state-visitation counts to explicitly encourage exploration into rarely-seen states; it benefits most in environments where many distinct states happen to share identical Q-values, forcing exploration to become effectively uniform
- C) Dueling DQN splits V(s) and A(s,a), merged as Q = V(s) + A(s,a) - mean_a A(s,a); it helps most in states where action choice barely matters — V updates from any transition there, faster than a single-head network
- D) Dueling DQN replaces the standard scalar Q-function output layer with a full distributional layer that instead outputs quantiles of the entire return distribution rather than a point estimate; it benefits most in states exhibiting unusually high reward variance
Q3. You are applying DQN to a robotic manipulation task where the reward is 1 only when the robot successfully places an object and 0 otherwise, with episodes of 200 steps. After 10M steps, the policy never achieves reward > 0. What is happening and what are your next steps?
- A) The issue is that DQN is architecturally incapable of learning any manipulation task at all; switching to a policy-gradient algorithm like PPO will automatically resolve the sparse-reward problem because PPO's on-policy data is always perfectly relevant to the current policy
- B) The network has overfit to the constant 0-reward signal seen in every episode; the fix is to regularise the Q-network more heavily with dropout and weight decay so it generalises correctly to the rare success state it has essentially never observed during training
- C) 10M steps is simply insufficient for a robotic manipulation task of this difficulty; continuing training for 100M steps will let random exploration accidentally stumble onto the success state often enough for ordinary Q-learning to slowly propagate the reward signal backward
- D) The buffer holds only 0-reward transitions, giving no gradient signal; next steps in order: curriculum learning (start near the target), Hindsight Experience Replay (relabel failures as their achieved goal), dense reward shaping, and demo-augmented RL
Try it interactively
ML Systems Lab is a free interview-prep platform for ML engineers — work through the full interactive module, quizzes, and drills.
Open ML Systems Lab →