RLHF: Reward Modeling, PPO, and Why DPO Is Replacing It

RLHF is the technique that turned GPT-3 into ChatGPT — aligning a language model to human preferences rather than just next-token prediction. It is one of the most influential ideas in modern AI, and also one of the most unstable pipelines to train. DPO (Direct Preference Optimisation) emerged in 2023 as a simpler alternative that achieves comparable results without RL at all.

Language model pretraining on next-token prediction creates a model that is a good statistical approximation of its training corpus. It is not aligned with human intentions: it will complete harmful prompts, give verbose unhelpful answers, and confidently confabulate. RLHF addresses this by fine-tuning the model on human preferences.

The three-stage RLHF pipeline (InstructGPT, 2022)

Stage 1 — Supervised Fine-Tuning (SFT): start with the pretrained model and fine-tune it on a dataset of high-quality (prompt, response) pairs written by humans. This teaches the model the desired response format and basic instruction-following. The SFT model is the foundation for the next stages.

Stage 2 — Reward Model Training: sample multiple responses from the SFT model for each prompt. Human raters compare responses and mark which is better. These preference pairs (prompt, response_A, response_B, label: A > B) train a reward model R_θ that outputs a scalar reward for any (prompt, response) pair. The reward model is typically the same architecture as the LLM with the final token prediction head replaced by a regression head. Training objective: Bradley-Terry model — maximise the log probability that the preferred response has higher reward than the rejected response: log σ(R_θ(x, y_w) - R_θ(x, y_l)), where y_w is the preferred response and y_l is the rejected one.

Stage 3 — RL Fine-Tuning with PPO: use the reward model as the reward signal in a reinforcement learning loop. The policy is the LLM; the action is generating tokens; the reward comes from the reward model at the end of the generated sequence. PPO (Proximal Policy Optimisation) is used because it is stable and efficient for large models. The PPO objective includes a KL penalty: reward = R_θ(x, y) - β * KL(π_RL || π_SFT), where π_SFT is the SFT model and β controls how far the RL policy can deviate. Without the KL penalty, the RL model learns to exploit the reward model — generating text that gets high reward scores but is incoherent or degenerate (reward hacking).

Why PPO is hard

PPO requires four models loaded simultaneously: the policy (LLM being trained), the SFT reference model (for KL penalty), the reward model (for rewards), and the value model (PPO critic). For a 7B parameter model, this is 4 × 7B × 4 bytes = ~112GB GPU memory just for model weights, before activations. Training instability is common: reward hacking, mode collapse, and sensitivity to hyperparameters. The PPO objective is non-stationary because the policy changes the distribution over which the reward model is evaluated.

DPO: Direct Preference Optimisation (Rafailov et al., 2023)

DPO's key insight: the optimal RL policy under KL-constrained reward maximisation can be expressed in closed form. This allows the RL objective to be rewritten directly in terms of the policy model, eliminating the need for a separate reward model and RL training. DPO loss: -log σ(β log(π_θ(y_w|x)/π_SFT(y_w|x)) - β log(π_θ(y_l|x)/π_SFT(y_l|x))). This is a supervised loss on preference pairs that implicitly trains the policy to increase the relative probability of preferred responses over rejected ones, without an explicit reward model. DPO requires only: the SFT model (reference) and the policy model being trained. No reward model, no value model, no RL loop. 2 models instead of 4.

DPO vs PPO: practical trade-offs

DPO advantages: simpler implementation, less memory, more stable training, no reward hacking. DPO disadvantages: less flexible (cannot incorporate non-preference feedback signals), no explicit reward model for analysis, less explored for very long-context alignment. In practice, DPO has matched or exceeded RLHF on many benchmarks (TruthfulQA, AlpacaEval) with significantly less engineering overhead. Most open-source fine-tuning pipelines (Axolotl, LLaMA Factory) now default to DPO.

Constitutional AI (CAI) and RLAIF

Anthropic's Constitutional AI (2022) replaces human raters with an AI critic. A set of principles (the "constitution") guides the AI critic to evaluate responses. RLAIF: Reinforcement Learning from AI Feedback — use a stronger model to generate preference labels, then run standard RLHF. Scales better than human annotation for large datasets.

Reward hacking: the core problem

The reward model is a proxy, not the true human preference. Optimising it too hard produces reward hacking: generating responses that are verbose (longer = rated higher by some raters), sycophantic (agreeing with the user), or formulaic (specific phrases that get high ratings). The KL penalty in PPO and the reference model in DPO both guard against this, but the problem is fundamental to any proxy optimisation.

Try on Colab: use the TRL library (Hugging Face) with a small GPT-2 or OPT model. Train a reward model on a subset of the Anthropic HH-RLHF dataset. Compare DPO training (DPOTrainer in TRL) vs PPO training (PPOTrainer) on the same preference data. Measure: training time, memory usage, and win rate of each fine-tuned model against the SFT baseline using the reward model as a judge.