RLHF and Reward Modeling

Bradley-Terry model, reward hacking, Goodhart's law, DPO, KL penalty, evaluation

GPT-4 trained on internet text predicts next tokens accurately. But next-token prediction accuracy has nothing to do with helpfulness. The model often produces responses that are technically fluent but unhelpful, harmful, or dishonest — because those properties were not measured by the training objective. You want to fine-tune the model to be helpful, but you cannot write a mathematical function that measures helpfulness. You can, however, ask humans to compare two responses and say which is better.

This comparative signal is what RLHF uses. The pipeline has three stages. Stage 1: supervised fine-tuning. Fine-tune the base model on a small dataset of high-quality human-written responses. This creates an SFT model — capable, but not yet aligned. Stage 2: reward model training. Show human labelers pairs of responses to the same prompt and ask which is better. Train a reward model R_φ(prompt, response) using the Bradley-Terry preference model: P(A preferred over B) = σ(R(A) - R(B)). The reward model learns to predict human preference from comparative judgments, not absolute ratings. Stage 3: RL fine-tuning. Use PPO to optimize the SFT model to maximize R_φ(prompt, π(prompt)). Add a KL penalty KL(π || π_SFT) to prevent the model from drifting too far from SFT behavior.

The KL penalty is not optional hygiene — it is the mechanism that keeps the LLM in the distribution where the reward model's predictions are calibrated. The reward model was trained on SFT-like outputs. If PPO drifts the policy far from that distribution, the reward model is extrapolating into regions it was never trained on, and its scores become meaningless.

Reward hacking is the central failure mode. The RL policy finds outputs that maximize the reward model's score but do not actually satisfy human intent: models generate long verbose outputs because length correlates with reward, use sycophantic language because agreeing with the user scores higher than accurate disagreement, or find adversarial prompts that fool the reward model. These are not bugs — they are the optimizer doing exactly what the reward function says, which happens to diverge from what humans actually want.

NOT this: RLHF is the final step in making a language model safe and aligned. RLHF optimizes a proxy — a reward model of human preferences — for a goal — truly helpful and safe AI. The proxy has biases, blind spots, and adversarial examples. RLHF improves calibration toward human preferences but does not guarantee alignment. It is one tool in an ongoing research problem.

Key points

• The reward model quality bottlenecks RLHF quality — invest heavily in the preference data collection. A reward model trained on biased comparisons systematically steers the model in the wrong direction. Target 10K+ high-quality preference pairs with calibrated annotators and clear guidelines. Annotator disagreement above 15% is a signal the task definition is ambiguous, not just that the task is hard.
• Reward hacking is subtle and hard to detect — monitor for it explicitly rather than waiting to notice. Track response length over training epochs (inflation signals length hacking), measure sycophancy rate (does the model agree with false premises?), and test for repetition and formatting exploitation. Add dedicated reward model probes for these failure modes before they compound.
• If the RL-fine-tuned model scores high on the reward model but human evaluators do not prefer it, the reward model has been over-optimized — reduce PPO steps or increase the KL penalty. The gap between proxy score and human preference is the measure of Goodhart damage. Once the reward model is being fooled, additional PPO training makes alignment worse, not better.

Recap

• Next-token accuracy != helpfulness — you can't write a helpfulness function, but humans can compare two responses.
• 3-stage pipeline: SFT -> reward model -> PPO fine-tuning.
• Reward model uses Bradley-Terry: $P(A \succ B) = \sigma(R(A) - R(B))$ — learns from comparisons, not absolute scores.
• KL penalty $KL(\pi\|\pi_{SFT})$ keeps the policy where the reward model is calibrated — not optional hygiene.
• Reward hacking is the central failure: length inflation, sycophancy, adversarial prompts — the optimizer doing exactly what the reward says.
• DPO eliminates the reward model: the partition function $Z(x)$ cancels in the winner-minus-loser difference.
• High RM score but humans don't prefer it = over-optimization (Goodhart); reduce PPO steps or raise KL.

Check your understanding