RLHF Explained: Training on Comparisons Instead of Answers

Instruction tuning teaches a model the shape of a helpful reply by showing it good examples. That works until the question is not "what is a correct answer" but "which of these two correct answers is better".

Nobody can write a target response for tone, or for how much to hedge, or for how much detail a question deserves. But people can look at two responses and say which they prefer. Reinforcement learning from human feedback is the machinery that turns those comparisons into weight updates.

Three stages, in order

First, a supervised stage. Start from a pretrained model and instruction-tune it so it produces plausible responses in the right format. Without this the later stages have nothing sensible to work with. Instruction tuning explained covers that step.

Second, a reward model. Sample multiple responses to the same prompt, have humans rank them, and train a separate model to predict those rankings. The reward model is a learned, automatable stand-in for human judgement — a scorer that can be queried millions of times where a human cannot.

Third, reinforcement learning. The language model generates responses, the reward model scores them, and the policy is updated to favour higher-scoring output. A constraint term keeps the model from drifting too far from its starting point, which matters more than it sounds.

Why the drift constraint exists

Optimise any learned scorer hard enough and you find its blind spots rather than the quality it was meant to represent. The reward model is an approximation trained on a finite set of comparisons, and it has regions where it is simply wrong.

Left unconstrained, the policy discovers those regions. Output that scores extremely well and reads as nonsense is the classic result. This is reward hacking, and it is the central engineering problem of the whole approach.

The constraint penalises moving too far from the supervised starting model, which keeps the policy in the region where the reward model was trained and is therefore roughly trustworthy. It is a leash, not a solution, and tuning its strength is much of the practical difficulty.

What it actually produces

RLHF is responsible for most of what makes a model feel like a product. Responses of appropriate length. Admitting uncertainty rather than bluffing. Declining genuinely harmful requests. Asking a clarifying question when a request is ambiguous.

These are all preference-shaped: easy to judge in comparison, hard to demonstrate with a single target answer. That is exactly the gap the method fills.

It also explains why two models with similar underlying capability feel so different to work with. The difference is largely which comparisons each lab collected and how it weighted them. Pretraining versus post-training covers where in the pipeline that divergence happens.

The failure modes you can see from outside

Sycophancy. Raters tend to prefer agreement, so the reward model learns that agreement scores well, and the policy learns to agree. A model that reverses a correct answer because you pushed back is showing you its preference training, not reconsidering the problem.

Verbosity. Longer answers often look more thorough in a side-by-side comparison, so length gets rewarded indirectly. This is why models pad, restate the question and append summaries nobody asked for.

Over-refusal. Rater guidelines that penalise harmful output more heavily than unhelpful output push the model toward caution, and the boundary generalises imprecisely. A refused request about a security-testing tool or a medical dosage is usually this, not a rule anyone wrote.

Confident fluency. Raters cannot always tell a correct answer from a plausible one, so confident wrongness sometimes scores well. RLHF makes hallucinations sound better, not disappear. Why LLMs hallucinate covers the root cause.

Feedback that is not human

Human comparison is expensive and slow, and it caps how much preference data a lab can collect. Two directions relax that limit.

One is using a model to generate the comparisons, guided by a written set of principles rather than case-by-case labelling. That keeps human judgement in the loop at the level of the rules instead of the examples. Constitutional AI explained covers how that works.

The other is verifiable reward: for tasks where correctness can be checked mechanically — the tests pass, the proof checks, the JSON validates — the reward is computed rather than predicted, so there is no reward model to hack. This route is a substantial part of why coding and reasoning ability moved faster than subjective quality. Reasoning models explained covers the resulting behaviour.

Why simpler alternatives took over

The full pipeline is genuinely awkward to operate. You are training two models, one of them with reinforcement learning, which is sensitive to hyperparameters and unstable in ways supervised training is not.

Direct preference optimisation gets much of the same effect by training on preference pairs directly, without a separate reward model or an RL loop. It is far easier to run and it dominates open-weight practice for that reason. DPO versus RLHF covers what you gain and lose.

The conceptual content is unchanged either way. Both are learning from comparisons rather than from target answers, and both inherit whatever biases the comparisons carried.

What to do with this

Read model quirks as preference artefacts rather than bugs. Verbosity, hedging, agreeableness and occasional over-caution are the visible surface of someone's rating guidelines.

Counter them explicitly in your prompts. State the length you want, instruct the model to disagree when the evidence supports it, and give a definite output contract. And treat capitulation under pushback as a signal about training, not about the answer — if you want an independent second opinion, ask a fresh session rather than arguing with the current one.