DPO vs RLHF: Two Routes From Preferences to Model Behaviour

Both methods solve the same problem: you have pairs of responses where a human preferred one, and you want the model to produce more output like the preferred one. They differ entirely in the route from that data to updated weights.

RLHF takes the long way, training a reward model and then optimising against it with reinforcement learning. DPO takes a shortcut that turns the same preference data into an ordinary supervised loss. The shortcut is why almost every open-weight release you can download was aligned this way.

What RLHF does with the data

The preferences train a separate reward model — a scorer that predicts which of two responses a human would pick. That scorer becomes an automatable stand-in for human judgement.

Then a reinforcement learning loop runs: the language model generates responses, the reward model scores them, and the policy is updated toward higher scores, with a constraint keeping it from drifting too far from where it started. RLHF explained covers the full pipeline.

The important property is that the policy generates fresh responses during training and gets scored on them. It explores. It can find behaviours nobody demonstrated in the original comparison data.

What DPO does instead

DPO observes that the optimal policy under that reward-plus-constraint objective can be expressed directly in terms of the policy itself, which means you can skip building the reward model and skip the RL loop entirely.

The result is a loss function computed straight from preference pairs: raise the likelihood of the preferred response, lower the likelihood of the rejected one, relative to a frozen reference copy of the starting model. That reference copy plays the role the drift constraint played.

Operationally it is just supervised training. One model in memory plus a frozen reference, a standard loop, standard hyperparameters, and no generation during training. A team that can fine-tune can run DPO; running the full RL pipeline is a different level of investment.

The real trade-off: on-policy versus off-policy

This is the distinction that matters and it gets lost in the discussion of implementation complexity.

RLHF is on-policy. The model is scored on responses it generates now, so as it changes, the training signal follows it. If it develops a new failure mode mid-training, the reward model can penalise it, because the reward model evaluates whatever appears.

DPO is off-policy. The preference pairs were collected in advance, usually from a different model. The loss can only push toward responses in that fixed dataset and away from others in it. If the model drifts into behaviour the dataset never covered, nothing in the objective notices.

In practice this shows up as DPO being excellent at instilling the preferences you collected and weaker at generalising beyond them. The dataset is more of a hard boundary than it is under RLHF.

Where each one wins

DPO wins on cost, stability and reproducibility. Training is cheaper, results vary less between runs, and there are far fewer knobs to get wrong. For a team with a clear set of preferences to encode — house style, refusal boundaries, output format discipline — it usually gets most of the available benefit.

RLHF wins where the preference space is broad and you cannot enumerate it, and where you want the model to discover good behaviour rather than imitate specific examples. It also composes more naturally with computed rewards, since a reward function that runs tests or validates a schema slots directly into the loop.

That last point is significant for coding models. When correctness is mechanically checkable there is no reward model to hack, and the on-policy loop can push hard without the usual risk. Reasoning models explained covers the behaviour that comes out.

Where the preference data comes from

Both methods live or die on the comparisons, and human comparison is slow and expensive. Most modern pipelines generate candidate pairs with models and filter them, keeping humans at the level of the rules rather than the individual labels.

That scales, and it imports the generating model's habits along with its judgement. If the generator is verbose or over-cautious, those preferences propagate into whatever you train on them. Synthetic training data covers the filtering that keeps this honest.

A related route replaces case-by-case human labels with a written set of principles that a model applies to produce the comparisons. Constitutional AI explained covers how that shifts the human effort from labelling to rule-writing.

What this means if you are not training models

Mostly it explains what you can expect from an open-weight release. Open-weight models are typically DPO-aligned because that is what a lab can run and publish repeatably, and the visible consequence is preference behaviour that is well-defined inside its training distribution and less predictable outside it.

You will notice this as a model that is consistent on ordinary requests and oddly variable on unusual ones — a refusal boundary that seems arbitrary, or a format rule that holds for common cases and lapses on rare ones. That is off-policy training showing its edges.

It also means fine-tuning an aligned open-weight model is easier than people assume. DPO is a small step from ordinary supervised fine-tuning, so if you have a few thousand real preference pairs from your own usage, encoding them is tractable. Fine-tuning versus prompting versus RAG covers whether you should.

A decision rule

If you can enumerate the preferences you want and collect pairs that demonstrate them, use DPO. It is cheaper, more stable, and good enough for the overwhelming majority of applied work.

Reach for the full RL pipeline when you have a computable reward — tests passing, a validator, a checker — or when the behaviour you want cannot be captured by a fixed dataset. And in either case, spend your effort on the quality of the comparisons rather than the choice of optimiser. Both methods faithfully learn whatever the data says, including its mistakes. Open-weight models compared covers what the current releases actually ship with.