Constitutional AI: Alignment From Written Rules, Not Labels

Preference-based alignment needs comparisons: pairs of responses where someone said which was better. Collecting those from humans is slow, expensive, and hard to keep consistent across a large pool of raters.

Constitutional AI moves the human contribution up a level. Instead of judging individual responses, people write down the principles — the constitution — and a model applies those principles to produce the comparisons. The human effort goes into the rules rather than the labels.

The two phases

The first phase is supervised. The model produces a response, is then asked to critique that response against a principle drawn from the constitution, and finally to revise it in light of the critique. The revised responses become training data.

What this produces is a model that has internalised the revision rather than needing to perform it. At inference time it does not critique and rewrite; it simply tends to produce output closer to what the revision step would have generated.

The second phase is preference-based. The model generates response pairs, and a model — guided by the constitution — picks the better one. Those comparisons feed the same preference machinery that human labels would have fed. RLHF explained covers what happens downstream of the comparisons.

Why the substitution works at all

It relies on an asymmetry that shows up throughout applied LLM work: judging is easier than generating. A model that would not spontaneously produce the ideal response can often tell which of two responses better satisfies a stated principle.

Handing it an explicit rule makes the judgement easier still. "Is this response condescending?" is a far more tractable question than the open-ended "which of these is better", because the criterion is supplied rather than inferred.

That is also the boundary of the method. Where judgement requires knowledge the model does not have, or where the principle is genuinely contested, model-generated comparisons are no better than the model itself. The method scales judgement; it does not create it.

What you gain

Consistency. Human raters disagree, get tired, and interpret guidelines differently. A model applying a written rule is at least uniform, and its errors are systematic rather than random — which makes them findable.

Auditability. The constitution is a document. You can read it, argue with it, and revise it. Behaviour encoded in a hundred thousand individual rating decisions is not inspectable in any practical sense.

Iteration speed. Changing a principle and regenerating the preference data is fast. Recollecting human labels for a changed guideline is a procurement exercise.

Rater welfare. Much preference work involves reviewing genuinely unpleasant content. Reducing the volume a human must read is a real benefit that rarely gets mentioned.

What you lose

Judgements are bounded by the model doing the judging. If it has a blind spot, the constitution does not reveal it — the same blind spot produces the comparisons and the trained behaviour, so nothing in the loop catches it.

There is also a self-reference risk: a model shaping its own successor tends to reinforce its existing tendencies. Verbosity, hedging patterns and stylistic tics can compound across iterations unless something external interrupts them. Synthetic training data covers the same dynamic in the pretraining and instruction-data context.

And written principles are ambiguous in ways their authors do not anticipate. A rule about avoiding harm has to be applied to a security research question, a medical dosage question and a fiction request, and the interpretation the model settles on may not be the one you meant. Over-refusal frequently starts here. Why benchmarks disagree covers the analogous problem of underspecified criteria in evaluation.

The connection to system prompts

There is a useful parallel for anyone building on these models. A system prompt is a constitution applied at inference time instead of during training — a written set of principles the model is asked to follow.

The difference is durability. Trained behaviour is baked into the weights and persists across a long conversation. A system prompt competes with everything that follows it, and its influence decays as the context fills. System prompts explained covers that decay and how to counter it.

The practical implication: do not expect a system prompt to override a strongly trained disposition. It shifts behaviour at the margin. If a model has been trained to refuse a category of request, telling it not to usually fails, and the attempts that succeed are the ones you would rather nobody discovered. Prompt injection and agent security covers that surface.

What it does not fix

Alignment methods shape behaviour, not knowledge. A model trained under any constitution still hallucinates, because fabrication comes from the next-token objective producing fluent continuations regardless of whether the facts exist.

A well-written constitution can encourage the model to express uncertainty more often, which helps at the margin. It cannot give the model access to whether it actually knows something. Why LLMs hallucinate covers why that limitation is structural.

Nor does it settle what the principles should be. The method faithfully implements whatever values are written down, and writing them down is the hard part. Constitutional AI is a mechanism, not an answer.

The takeaway for builders

The transferable idea is not the training method — it is that explicit written criteria beat implicit judgement, and that a model applying stated criteria is more consistent than one asked to judge in the abstract.

Use that in your own evaluation and review pipelines. If you are using a model to grade output, give it a written rubric rather than asking whether the answer is good. The improvement in consistency is immediate and it costs nothing but the time to write the rubric down. How to benchmark LLMs on your own work covers building that harness.