Pretraining vs Post-Training: Where a Model Gets Its Behaviour

Model training splits into two stages that do very different jobs. Pretraining builds a general model of language and the world from an enormous unlabelled corpus. Post-training takes that model and shapes it into something that follows instructions and behaves acceptably.

The split matters practically because it tells you which problems prompting can fix. A behaviour problem usually can be worked around. A knowledge problem usually cannot, and needs retrieval instead.

What pretraining does

Pretraining is next-token prediction over a very large text corpus. The objective is mechanically simple: given a prefix, predict what follows, and adjust the weights when the prediction is wrong. Repeat across a corpus far larger than any human could read.

Predicting the next token well requires a great deal of incidental structure. To continue a sentence about a chemical reaction the model must represent something about chemistry. To close a bracket correctly it must track nesting. Capability emerges as a side effect of compression, not because anyone specified it. How LLMs actually work covers the prediction loop.

This stage dominates the compute budget and the cost. It is also where nearly all the knowledge enters. If a fact was not represented in the pretraining corpus, no later stage puts it there in any reliable way.

What a pretrained model is like to use

Badly, if you have not seen one. A raw pretrained model is a text continuer, not an assistant. Ask it a question and it may continue with a list of similar questions, because in the corpus, questions are frequently followed by more questions.

It has no notion of a conversational turn, no notion of being helpful, and no reason to stop. It has enormous latent capability and no interface to it.

That gap is exactly what post-training closes. The capability is largely already present after pretraining; what is missing is the habit of applying it in the shape a user expects.

What post-training does

Post-training is a sequence of much smaller, much more targeted stages on curated data. The first is typically instruction tuning: supervised training on examples of instructions paired with good responses, which teaches the model the shape of a helpful reply. Instruction tuning explained covers that stage.

The second is preference optimisation, where the model is trained on comparisons between responses rather than on single target outputs. This handles qualities that are easy to judge and hard to demonstrate — tone, hedging, refusal behaviour, how much detail to give. RLHF explained covers the reinforcement-learning route and DPO versus RLHF covers the simpler alternative.

Increasingly there is a third stage aimed at reasoning, where the model is trained on problems with verifiable answers and rewarded for reaching them. This is what distinguishes models that reason at length before answering. Reasoning models explained covers the difference in behaviour.

Which stage owns your problem

This is the useful diagnostic. If the model does not know something — your internal API, a library released after its data was collected, a private document — that is a pretraining gap. Prompting cannot fix it, and neither can fine-tuning in any dependable way. Supply the information in context.

If the model knows the thing but presents it badly — wrong format, wrong length, hedging when you want a decision, refusing something innocuous — that is post-training behaviour. Prompting moves it, because you are working with a disposition rather than an absence.

The test is simple: give the model the fact and see whether the answer becomes correct. If yes, it was a knowledge gap. If it still gets it wrong, the problem is elsewhere. Fine-tuning versus prompting versus RAG covers choosing the intervention.

Why open weights come in two flavours

Labs releasing open weights often publish both a base model and an instruction-tuned one. The base model is the pretrained artefact before post-training; the instruct model has been through it.

If you plan to do your own post-training, the base model is the right starting point — you are not fighting someone else's preference tuning. If you want something usable immediately, take the instruct model, which is what almost everyone actually wants.

GLM-5.2 and both DeepSeek V4 variants ship under MIT licences; Kimi K3 published weights on 27 July 2026 under Moonshot's own terms with a revenue threshold for model-as-a-service. The licence governs what you may do with either artefact, and it is worth reading before building on one. Open weights versus open source covers the distinction.

Why post-training explains model personality

Two models pretrained on broadly similar public data can feel completely different to use. One is terse, one is verbose. One refuses readily, one complies with almost anything. One reliably emits JSON, one wraps it in commentary.

Those differences come overwhelmingly from post-training, because that is where the lab expresses its judgement about how a model should behave. The pretraining corpora are more alike than the products are.

This is also why a model can regress on your task after a minor version update. The lab adjusted its post-training and shifted a behaviour you were depending on, without touching the underlying knowledge. Pinning model versions covers protecting yourself from that.

The takeaway

Ask which stage owns the problem before choosing a fix. Missing knowledge goes in the prompt. Unwanted behaviour goes in the system prompt or the output contract. Neither is helped by the other.

And when evaluating a model, evaluate the post-trained artefact you will actually call. Parameter counts and pretraining scale describe potential; the post-trained behaviour is what your code has to work with. What is a parameter count covers why the headline size tells you less than it appears to.