Layer Normalisation: The Part That Keeps Training Stable

Layer normalisation is the least glamorous component in a transformer and one of the few that no serious model omits. It does nothing clever with language. Its entire job is to stop numbers from drifting to scales the rest of the network cannot handle, and the way that job is done has changed twice in ways that matter to anyone serving models at low precision.

The problem: scale drift with depth

A transformer is a stack of blocks, each transforming the output of the last. A hundred layers deep, small systematic effects compound.

If each block tends to enlarge the magnitude of its output slightly, then by layer 80 the values are enormous. If each shrinks them slightly, by layer 80 they have collapsed towards zero. Neither state is recoverable by the layers that follow.

During training this is fatal. Gradients flowing backwards through a stack with drifting scale either explode into numerical overflow or vanish into nothing, and the model either diverges or stops learning. Depth was hard to achieve for exactly this reason before normalisation became standard.

What normalisation actually does

Layer normalisation takes the vector representing one token at one point in the network, measures its typical magnitude, and rescales it to a consistent size. It then applies a small set of learned parameters so the network can still choose a scale where it wants one.

The crucial detail is what it normalises across. Batch normalisation, the older technique from computer vision, normalises across the examples in a batch — every image in a batch influences the statistics of every other. That is unusable for autoregressive language models, where sequences have wildly different lengths and inference frequently runs one request at a time.

Layer normalisation operates within a single token's own vector. It needs no other examples, behaves identically at batch size one and batch size 512, and is unaffected by sequence length. That independence is why it, rather than batch norm, became the transformer default.

Pre-norm versus post-norm

The original transformer applied normalisation after each sub-layer, on the sum of the block's output and its input. Every model since roughly 2020 applies it before instead, on the input to the sub-layer.

The reason is what it does to the residual path. In post-norm, the skip connection running through the network passes through a normalisation at every layer, so a gradient travelling backwards is rescaled a hundred times over. In pre-norm, the residual path is clean end to end and normalisation sits only on the branch feeding each sub-layer.

The consequence is practical: post-norm models need careful learning-rate warm-up and are prone to diverging, while pre-norm models train stably at depth with far less babysitting. Residual connections explained covers why that unobstructed path matters so much.

Why RMSNorm replaced it

Standard layer normalisation does two things: it subtracts the mean, then divides by the standard deviation. RMSNorm skips the first step and only rescales by magnitude.

Empirically the mean subtraction contributes very little to quality, and dropping it removes a pass over the vector and a synchronisation point in the kernel. On a component that runs twice per layer across a hundred layers for every token, that adds up.

Nearly every recent open-weight model uses RMSNorm. It is a good illustration of how architecture actually evolves: not through dramatic reinvention, but by finding that a step everybody assumed was load-bearing turns out not to be.

The connection to low-precision serving

This is where normalisation stops being trivia. Quantisation works by representing weights and activations in fewer bits, which means a much narrower range of values can be expressed.

Transformers develop outlier features — a handful of dimensions that carry values orders of magnitude larger than the rest. Normalisation layers sit right where those outliers are handled, and they are among the components most sensitive to being quantised naively. This is why serious quantisation schemes keep normalisation parameters at higher precision rather than compressing everything uniformly.

It also explains a class of confusing bug reports. A quantised model that produces fluent text on short prompts and degenerates into repetition on long ones is often suffering from accumulated numerical error that normalisation would have contained at full precision. If you self-host and see behaviour that degrades with length rather than being uniformly worse, precision is the first thing to check.

Why identical inputs can give different outputs

Normalisation involves summing across a vector, and floating-point addition is not associative — adding numbers in a different order gives slightly different results.

Inference servers batch requests together and split work across devices in ways that depend on what else is in flight, so the summation order for your request can vary between calls. Those tiny differences propagate through a hundred layers and occasionally flip which token wins.

That is a large part of why a temperature-zero request is not reliably reproducible across calls even on the same provider. Determinism and seeds covers what you can and cannot pin down.

The takeaway

You will never configure a normalisation layer. What it gives you is a set of explanations: why depth is possible at all, why quantised models fail in the specific ways they do, and why identical requests do not always return identical text.

The one operational rule worth carrying: when a self-hosted model degrades with prompt length rather than failing outright, suspect numerical precision before you suspect the weights. The transformer architecture overview puts the component in context.