Residual Connections: Why Deep Models Train at All

A residual connection is arithmetically the simplest thing in a transformer: take the input to a block, add it to the block's output, pass the sum onward. It is also the change that made deep networks trainable, and the reason a modern model can be a hundred layers deep instead of twelve.

The problem it solved

Before residual connections, adding layers past a certain depth made networks worse — not just harder to train, but worse on the training data itself. That is a strange result. A deeper network can always represent whatever the shallower one did, by having its extra layers do nothing.

The issue was that "do nothing" is difficult to learn. A layer that transforms its input has to hit an exact configuration to reproduce it unchanged, and gradient descent has no particular reason to find that configuration.

So extra layers did not sit out politely. They contributed noise, and every layer of noise had to be undone by the ones above it. Depth became a liability rather than an asset.

Making the identity the default

The fix flips the default. Instead of asking each block to produce the next representation, ask it to produce a change to the current one, then add that change to what came in.

Now doing nothing is easy: output zero. A block with nothing useful to contribute can converge on a small output and become effectively transparent, and gradient descent finds that state readily because zero is where the weights start out near anyway.

Each block therefore learns a refinement rather than a replacement. That is why they are called residual connections — the block computes the residual, the part that still needs adding.

The gradient highway

The training benefit is more direct still. During backpropagation, gradients travel from the output back towards the input, and at each step they are multiplied by whatever that layer does to them.

Multiply a hundred numbers slightly below one and you get approximately zero. Multiply a hundred slightly above one and you get an overflow. This is the vanishing and exploding gradient problem, and it is what capped useful depth.

An addition passes gradient through unchanged. The residual path therefore forms an unobstructed route from the loss all the way back to the first layer, alongside the multiplicative route through the blocks. Even if the blocks attenuate the signal, the direct path keeps it alive. This is also why pre-norm layer normalisation won: it keeps normalisation off the residual path and leaves the highway clean.

The residual stream as a shared bus

Once every block adds to a running total, a useful way to see the architecture emerges. Rather than a pipeline where data is transformed stage by stage, a transformer looks more like a shared bus that every component reads from and writes to.

An attention head reads the current state, decides what to contribute, and adds it. A feed-forward block does the same. Nothing overwrites what came before; contributions accumulate.

This is not just a metaphor — it is how interpretability work actually treats these models, tracing which components wrote which information into the stream at which layer. It also explains a practical observation: attention tends to handle moving information between positions while feed-forward blocks handle recalling facts, and both write into the same place.

What it explains about pruning and distillation

Because each block contributes an increment rather than a transformation, removing one does not break the chain. The stream simply misses that increment.

This is why layer pruning degrades models gracefully rather than catastrophically, and why middle layers are usually the most removable — early layers build basic representations and late layers commit to an output, while the middle does incremental refinement that partially overlaps.

The same property underpins early-exit schemes, where easy tokens are decided at a shallow layer, and it is part of why distillation into a shallower student works as well as it does. A network of increments has redundancy that a strict pipeline would not.

Where it shows up in your work

You will not configure this. What you get is a better model of what a transformer is doing, and a few concrete consequences.

Smaller variants of a model family are often the same architecture with fewer layers, which is why they tend to feel like the larger model with less depth of reasoning rather than a different model entirely. The increments are similar; there are simply fewer of them. Small versus large models covers where that gap bites.

It also sets expectations for quantised and pruned deployments. Because contributions accumulate additively, small per-layer errors add up along the stream rather than cancelling, which is why aggressive compression tends to show as gradual drift in quality rather than an obvious break.

The takeaway

Residual connections are the clearest example of a pattern worth internalising about this field: the change that unlocked the most capability was not a new kind of computation, but a rearrangement that made an existing computation trainable.

The same story repeats elsewhere. Flash attention rearranged memory access without touching the mathematics. Pre-norm moved a normalisation by one position. When you read that a new model has an architectural improvement, it is usually this kind of change rather than a new mechanism. The transformer architecture overview puts the pieces together.