Transformer Architecture Explained for Working Developers

Nearly every model you use is a transformer, and the architecture has been stable enough since 2017 that understanding it once pays off indefinitely. You do not need the mathematics to get the useful parts.

The overall shape

Text becomes tokens. Tokens become vectors through an embedding lookup. Those vectors pass through a stack of identical blocks — dozens in a small model, over a hundred in a large one. The final vectors are projected back into a distribution over the vocabulary, and one token is sampled.

Then the whole thing repeats with that token appended. Generation is this loop, one token at a time, which is why output arrives sequentially and why generation cannot be parallelised the way prompt processing can.

What is inside a block

Each block has two parts, and every block has the same structure.

Attention lets each position gather information from other positions. This is where a pronoun connects to its referent and a variable connects to its declaration. It is the part that moves information between positions.

A feed-forward network processes each position independently, transforming what attention gathered. It is typically several times wider than the model's hidden dimension, and it holds most of the parameters. Where attention routes information, the feed-forward layer computes on it.

Both are wrapped in residual connections and layer normalisation. The residual connection adds the input back to the output, so each block modifies a running representation rather than replacing it. This is what makes very deep stacks trainable — gradients flow through the additions rather than having to survive every transformation. Residual connections explained covers why that matters.

Where the parameters live

Most parameters are in the feed-forward layers, not attention. That surprises people, because attention gets the conceptual attention.

This is what mixture-of-experts exploits. Instead of one feed-forward network per block, an MoE model has many, and routes each token to a small subset. Total parameters rise enormously while computation per token stays modest — which is why a 1.6T model can activate only 49B per token. Mixture-of-experts explained covers the routing.

Why position needs encoding

Attention has no inherent notion of order. It computes relevance between pairs, and a pair does not know which came first. Without positional information, "the test broke the build" and "the build broke the test" look identical.

Position is therefore injected explicitly. Modern models mostly use rotary embeddings, which encode position by rotating query and key vectors by an angle proportional to their index. Relative distance falls out of the geometry, which turns out to extrapolate better to longer sequences than earlier schemes.

This matters practically because context-window extension techniques usually work by manipulating positional encoding. When a model advertises a longer window than it was trained on, that is generally what has been adjusted, and it is why the extended range often performs worse than the native one.

Why models predict rather than know

The output layer produces a score for every token in the vocabulary. Those scores become a probability distribution, and a token is sampled from it.

That is the entire objective, at every step. There is no lookup, no database, no verification. The model produces a plausible continuation given what came before.

Most surprising behaviour follows directly from this. Hallucination is not a malfunction — a fluent, plausible, wrong continuation is exactly what the objective rewards when the model lacks the fact. Sensitivity to phrasing follows because different prefixes genuinely have different likely continuations. Inconsistency between runs follows because sampling is stochastic unless you make it otherwise. Why LLMs hallucinate covers this properly.

Depth and width, and what each buys

Depth is the number of blocks; width is the hidden dimension. Both increase capacity differently.

Deeper stacks allow more sequential transformation — more steps of refinement applied to the representation. Wider layers allow more to be held at each step. Labs tune the ratio empirically, and the balance has shifted over time as training techniques improved.

Depth also has a direct cost consequence: every block must run in sequence for every token, so a deeper model has higher latency per token even at equal parameter count. That is part of why the cheap tier of a family often feels noticeably snappier rather than merely marginally so.

What to take from this

Three things generalise beyond any particular model.

Attention explains the context behaviour — quadratic cost, degraded middles, the value of caching. Feed-forward layers explain where capacity and parameters live, and therefore what mixture-of-experts is doing. And the sampling objective explains hallucination, phrasing sensitivity and run-to-run variation, which together account for most of the frustration people have with models.

None of that requires the mathematics. It requires knowing that the model is producing a probability distribution over next tokens, and everything else is machinery for doing that well.