Positional Encoding: How a Transformer Knows Token Order

A transformer has no built-in sense of order. Attention compares every token against every other token, and that comparison is identical whether a word appeared first or last. Feed the same tokens in a different order and, with nothing else added, the model computes the same scores. Positional encoding is what fixes that, and the specific fix a model uses determines how far its context can be stretched and where it starts to degrade.

Why the problem exists at all

Older sequence architectures processed tokens one at a time, so order was implicit in the processing itself. Position was never a separate concern because the machinery could not avoid it.

Transformers threw that away deliberately. Processing every position in parallel is the reason they train efficiently on modern hardware, and it is the reason the attention mechanism can look directly at distant tokens without information passing through a chain of summaries.

The cost of that parallelism is that order has to be supplied explicitly. The model receives a bag of token embeddings and, unless position is encoded into them somehow, "the cache invalidated the request" and "the request invalidated the cache" are the same input.

Absolute encodings: the first answer

The original approach adds a position-specific vector to each token embedding before the first layer. Position 1 gets one vector, position 2 another, and the sum carries both what the token is and where it sits.

Two variants dominated early. A learned table treats each position as a trainable parameter, exactly like a vocabulary entry. A sinusoidal scheme computes the vector from a fixed pattern of sine and cosine waves at different frequencies, so nothing needs training and any position has a defined value.

Both work, and both share a weakness that only becomes obvious at scale: they describe where a token sits in absolute terms, when what attention actually needs is how far apart two tokens are. Whether a subject and its verb are adjacent matters; whether they sit at positions 40 and 41 or 40,000 and 40,001 usually does not.

The extrapolation wall

A learned table has entries only for the positions seen during training. Ask for position 200,000 when the table stops at 8,192 and there is no entry — the model has literally never been given a representation for that slot.

Sinusoidal encodings at least produce a value for any position, but producing a value is not the same as producing a useful one. The pattern of interactions at distances far beyond training is one the weights never learned to interpret, and quality falls off a cliff rather than degrading gently.

This is the underlying reason a model cannot simply be told to accept a longer prompt. The context window is not an arbitrary configuration limit; past a point, the positional scheme stops carrying meaning that the rest of the network knows how to use. Context windows explained covers what that limit actually represents.

Relative positions and rotary embeddings

The better framing is relative: encode the distance between the query token and the key token, rather than the absolute index of each.

Rotary position embedding, the scheme in nearly every current open-weight model, achieves this elegantly. Instead of adding a vector to the embedding, it rotates the query and key vectors by an angle proportional to their position. When two rotated vectors are compared, the rotations partially cancel and what survives depends on the difference between the positions.

The result is a relative encoding that costs nothing extra to compute and slots into attention without changing its shape. Rotary embeddings explained goes through the mechanism in more detail and covers why it extends more gracefully than the alternatives.

How 1M-token windows were reached

Rotary encodings extend, but not for free. A model trained at 32K tokens does not automatically handle 1M just because the rotation formula defines an angle for every position.

The common approach is to rescale the rotation frequencies so that a long sequence is mapped into the range of angles the model was actually trained on, then fine-tune briefly at the longer length so the weights adapt. Several published variants of this idea exist, and vendors do not always say which one they used.

That matters when you read a spec sheet. Kimi K3, GLM-5.2, both DeepSeek V4 variants and MiniMax M3 all advertise 1M context, but the advertised number describes what the model accepts, not the length at which it was extensively trained. Reliable recall at the top of that range is an empirical question, and worth measuring on your own material rather than assuming. The 1M-context comparison covers how the current crop differ.

What you observe when it strains

Positional weakness rarely announces itself. It shows up as a model that answers confidently while ignoring a constraint stated 300,000 tokens earlier, or one that loses track of which of two similar functions you were discussing.

It compounds with the dilution of attention weight across many positions, which is why material buried in the middle of a very long prompt is used less reliably than material at either end. The lost-in-the-middle problem covers mitigations, and most of them amount to not relying on extreme distances in the first place.

The practical rule is unglamorous. Treat the advertised window as a ceiling rather than a working range, keep the instructions that must be obeyed near the start or the end, and test recall at the length you actually intend to run at before designing a system around it.