Rotary Embeddings (RoPE) Explained With Clock Hands

Rotary position embedding is the positional scheme in almost every open-weight model shipping today, and it is the reason context windows could be stretched from a few thousand tokens to a million without retraining from scratch. The mechanism is genuinely simple once you stop thinking about vectors and start thinking about clock hands.

The idea in one move

Earlier schemes added a position vector to the token embedding, mixing "what this token is" and "where it sits" into one sum. RoPE does not add anything. It rotates.

Take the query and key vectors, split them into pairs of numbers, and treat each pair as a point on a circle. To encode position 40, rotate every pair by an angle proportional to 40. To encode position 41, rotate slightly further.

The token's identity is preserved because rotation does not change a vector's length — it only changes its direction. Position becomes an angle rather than an additive offset, and that turns out to matter enormously.

Why rotation gives relative distance for free

Attention scores compare a query against a key by measuring how aligned they are. Alignment between two rotated vectors depends only on the angle between them, and that angle is the difference of the two rotations.

So if the query sat at position 40 and the key at position 35, the comparison behaves as though the distance were five — regardless of whether those tokens sat at positions 40 and 35 or 40,040 and 40,035.

That is the whole trick. Nothing in the architecture had to be changed to make attention relative; rotating the inputs made it relative automatically. It is the property absolute positional encodings conspicuously lack, and the reason they extrapolate so badly.

Fast hands and slow hands

Not every pair rotates at the same rate. RoPE assigns each pair its own frequency, spanning from very fast to very slow — like a clock with a second hand, a minute hand, an hour hand and a hand that completes one turn per year.

The fast pairs distinguish adjacent tokens sharply: one step moves them a long way round the circle, so "the token immediately before" is unmistakably different from "two tokens before". They are useless at distance, because after a few hundred steps they have wrapped many times and carry no usable signal.

The slow pairs are the opposite. They barely move between neighbouring tokens, so they cannot resolve fine order, but they still discriminate between position 1,000 and position 100,000 because they have not completed a full turn. Together the set gives the model both fine local ordering and coarse global placement.

How context windows got extended

Here is where the design pays off. A model trained at 32K tokens has learned to interpret angles in a certain range. Ask for position 500,000 and the slow hands are now in territory the weights never saw, and quality falls apart.

But because position is an angle, you can rescale. Slow the rotation down so that 500,000 tokens sweep through the same range of angles the model trained on, then fine-tune briefly at the longer length so the weights adapt to the compressed spacing. Several published variants of this idea differ in how much they slow the fast hands versus the slow ones.

You cannot do this with a learned lookup table, because there is nothing to interpolate — position 500,000 simply has no entry. Rotation is continuous, so intermediate values are meaningful by construction. That is the concrete reason RoPE won.

What the rescaling costs

Nothing is free. Compressing a million positions into the angular range a model learned at 32K means neighbouring positions are now closer together in angle than they were during training.

Fine local ordering therefore gets slightly blurrier. In practice this is rarely what you notice, because the dominant long-context failure is a different one: attention weight spread thin across an enormous number of candidate positions. The lost-in-the-middle problem covers that effect.

The more useful implication is that a stated window describes what the model accepts after extension, not the length at which it was heavily trained. Kimi K3, GLM-5.2, both DeepSeek V4 variants and MiniMax M3 all advertise 1M context; Kimi K2.6 advertises 256K. Those numbers say nothing about how reliably each recalls a detail buried at 800K, which is an empirical question. The 1M-context comparison covers what differs between them.

What to do with this

Test recall at the length you intend to run at, using your own material, before designing a system that depends on it. A synthetic needle-in-a-haystack test is easy to pass and a poor proxy for finding a subtly relevant function in a large repository.

Assume degradation is gradual rather than a cliff. There is rarely a sharp point where a long-context model stops working; there is a range where it silently gets less reliable, which is worse because nothing errors.

And when a long context is available, that is not a reason to fill it. Retrieving the twenty thousand tokens that matter beats supplying eight hundred thousand that mostly do not — it is cheaper, faster and more accurate. RAG versus long context works through where that line sits.