Expert Routing in MoE: How a Token Picks Its Path

In a mixture-of-experts model, the experts are the easy part. They are ordinary feed-forward blocks. The interesting component is the router: a small network that looks at each token and decides which handful of experts will process it.

That decision is made independently for every token at every MoE layer. It is the mechanism that lets a model hold trillions of parameters while computing with only tens of billions per token, and it is also the source of most of the odd behaviour people notice in MoE models.

What the router actually does

At each MoE layer, the router takes the token's current representation and produces a score for every expert. The top few scores win, the token is sent to those experts, and their outputs are combined weighted by the router's confidence.

The number chosen is the top-k. Kimi K2.6 holds 384 experts and routes each token to eight of them, plus one shared expert that every token passes through regardless. That shared expert is a common design: it carries the general-purpose behaviour so the routed experts can afford to be more specialised.

Nothing about this is hand-designed. The router is trained jointly with everything else, so the specialisation that emerges is whatever reduced loss during training, not a taxonomy anyone chose. The broader MoE picture covers why labs build models this way.

Experts are not topics

The most persistent misconception is that one expert handles Python, another handles French, another handles arithmetic. Interpretability work consistently finds something messier.

Specialisation tends to be at the level of token-scale patterns rather than subject matter — punctuation and structure, certain syntactic positions, particular kinds of continuation. Experts do develop preferences, but they rarely map onto categories a person would name.

This matters practically because it kills the intuition that you could pick experts yourself, prune the ones you do not need, or predict which will fire for a given prompt. A single sentence typically routes across a wide spread of experts, and consecutive tokens in the same sentence often go to different ones.

Token-choice and expert-choice

There are two ways to frame the assignment, and they behave differently.

Token-choice routing lets each token pick its top experts. It is the intuitive design and the more common one, but it offers no guarantee that load is even — nothing stops most of a batch selecting the same expert, leaving others idle.

Expert-choice routing inverts it: each expert selects the tokens it scores highest, up to a fixed capacity. Load is then balanced by construction, at the price of some tokens not being selected by anyone and passing through with less processing.

Token-choice with an explicit balancing pressure during training is the mainstream answer. It keeps the intuitive semantics and pushes the imbalance problem into the training objective, which is where most of the engineering effort now sits.

Capacity limits and dropped tokens

Serving imposes a hard constraint the router does not know about. Each expert runs on a device with finite memory and a fixed buffer for the tokens it will process in a step.

When more tokens route to an expert than its capacity allows, something has to give. Excess tokens may be dropped — passed through the layer via the residual connection without expert processing — or the batch may be reshaped, or capacity raised at the cost of memory.

Dropping is not catastrophic because the residual path preserves the token, but it does mean a token can receive less computation than the architecture nominally provides. Whether it happens depends on the batch it was unlucky enough to share a step with.

Why identical prompts can differ

This is the consequence developers actually run into. Routing is computed per token, but capacity is enforced per batch — so what happens to your token can depend on the other requests batched alongside it.

Two identical requests sent minutes apart may be batched with different neighbours, hit different capacity pressure, and produce slightly different output. Nothing about your request changed. The batch did.

This adds a source of nondeterminism that dense models simply do not have, on top of the floating-point and scheduling variation every model exhibits. If you have been chasing reproducibility on an MoE endpoint and finding a temperature of zero insufficient, this is frequently the reason. Determinism and seeds covers what you can and cannot pin down.

The practical response is to design for it rather than to fight it. Validate structured output instead of assuming byte-identical responses, and treat exact-match regression tests against a hosted MoE endpoint as unreliable by construction.

What routing costs at serving time

The router itself is tiny and its compute is negligible. The expense is what its decisions imply for data movement.

When experts are distributed across devices, each routing decision becomes a network transfer — the token goes to whichever device holds its expert and the result comes back. That all-to-all exchange happens at every MoE layer, and its volume depends on how the batch happened to route. Expert parallelism is built around managing exactly this.

The result is throughput that varies with traffic in ways dense serving does not. Even routing keeps every device busy; concentrated routing creates stragglers that hold up the whole step.

What to take from this

If you call an API, the routing is invisible and the model is simply a model. The two things worth carrying are that reproducibility is weaker on MoE endpoints, and that the headline parameter count describes memory rather than the computation your token receives.

If you self-host or fine-tune, routing is the fragile part. Fine-tuning on a narrow dataset can collapse traffic onto a few experts and quietly turn an expensive model into a small one, which is a large part of why dense models remain easier to adapt. Check expert utilisation before and after any tuning run, and treat a sharply skewed distribution as a failure signal rather than a curiosity.