Load Balancing in MoE: Stopping Experts Going Idle

A mixture-of-experts model is a bet that its parameters will be used. Hold 384 experts and route to eight per token, and the arithmetic only works out if the traffic actually spreads — if most tokens funnel through the same dozen experts, you have paid for a trillion parameters and are running a much smaller model.

Keeping the load spread is not automatic. It has to be forced during training and managed during serving, and both stages fail in ways that are easy to miss because the model keeps producing plausible output either way.

Why routing collapses on its own

The router learns which experts to pick, and the process has a feedback loop built in. An expert that gets chosen slightly more often early in training receives more gradient, improves faster, and becomes a better choice — so it gets chosen more often still.

Left alone, this converges on a small set of well-trained experts and a long tail that never develops. The model trains, the loss falls, and the capacity you were paying for never materialises. This failure is called expert collapse, and it is the reason MoE training needs explicit counter-pressure.

The standard remedy is an auxiliary objective that penalises uneven expert usage during training, nudging the router towards spreading tokens. It works, but it competes with the main objective — pressure to balance is pressure away from routing each token to the expert that would handle it best.

Tuning that balance is one of the genuinely difficult parts of training an MoE model, and it is why lab write-ups spend disproportionate space on it. Too little and the model collapses; too much and routing becomes arbitrary and quality suffers.

Approaches that avoid a competing loss

Because the auxiliary loss fights the main objective, recent designs try to balance without it.

One family adjusts a per-expert bias term that shifts routing scores. An overloaded expert has its bias lowered so it wins fewer tokens; an underused one has it raised. Since the bias affects selection rather than the loss, balance is achieved without adding a gradient that pulls against quality.

Another approach constrains balance structurally — restricting how many devices a token's experts may span, or reserving a shared expert that every token uses so the routed experts carry less of the general load. Kimi K2.6's single shared expert alongside eight routed ones is an example of that shape.

None of these makes the problem disappear. They move it from being an explicit loss term to being a scheduling or architectural constraint, which is generally a better place for it.

The serving problem is different

A model can be perfectly balanced on average across a training corpus and badly skewed on a particular batch. Training balance is a statistical property; serving load is whatever this second's traffic asks for.

Real traffic is not uniform. A batch of requests that all look similar — the same coding agent, the same system prompt, the same file type — will route similarly, because routing depends on token content. Homogeneous traffic produces homogeneous routing, and homogeneous routing produces skew.

When experts are spread across devices, that skew becomes a straggler problem. The devices holding hot experts do more work while others wait, and since the layer cannot complete until every device is done, the whole batch runs at the pace of the busiest device. Expert parallelism is where this cost lands.

The observable consequence is throughput that varies with the shape of your traffic, not just its volume. This is why MoE tokens-per-second figures are harder to reproduce than dense ones, and why a benchmark using varied prompts flatters a deployment that will serve one narrow workload.

Capacity, dropping and the memory bill

Serving stacks bound the problem with a capacity limit: a maximum number of tokens each expert will process per step. The buffer has to be allocated in advance, so capacity is a memory decision.

Set it generously and you reserve memory that is idle whenever routing is even. Set it tightly and tokens routed to a saturated expert are dropped — passed forward through the residual connection without that expert's processing.

Dropped tokens are not lost, but they receive less computation than the architecture nominally provides, and which tokens get dropped depends on batch composition rather than anything about the request. It is another reason identical prompts to an MoE endpoint are not reliably identical, as the routing mechanism makes concrete.

Fine-tuning is where teams break it

If you tune an MoE model on your own data, you can undo the balance the original training established.

Narrow data routes narrowly. Tune on one codebase in one language and the router learns that a small set of experts suffices, because for that data it does. Gradient flows almost entirely to those experts, the rest drift, and the model quietly becomes far smaller than the one you started with — while still costing full memory to host.

The symptom is a model that performs well on data resembling the tuning set and noticeably worse than the base model on anything else. It is easy to attribute that to ordinary overfitting and miss the structural cause.

Instrument expert utilisation before and after any tuning run, and treat a sharply concentrated distribution as a failure signal. Freezing the router is one mitigation. Choosing a dense model is often the simpler one, and the architecture comparison makes that case in more detail.

What to do with this

As an API consumer, the practical takeaway is modest: expect throughput and latency on MoE endpoints to vary with traffic shape in ways dense models do not, and do not build exact-match tests against them.

As an operator, export per-expert utilisation and watch its distribution the way you watch queue depth. A flat distribution means the model you are hosting is the model you are paying for. A spiky one means some of that memory is decoration, and it is worth knowing which before you scale the deployment.