Sparse vs Dense Activation: What Runs on Every Token

DeepSeek V4 Pro holds roughly 1.6T parameters and activates about 49B of them per token. GLM-5.2 holds around 744B and activates about 40B. Kimi K3 holds 2.8T and activates 104B. In each case, well over ninety percent of the model does nothing for any particular token.

That is sparse activation, and it is the defining property of every frontier open-weight model in 2026. Dense activation — running every parameter on every token — is what smaller models still do, and what most people picture when they read a parameter count.

The distinction in one paragraph

A dense model applies its entire weight set to each token. A 27B dense model such as Qwen 3.6 27B does 27B parameters' worth of arithmetic per token, every token, with no exceptions.

A sparse model routes each token through a subset. The unused parameters still have to be held in memory, because any token might need them, but they cost nothing in computation for the tokens that do not.

So sparsity separates two things a dense model welds together: how much the model knows, and how much work it does per token. That separation is the entire reason the architecture exists.

Why memory and compute stop tracking each other

In a dense model, doubling parameters doubles both memory and per-token compute. Sparsity breaks that link and the two costs behave completely differently afterwards.

Memory scales with total parameters. Every expert must be resident because routing is decided per token at run time, and you cannot know in advance which will be needed. This is a fixed cost paid whether the deployment serves one request or a million.

Compute scales with active parameters. The per-token arithmetic is set by what actually runs, which is why a 1.6T model can be priced near a much smaller one — DeepSeek V4 Pro is around $0.44 in and $0.87 out per million tokens, well below what its headline size would suggest.

Once you see the split, provider economics become obvious. Memory is amortised across every request the fleet serves, so at scale it is nearly free per request. Compute is not amortised at all. The architecture comparison works through what that means when you are the one hosting.

Bandwidth is the part people miss

Sparse activation reduces arithmetic cleanly. It reduces memory traffic much less cleanly, and memory traffic is what actually limits generation speed.

Producing one token means reading the weights it needs out of memory. If a batch of tokens routes to many different experts — and with hundreds of experts it usually does — the union of weights read for that batch can be a large fraction of the model, even though each individual token touched only a sliver.

The activation sparsity that looks so favourable per token therefore dilutes as batch size grows. This is one reason sparse models benefit less from large batches than the active-parameter figure implies, and why their throughput is harder to predict. Uneven expert load compounds it.

Reading a spec sheet correctly

The headline parameter count of a sparse model is a memory figure wearing a capability costume. Treating it as a capability figure produces comparisons that are wrong by an order of magnitude.

Compare active counts when you want an intuition for per-token computation. DeepSeek V4 Pro's 49B active against Qwen 3.6 27B dense is under two to one, which is a far better predictor of the observed gap — 80.6 percent SWE-bench Verified against 77.2 — than 1.6T against 27B would ever be.

Compare total counts when you want an intuition for what it takes to host. That is where the 1.6T matters, and Kimi K3's roughly 1.6TB of weights is the number that decides whether self-hosting is on the table.

Neither figure alone tells you much. What a parameter count means is worth settling before comparing models across architectures at all.

Does sparsity cost quality

Per active parameter, sparse models are generally not as efficient as dense ones. A dense 40B trained comparably would tend to beat the 40B-active slice of a large sparse model on narrow, bounded tasks.

What sparse models win on is breadth. The extra parameters store knowledge that only surfaces sometimes, and that is exactly the shape of real work — occasional obscure library, unusual language, rare failure mode. A dense model of the same active size has nowhere to keep that.

Which explains a pattern that confuses people: a good dense model in the tens of billions can sit within a few points of a trillion-parameter sparse model on a benchmark and still feel weaker in daily use. Benchmarks are bounded; the work is not.

Sparsity also lets a lab add capability without adding per-token cost. DeepSeek V4 Flash activates 13B and prices around $0.14 in and $0.28 out, offering coverage a 13B dense model could not while charging like one. Whether you need the coverage is the question worth asking before paying for it.

Practical implications

If you buy tokens, ignore the architecture and buy on capability, price, context and latency. Sparsity is why the prices look the way they do, and its operational costs are the provider's.

If you host, the total parameter count is your constraint and the active count is a distraction. A sparse model does not become affordable to run because it computes cheaply — it becomes affordable when you serve enough traffic to amortise the memory. Below that volume, dense wins on every axis that touches your budget.

If you fine-tune, sparsity is a liability. Narrow data concentrates routing and lets unused experts drift, so you can end up hosting a trillion parameters and effectively running a fraction of them. Routing behaviour is what to instrument if you go ahead anyway.