Dense vs Mixture-of-Experts: Which You Can Actually Run

Almost every model at the top of the open-weight field in 2026 is a mixture-of-experts design. Almost every model you can run on one GPU is dense. That is not a coincidence, and understanding why explains most of what is confusing about model specifications.

The core difference in one paragraph

A dense model runs every parameter on every token. A 27B dense model does 27B parameters' worth of computation per token and needs 27B parameters' worth of memory.

A mixture-of-experts model holds many parameters but routes each token through a small subset. DeepSeek V4 Pro holds roughly 1.6T and activates about 49B per token. It needs memory for all 1.6T and compute for 49B.

So MoE decouples two things that are joined in a dense model: how much the model knows and how much it thinks per token. That is the whole trick.

Why this makes specifications misleading

The headline parameter count of an MoE model tells you about memory, not capability. Comparing a 1.6T MoE against a 27B dense model as "sixty times bigger" is wrong in the way that matters — the honest comparison is 49B active against 27B active, which is under two to one.

This is why a well-trained dense 27B can land within a few points of a trillion-parameter MoE on bounded benchmarks. Qwen 3.6 27B reports 77.2 percent on SWE-bench Verified against DeepSeek V4 Pro's 80.6. The gap is real, and it is nothing like sixty to one.

Whenever you see a trillion-parameter figure, find the active count before forming an opinion. Mixture-of-experts explained covers the routing mechanism in detail.

What each architecture costs to run

Dense. Memory and compute scale together and both are predictable. Throughput is flat and easy to plan: every token costs the same, so tokens per second stays stable until you saturate the hardware. A dense 27B fits on one accelerator when quantised.

MoE. Memory is the binding constraint. Kimi K3's weights are roughly 1.6TB — a multi-node deployment with fast interconnect between nodes, because tokens routed to experts on other nodes require communication.

MoE throughput is also less predictable. If traffic routes unevenly across experts, some are overloaded while others idle, and real throughput lands below the theoretical figure. Published tokens-per-second numbers for MoE models are correspondingly harder to reproduce under production traffic.

Why labs build MoE anyway

Because capability tracks total parameters more than active ones, and inference cost tracks active parameters more than totals.

That gives a lab a lever nothing else provides: add knowledge without adding per-token cost. For an API provider serving millions of requests, holding 1.6T in memory once and paying 49B of compute per token is a very good trade. The memory is a fixed cost; the compute is per-request.

The economics invert for a single team self-hosting. You pay the same enormous memory cost and amortise it over far fewer requests.

Which yields the rule: MoE favours the serving provider, dense favours the self-hoster. If you are using an API, MoE's downside is somebody else's problem and you get the capability. If you are running it yourself, dense is frequently the only practical option.

Quantisation interacts with this

Quantisation reduces weight precision to cut memory. It helps both architectures but disproportionately helps dense models get onto single accelerators, which is exactly the threshold that matters.

The caution: quantisation degrades quality unevenly. It often leaves conversational fluency intact while eroding precise multi-step reasoning — which is the capability coding work depends on. A quantised model can feel fine in casual testing and underperform on real tasks. Evaluate the quantised build you will actually deploy. Quantization explained covers which formats hold up.

Choosing

• Using an API — ignore architecture entirely. Compare capability, price and context. MoE's costs are the provider's.
• Self-hosting on limited hardware — dense, almost certainly. Qwen 3.6 27B is the current reference point.
• Self-hosting at scale with real infrastructure — MoE becomes viable and the capability ceiling is higher.
• Comparing spec sheets — always find the active parameter count. The total is a memory figure wearing a capability costume.

Fine-tuning behaves differently too

If you intend to fine-tune rather than just serve, the architectures diverge again.

Fine-tuning a dense model is well-trodden. The tooling is mature, the memory requirements scale predictably with model size, and parameter-efficient methods like low-rank adaptation work exactly as documented. A dense 27B is something a single well-equipped machine can tune.

Fine-tuning a mixture-of-experts model is harder in ways that are not obvious upfront. You must decide whether to tune the router, the experts, or both. Tuning experts on a narrow dataset can cause routing collapse, where traffic concentrates on a handful of experts and the rest of the model stops contributing — which quietly converts your expensive MoE into a much smaller model. The tooling is also less mature and the failure modes are less documented.

For most teams the practical answer is that fine-tuning favours dense models by a wide margin, and that this should factor into the architecture choice at the start rather than being discovered halfway through a project. Fine-tuning vs prompting vs RAG covers whether you need to tune at all, which is worth settling first.

The trend to watch

Dense models have been holding their ground better than the parameter-count race suggests, because training quality has improved faster than scale. A 27B trained in 2026 is a substantially different object from a 27B trained in 2024, on the same architecture and the same parameter count.

The practical implication is that you should re-test the small-model option periodically rather than assuming last year's conclusion holds. Teams that ruled out dense models on a 2024 evaluation are frequently still paying for capability they no longer need.