Model Parallelism: How Huge Models Are Split to Run

Kimi K3 ships weights of roughly 1.6TB. No single accelerator holds that, so the model has to be cut into pieces that run on different devices and coordinate through the network between them.

There are several ways to make that cut, and they are not interchangeable. Each one puts communication in a different place, and where the communication lands determines whether your deployment is latency-friendly, throughput-friendly, or quietly bottlenecked on interconnect.

Data parallelism is not model parallelism

Worth clearing up first, because the terms get mixed. Data parallelism replicates the whole model on every device and sends different requests to each. It needs no coordination during a forward pass and scales throughput perfectly.

It also requires the model to fit on one device, which is exactly the case that fails for frontier models. Data parallelism is how you scale a deployment once it fits; model parallelism is how you make it fit at all.

In practice, large deployments use both — the model is split across a group of devices, and that group is replicated. Understanding which axis you are scaling is the difference between adding capacity and adding latency.

Tensor parallelism: split every layer

Tensor parallelism cuts individual weight matrices across devices. Each device holds a slice of every layer, computes its portion of the operation, and the partial results are combined before the layer can finish.

The consequence is communication inside every layer, on every token. Devices must exchange and reduce partial results many times per forward pass, so the interconnect between them is on the critical path constantly.

That makes tensor parallelism excellent within a single machine, where devices share a fast local link, and poor across machines, where every reduction crosses a slower network. The usual rule is to keep the tensor-parallel group inside one node and use something else beyond it.

What it buys is latency. All devices work on the same token simultaneously, so a single request gets the whole machine, which is why it is the default choice for interactive serving.

Pipeline parallelism: split by depth

Pipeline parallelism assigns whole layers to devices. The first device runs layers one through twenty, hands its output to the second device for the next twenty, and so on.

Communication is minimal — one activation transfer at each boundary rather than a reduction inside every layer — so it survives slower links and crosses machine boundaries comfortably.

The cost is idle time. While the first device works on a token, the later ones have nothing to do until the handoff arrives. That gap is the pipeline bubble, and with a single request in flight most of the hardware is idle most of the time.

The fix is to keep many requests in flight so that each stage always has something to process. Pipeline parallelism therefore favours throughput and tolerates latency, which is the opposite profile to tensor parallelism and another instance of the same trade-off appearing in a new place.

Expert parallelism: split by expert

Mixture-of-experts models add a third axis. Because only a few experts run per token, the experts themselves can be distributed across devices, with each device holding a subset.

Kimi K2.6 holds 384 experts, eight routed plus one shared per token. Spreading those across devices means each holds a fraction of the total parameters, which is what makes a model of that size hostable at all.

The communication pattern is distinctive. After routing, tokens must be sent to whichever devices hold their chosen experts, and the results sent back. This is an all-to-all exchange whose volume depends on how the router distributed the batch, and it happens at every MoE layer.

That data-dependent communication is why MoE serving throughput is harder to predict than dense serving. If routing concentrates traffic on a few devices, those become stragglers and everyone waits. Keeping expert load even is a serving problem as much as a training one, and how the router chooses determines the traffic pattern.

How they combine

Real deployments layer these. A common shape is tensor parallelism within a node, expert or pipeline parallelism across nodes, and data-parallel replication of the whole arrangement for capacity.

The design constraint is always the same: put the chattiest parallelism where the fastest link is. Tensor parallelism across a slow network will underperform badly regardless of how much aggregate compute you have assembled.

This is also why published throughput figures for large models are so hard to reproduce. They assume a specific topology and interconnect, and the same model on the same nominal hardware with a different arrangement can behave very differently.

What this means for self-hosting

The first question is not whether you can afford the compute but whether you have the interconnect. A collection of machines with ordinary networking between them can host a large model in the sense of holding the weights, and still deliver latency that makes it unusable interactively.

This is a large part of why dense models remain the practical self-hosting choice for most teams. A dense 27B such as Qwen 3.6 27B fits on a single device when quantised, needs no model parallelism at all, and therefore has no interconnect story to get wrong. The dense versus MoE comparison comes down to this more often than to benchmark scores.

Quantisation interacts here too, because reducing weight precision can move a model from needing several devices to needing one, and crossing that threshold removes an entire class of problem. The quality cost of quantisation is the thing to weigh against it.

The takeaway

If you consume an API, none of this is yours to manage, but it explains why frontier open-weight models are cheap to call and expensive to run yourself. You are buying access to a topology, not just to weights.

If you are planning a deployment, decide the parallelism shape from your latency requirement first. Interactive means tensor parallelism inside one machine and a model small enough to make that possible. Batch throughput tolerates pipelines and multiple nodes. Choosing the model before choosing the topology is how self-hosting projects end up rebuilt.