Models That Fit on One GPU: What You Can Actually Run

There is a hard cliff in self-hosting, and it sits at the boundary of one accelerator. On one side you have a container, a model file and a serving process. On the other you have multi-node orchestration, an interconnect fabric, and a person whose job is now keeping that fabric healthy.

Almost every model discussion online is about capability. If you are hosting the model yourself, capability is the second question. The first is whether it fits.

The arithmetic that decides it

Weight memory is the easy part and you can do it on the back of an envelope. Multiply the parameter count by the bytes per parameter at your chosen precision: two bytes at 16-bit, roughly one at 8-bit, roughly half at 4-bit. A 27B model is therefore about 54GB at 16-bit, 27GB at 8-bit and 14GB at 4-bit.

That is only the floor. You also need the KV cache, which grows with context length and concurrency, plus activation memory and framework overhead. The cache is the part people forget, and it is the part that scales with your traffic rather than with your model.

A practical rule: budget the weights, then assume you need meaningful headroom on top for cache and overhead, then measure the real number under your own concurrency rather than trusting the estimate. How the KV cache works explains why that second term grows the way it does.

Dense models are the ones that clear the bar

The models that fit on one accelerator in 2026 are overwhelmingly dense, and that is structural rather than accidental. A dense model runs every parameter on every token, so its memory footprint and its compute footprint are the same number.

A mixture-of-experts model breaks that link. It holds a very large number of parameters and routes each token through a small subset, which makes it cheap per token and expensive to hold. Kimi K3's weights are roughly 1.6TB. The active-parameter count is a fraction of that, but you cannot load a fraction of a model — every expert has to be resident somewhere.

So the trillion-parameter open-weight models are open in the licensing sense and closed in the practical sense for anyone without a cluster. Dense against mixture-of-experts covers why labs keep building the ones you cannot run.

Qwen 3.6 27B is the current reference point

The clearest single-GPU option in the current field is Qwen 3.6 27B, released in April 2026. It is a dense 27B designed to run on one accelerator, and it reports 77.2 percent on SWE-bench Verified.

That figure is worth sitting with, because it undercuts the intuition that small models are toys. DeepSeek V4 Pro — a 1.6T mixture-of-experts activating roughly 49B per token — reports 80.6 percent on the same benchmark. The gap is about three points. The difference in what it takes to run them is not three of anything.

The honest framing is that you are trading a few points of benchmark performance for the ability to deploy without a distributed serving stack. For a lot of workloads that is an obviously correct trade, and for some it is not. The Qwen 3.6 guide goes deeper on where it holds up.

Quantisation moves the line, and costs something

Quantisation is what actually gets most models onto a single card. Dropping from 16-bit to 8-bit roughly halves weight memory; 4-bit halves it again. That is the difference between needing two accelerators and needing one, which is the only threshold that changes your architecture.

The cost is not free and it is not uniform. Quantisation tends to preserve conversational fluency well while eroding precise multi-step reasoning first — which is exactly the capability that coding and agent work depend on. A quantised build can feel completely fine in casual testing and then underperform on the tasks you deployed it for.

The discipline is simple: evaluate the exact quantised build you intend to ship, on your own tasks, rather than reading the benchmark for the full-precision original. Quantization explained covers which formats degrade gracefully and which do not.

Throughput is the constraint nobody budgets for

Fitting the model is necessary and not sufficient. A single accelerator serving a single request is fast. The same accelerator serving twenty concurrent agent loops, each holding a long context, is a different machine.

Every concurrent request holds its own KV cache, so concurrency and context length multiply against the same fixed memory. Teams routinely size their hardware against the model and then discover the real limit was how many simultaneous sessions the cache could hold.

Measure this before committing. Run your realistic context length at your realistic concurrency and watch where latency degrades. That number, not the parameter count, is what determines whether one card serves your team.

When single-GPU is the wrong goal

Self-hosting earns its keep for three reasons: data that cannot leave your network, sustained volume high enough to beat per-token pricing, and a need for guaranteed availability independent of a vendor.

If none of those apply, the single-GPU question is a distraction. An API gives you frontier-class models with no memory arithmetic at all, and the crossover volume at which self-hosting is cheaper is higher than most teams estimate. The self-hosting cost comparison works through where the line actually falls.

The decision rule: host it yourself if one of those three reasons applies, and if so, start from dense models sized to your hardware rather than from a leaderboard. Otherwise use an API and spend the effort elsewhere.