Which Open Model Fits Your GPU: A Memory-First Guide

The usual way to choose a self-hosted model is to read benchmark tables, pick a winner, and then discover it does not fit. Reverse the order. Hardware is the binding constraint, and it eliminates most of the field before quality enters the discussion.

The single most useful idea here is that mixture-of-experts models have two parameter counts and they answer two different questions. Total parameters tell you how much memory you need. Active parameters tell you how fast generation will be. Confusing them is why so many capacity plans are wrong by an order of magnitude.

Total parameters set the memory floor

A mixture-of-experts model routes each token through a small subset of its experts. Only that subset computes. But the router can select any expert on any token, so every expert must be resident and reachable, which means the whole model sits in memory whether it is busy or not.

Work through the 2026 field with that in mind. Kimi K3 is 2.8T total parameters with 104B active, and the published weights run to roughly 1.6TB. DeepSeek V4 Pro is 1.6T total with 49B active. GLM-5.2 is around 744B total with roughly 40B active. Kimi K2.6 is 1T total with 32B active.

None of those are single-GPU propositions. Even at aggressive quantisation, a model whose weights measure in the hundreds of gigabytes needs a multi-GPU node at minimum and often a multi-node setup with a fast interconnect. Mixture of experts explained covers why the architecture trades memory for compute in exactly this way.

Active parameters set the speed

Once the weights are resident, generation speed tracks active parameters rather than total. This is the whole point of the architecture: DeepSeek V4 Flash activates around 13B parameters per token, so it generates at roughly the speed you would expect from a 13B dense model while carrying the knowledge of something far larger.

That is why the sparse models feel fast despite their size. It also means throughput comparisons between a sparse and a dense model of the same total size are meaningless — the sparse one will be dramatically faster and require dramatically more memory.

For latency planning, active parameters plus your batch size and context length are the inputs that matter. Inference latency explained breaks down where the milliseconds actually go.

The dense exception

Qwen 3.6 27B is the model that changes the shape of this decision, because it is dense at 27B and designed to run on a single GPU. It reports 77.2 percent on SWE-bench Verified, which puts it in serious territory for a model you can host without a cluster.

Dense means every parameter computes on every token. That is less efficient per unit of capability than a sparse design, but it is enormously simpler to deploy: one card, one process, no expert routing, no cross-device communication to tune.

For a team with a single workstation-class GPU or one accelerator in a cloud instance, this is usually the only realistic option, and the gap to the giant sparse models is smaller than the parameter counts imply. Dense versus MoE models compared covers the trade in detail, and the Qwen 3.6 guide covers the model itself.

What quantisation buys and what it costs

Quantisation reduces the precision of stored weights, cutting memory roughly in proportion. Going from 16-bit to 8-bit approximately halves the footprint; going to 4-bit roughly halves it again.

The cost is quality, and it is not uniform. Degradation from moderate quantisation is often invisible on conversational tasks and clearly visible on precise ones — code that must compile, structured output that must parse, arithmetic that must be right. Long-context behaviour tends to degrade before short-context behaviour does.

The practical rule is to quantise only as far as you must to fit, then measure on your actual workload rather than trusting a perplexity number. Quantisation explained covers the formats and where each one breaks down.

The KV cache is the second memory budget

Weights are the fixed cost. The key-value cache is the variable one, and it is the part that surprises people in production.

Every token in every active request holds cache entries proportional to context length. Kimi K3, GLM-5.2, both DeepSeek V4 variants and MiniMax M3 all ship 1M context windows. Actually using that window, across several concurrent requests, can consume more memory than the weights themselves.

Attention design matters here. Kimi K2.6 uses multi-head latent attention specifically to compress the cache, which is why its 256K window is cheaper to serve than the raw number suggests. The KV cache explained covers the arithmetic; plan for peak concurrency at your real context length, not for a single idle request.

A sizing checklist

Start with total parameters and your quantisation target to get a weight footprint. Add headroom for the KV cache at your expected concurrency and context length — a third of total memory is a reasonable starting reservation. Add a little more for activations and fragmentation.

If the result exceeds what you have, you have three moves: quantise harder, pick a smaller model, or accept a lower concurrency ceiling. There is no fourth move, and pretending otherwise produces out-of-memory failures under load rather than in testing.

Then ask the question most self-hosting plans skip: is the utilisation there? A GPU costs the same idle as busy. Below roughly steady, sustained load, a per-token API is cheaper and considerably less work. Self-hosting versus managed API cost and the open-model price comparison give you both sides of that calculation.

The short version: one GPU means Qwen 3.6 27B. A multi-GPU node opens up the mid-sized sparse models with quantisation. The largest frontier open models are a cluster commitment, and for most teams the honest answer is to rent them through an API instead.