Best Model for Self-Hosting: What Fits on What You Have

Self-hosting inverts model selection. Through an API you compare capability and price. On your own hardware the first question is what physically fits, and everything else is downstream.

Start with memory, not benchmarks

The binding constraint is holding the weights. For a mixture-of-experts model that means total parameters, not active ones — you must hold all of them even though each token uses a fraction.

That single fact eliminates most of the leaderboard. Kimi K3's weights are roughly 1.6TB. That is a multi-node deployment with fast interconnect, not a machine. DeepSeek V4 Pro at 1.6T total has the same problem despite activating only about 49B per token.

Meanwhile a dense 27B like Qwen 3.6, quantised to 8-bit, fits comfortably in the memory of a single modern accelerator. The capability gap between those two options is a few points on bounded benchmarks. The deployment gap is between a cluster and a rented GPU.

The realistic tiers

One consumer or mid-range accelerator. Dense models up to roughly 30B at 8-bit quantisation, or larger at 4-bit with more quality loss. Qwen 3.6 27B is the current reference point — 77.2 percent on SWE-bench Verified from something that runs on one card is the headline result of 2026.

One high-memory data-centre accelerator. Larger dense models, or smaller MoE models at aggressive quantisation. More headroom, meaningfully more cost per hour.

Multi-node with fast interconnect. The frontier MoE models become possible. This is an infrastructure project with ongoing operational burden, and it is where most self-hosting plans quietly die.

Be honest about which tier you are in before shortlisting models. Most teams are in the first, and most model comparison content is written about the third.

Quantisation, and its uneven cost

Quantisation reduces weight precision to cut memory. It is what makes single-GPU deployment possible at all, and it is not free.

Eight-bit is usually close to lossless for practical purposes. Four-bit saves substantially more memory and costs noticeably more quality. Below that, quality falls off sharply.

The important detail: the loss is not uniform across tasks. Quantisation frequently leaves conversational fluency intact while eroding precise multi-step reasoning — exactly the capability coding depends on. A quantised model can pass casual testing and fail on your real work.

So evaluate the quantised build you intend to run, not the full-precision weights. Quantization explained covers which formats hold up.

The costs nobody puts in the spreadsheet

Self-hosting comparisons usually pit GPU rental against API token spend and stop there. The missing terms are the ones that decide it.

• Engineering time. Somebody sets it up, keeps it running, handles OOMs, upgrades the serving stack, and is paged when it breaks. This is the largest omitted cost, routinely.
• Idle capacity. A rented GPU costs the same at 3am. Unless utilisation is high, you are paying for hours you do not use — which is exactly what per-token pricing avoids.
• Upgrade churn. New models arrive constantly. Each one is a redeployment and a re-evaluation.
• Throughput under real load. Single-request benchmarks flatter self-hosting. Measure at your actual concurrency, where batching and memory pressure behave differently.

Self-hosting vs managed API cost works through the arithmetic properly.

When self-hosting is right regardless

Some constraints make the cost comparison irrelevant.

Data residency, where tokens legally cannot leave your network. Air-gapped environments, where no API is reachable. Regulatory requirements about where inference occurs. In all of these, the question is not whether self-hosting is cheaper but which model fits on hardware you are permitted to use.

Sustained very high volume is the other case. Above a crossover point a saturated GPU beats per-token pricing outright — but the crossover is higher than most teams estimate, because they omit the engineering time.

Recommendation

• One GPU, coding work — Qwen 3.6 27B, quantised to 8-bit. Best capability-per-gigabyte available.
• Permissive licence required — Qwen or DeepSeek V4. Avoid Kimi K3, which needs a separate agreement above a revenue threshold.
• Real cluster available — DeepSeek V4 Pro. MIT, strong, and the family gives you Flash for cheap bulk work on the same licence.
• Undecided — start with an API, measure your actual volume for a month, then compute the crossover with engineering time included. Most teams discover they are below it.