Quantization Cost Savings and the Quality You Trade Away

Quantization reduces the numerical precision of model weights. Less precision means less memory, which means cheaper hardware and more concurrent sessions on the same card. It is the technique that turns "this model needs a cluster" into "this model runs on one accelerator".

It is not free, and the cost is distributed in a way that makes it easy to underestimate.

Where the saving comes from

Memory is the binding constraint for most self-hosted deployments. A model must be held in accelerator memory to serve, and the KV cache for each concurrent session sits alongside it.

Halving weight precision roughly halves the memory the weights occupy. That has three compounding effects: cheaper hardware qualifies, more sessions fit alongside the weights, and memory bandwidth pressure falls, which raises throughput during generation.

Since decode is memory-bandwidth-bound rather than compute-bound, that last effect is real rather than incidental — a quantised model frequently generates faster, not just cheaper. KV cache explained covers why bandwidth dominates decode.

The quality cost is uneven, and that is the trap

The standard framing is a smooth trade: less precision, slightly worse output. The reality is lumpier.

Eight-bit quantisation is usually close to lossless for practical purposes. Four-bit saves considerably more memory and costs noticeably more quality. Below that, degradation is sharp.

The important part is which quality degrades. Quantisation tends to leave conversational fluency and general knowledge largely intact while eroding precise, multi-step, structured reasoning — which is exactly what coding work depends on.

So a quantised model can pass casual testing convincingly. It answers questions well, explains concepts clearly, and feels unchanged. Then it underperforms on a long refactor or a multi-step agent run, and the cause is not obvious because the model seems fine when you talk to it.

Evaluate the build you will run

The practical consequence is simple and widely ignored: benchmark the quantised weights at the precision you intend to deploy, on tasks that resemble your real work.

Published benchmark figures are almost always for full-precision weights. Qwen 3.6 27B reporting 77.2 percent on SWE-bench Verified is a full-precision result; the four-bit build you fit onto one card is a different artefact and deserves its own measurement.

Test multi-step tasks specifically, because that is where the loss concentrates. A single-turn evaluation will tell you the quantised model is fine, and a twenty-turn agent run will tell you it is not.

The saving only exists if you self-host

This is worth stating plainly because it is a common confusion. If you use a model through an API, quantisation is the provider's implementation detail. You pay their per-token rate regardless of what precision they serve at, and you generally cannot choose.

Quantisation is a lever for people running weights themselves. For everyone else the equivalent lever is model tier selection — choosing DeepSeek V4 Flash over V4 Pro is the API-side version of the same trade, and it comes with published figures rather than requiring your own measurement. DeepSeek V4 Pro vs V4 Flash covers that comparison.

Where it clearly pays

When it is the difference between fitting on hardware you have and not fitting at all. A four-bit model running on one accelerator beats a full-precision model you cannot deploy, whatever the quality gap.

When the workload is bounded and single-turn — classification, extraction, short transformations — where the degraded capability is not the one being used.

And when throughput matters more than peak quality, because the bandwidth saving raises tokens per second at the same time as it cuts memory.

Measuring the saving honestly

The number worth tracking is not memory footprint, it is cost per successfully completed task. A four-bit deployment that halves your hardware bill but needs a second attempt on a third of requests has not saved you a third of anything — it has moved spend from capacity into retries, where it is harder to see.

Build the comparison from a fixed set of real requests replayed against both precisions. Record tokens consumed, wall-clock time, and whether the output passed whatever check you already apply — tests, a schema validator, a reviewer. Divide total cost by passes, not by requests. That single ratio makes the tradeoff legible in a way that a benchmark delta never does.

Expect the answer to differ by workload rather than by model. The same quantised weights can be an obvious win for summarisation and an obvious loss for multi-file refactoring, which is why a blanket policy is usually worse than a per-workload decision.

Where to be careful

Long unattended agent runs, where per-step degradation compounds across many steps and a small quality loss becomes a large completion-rate loss.

Anything where output correctness is checked downstream by a human rather than mechanically, because degradation that produces plausible wrong answers is the most expensive kind.

And any deployment where you did not measure. The genuine risk with quantisation is not that it degrades quality — it is that it degrades quality invisibly, in a way that casual testing does not reveal. Quantization explained covers which formats hold up best.