Where Quantised Models Actually Lose Quality

The standard defence of aggressive quantisation is a benchmark table showing a one or two point drop, offered as proof that the smaller checkpoint is fine. The table is usually accurate and the conclusion usually is not.

Quality loss from quantisation is not spread evenly across what a model can do. It concentrates in specific capabilities, and an aggregate score averages the damage away. A model that loses two points overall can have lost fifteen on the one thing you actually rely on.

The loss is uneven by construction

Quantisation reduces the precision available to represent each weight. Whether that matters depends on how much a given computation depended on precision in the first place.

Some capabilities are robust because they are broadly distributed across the network and reinforced by many redundant paths. General language fluency and common factual recall sit here, which is why quantised models still sound completely normal — the first thing you notice about a badly quantised model is that nothing seems wrong.

Other capabilities depend on narrow, precise distinctions: a small logit gap between two tokens, an attention head that has to attend to exactly one position, a routing decision that separates two similar experts. Those are where the rounding error lands, and they fail in ways that look like the model getting something subtly wrong rather than the model breaking.

What erodes first

Four categories show up repeatedly, and they are worth testing directly rather than trusting to an average.

Long-context retrieval is the most commonly reported. Pulling one specific fact out of a large context depends on sharp attention, and blunting the precision blunts the attention. A quantised model that answers short questions identically to the original can be measurably worse at finding a definition on page four hundred. Long-context degradation covers how to test that axis, and needle-in-a-haystack tests covers the standard probe.

Multi-step tool use is the second, and it degrades through compounding rather than through any single failure. A small per-step increase in the chance of a malformed argument or a wrong tool choice multiplies across a twenty-step run, so an agent trajectory can fail far more often than the per-token difference suggests.

The third is exact-format output — strict JSON, precise code syntax, arithmetic. These have no partially-correct region. The fourth is anything rare in training: low-resource languages, unusual libraries, uncommon token sequences. Capabilities that were already thin get thinner first.

Outliers explain most of the damage

The mechanism behind all of this is that weight and activation magnitudes are not evenly distributed. A small number of channels carry values far larger than the rest, and those outliers matter disproportionately to the output.

A naive quantiser has to choose a scale that covers the whole range. Cover the outliers and you waste most of your representable values on a range nothing occupies, leaving the ordinary weights coarsely represented. Clip the outliers and you destroy the signal they carried. Either way the error is concentrated rather than random.

Every good quantisation method is a response to this. Finer-grained scales per group of weights, calibration passes that measure which channels matter under real activations, and mixed precision that keeps the sensitive parts at higher width. The differences between formats are mostly differences in how they handle outliers.

Why a benchmark delta hides it

A published benchmark comparison between a full-precision and a quantised checkpoint is a single number covering a task pool that probably does not resemble your work.

Two problems compound. First, aggregation: if a benchmark contains ninety percent tasks that are robust to quantisation and ten percent that are sensitive, a catastrophic loss on the sensitive tenth shows up as a small overall drop. Second, distribution: most published quantisation evaluations lean on short-context, single-turn, English tasks, which are precisely the conditions under which quantisation looks best.

The result is a systematically optimistic figure. It is not dishonest — it is the natural consequence of measuring on the tasks that are easiest to measure. Why benchmarks disagree covers the general version of this problem.

How to measure it on your own workload

The measurement that answers your question is a paired comparison, and it is a day of work rather than a research project.

Take fifty to a hundred real tasks from your own history, weighted toward the hard end rather than sampled uniformly. Run each through the full-precision model and the quantised one at temperature zero with identical prompts and seeds, and score them with whatever check you already trust — the test suite, the schema validator, a rubric, a diff against the known-good result. Evaluating models on your codebase covers assembling the set.

Then look at the failures individually rather than at the score. What you are hunting for is a pattern: all the failures involve long inputs, or all of them involve the same tool, or all of them are the tasks with strict output formats. A pattern means a specific capability broke and you can decide whether you need it. A scatter of unrelated failures usually means you are looking at ordinary sampling noise instead.

Run the agentic tasks end to end rather than step by step. A per-step comparison will understate the loss, because compounding is exactly the effect you are trying to detect.

Which formats hold up

As a rule, the further you go below the native precision of the release, the more you should verify rather than assume.

Eight-bit formats are generally the least disruptive step, because the representable range still comfortably covers ordinary weight distributions. Four-bit is where calibration starts to matter a great deal: methods that measure channel importance against real activation data hold up substantially better than round-to-nearest, and per-group scales hold up better than per-tensor. Below four bits, treat every claim as requiring evidence on your own tasks.

Two practical notes. Keeping embeddings, the output head and attention projections at higher precision costs little disk and preserves a surprising amount of quality, which is why most good mixed-precision recipes do it. And a checkpoint already released at low native precision has far less headroom left — check the published size against the parameter count first, as storage planning covers, before quantising further.

A decision rule

Quantise for a reason, not by default. If the model already fits with room for the KV cache you need, the smaller checkpoint buys you nothing and costs you an unknown amount of quality.

If you do need it, take the mildest step that makes the model fit, verify on your own paired set, and specifically test the four fragile categories: long context, multi-step tool use, strict formats, and anything rare in your domain. If the paired comparison shows a pattern rather than noise, either step back a level or keep the sensitive parts at higher precision.

And weigh the result against the alternative, which is a smaller model at full precision. That frequently beats a large model quantised hard, and it is the comparison most people skip. Quantisation cost savings covers what the trade is actually buying you.