LLM Summarisation: Omission Hurts More Than Invention

Summarisation looks like the easiest thing to hand to a model, and it is the task where bad output is hardest to notice. A fabricated fact is at least detectable if you check. A missing fact leaves no trace at all — the summary reads perfectly and the one clause that mattered is gone.

That asymmetry should shape the whole design. The interesting question is not "is anything here wrong" but "is anything important missing", and answering it requires knowing what important means before you generate.

Define the extraction target, not the length

"Summarise this in three paragraphs" gives the model complete discretion over what survives, and it will optimise for a summary that reads well. Readable summaries systematically drop the awkward specifics: the caveat, the dissenting comment, the number that contradicts the headline.

Replace length instructions with a content contract. For a support thread: the reported problem, what was tried, the current state, the next action and its owner. For a meeting: decisions made, decisions deferred, and open questions with names attached. For a document: the claim, the evidence, and the stated limitations.

A named-slot structure changes the failure mode from silent omission to a visible empty field. An empty "next action" slot is information. A flowing paragraph that happens to not mention the next action is indistinguishable from one where there was no next action.

{"problem": "...", "attempted": ["..."],
 "state": "...", "next_action": null, "owner": null}

Position bias decides what gets kept

Given a long input, content at the beginning and end has disproportionate influence on the summary. Material in the middle gets compressed harder, and this effect is strong enough to be the single biggest determinant of what your summary contains.

This matters most for the document types where the important content is structurally in the middle: a long email thread where the resolution is at message twelve of twenty, a transcript where the decision happens forty minutes in, a report where the limitations section sits before the appendices. The general mechanism is covered in why models attend less to the middle of a long input, and summarisation is where it does the most damage.

Two practical responses. Chunk so that no single call has a long middle, which is the strongest mitigation. And test for it directly: place a known distinctive fact at the start, middle and end of the same document and check whether it survives in all three positions. If it does not, your chunk size is too large regardless of what the context window allows.

Map-reduce versus single-pass

With a million-token context window available on several current models, the single-pass option is real for most documents. It is not automatically the right one.

Single-pass preserves cross-references. If paragraph three qualifies a claim made in paragraph forty, only a model seeing both can reconcile them. Map-reduce breaks that: the chunk containing the claim is summarised without the qualification, and the qualification is summarised without the claim, and neither summary is wrong while the combination is misleading.

Map-reduce gives better coverage of detail, because each chunk gets the model's full attention rather than competing with fifty others, and it parallelises so wall-clock latency stays flat as the document grows. It also costs more in total tokens, since the reduce stage re-reads the intermediate summaries.

The hybrid that usually wins: map over structural chunks to extract facts into the slot structure, then reduce by merging slots rather than by re-summarising prose. Merging structured data is deterministic where prose merging is another lossy generation step. If you are choosing between stuffing the whole document in and retrieving parts of it, the trade-offs are set out in when retrieval beats a very large context window.

Refinement chains drift

The sequential pattern — summarise chunk one, then pass that summary plus chunk two, and so on — is appealing because it preserves ordering and stays within a small context.

It also compounds error. Each step re-summarises an already-summarised summary, so early content is compressed repeatedly while late content is compressed once. By chunk twenty, the opening of the document has been through twenty rounds of lossy compression and is typically reduced to a sentence, whether or not it deserved that.

Use refinement only when the document has genuine narrative dependency and you need ordering preserved. Otherwise map-reduce with structured merge does the same job without the recency skew.

Evaluate faithfulness with cheap deterministic checks first

Before reaching for a model judge, run the checks a program can do. They catch a surprising proportion of real failures at near-zero cost.

Every number in the summary should appear in the source. Every named entity in the summary should appear in the source. Every date should appear in the source. These three checks catch most fabrication, because fabricated content usually introduces a specific that was not there. They produce false positives on legitimate arithmetic and paraphrase, so treat a hit as a flag for review rather than a failure.

Omission needs the inverse check and it is harder. Extract a list of salient items from the source with a separate call, then verify each appears in the summary. This is expensive but it measures the thing that matters, and running it on a fixed evaluation set rather than in production keeps the cost bounded. The statistics of comparing two summarisation prompts are easy to get wrong, and worth doing properly per measuring whether a prompt change actually helped.

Make the summary auditable

A summary nobody can check is a claim. Requiring the model to attach a source reference to each extracted slot — a message ID, a line range, a timestamp — changes it into something verifiable, and it measurably reduces fabrication because the model has to point at something.

References also make review fast. A human checking a five-slot summary with citations can verify it in under a minute; the same person checking a three-paragraph prose summary against a twenty-message thread will not bother. Where the summary feeds a decision, that difference is the whole value.

Validate the references programmatically. A cited message ID that does not exist is a hard failure you can catch without human involvement, and it happens often enough to be worth the check.

Cost scales with input, so batch and cache

Summarisation is input-heavy and output-light, which is the cheap direction on most pricing. The cost driver is how many times you read the same source text.

If several summaries are produced from one document — a short version, a technical version, a version for a different audience — put the document in a cached prefix and vary only the instruction at the end. That turns three full reads into one, and the mechanics of what counts as a cache hit are in how prefix caching matches tokens.

Summarisation is also a task where a smaller model often suffices, because extraction against a clear slot contract is much easier than open-ended reasoning. Test the cheap option on your evaluation set before defaulting to the expensive one; the guidance in deciding when a cheap model is good enough applies directly here.