LLM Data Extraction: Design the Schema Before the Prompt

Extraction is the task where models are most immediately useful and most quietly wrong. Pull vendor, date and total from ten thousand invoices and you will get ten thousand well-formed JSON objects. Some percentage of them contain a value that is not in the document, and there is nothing in the output to distinguish those from the rest.

Almost every one of those errors traces back to a schema that did not give the model a correct option. If there is no way to say "not present" or "two candidates", the model has to pick something, and it will.

Absence must be representable and rewarded

The first rule of an extraction schema is that every field can be null and the model is told, explicitly and repeatedly, that null is the correct answer when the value is not stated.

This sounds obvious and is routinely violated. A required string field with a description like "the vendor name" is an instruction to produce a vendor name. When the document has no vendor, the model produces the most plausible one, drawn from the letterhead, the footer, or one of your few-shot examples. Schema-constrained decoding makes this worse rather than better, because it guarantees a string arrives.

Distinguish the cases in the type, not in a comment. "Not stated in the document", "stated but unreadable" and "multiple conflicting values" are three different situations with three different downstream handlings, and collapsing them into a single null throws away the information a human reviewer needs.

{"vendor": {"value": null, "reason": "not_stated"},
 "total": {"value": "1240.00", "currency": "EUR"},
 "due_date": {"value": null, "reason": "unreadable"}}

Normalise in code, not in the prompt

Asking for an ISO date is asking the model to do two jobs: find the date and convert it. The second job is where errors enter, and it is a job a library does perfectly.

Extract the surface form and normalise afterwards. "03/04/2026" extracted verbatim can be parsed with an explicit locale assumption that you control and can change; the same string converted by the model becomes March or April depending on nothing you can inspect. The same applies to currency amounts, phone numbers, addresses and units.

Where normalisation genuinely needs judgement — mapping a free-text job title to your taxonomy, say — that is a classification problem and deserves its own step with its own evaluation, along the lines of running classification with a fixed label set and an abstain option. Bundling it into extraction makes both harder to debug.

Provenance turns claims into checkable facts

Require a source span with every extracted value: a character offset, a line number, or the verbatim quoted text. Then verify the span in code. If the quoted text does not appear in the document, the extraction is a fabrication and you can reject it automatically.

This is the single most valuable check available for extraction, and it costs a string comparison. It catches the entire class of errors where the model produced a plausible value from context rather than reading it, which is the class that matters.

Character offsets are the strongest form because they are unambiguous, and they are also the form models are worst at producing accurately. Verbatim quotes are more reliable in practice — the model reproduces text well even when it counts characters badly — and a substring search recovers the offset for you. Use quotes, search for them, and treat a miss as a hard failure.

Document structure defeats naive text extraction

The hard part of extraction from real documents is rarely the language. It is that the value lives in a table cell whose meaning depends on a header three rows up, or in a two-column layout that linearises into interleaved nonsense, or in a scanned image where OCR turned a 5 into an S.

Fix the input before you tune the prompt. Preserve table structure in the serialisation you hand the model — markdown tables or a cell-addressed representation both work far better than flattened text. Keep reading order correct for multi-column layouts, which usually means a layout-aware parser rather than a naive text dump. Where the document is a scan, decide deliberately whether you send the image to a vision-capable model or run OCR first, since how models process images alongside text determines which errors you inherit.

A useful diagnostic: when extraction fails, look at exactly what the model saw. Most of the time the value was not legible in the input, and no prompt change would have helped.

Long documents need retrieval, not a bigger window

Several current models take a million tokens of context, which makes it tempting to feed an entire contract in and ask for twenty fields. It works, and it is usually the wrong shape.

Attention over a very long document dilutes, and recall of a specific clause degrades with distance from the ends. Cost scales linearly with input, so re-reading the whole document for each field group is expensive at volume. And you have no way to tell whether a null means the clause is absent or the model missed it.

Locate first, extract second. Find the candidate sections for a field group with retrieval or with structural rules, then run extraction on those sections only. You get shorter context, a checkable answer to "where did it look", and a cost that scales with the relevant part of the document rather than its size. The general shape of that decision is covered in choosing between retrieval and a very large context window.

Validate, then route the uncertain cases to people

Schema validation is the floor, not the ceiling. Beyond types, check the business rules: line items sum to the total, the due date is after the invoice date, the currency is one you trade in, the tax rate is in a legal range. These catch real errors that are type-correct.

Then decide what happens to failures. A pipeline that silently drops invalid extractions is worse than one with no validation, because it hides its own error rate. Route them to a review queue with the document, the extracted values and the source spans, so a human resolves in seconds rather than re-reading.

Self-consistency is a cheap uncertainty signal where you have budget: run the extraction twice and flag fields that disagree. Agreement is not proof of correctness, but disagreement is a strong signal of ambiguity and it correlates well with the cases a human needs to see. Whether the checking model should be the same one that generated the extraction is a real question, discussed in where a human review step actually belongs.

Measure per field, never in aggregate

A pipeline reported as 94 percent accurate is hiding something. Aggregate accuracy is dominated by the easy fields — the ones with a fixed format and a reliable position — and the field that matters is usually the one at 60 percent.

Track precision and recall per field, and separate the two kinds of error: extracting a wrong value, and returning null when a value was present. They have different causes and different fixes, and a metric that merges them tells you nothing about which to work on.

Build the evaluation set from documents that already went through review, so you have ground truth for free, and keep adding the ones that failed. Fifty hand-labelled documents covering your genuinely awkward formats will teach you more than five thousand easy ones.