Few-Shot Example Management: Curation Beats Quantity

Few-shot examples are the highest-leverage part of most prompts and the least maintained. They get added when something breaks, never removed, and nobody notices when two of them start disagreeing with each other.

The reason they work is boring: an example demonstrates the output format, the level of detail, the tone and the edge-case policy all at once, in a form the model can pattern-match against. The reason they fail is equally boring. They are data, they go stale, and a stale example is an instruction to do the wrong thing.

Examples teach format faster than instructions do

If you want a specific shape of output, showing two of them beats a paragraph describing the shape. This is why few-shot works well for extraction, classification and rewriting, and less well for open-ended reasoning where there is no single correct shape.

The corollary is that if your output is already constrained by a schema, examples buy you much less. When the provider enforces a JSON schema, the format is guaranteed and your examples should stop demonstrating braces and start demonstrating judgement: which field gets the value when the document is ambiguous, what null means, when to abstain. The interaction between the two is worth understanding before you add examples at all, and how schema-constrained decoding actually enforces shape covers where the boundary sits.

A useful test: delete an example and see whether the failure it was added to fix comes back. If it does not, the example is costing you tokens on every call for nothing.

Contradiction is the dominant failure mode

Examples accumulate one incident at a time. Someone adds an example where an ambiguous address is left null. Six weeks later someone else adds an example where a similar ambiguous address is guessed. Now the prompt contains two policies and the model picks one at random depending on the input.

This is invisible in review because nobody reads all twelve examples when adding the thirteenth. It shows up as increased variance, which people misdiagnose as temperature or as model flakiness.

The fix is procedural, not clever. Write the policy down as a short rule list above the examples, and require that every new example be justified against a rule. If an example cannot be justified, either the rule is wrong or the example is. Then run a periodic consistency pass: take each example input, run it through the current prompt with that example removed, and check the output still matches the example output. Disagreements are your contradictions.

Order and recency effects are real

The position of an example changes its influence. Examples near the end of the prompt, closest to the actual query, tend to have more pull than examples buried in the middle. This is the same positional weakness that shows up in long-context retrieval, described in why models attend less to material in the middle of a long input.

Practical consequences. Do not put your most important edge case first and assume it dominates. Do not order examples by when they were added, which is what happens by default and means your oldest policy sits closest to the top. If your labels are imbalanced, do not cluster all the examples of one class together, because the last few examples set an expectation about what the next answer probably is.

Interleave classes, put the case you most want respected last, and check whether shuffling the order changes your evaluation results. If it does, you have fewer robust examples than you think.

Static sets, dynamic selection, and the cost line

A static set of six to ten well-chosen examples is the right starting point for almost everything. It is cacheable, reviewable, and cheap to reason about.

Dynamic selection — retrieving the nearest examples to the current input from a larger pool — helps when your inputs span genuinely different domains and one set of examples cannot cover them. It costs you two things. The prompt prefix now changes per request, so prefix caching stops helping, which matters more than people expect once volume is real. And your prompt is no longer reproducible from the repository, so you must log which examples were selected or you cannot debug an output.

If you do go dynamic, retrieve on the input representation you actually care about rather than raw text similarity, and keep a small fixed core set that is always present regardless of retrieval. That gives you a floor on behaviour when retrieval returns something unhelpful.

Examples leak

Models copy from examples. Usually that is the point; sometimes it is a bug. Specific values from an example — a company name, a date, a currency code, an ID format — reappear in outputs for inputs that never contained them.

This gets worse when the input is sparse. Give a model an invoice with a missing vendor field and a set of examples where the vendor is always present, and it will sometimes supply a vendor from an example rather than emit null. That is a hallucination with a traceable source.

Two mitigations. Use obviously synthetic values in examples — placeholder names, dates far from the present, IDs that could not be real — so leakage is detectable rather than plausible. And include at least one example where the correct answer is null or an abstention, because a set where every example produces a confident full answer teaches the model that confident full answers are always available.

Keep examples in sync with the schema

When a field is added, renamed or made optional, every example containing that field is now wrong. Nothing enforces this, and the model will happily follow the outdated shape while your parser rejects it.

Store examples as structured data rather than as prose inside the template, and validate them against the live schema in CI. An example whose output does not validate is a build failure, exactly like a test that references a deleted function.

for ex in load_examples("extract/invoice"):
    Invoice.model_validate(ex.output)   # fails CI on schema drift

Render them into the prompt from that structured source. This also means you can generate the same examples in a different serialisation if you change formats, without re-authoring anything.

Decide with measurement, not intuition

Every example costs input tokens on every single request. Ten examples of two hundred tokens each is two thousand tokens of overhead per call, forever. That is often worth it and often not, and the only way to know is to measure with and without.

Run the same evaluation set at zero-shot, three-shot and your current set. If three-shot matches ten-shot, ship three. The methodology for making that comparison mean something — sample size, paired runs, variance — is the subject of measuring whether a prompt change actually helped, and it applies to example sets more than to any other part of the prompt.

Once the set is settled, treat it as a versioned artefact like the rest of the template, using the conventions in versioning prompts so you know which one produced an output. An example set that changes without a version bump is the same class of bug as an untracked config edit.