LLM Classification: Fix the Taxonomy Before the Model

Classification is the task where a model looks most obviously superior to the alternative, and where it most often loses to a logistic regression that took an afternoon. It is also the task where the accuracy number you report is most likely to be meaningless.

Both problems have the same root. Classification quality is bounded by how well the labels are defined, and a model cannot apply a distinction that humans do not agree on.

Measure human agreement before you measure the model

Take a hundred items and have two people label them independently. If they agree 75 percent of the time, then 75 percent is roughly your model ceiling, and an 80 percent model score means your evaluation labels are noisy rather than that the model is superhuman.

Low agreement almost always means the taxonomy has overlapping categories or a missing one. "Bug", "Performance issue" and "Unexpected behaviour" are not three categories; they are one category and two ways of describing it, and every labeller will split them differently.

Fixing the taxonomy is unglamorous work that pays better than anything you can do to the prompt. Merge categories that annotators confuse. Split categories where the disagreement is about two genuinely different things sharing a name. Add the category that people keep forcing into "other". Then re-measure agreement.

Definitions with boundaries, not synonyms

A label description that reads "Billing: issues related to billing" is a synonym, not a definition. It tells the model nothing it did not already infer from the label name.

Useful definitions state the boundary. What is included, what is specifically excluded and where it goes instead, and one concrete example of the near-miss case. "Billing: charges, invoices, refunds and payment failures. Does not include subscription plan changes, which are Account. A failed card on an upgrade attempt is Billing."

The exclusions do most of the work, because misclassification happens at boundaries rather than in the middle of a category. Write the definition by looking at the pairs that annotators confused, and put the disambiguation directly in the definition rather than in a general instruction that applies to everything.

Abstention is a first-class outcome

Force a model to choose from a fixed list and it will always choose, including for items that belong to none of them and items that are genuinely ambiguous. Those forced choices are indistinguishable from confident correct ones in your output.

Give it an explicit way out and say when to use it. Not a vague "other" bucket, which becomes a dumping ground, but a specific abstention with a reason: outside taxonomy, insufficient information, or multiple categories apply equally.

{"label": "abstain", "reason": "insufficient_information",
 "candidates": ["billing", "account"]}

The abstention rate is then a live quality metric. A sudden rise means your input distribution shifted, which is exactly the signal you want and which a forced-choice classifier hides entirely by continuing to emit confident labels. Route abstentions to a review queue, and use what comes back to extend the taxonomy, following the pattern in deciding where a human review step belongs.

Class imbalance breaks your metric and your prompt

If 90 percent of items are one class, a classifier that always predicts that class scores 90 percent. Accuracy is worthless here and yet it is what gets reported.

Use per-class precision and recall and a confusion matrix. The matrix tells you the specific thing you need: which pairs get confused, in which direction. A model that misses 40 percent of the rare urgent class while scoring 92 percent overall is a failure, and only the per-class numbers show it.

Imbalance also affects the prompt. If your few-shot examples mirror the real distribution, nine of ten demonstrate the majority class and the model learns that the majority class is usually right. Balance the examples across classes rather than sampling them naturally, and be aware that the last examples in the prompt carry extra weight, for the reasons in managing few-shot example sets.

Beat a cheap baseline or do not ship

Before deploying a model classifier, build the boring version: TF-IDF features into logistic regression, or a small fine-tuned encoder. An afternoon of work, milliseconds per item, no per-call cost, fully deterministic and inspectable.

On tasks with a few thousand labelled examples and a stable taxonomy, the boring version is frequently competitive and sometimes better. It is always cheaper by orders of magnitude, and cost per item matters enormously at classification volumes, where you might be running millions of items rather than thousands.

Where the model genuinely wins is the cold start, when you have a taxonomy and no labelled data at all, and on tasks requiring real world knowledge or nuance that surface features cannot capture. A good pattern is to use the model to label a training set, verify a sample by hand, then train the cheap classifier on it and keep the model for the low-confidence tail. The economics of that split are the subject of working out when a cheap model is enough.

Confidence, and why the model's own number is not it

Ask a model how confident it is and it will produce a number that looks like a probability and behaves like a style choice. Self-reported confidence clusters at round values, shifts with phrasing, and correlates weakly with correctness.

Token logprobs on the label token are a better signal where your provider exposes them, because they reflect the actual distribution over choices rather than a verbalised guess. They are still uncalibrated, so map them to real probabilities using a held-out set before you threshold on them.

The cheapest usable signal is agreement across samples. Run the classification three times at nonzero temperature and use the disagreement rate. Items where all three agree are reliable; items where they split are the ones a human should see. This costs three times as much and it is applied only to the items where it matters if you gate it on a first-pass signal.

Multi-label and hierarchy need different handling

If an item can carry several labels, do not ask for a list from a flat taxonomy and hope. Ask a yes-or-no question per label, or per group of related labels, which gives you an independent decision for each and avoids the model settling on a plausible-looking set of two.

For hierarchical taxonomies, classify top-down. Choose the top-level category, then choose within it using only that branch's definitions. This keeps the number of options in each decision small, keeps the definitions in context focused, and lets you report accuracy at each level, which tells you whether errors are coarse or fine.

The cost is more calls per item, so batch where you can and cache the shared taxonomy prefix, since the definitions dominate the input tokens and are identical across every item.

A shipping checklist

Measure inter-annotator agreement and fix the taxonomy until it is acceptable. Write definitions with explicit exclusions drawn from real confusions. Add an abstention path with reasons and monitor its rate. Report per-class precision and recall with a confusion matrix, never aggregate accuracy. Benchmark against a cheap classifier and justify the difference in cost.

Then pin the model and the prompt version together, and re-run the evaluation set whenever either changes, because a taxonomy that was correctly applied by one checkpoint is not automatically applied the same way by the next. The mechanics of that are covered in pinning model versions rather than tracking a floating alias.