Best Model for Structured Output: Constrain, Do Not Hope

The usual approach to reliable JSON is to pick a stronger model and ask nicely in the prompt. That improves the failure rate without eliminating it, and for a pipeline processing thousands of items a residual failure rate is still a lot of failures.

The better approach makes invalid output structurally impossible, at which point model choice shifts to a different question entirely.

Two kinds of correctness

Separate these, because they need different solutions.

Structural correctness is whether the output parses and matches the schema. Valid JSON, required fields present, types right, enums respected.

Semantic correctness is whether the values are right. Correct sentiment, accurate extraction, sensible categorisation.

Structural correctness is a solved problem you should stop paying for with model quality. Semantic correctness is where model capability genuinely matters.

Constrained decoding solves the structural half

When a provider supports JSON mode or schema-constrained output, the decoder is restricted at each step to tokens that can continue a valid document. Invalid output is not unlikely — it cannot be produced.

That converts a probabilistic problem into a guarantee, and it means a small cheap model with constrained decoding produces structurally valid output at the same rate as a frontier model: always.

So if you are choosing a model to get better JSON, check whether your provider supports constrained output first. It is usually a larger improvement than any model upgrade, and it is free. Structured outputs and JSON mode covers the mechanics and the caveats.

Where model quality still decides

Constrained decoding guarantees shape, not sense. A model can emit perfectly valid JSON with wrong values, and that failure is worse than a parse error because nothing downstream catches it.

Model capability matters for:

• Judgement fields. Sentiment, priority, category — anything requiring an actual decision.
• Extraction from messy input. Pulling structured data out of unstructured text, where the hard part is knowing what counts.
• Deep or conditional schemas. Nested objects where one field's validity depends on another's value.
• Knowing when to abstain. Returning null or an explicit "not found" rather than inventing something to fill a required field. This is the failure that hurts most and the one weaker models commit most.

That last point deserves emphasis. Constrained decoding forces the model to produce a required field. If the answer is not in the input, the model must invent one. Make such fields nullable and say explicitly in the prompt that null is the correct answer when the information is absent.

Schema design does more than model choice

• Flat beats nested. Every level of nesting adds failure modes. Flatten where you can.
• Enums beat free text. A closed set eliminates a whole class of semantic error and makes validation meaningful.
• Describe every field. Schema descriptions are read by the model. A field called status with no description invites guessing.
• Fewer fields per call. Two focused calls usually beat one call with twenty fields, because attention spreads thin across a large schema.
• Make optional things optional. Required fields the model cannot fill are an invitation to hallucinate.

Recommendation

• Provider supports constrained output — use the cheapest model that gets the semantics right. DeepSeek V4 Flash is a strong default for extraction and classification at volume.
• No constrained output available — a stronger model helps, but switching providers helps more. Validate and retry as a fallback, and log the failure rate so you know what it actually is.
• Judgement-heavy fields — spend on model quality here, and only here. This is where capability converts into accuracy.

Always validate the output against your schema even with constrained decoding enabled, and always log rejects. A silent rise in the reject rate is the earliest signal that a provider changed something underneath you.

Retries are a fallback, not a strategy

Where constrained decoding is unavailable, the standard pattern is to validate the output and retry on failure. That works, and it is worth understanding what it actually costs before relying on it.

A retry doubles the cost and the latency of every failed item. At a five percent failure rate across a large batch that is a modest overhead. At twenty percent it is substantial, and it arrives unevenly — the items that fail once are more likely to fail again, because whatever made them hard has not changed.

Retries also mask the underlying problem. A pipeline quietly retrying a fifth of its items looks healthy from the outside while doing far more work than necessary. Log the first-attempt success rate separately from the final success rate, or you will not notice the difference growing.

When you do retry, change something. Feeding the identical prompt back usually produces the identical failure. Including the validation error in the retry — naming the field and what was wrong with it — converts a coin flip into a correction.

Where the schema meets the domain

Most structured-output failures that survive constrained decoding are not model failures at all. They are schema failures, and they show up as the model doing something reasonable with an underspecified field.

A field called priority with no description and no enum will be filled with whatever vocabulary the model favours — high, urgent, P1, critical — varying between calls. That is not hallucination; it is an unanswerable question being answered plausibly.

The fix is always the same: constrain the field, describe it, and give an example. An enum of three values with a one-line description of when each applies eliminates the entire failure class, and it costs nothing at inference time.

Spend your effort here before spending it on model selection. A well-specified schema on a cheap model beats a vague schema on an expensive one, consistently and at a fraction of the price.