Synthetic Training Data: What It Does and Where It Fails

Synthetic data has a reputation problem. Half the discussion treats it as the obvious answer to running out of web text, and the other half treats it as a contamination risk that will eventually degrade every model trained on it.

Both positions are too broad. Synthetic data is a family of techniques with very different reliability characteristics, and the difference between the ones that work and the ones that do not comes down almost entirely to whether the generated examples were verified before being used.

Synthetic does not mean fake

The term covers any training example produced by a model or a program rather than harvested from human-written text. That is a much wider category than it sounds.

At one end sits a model generating a plausible-looking question and answering it from its own parameters, which is exactly as unreliable as it sounds. At the other end sits a compiler producing labelled examples of a program that does or does not build, or a test suite deciding whether a generated patch is correct. Both are synthetic. Only one of them can be wrong in an interesting way.

The useful distinction is not human versus generated but verified versus unverified. A generated example checked against an external source of truth carries genuine information. An unverified one carries whatever the generator already believed, which is why the two behave so differently at scale.

What it is actually used for

Three jobs dominate, and they sit at different stages of the pipeline.

The first is instruction and preference data for post-training. Collecting human demonstrations of good responses is slow and expensive; generating candidates and having them ranked or filtered is much faster, and the resulting dataset shapes behaviour rather than knowledge. Instruction tuning and the pretraining and post-training split cover where this sits.

The second is coverage of cases that are rare or absent in scraped text. Tool-calling traces with exact schemas, multi-step agent trajectories, long-context tasks with a known correct answer, structured output in specific formats — the open web contains very little of any of it, and generating it is the only practical route.

The third is verifiable domains. Code, mathematics and anything with an executable check let you generate large volumes of candidates and keep only the ones that pass. This is the most reliable form of synthetic data precisely because the filter does not depend on a model's judgement.

Distillation transfers behaviour from a stronger model

The best-known pipeline is distillation: run a capable teacher model over a set of prompts, keep its outputs, and train a smaller student on them.

What transfers is behaviour rather than knowledge in the general sense — response structure, reasoning style, formatting habits, the shape of a good answer. This is why a distilled small model can feel dramatically more useful than its size suggests while still failing on questions requiring knowledge the student never had capacity to store. Model distillation explained covers the mechanics.

The limits are worth stating plainly. A student trained on a teacher's outputs inherits the teacher's errors along with its strengths, and it very rarely exceeds the teacher on the distribution it was distilled over. Distillation is also a licensing question as much as a technical one, since many model terms restrict using outputs to train competing models.

Self-instruct pipelines bootstrap from a seed set

The other common shape does not need a stronger teacher. Start with a small set of human-written examples, have a model generate many more in the same style, filter aggressively, and add the survivors to the pool.

Every serious version of this puts most of its engineering into the filter rather than the generator. Deduplication so the set does not collapse onto a handful of templates, difficulty filtering so trivial examples do not dominate, and — where the domain allows it — an execution or verification step that discards anything demonstrably wrong.

Without those filters the pipeline amplifies whatever biases the generator had, including its preference for certain phrasings and its blind spots. With them it can produce genuinely useful coverage of cases nobody would have thought to write by hand. The filter is the product.

What the model-collapse evidence actually shows

Model collapse is the concern that training on generated data recursively causes distributions to narrow until models lose the tails of what they once represented. The effect is real and it has been demonstrated, but the conditions matter enormously.

The demonstrations that show severe collapse generally involve replacing the training data at each generation — training on the previous model's output alone, then repeating. Under that setup variance shrinks each round and rare cases disappear, which is exactly what you would expect from repeatedly sampling from a sample.

Work examining the more realistic setup, where synthetic data accumulates alongside the original human data rather than replacing it, finds the degradation far less severe. Real pipelines look like the second case: generated data is mixed with a large retained corpus, filtered, and weighted. That is not a reason to dismiss the concern, but it does mean the doomsday framing does not describe how anyone actually trains.

The problem it does create for evaluation

There is a second-order effect that gets less attention and matters more day to day. As synthetic data becomes common, the boundary between training data and benchmark data blurs.

Generating training examples from public benchmarks, or from web text that discusses them, is easy to do accidentally. The result is a model that scores well on a published evaluation without the corresponding capability, which is a harder problem to detect than deliberate contamination because nobody involved intended it. Data contamination in training and benchmark contamination cover the detection side.

The practical implication is a familiar one: weight private and recently constructed evaluations more heavily than long-published benchmarks, and keep a task set drawn from your own work, which by construction cannot appear in anyone's pipeline.

What this means when choosing a model

You will almost never know what a given model was trained on. Labs disclose very little about data composition, and where they do disclose it, the description is high level. Treat any confident claim about a specific model's synthetic data proportion as speculation unless the lab published it.

What you can observe is the signature. Models trained heavily on generated instruction data tend to be strongly formatted, consistent in tone, and reliable at following structure — and sometimes brittle just outside the distribution they were shaped on, where a slightly unusual request produces a confidently wrong answer in an impeccably tidy format.

So test the edges rather than the centre. Unusual libraries, your internal conventions, awkward multi-step tasks, inputs that do not resemble a tutorial. Those probe whether the model has coverage or only style, and no published figure will answer that for you. Benchmarking on your own work covers building the probe.