How does benchmark data get into a training set?

Benchmarks are public, so their problems and solutions appear across repositories, papers, tutorials and forum posts. Web-scale crawling picks those up, and decontamination filters that match exact strings miss paraphrased copies.

Can I tell whether a specific model is contaminated?

Not from outside. Training corpora are not published, so you only see symptoms such as a gap between public scores and private ones, or sharp degradation when a problem is superficially reworded.

Does contamination change which model I should pick?

It distorts absolute scores more than relative ranking, since all major models are exposed to similar data. Use leaderboards for a shortlist, then decide on thirty to fifty tasks from your own recent work.

Data Contamination in Training: Why Scores Overstate Skill

A model that has seen the answers before the exam will score well and tell you nothing. That is contamination, and it is the most common reason a model that looks excellent on a leaderboard feels ordinary on your codebase.

The word gets thrown around as an accusation. It is more useful as a mechanism to understand, because knowing how it happens tells you which scores to discount and which to trust.

How test data ends up in training data

Pretraining corpora are assembled by crawling very large amounts of public text. Benchmarks live in public repositories, get quoted in papers, get discussed on forums, get reproduced in blog posts and get copied into tutorial notebooks.

By the time a benchmark is two years old, its problems and often its solutions exist in dozens of paraphrased forms across the open web. A crawler cannot reliably tell a benchmark item from ordinary text, because textually it is ordinary text.

Nobody has to cheat for this to happen. The default outcome of scraping the internet is that popular evaluation sets are in the corpus. Labs run decontamination filters, but those filters match on exact or near-exact strings, and a rewritten version of the same problem slips through untouched.

The problem compounds with derived data. Synthetic training data is frequently generated by prompting an existing model, and if that model absorbed a benchmark, its generations can carry the pattern forward into the next model's training set.

What contamination looks like from outside

You almost never see contamination directly. You see symptoms.

The clearest one is a gap between a public benchmark score and a private, structurally similar evaluation. If a model scores near the top on a well-known coding benchmark and mid-pack on the same kind of task drawn from your own repository, something is inflating the public number.

Another symptom is fragility to superficial rewording. A model with genuine capability on a class of problems handles a renamed variable, a changed constant or a reordered clause without much loss. A model relying on memorised structure often degrades sharply for changes that should be irrelevant.

A third is suspiciously specific output. A model that reproduces a canonical solution's exact helper function names, exact comment wording or exact edge-case ordering has probably seen that solution rather than derived it.

Why proving it is nearly impossible

Training corpora are almost never published in full, so an outside observer cannot search them. What remains is inference from behaviour, and behavioural evidence is weak.

Memorisation and competence produce similar outputs on the items where they agree. A model may have genuinely learned the pattern that a benchmark tests, and score highly for legitimate reasons, and look identical to a contaminated model on that benchmark.

The reverse mistake is common too. Contamination gets invoked whenever a score seems too good, which converts a real phenomenon into an unfalsifiable dismissal. Why benchmarks disagree covers the other explanations that usually account for more of the variance than leakage does.

The honest position is that contamination is one of several forces inflating public scores, it is rarely the largest one, and you can neither confirm nor rule it out from outside.

Contamination of the harness, not just the data

There is a subtler version worth naming. A benchmark is a dataset plus a harness: a prompt format, a scoring rule, a retry policy, an allowed toolset.

Labs optimise against public harnesses because that is what gets reported. Prompts get tuned, tool schemas get shaped, output formats get aligned to what the scorer accepts. None of that is leakage of test items, and all of it inflates the score relative to what you will get in your own harness.

This is why the same model can post a strong figure on an agentic coding benchmark and behave differently inside your editor. Your harness is not the benchmark's harness. SWE-bench explained goes through how much the harness decides.

Why it matters less than it sounds

Contamination inflates the absolute number. It distorts relative ordering less, because every major model is trained on broadly similar web-scale data and every major model is exposed to broadly the same public benchmarks.

So a leaderboard remains weakly informative about ranking while being badly wrong about magnitude. Treating a benchmark percentage as a probability that the model will solve your next ticket is the actual error. Treating it as a rough tier signal is defensible.

The exception is a newly published benchmark. For the first months of its life a benchmark is genuinely uncontaminated for models trained before it existed, which is why fresh benchmarks tend to show wider spreads and lower absolute scores than mature ones.

Building an evaluation contamination cannot reach

The only reliable defence is private data the model has not seen, and it is easier to assemble than people expect.

Pull thirty to fifty tasks from your own recent history: closed pull requests, resolved incidents, real bug reports with their real fixes. These have never been crawled, they match your actual distribution of work, and they carry a ground truth you already trust.

Score them the way you would judge a colleague — did the change work, did it need rework, did it break something else — rather than by string match against a reference answer. Benchmarking LLMs on your own work covers building this without turning it into a project.

Re-run it whenever you change models. Provider-side model updates shift behaviour without changing the name you call, which is a separate reason to pin model versions and re-measure deliberately rather than discovering drift in production.

A working rule

Use public benchmarks to build a shortlist of two or three candidates and never to make the final choice. Assume every published figure is optimistic by an unknown margin, and assume that margin is larger for older benchmarks than newer ones.

Then decide on your own tasks. A model that wins on fifty examples from your repository has demonstrated something no leaderboard can, and the measurement costs less than a day. Choosing a model for coding covers what else belongs in that decision.