Benchmark Contamination Explained, and What to Do About It

Contamination is what happens when the answers end up in the training data. The model does not solve the task, it recognises it, and the resulting score measures memory rather than capability.

It is a real problem and also the laziest available explanation for any benchmark result someone dislikes. Both of those things are worth holding at once.

How the leak actually happens

The obvious route is direct: a benchmark is published to a public repository, the repository gets crawled, and the test items are somewhere in a training corpus measured in trillions of tokens. Nobody has to intend it.

The subtler route is discussion. A benchmark becomes popular, people write blog posts analysing specific items, solutions appear in notebooks, and papers quote examples. The test set gets reconstructed piecemeal across thousands of documents that no filter would flag as the benchmark itself.

Then there is derived leakage, which is the hardest to reason about. Training on the linked GitHub issue, the merged pull request, or the release notes for the version that fixed a bug can carry the answer without the benchmark file ever appearing anywhere.

This is why age matters more than anything else. A benchmark published three years ago has had three years of crawls, discussion and derivative content; a benchmark released last month has had almost none.

What it looks like from the outside

You rarely get proof, so you look for signatures. The clearest one is a large gap between a model's score on an established benchmark and its score on a fresh or private set of comparable difficulty.

Another is brittleness under perturbation. Rename the variables, change the numbers, restate the problem in different words, and a genuinely capable model barely moves while a memorising one drops sharply.

A third is suspiciously uniform performance across a benchmark's difficulty range. Real capability degrades as tasks get harder; recall does not, so a flat profile across easy and hard items is a signal worth chasing.

None of these is conclusive alone. Treat contamination as one hypothesis among several — harness differences, prompt tuning and task-pool fit explain many surprising results, as why benchmarks disagree sets out.

Why it is over-diagnosed

Contamination is a satisfying accusation because it requires no evidence and cannot easily be disproved. When a model from an unfamiliar lab posts a strong number, it is the first thing said, and it is usually said by someone who has run no perturbation test at all.

Most score inflation has more mundane causes. A vendor evaluates with a harness they built, a prompt they iterated on, a reasoning effort setting turned up, and several attempts allowed. That is worth several points without anyone touching the test set, which is the subject of vendor-claimed versus independent benchmarks.

Task-pool fit explains more still. A model trained heavily on Python web repositories will look strong on a benchmark drawn from Python web repositories. That is a transfer limitation, not cheating, and it produces the same shape of surprise.

The useful posture is to stop arguing about intent and start weighting evidence by how contaminable it is.

Defences that work

Prefer recent benchmarks over established ones, and prefer held-out or rotating sets over fixed ones. A benchmark that refreshes its items periodically decays much more slowly than one frozen in 2023.

Prefer live preference data for anything a human reads. Arena-style pairwise voting uses prompts written fresh by users and never published as a set, which makes memorisation close to impossible even though it brings its own presentation biases.

Perturb before you believe. If a published figure matters to a decision, take five items from that benchmark, rewrite them so the substance is identical and the surface is different, and check whether the model holds up. Ten minutes of this settles most arguments.

Most of all, keep a private set. Tasks from your own repository that were never public cannot be in anyone's training corpus by construction, and that property is worth more than any methodology argument.

Building the private set

Pull tasks from your own history rather than writing them. Closed issues where you have the real fix, incidents where you know the root cause, refactors you already completed — the ground truth exists and cost you nothing to produce.

Keep the set out of public repositories, out of pull request descriptions, and out of anything you paste into a tool that logs prompts for training. A private benchmark stops being private the first time it is convenient to share it.

Refresh it as your codebase moves. A set built from two-year-old tasks measures a codebase you no longer have, and it slowly stops predicting anything. Building a model eval harness covers the mechanics of keeping one alive.

The decision rule is simple. Use public benchmarks to build a shortlist and to spot profile differences, discount them in proportion to their age and publicity, and let your own uncontaminated set decide — because it is the only measurement that is definitely about your work.