Repetition Penalties: Why They Break Code Generation

Repetition penalties are the standard response to a model that gets stuck repeating itself, and they are one of the few sampling parameters that can actively break a working system. On prose they are a blunt but tolerable fix. On code they punish exactly the tokens that must repeat, and the failure they cause is subtler than the one they were meant to solve.

Why models loop in the first place

A model predicts the next token from everything before it. Once a phrase appears in the context, the pattern "that phrase, then that phrase again" becomes a pattern the model has seen — in lists, in tables, in poetry, in generated boilerplate.

So repetition is mildly self-reinforcing. Each occurrence raises the probability of the next, and if a few land in a row the loop can become the highest-probability continuation available. From inside the distribution there is no signal that this is pathological.

The tendency is much stronger at low temperature, where the model always takes the leading token and never gets the nudge that would break the cycle. It is also stronger in quantised or heavily compressed deployments, where numerical error flattens the distinctions that would otherwise separate candidates.

The three knobs and what they differ on

Providers expose up to three parameters, and they are not interchangeable.

Presence penalty subtracts a fixed amount from the score of any token that has already appeared, once, regardless of how many times. It pushes towards introducing new vocabulary rather than towards avoiding loops specifically.

Frequency penalty subtracts an amount proportional to how often the token has appeared. A token used twenty times is penalised ten times as hard as one used twice, which targets runaway repetition much more precisely.

Repetition penalty, common on open-weight serving stacks rather than hosted APIs, divides the logit instead of subtracting from it. That is a meaningfully different operation, and because negative logits behave inversely under division, values above 1.0 can produce surprising results. Treat it as the most dangerous of the three.

Why this is destructive on code

All three operate on tokens without any notion of what a token means, and code repeats deliberately.

Write a Python file with forty functions and the token for a four-space indent appears hundreds of times. A frequency penalty escalates against it with every occurrence, and eventually a less-indented alternative outscores the correct one. The output is syntactically broken in a way that looks like the model simply lost track.

The same applies to closing braces, to return, to const, and above all to identifiers. A variable named userRepository used consistently across a class is exactly what good code looks like and exactly what a frequency penalty attacks. The model starts reaching for a synonym, and you get a reference to something that does not exist.

The failure is worse than the loop it prevented, because it produces plausible code that does not compile or, occasionally, code that compiles and is wrong. Set both penalties to zero for code generation. This is not a tuning preference; it is a correctness requirement.

What to do instead about looping

Check temperature first. Repetition at temperature zero is common and often disappears entirely at 0.3, without touching a penalty. That is the cheapest fix available and the one most often skipped.

Check the prompt second. Models loop when they have run out of things to say but have not been given permission to stop — an instruction to produce ten items when only four exist, or an open-ended request with no terminating condition. Telling the model it may return fewer items, or giving it an explicit end marker, removes the pressure that caused the loop. Stop sequences explained covers the mechanics of ending generation cleanly.

Check the deployment third if you self-host. Repetition that starts only on long outputs is a classic sign of accumulated numerical error in an over-aggressively quantised model. Quantisation explained covers where those thresholds sit.

Loops in agents are a different problem

An agent that calls the same tool with the same arguments five times running is not experiencing token-level repetition. Each call may be generated with perfectly healthy sampling.

It is stuck because its context contains a failed action and no new information, so the most reasonable next step genuinely is to try that action again. No sampling parameter addresses that, and adding a frequency penalty will only corrupt the tool arguments.

The fix belongs in the loop, not in the decoder: track recent action signatures, detect the repeat, and change the situation — inject the observation that this has already been tried, escalate to a different strategy, or stop. Detecting agent loops covers the detection, and agent error recovery patterns covers what to do once you have caught one.

When a penalty is the right tool

There is a narrow band where these parameters earn their place. Long-form creative prose where the model reaches repeatedly for the same adjective, or generating variety across a batch of marketing copy, both benefit from a small presence penalty.

Keep the values low — well under half the maximum most APIs allow — and read a full output rather than a snippet, because the damage from an over-aggressive penalty shows up late, once enough tokens have accumulated to trigger it.

Anywhere output feeds a parser, a compiler or another program, leave them at zero. Structured outputs and JSON mode covers the better tool for those cases: constrain the grammar so invalid tokens cannot be sampled at all.

The takeaway

The decision rule is short. Code, JSON and any machine-consumed output: penalties at zero, always. Creative prose with genuine word-choice repetition: a small presence penalty, checked on full outputs. Agent loops: fix the loop, never the sampler.

And when repetition appears, work through temperature, prompt and precision before reaching for a penalty. In most cases one of those three is the actual cause, and the penalty would only have masked it.