Exponential Backoff That Works on LLM APIs

Exponential backoff is four lines of code that almost everyone gets subtly wrong. The exponent is the easy part. The parts that decide whether your retry layer helps or hurts are the jitter distribution, the deadline, the classification of what deserves a retry at all, and whether the call is safe to repeat.

The policy question — how retries interact with rate limits and quota — is covered in pacing requests under a rate limit. This is the algorithm itself.

Why the naive version amplifies

Plain exponential backoff computes a delay of base times two to the attempt number, and sleeps exactly that long. Every client that failed at the same moment now retries at the same moment, one second later, then two, then four. The load pattern is a series of synchronised spikes rather than a smooth recovery.

Inference capacity is genuinely scarce, so this matters more than it does against a stateless web service. A provider shedding load to recover gets hit by the entire fleet again at each interval boundary, and the outage lasts longer than the fault that caused it.

The fix is randomisation, and the specific distribution you pick has a measurable effect on both completion time and load on the upstream.

Full jitter and decorrelated jitter

Full jitter sleeps a uniformly random duration between zero and the current ceiling. It maximally spreads retries, which is what a struggling upstream wants, at the cost of sometimes retrying almost immediately.

def full_jitter(attempt, base=0.5, cap=30.0):
    return random.uniform(0, min(cap, base * (2 ** attempt)))

Decorrelated jitter derives the next delay from the previous one rather than from the attempt number, which produces a random walk that spreads well while trending upward more smoothly.

def decorrelated(prev, base=0.5, cap=30.0):
    return min(cap, random.uniform(base, prev * 3))

Either is fine. Both are enormously better than no jitter. Pick one, cap it, and stop tuning — the difference between them is small compared with the difference between having jitter and not having it.

Retry-After beats your arithmetic

When the response carries a Retry-After header, use it. It is the provider telling you the minimum wait, and it is better information than an exponent that knows nothing about the upstream state.

Read it defensively. The header may be an integer number of seconds or an HTTP date, and some deployments send retry-after-ms in milliseconds instead. Clamp whatever you parse to a sane ceiling so a misconfigured gateway advertising a 3,600-second wait does not park your worker for an hour.

Then still add a small random offset on top. A hundred clients all honouring the same Retry-After value will otherwise resynchronise perfectly at the end of the window, which is the exact problem jitter exists to solve.

Classify before you retry

The retryable set is small and worth writing down explicitly: connection errors, read timeouts, 429 rate limits, and 5xx server errors including 503 and the 529 overload status Anthropic uses.

Everything else is a bug in the request. A 400 stays a 400. A 401 will not authenticate itself on the third attempt. A 429 carrying an insufficient_quota code is a billing state rather than congestion, and retrying it wastes latency and replaces a clear error message with a vague one. The full breakdown is in the LLM API error code reference.

The awkward case is a 500 that reproduces every time with the same body. That is not a transient fault, it is an input the provider cannot handle, and no backoff schedule will fix it. Track repeat failures by request fingerprint so you can tell the two apart instead of retrying a poisoned request forever.

Deadlines, not attempt counts

Attempt counts are the wrong budget. Five attempts with a thirty-second ceiling can hold a request open for well over a minute, and if a user is waiting you have chosen the worst outcome available: slow, then failed.

Set a deadline when the request enters your system and pass it down. Before each retry, check how much of it remains; if the remaining time is less than the expected duration of another attempt, give up now and return the error you already have.

deadline = time.monotonic() + budget_seconds
attempt = 0
while True:
    remaining = deadline - time.monotonic()
    if remaining <= 0:
        raise last_error
    try:
        return call(timeout=min(per_try_timeout, remaining))
    except Retryable as e:
        last_error = e
        delay = full_jitter(attempt)
        if delay >= deadline - time.monotonic():
            raise
        time.sleep(delay)
        attempt += 1

Deadline propagation also solves the nested-timeout problem, where an inner retry loop happily exceeds the timeout of the caller that is already gone. The interaction between per-attempt and overall limits is worked through in tuning timeouts for LLM calls.

Only retry what is safe to repeat

A chat completion is usually idempotent in the sense that repeating it does not corrupt state — but it is not free, and it is not idempotent if a tool call in the loop wrote something.

Two rules keep this straight. Never retry a request whose side effects have already been committed; retry the outermost operation instead, with the tool results you already have. And when the provider supports an idempotency key, send one, so a request that timed out on your side but succeeded on theirs is not billed and executed twice.

Streaming needs its own rule. A stream that died after emitting most of its tokens has already been paid for, and restarting pays again from zero. Decide explicitly between restarting and continuing from the partial text, as described in handling SSE disconnects.

One retry layer, and it must be observable

The most damaging configuration is the accidental one. Provider SDKs retry by default, HTTP clients can retry, service meshes retry, and your own decorator retries. Three layers of three attempts is twenty-seven upstream calls for one logical request.

Pick the layer you can instrument, and disable the rest. Then export attempt counts, the distribution of final outcomes, and the total time spent sleeping in backoff. A retry layer you cannot measure is indistinguishable from an amplification bug, and the money it burns is discussed in the hidden cost of retries.

Test it before production does

Backoff code is rarely exercised until the day it matters, which is the worst possible time to discover a sign error. Three tests are enough to keep it honest.

First, a fake clock and a seeded random source, asserting that the delay sequence stays under the cap and respects the deadline. Second, a stub client that returns a fixed sequence of statuses, asserting that a 400 is not retried and a 503 is. Third, a load test where the stub fails everything, asserting that total upstream calls stay within your retry budget rather than growing with concurrency.

If you only write one, write the third. It is the one that catches stacked retry layers, and stacked retries are how a minor upstream wobble becomes your incident.