Rate Limiting Agent Fleets Without Killing Throughput

A single agent hitting a rate limit is a nuisance you solve with backoff. A fleet hitting one is a different problem, because every agent is drawing from the same bucket and none of them knows the others exist.

The symptom is distinctive. Throughput is fine, then a batch job starts, then every interactive request in the system begins failing — including the ones from users who have nothing to do with the batch. The provider is enforcing an organisation-level limit, and your architecture has no concept of it.

Fleet-wide rate limiting is the piece that turns a working prototype into something you can run for other people.

Know which limit you are actually hitting

Providers typically enforce more than one dimension at once: requests per minute, input tokens per minute, output tokens per minute, sometimes concurrent requests. They usually apply per key or per organisation, so splitting work across processes does not split the quota.

These bind differently. A fleet doing many short classifications hits requests per minute. A fleet doing long-context analysis hits tokens per minute at a small fraction of the request rate, because a single request carrying a large prompt can consume a meaningful slice of the minute's budget on its own.

Read the response headers rather than guessing. Most OpenAI-compatible providers return remaining quota and reset time on every call, and those headers are the only honest picture of where you stand. If you are not logging them, start there before building anything. The API error code reference covers what the rejections themselves look like.

A shared token bucket, not per-process limits

The common mistake is limiting each worker process independently. Ten processes at ten requests per second each is a hundred per second at the provider, and the per-process limit gives you a comforting sense of control while enforcing nothing.

The limiter has to be shared. A token bucket in Redis, keyed per provider and per limit dimension, with every agent acquiring before it calls, is the standard shape and is not much code. Refill at the provider's stated rate, allow a modest burst, and make acquisition block rather than fail.

Crucially, acquire for the estimated cost, not for one unit. A request with a 200,000-token prompt should draw 200,000 tokens from the token bucket, not one. Estimate from the prompt you are about to send, then reconcile with the usage the response reports, so persistent under-estimates get corrected rather than compounding.

Keep the buckets slightly below the published limits. Running at ninety-five percent leaves headroom for estimate error and for whatever else shares the key.

Admission control beats retry

The difference between a fleet that degrades gracefully and one that collapses is whether excess work waits at the front door or discovers the limit by being rejected.

Rejections are expensive in a way that is easy to miss. You have already paid the latency, you often have partial work in flight, and the retry re-sends the same large prompt. A fleet in a rejection loop can spend a large share of its budget on calls that never returned anything, which is the dynamic described in the hidden cost of retries.

Waiting at admission costs only time. The agent asks for capacity, the limiter says wait four hundred milliseconds, the agent waits and then succeeds. No wasted tokens, no thundering herd, and a queue depth you can actually observe.

Priority, or the batch job eats everything

An undifferentiated fleet is unfair by construction. Whoever queues most work gets most capacity, which is precisely backwards from what you want.

Give the limiter classes. Interactive traffic gets a reserved share of the bucket that batch work cannot touch. Background jobs get the remainder and are expected to be slow. Retries should go in a lower class than first attempts, so a struggling workload cannot crowd out healthy traffic while it thrashes.

If you serve multiple tenants from one key, per-tenant ceilings become mandatory rather than optional. One customer's runaway agent should degrade that customer, and nobody else. This is the same isolation argument that shapes sensible agent cost controls.

Backoff that does not synchronise

When rejections do happen, the fleet must not recover in lockstep. Twenty agents that all back off for exactly one second all return at exactly the same moment and reproduce the spike.

Exponential backoff with full jitter is the well-established answer: sleep a random duration between zero and the current ceiling, doubling the ceiling each attempt. It spreads the return across the window instead of concentrating it.

Honour the retry-after header when the provider sends one — it is better information than any local heuristic. And cap total attempts at the fleet level, not just per request, or ten agents retrying five times each becomes fifty extra calls nobody authorised. Rate limits and retries covers the per-request mechanics in more depth.

Reduce demand before you optimise the limiter

The best fix for a rate limit is often needing less of it.

Caching the stable prefix of a prompt cuts input tokens substantially on repeated calls, and input tokens are usually the dimension that binds first for agent workloads. Prompt caching is the highest-leverage change available to most fleets.

Routing mechanical steps to a smaller model spreads load across a second quota and costs less per call. So does trimming what agents send: a tool that returns ten relevant lines instead of a whole file reduces every subsequent turn in that conversation, and the compounding is larger than it looks.

What to monitor

Track the rejection rate as a percentage of total calls, the time agents spend waiting at admission, remaining quota from the response headers, and the share of spend attributable to retried calls.

The last one is the number that changes behaviour. A team that discovers a tenth of its bill went to calls that returned nothing usually finds the engineering time for admission control immediately, where the same team will happily ignore a rejection-rate graph for months.

The shape to build

One shared limiter per provider key, acquiring by estimated token cost across every dimension the provider enforces. Classes with reserved capacity for interactive work. Jittered backoff honouring retry-after. Fleet-level attempt caps. Headers logged from every response so the limiter can be calibrated against reality rather than documentation.

Build it before you scale the fleet, not after. Retrofitting a limiter means finding every call site, and in a system where agents spawn agents there are always more of those than the diagram suggests.