Rate Limit Tiers Explained: RPM, TPM and What Binds First

Most teams meet rate limits as a 429 in a log and treat it as a transient error to retry through. That works until it does not, and the reason it stops working is that a rate limit is not one number. It is at least two meters running at once, measuring different things, and knowing which one you hit changes the fix entirely.

If you retry harder against a token limit you make it worse. If you batch harder against a request limit you fix it. Guessing wrong is common because the error looks identical either way.

Two meters, not one

Requests per minute counts calls. It does not care whether a call carried fifty tokens or fifty thousand. It exists to protect the provider from connection churn and scheduling overhead, and it is the limit that binds on workloads made of many tiny requests — classification, embedding, short structured extraction.

Tokens per minute counts work. It usually counts input plus output together, though some providers meter them separately, and it exists to protect the actual GPU capacity behind the endpoint. It binds on workloads made of few large requests, which describes essentially all agent traffic.

Both are typically enforced over a rolling window rather than a wall-clock minute, which is why the limit can bind at second 12 of a minute that has barely started. And they are enforced independently: you can be at 5 percent of your request limit and fully saturated on tokens.

Why the token limit usually binds first

Divide your token limit by your request limit and you get the average request size the tier was designed for. For most published tiers that figure lands somewhere in the low thousands of tokens. Chat messages fit inside it comfortably. Agent turns do not.

The reason is that models are stateless, so every turn resends the whole conversation. An agent on turn fifteen is sending fifteen turns of history plus tool definitions plus file contents, and that single request can be twenty or thirty times the size the tier assumed. Fifteen of those in a minute can exhaust a token budget while using a trivial fraction of the request budget.

Reasoning models push this further, because the tokens they generate internally count against output even when you never see them. The token cost of reasoning models covers that accounting, and it is the usual explanation when a workload hits token limits at a request volume that looks obviously safe.

Long context makes it worse again in a way that is easy to miss. Doubling the context you send doubles input tokens on every single turn, so a config change that looks like a quality improvement can halve your effective throughput. The hidden cost of context bloat is the same phenomenon viewed through spend instead of throughput.

Burst versus sustained

Published limits are usually sustained rates, but the enforcement is often a token bucket, which means you can briefly exceed the stated rate if you have been under it. That is why a workload can run fine for ten minutes and then start throttling with no change in average load — the bucket that was absorbing your bursts has drained.

The practical consequence is that average throughput is the wrong thing to plan against. Plan against your peak minute, and if your traffic is bursty by nature, smooth it deliberately rather than relying on a burst allowance you cannot see the size of.

The pathological case is a fan-out. Twenty subagents launched simultaneously present as one enormous burst, exhaust the bucket, and then all twenty retry at the same moment and exhaust it again. Bounding agent fan-out covers the concurrency side; from the rate-limit perspective the fix is to stagger the launch, not to raise the ceiling.

How tiers get assigned and raised

Providers assign tiers on some combination of account age, cumulative spend and payment history, and they promote automatically when you cross a threshold. The mechanism is fraud and abuse control more than capacity allocation, which is why a brand-new account with a large budget still starts low.

Two things follow. First, promotion is usually gradual and automatic, so the answer to a limit you will outgrow in a month is often to wait rather than to file a ticket. Second, prepaying can accelerate it where the tier depends on spend to date, which is a genuine reason to prepay beyond the usual cash-flow ones.

When you do request an increase, come with numbers: your current utilisation against each meter separately, your peak minute, the workload shape, and what you have already done to smooth it. A request that shows you understand which meter binds gets a materially better response than one that asks for more of everything.

Engineering around limits

Backoff is the baseline and the part most often done badly. Retrying immediately on a 429 adds load to a system that just told you it is saturated, and a fleet retrying in lockstep produces a thundering herd that keeps the limit permanently binding. Exponential backoff with jitter is the minimum; exponential backoff done properly covers the parameters that matter.

Better than backoff is not hitting the limit. Read the rate limit headers most providers return — remaining requests, remaining tokens, and seconds until reset — and throttle proactively against them. That converts a reactive error-handling problem into a scheduling problem, and scheduling problems are much easier to reason about.

Batching addresses request limits specifically. If you are throttled on RPM with token headroom to spare, combining many small items into fewer larger calls trades directly along the axis you have room in. It does not help at all against a token limit, which is why identifying the binding meter first matters so much. Batching requests covers the trade-offs, including the latency cost.

Queueing addresses both. Put a bounded queue in front of the provider, drain it at a rate you know is under the limit, and let backpressure propagate to the callers. This gives you one place to enforce the rate rather than every caller guessing, and it makes utilisation observable. Rate limiting an agent fleet covers the shared-budget version of the problem.

Instrumenting it properly

The single most useful thing you can log is which meter caused each 429, taken from the response headers rather than inferred. Without that, every rate-limit investigation starts with an argument.

Then track utilisation against both meters as a percentage of the limit, sampled per minute rather than averaged per hour. An hourly average hides exactly the burst that is causing your problem. And track throttled request count separately from failed request count, because a request that succeeded after backing off is a capacity signal, not an error, and burying it in the error rate loses that signal entirely.

Finally, watch the ratio between your token utilisation and your request utilisation over time. When that ratio moves, your workload shape has changed — usually because context grew or an agent got a new tool — and it will tell you which limit you are about to hit before you hit it.

A decision rule

Find the binding meter first. If requests are saturated and tokens are not, batch. If tokens are saturated and requests are not, reduce what you send: trim context, cache prefixes, use a smaller model for the easy portion of the work. If both are saturated, you need a higher tier and you now have the evidence to ask for one.

In every case, put a bounded queue in front of the provider before you tune anything else. Most rate-limit pain is not a capacity problem, it is an admission-control problem, and the queue is where admission control lives.