Circuit Breakers for LLM APIs: Failing Fast on Purpose

A circuit breaker is a bet that when a dependency has failed the last twenty times, the twenty-first call is not worth making. For ordinary services that bet saves latency. For inference providers it also saves money, because a request that fails after forty seconds of streaming has still consumed prefill on their side and a worker on yours.

Retries protect a single request. A breaker protects the system, by noticing that a provider is down and refusing to keep discovering it once per request. The two mechanisms are complementary, and the retry side is covered in the mechanics of exponential backoff.

The three states

Closed means traffic flows and the breaker counts outcomes. Open means every call is rejected immediately without touching the network. Half-open is the recovery probe: after a cooldown, a small number of calls are allowed through, and the result decides whether the breaker closes again or reopens for another cooldown.

The half-open state is the part people skip and the part that matters. Without it, a breaker either stays open forever or dumps the full production load onto a provider that has just come back, which knocks it over again. Allow one or two concurrent probes, no more.

The transition back to closed should require more than one success. A single lucky response from a provider running at ten percent capacity will flap the breaker straight back open. Requiring three consecutive successes is a common and reasonable choice.

Count the right failures

This is where LLM breakers differ from generic ones. Not every error means the provider is unhealthy, and a breaker that counts the wrong things trips on your bugs instead of on theirs.

Count connection failures, read timeouts, 5xx responses, and the 529 overload status. Do not count 400s, 401s, 404s or schema validation failures — those are your requests being wrong, and opening the circuit because of a malformed prompt template takes down a healthy provider for everyone.

429s need a decision. A rate limit means the provider is fine and you are over your allowance, so tripping a breaker is arguably wrong; but continuing to hammer a limit you cannot clear is also wrong. The workable compromise is to feed 429s into your pacing layer rather than the breaker, and only count them as failures if they persist past the advertised Retry-After window. The distinction between a rate limit and an exhausted quota is set out in the error code reference.

Thresholds that survive low traffic

A percentage threshold on its own is unusable at low volume: one failure out of two requests is a fifty percent error rate, and it means nothing. Every breaker needs a minimum-volume guard.

The shape that works is a rolling window with two conditions — at least twenty requests in the last thirty seconds, and at least fifty percent of them failed. Below the volume floor the breaker stays closed regardless of the ratio.

class Breaker:
    def allow(self):
        if self.state == "open":
            if time.monotonic() >= self.opened_at + self.cooldown:
                self.state = "half_open"
                self.probes = 0
            else:
                return False
        if self.state == "half_open":
            if self.probes >= self.max_probes:
                return False
            self.probes += 1
        return True

Consecutive-failure counting is the simpler alternative and it is often better for a dependency called a few times a minute. Ten failures in a row is unambiguous at any volume, and it needs no window bookkeeping.

Slow is a failure too

The classic brownout is a provider that never returns an error and just gets slower. Error-rate breakers sit closed through the entire incident while every request burns its full timeout.

Count a call as a failure when it exceeds a latency threshold, not only when it throws. Pick the threshold from your own observed distribution rather than a round number — something around three times the normal high-percentile latency for that model is a defensible starting point, and it needs to be per-model because a reasoning model and a small fast model have completely different baselines.

Concurrency limiting achieves something similar with less configuration. A bounded semaphore in front of the provider means a slowdown produces immediate rejections once the pool is full, which is a breaker by another name. The latency baselines you need for either are discussed in what actually drives inference latency.

Scope the breaker correctly

One global breaker per process is too coarse. Providers fail per model, per region and per deployment far more often than they fail entirely, and a breaker keyed only on the provider name will either trip when one model is degraded or fail to trip at all.

Key on the tuple you would actually route on: provider, model, and region if you use more than one. That keeps a single degraded model from blocking traffic to healthy ones.

Do not go finer than that. A breaker keyed per API key or per user never accumulates enough samples to be meaningful, and you end up with thousands of breakers that each know nothing.

Open circuit, then what?

A breaker that only returns errors converts an upstream outage into your outage slightly faster. The value comes from what you do in the open state.

Fallback to a second provider is the strongest option when your prompts are portable, and it is the main argument for keeping requests provider-neutral in the first place — see routing between models and falling back. The caveat is that fallback multiplies your blast radius if you fail over into a provider already at capacity, so the fallback path needs its own breaker.

Degradation is the second option: serve a cached answer, a smaller model, or a clearly-labelled reduced response. Queueing is the third, and it is the right one for background work where nobody is waiting — accept the job, park it, and drain when the breaker closes.

The wrong option is silently returning something that looks like a real answer. If you fall back to a weaker model, record which model served the response, because the difference will show up in your evaluations and you will want to know why.

Making it debuggable

A breaker is a stateful thing that changes behaviour without any code path being obviously different, which makes it a common source of confusing incidents. Instrument it as a first-class object.

Export the current state as a gauge per key, count state transitions, and count requests rejected by an open breaker separately from requests that failed upstream. When an engineer asks why error rates spiked with zero provider calls in the logs, that last metric is the answer.

Log every transition with the counts that caused it. "Opened after 12 consecutive timeouts on model X" is actionable; "circuit open" is not. These signals belong in the same traces as your request spans, as described in tracing agent and LLM calls.

When not to bother

If you make a handful of LLM calls a minute from a single process, a breaker adds state and a new failure mode for very little benefit. A timeout and a retry cap cover you. Breakers earn their keep when many concurrent requests share a dependency, because that is when discovering failure independently per request becomes expensive.