Agent Queueing and Backpressure: When to Say No

Queues feel like the safe choice. Work arrives faster than agents can process it, so you buffer, and nothing is rejected. Everyone gets served eventually.

Eventually is the problem. An agent task that sat in a queue for forty minutes is often worthless by the time it runs — the branch moved on, the user gave up, the incident it was investigating resolved itself. You did not avoid failure, you deferred it and paid full token price for the deferral.

Backpressure is the discipline of admitting that the system is over capacity while it can still do something about it. For agent fleets it matters more than for ordinary services, because each unit of work is slow, expensive and non-refundable once started.

Why agent queues behave differently

Ordinary request queues drain fast. An agent task takes seconds to minutes, so a queue of a hundred tasks in front of a pool of five is not a brief buffer, it is an hour.

Service time also varies enormously. One task finishes in three tool calls and another runs to its step ceiling, so tail latency is dominated by a handful of long runs. Averages are actively misleading here; a queue sized on mean service time will be wrong by an order of magnitude on bad days.

And the work is not free to abandon. Cancelling a task mid-run has already cost you every token it spent, and it may have left half-finished side effects behind. That makes admission decisions far more consequential than in a stateless service, and it is why idempotency in agent actions is a prerequisite for aggressive shedding.

Bound the queue, always

An unbounded queue is a way of converting an overload problem into a memory problem plus a staleness problem. Give it a hard depth limit and decide explicitly what happens when it is full.

Size the bound in time, not items. If the pool completes roughly four tasks a minute and you are willing to make someone wait five minutes, the queue holds twenty. Expressing it that way makes the number defensible and makes it obvious when service time changes invalidate it.

Then reject at the door with a clear signal: a 429 with a retry-after, or an explicit queue-full response the caller can act on. Rejecting is not a failure of the system, it is the system telling the truth about its capacity, which is the only thing that lets callers adapt.

Deadlines beat positions

Every task should carry a deadline: the time after which running it is pointless. Attach it at submission, check it at dequeue, and drop anything that has expired without running it.

This one mechanism removes most of the pathology. Under a spike, stale work evaporates instead of consuming capacity, and the fleet spends its tokens on requests that someone still cares about. Without deadlines a queue that fell behind stays behind, because it is always working on the past.

Deadlines also give you an honest health signal. The rate of expiry is a direct measure of how far under-provisioned you are, and it is far more actionable than average queue depth.

Propagate the deadline into the run. An agent given a wall-clock ceiling derived from its remaining deadline will not spend twenty minutes on a task that was due in five, and it pairs naturally with the step budgets discussed in agent token budgets.

Shed load deliberately

When you must drop work, choose what to drop rather than dropping whatever arrives when the buffer is full.

Shed by class first: background enrichment before interactive requests, speculative work before requested work, retries before first attempts. A fleet that sheds its own retries under load recovers; one that sheds new user requests to keep retrying old ones does not.

Degrade rather than drop where the task allows it. A cheaper model, a smaller step budget or a single-pass answer instead of a full agent loop still returns something useful. Half a result now is usually worth more than a complete result after the deadline, and knowing when a cheap model is enough is what makes this option available.

Whatever you shed, record it. Silently discarded work is how a fleet develops a reputation for being unreliable in ways nobody can reproduce.

Backpressure has to travel upstream

A limiter that only slows the worker pool leaves the pressure sitting in the queue. The signal has to reach whatever is generating work.

For human-facing entry points that means visible feedback: a queue position, an estimated wait, a disabled submit button. For programmatic callers it means a rejection with retry-after that their client actually honours.

For internal producers — the orchestrators spawning subagents — it means the spawn call itself must be able to block or fail. This is the case that gets missed, and it is the one that matters most, because an orchestrator that keeps fanning out into a saturated pool is generating work that is already doomed. The interaction with fan-out width is direct: under pressure, narrow the fan-out rather than lengthening the queue.

What to measure

Queue depth alone tells you very little. Track time-in-queue at the ninety-fifth percentile, the expiry rate, the rejection rate, and the ratio of time-in-queue to service time.

That last ratio is the one worth putting on a dashboard. When waiting time exceeds working time, the system is over capacity regardless of what utilisation says, and adding buffer will make the experience worse rather than better.

Watch the composition of the queue too. If most of it is retries, the problem is upstream of capacity, and adding workers will simply let you fail faster — the failure taxonomy in agent failure modes is usually the more productive place to look.

A working default

Bound the queue in time units. Attach a deadline to every task and drop expired work at dequeue. Separate classes so interactive traffic has reserved capacity. Shed retries and background work first. Make the spawn path itself subject to backpressure so orchestrators feel the limit. Emit rejections that callers can act on, and log everything you drop.

None of this makes the fleet faster. It makes overload visible and survivable, which is the difference between a system that degrades and one that quietly stops being useful while every metric still looks green.