Agent Concurrency Control: Pools, Locks and Fair Slots

The first version spawns a worker per subtask and awaits them all. It is three lines of code and it works beautifully at four subtasks.

At sixty it produces a burst of rejected requests, a wave of retries that collide with the workers still running, a run that takes longer than the sequential version, and a bill that includes every failed attempt. Nothing crashed. The system simply had no idea how much work it was allowed to do at once.

Concurrency control is the part of agent engineering that looks like ordinary backend work, and it is, right up until the moment you notice that agents contend for things ordinary jobs do not.

What agents contend for

Provider capacity. Requests per minute and tokens per minute, usually enforced per key or per organisation, so every concurrent agent draws from one pool. This is the constraint most fleets hit first.

Tool and API capacity. The search index, the database, the internal service the agent calls. These often have far lower limits than the model provider and far less forgiving behaviour when you exceed them.

Mutable resources. A working directory, a branch, a staging environment, a record being updated. Two agents writing the same file is not a rate problem, it is a correctness problem, and no amount of retrying fixes it.

Your own attention. Twenty agents running concurrently produce twenty interleaved traces. If you cannot follow one run through the logs, you have exceeded the concurrency your observability supports, which is a real limit even though nothing rejects anything.

The worker pool

The baseline is a bounded pool: N slots, tasks queue for one, a task runs when a slot frees. Simple, predictable, and it makes the concurrency level an explicit number you can tune rather than an emergent property of the task list.

Size the pool against the tightest downstream limit, not against the number of subtasks. If the search API allows ten requests per second and each agent step makes one call, your ceiling comes from there, whatever the model provider allows.

The queue in front of the pool needs a depth limit too. An unbounded queue converts a load problem into a memory problem and a very long tail of tasks that will be stale by the time they run. Agent queueing and backpressure covers what to do when the queue fills.

Expose the pool's state: slots busy, queue depth, oldest waiting task. These three numbers tell you immediately whether a slow run is slow because agents are slow or because they are waiting.

Weighted slots, because agent tasks are not uniform

A pool with equal slots assumes every task costs the same. Agent tasks vary by an order of magnitude — one worker makes three tool calls and returns, another grinds through forty.

Weighting by expected cost works better than counting tasks. Give the pool a token budget per interval rather than a slot count, and admit a task when its estimated spend fits. Estimates can be crude; even a two-tier cheap-and-expensive split beats treating all tasks alike.

This also gives you a natural place to enforce spend ceilings, since the admission check is already looking at cost. The wider framing is in agent token budgets.

Locks for mutable resources

Rate limiting protects a shared service from load. Locking protects shared state from corruption, and the two are not substitutes.

Whatever an agent can mutate needs a single owner at a time: a branch, a directory, a customer record. Acquire before the work, release after, and give the lock a timeout well above the slowest legitimate run — an agent that dies mid-task must not hold a resource forever.

Prefer partitioning to locking where you can. Give each agent its own worktree or its own record range and contention disappears rather than being managed. The same instinct that makes namespaced shared memory between agents workable applies here.

Where locks are unavoidable, always acquire in a fixed global order. Two agents each holding one of two resources and waiting for the other is a deadlock, and an agent that decides which resource to take second based on what it read is very good at producing one.

Fairness and starvation

A shared pool serving several workloads will starve someone. A batch job that queues four hundred tasks fills every slot, and the interactive request behind it waits behind all of them.

Separate queues per class with reserved capacity is the usual answer: interactive work gets guaranteed slots, batch work gets the rest. It costs some utilisation and buys predictable latency for the traffic where latency is visible.

Long-running agents make this worse, because a slot held for twenty minutes is a slot nobody else gets. Cap the wall-clock time per task and requeue anything that exceeds it, so a stuck agent cannot hold capacity indefinitely — a concern that shows up throughout long-running agent design.

Retries interact badly with pools

A retry inside a task holds its slot while it waits. Under load, most slots end up occupied by agents sleeping through a backoff, and throughput collapses while utilisation looks like a hundred percent.

Release the slot before a long backoff and re-acquire after, or move retries out of the pool entirely onto a delay queue. Either way, add jitter — synchronised retries from a burst of workers reproduce the exact spike that caused the rejections, and the cost of those wasted attempts is real, as the hidden cost of retries lays out.

Cap total attempts across the run, not just per request. Ten workers each retrying five times is fifty extra calls, and a fleet-level ceiling is the only thing that bounds it.

Where to start

Put every agent spawn behind a bounded pool from the first version, even if the bound is generous. Retrofitting concurrency control after a fleet exists means finding every spawn site, and there are always more than you remember.

Then size the pool from the tightest downstream limit, add a queue depth cap, lock anything mutable or partition it away, and give every task a wall-clock ceiling. Those five things cover almost every concurrency incident an agent fleet will actually produce.