Batch API Savings: Trading Latency for a Real Discount

Many providers offer a batch mode: submit a set of requests, receive results within some window rather than immediately, pay less. The discount is real and the qualifying criteria are narrower than they first appear.

Why the discount exists

Interactive serving requires holding capacity in reserve. A provider must be able to answer a request now, which means running below full utilisation and accepting idle hardware during quiet periods.

Batch work has no such constraint. It can fill the gaps — run overnight, absorb spare capacity, be scheduled around interactive load. That improved utilisation is what the discount reflects, and it is why the trade is genuine rather than promotional.

It also explains the terms. The provider offers a completion window, not a completion time, because the value to them is precisely the freedom to schedule.

What qualifies

The requirement is that nobody is waiting. That is stricter than "not urgent".

Genuinely qualifying work includes bulk classification and extraction over a corpus, generating embeddings for a large index, offline evaluation runs, periodic summarisation of accumulated data, and backfilling a feature over historical records.

Work that does not qualify includes anything in a request path, agent loops where each step depends on the previous result, and interactive tools. An agent cannot use batch mode at all — the loop is inherently sequential, and a completion window measured in hours per step makes it meaningless.

The borderline case worth thinking about is scheduled work with a deadline. A nightly job that must finish before the working day starts can use batch mode only if the window comfortably fits inside the available time, with room for the provider to use the whole window.

Working the trade

Take a classification workload: 500,000 documents, roughly 800 input tokens each and 50 output tokens, on DeepSeek V4 Pro at around $0.44 input and $0.87 output.

Synchronously, that is 400 million input tokens and 25 million output — about $176 and $22, so roughly $198. A batch discount in the region of half would take it to around $99.

Whether saving $99 justifies building batch submission, polling, result reconciliation and failure handling depends entirely on whether this runs once or monthly. Once, almost certainly not. Monthly, comfortably yes.

That is the actual decision, and it is an engineering-cost question rather than a token-cost question. Compute your own figure and compare it to a realistic estimate of the implementation work.

The engineering is not trivial

Batch mode changes the shape of your code in ways worth planning for.

Requests are submitted as a set and identified by a key you supply, so you need reconciliation logic to match results back to inputs. Results arrive asynchronously, so you need somewhere to put them and something to poll or receive a callback. Partial failure is normal — some requests in a batch succeed and others do not — so you need per-item error handling rather than per-request.

None of this is hard. All of it is more than a configuration flag, and teams that estimate the switch as an afternoon are usually surprised.

Combine with caching carefully

Batch and caching interact in a way that is easy to get wrong.

If every item in your batch shares a long system prompt, that prefix is a strong caching candidate — but cache behaviour in batch mode varies by provider and is not always the same as the interactive path. Verify rather than assume, because the two discounts compounding is a substantially different figure from either alone.

Order matters too. Submitting items that share a prefix contiguously gives a cache a better chance than interleaving unrelated work. Prompt caching savings math covers the prefix rules.

The decision rule

Use batch mode when the work is genuinely offline, recurring, and large enough that the saving exceeds the engineering cost with margin.

Skip it when the volume is small, the work is one-off, or anything downstream is waiting. And never plan an agent workload around it — the sequential dependency makes it structurally impossible, regardless of how patient the user is. Batching requests to save money covers the simpler in-request batching that does work for sequential flows.

Measure the window you actually get

Providers quote a maximum completion window, and actual turnaround is usually well inside it. That gap is tempting to plan around and dangerous to depend on.

The whole reason the discount exists is that the provider may use the full window when capacity is tight — which correlates with exactly the busy periods when your own deadline is most likely to matter. A batch job that normally returns in twenty minutes and occasionally takes the full window will fail in the worst way: rarely, unpredictably, and under load.

So design against the quoted maximum, not the observed median. If your deadline cannot absorb the full window, the workload does not qualify, however good the typical turnaround looks in testing.

Instrument actual turnaround anyway. A shift in the distribution is useful early warning that the provider's capacity situation has changed, and it is the kind of thing that moves quietly.

Keep a synchronous fallback

Batch submission can fail, and a job that must complete needs a path that does not depend on it. Keeping the synchronous code path working — even if it is only used for retries and small volumes — means a batch outage degrades cost rather than availability.

This also makes the migration safer to attempt, since you can move a fraction of the volume to batch and compare, rather than switching wholesale and discovering the reconciliation logic has a bug on real data.