How should I split input into shards for agents?

Along semantic boundaries — per file, module, endpoint or incident — rather than by token count. Coherent shards produce answers that merge cleanly; arbitrary cuts split a unit of meaning across two contexts.

Why does the reduce step lose information?

Because merging prose summaries is itself a lossy summarisation. Have the map step emit structured records so deduplication, grouping and counting happen in code, and reserve the model for reconciliation and narrative.

Is map-reduce cheaper than a single long-context call?

Not usually. It multiplies the standing context across every shard. It buys parallel wall-clock time and lets you use a cheap model on the map step, but if the input fits in one context, one call is generally cheaper and more accurate.

Map-Reduce With Agents: Splitting Work That Actually Splits

You have four hundred files to audit, or two thousand log lines to classify, or a monorepo to summarise. The obvious move is map-reduce: split the input, run an agent over each shard, merge the results.

The map step usually works. The reduce step is where people get burned, because reducing LLM output is not like reducing numbers. Two shard summaries do not add. They overlap, contradict, use different vocabulary for the same thing, and omit whatever the shard boundary cut in half.

Map-reduce with agents is worth doing. It is just that the interesting engineering is all in the reduce, and most implementations spend all their effort on the map.

When the shape fits

The precondition is independence. A shard must be answerable without knowing what is in the other shards. Audit each file for hardcoded secrets: independent. Determine whether this codebase has a consistent error-handling strategy: not independent, because the answer is a property of the whole.

The second precondition is that the reduce is cheaper than the map. If merging four hundred summaries requires reading all four hundred in one context, you have not reduced anything, you have just moved the context problem downstream and paid for the shards as well.

The third is that shard-level errors are tolerable. Map-reduce over agents will have shards that fail, time out, or return something malformed. If a single bad shard invalidates the whole answer, you want a design with tighter guarantees, and you should read partial failure recovery before building it.

Sharding is a design decision, not a chunk size

Splitting by token count is the default and it is usually wrong. It cuts functions in half, separates a class from its interface, and puts a test in a different shard from the code it tests.

Split along the seams the work actually has: per file, per module, per endpoint, per incident, per customer. Semantic boundaries mean each shard is a coherent unit, and coherent units produce answers that merge without contradiction.

Where boundaries must cut through something, overlap deliberately. Give each shard a little of its neighbours, and accept that you will see duplicate findings — deduplicating is far easier than recovering a finding that fell in the gap.

And send each shard the same standing context: the objective, the conventions, the definitions. Shards that use different vocabulary for the same phenomenon are the single biggest source of reduce-step pain, and consistent framing is nearly free.

Designing the map output

The map step should emit records, not prose. One finding per record, with fixed fields: what, where, severity, evidence, confidence.

Records reduce mechanically. You can group by category, sort by severity, count occurrences and deduplicate on location without a model in the loop at all. Prose summaries force you to spend a second reasoning pass just to work out what the first pass said. The general case for this is in structured outputs and JSON mode.

Include a field for nothing found. Models under-report negatives, and a shard that returns an empty array is meaningfully different from a shard that failed. Without that distinction your coverage numbers are fiction.

Have every record carry its shard identifier. When the final report says something surprising, you want to jump straight to the shard that produced it, and that trace is the same discipline that makes agent tracing useful.

The reduce step

Do as much of the reduce in code as you possibly can. Deduplication, grouping, counting and ranking are deterministic operations, and a model doing them by hand will drop items and invent totals.

Use a model only for the part that genuinely needs judgement: reconciling records that disagree, and writing the narrative over an already-grouped set. Feeding it a structured, deduplicated set of forty groups instead of four hundred raw records is what keeps the reduce affordable.

When the reduced set is still too large for one context, reduce in a tree: merge shards in small batches, then merge the merges. Each level compresses again, so keep the tree shallow and make sure counts and identifiers survive every level — those are exactly what a summarising model discards first.

Be explicit about conflicts. Two shards disagreeing is a finding, not noise to be smoothed over, and it usually means the instructions were ambiguous rather than that one shard was wrong.

Cost and concurrency

Map-reduce multiplies token spend by the number of shards, plus the standing context you repeat in every one. That repeated preamble is often a third of the total bill, which makes it the obvious target for prompt caching if your provider supports it on a stable prefix.

It also multiplies request volume, which is how people discover their rate limits. Four hundred concurrent shards is not four hundred times faster; it is four hundred requests hitting a per-minute ceiling and retrying into each other. Run a bounded worker pool, and see agent concurrency control for the mechanics.

Pick the model per step. Mapping is usually mechanical extraction that a cheap model handles well, while reducing is judgement work worth a stronger one. Splitting the tiers this way is one of the few savings that does not degrade the result.

A worked shape

Auditing a repository for unsafe SQL construction: shard per source file, not per token window. Map with a cheap model emitting one record per suspicious call site, with the file, line, snippet and a confidence value. Deduplicate in code on file and line. Group by directory. Reduce with a strong model over the grouped set, asking specifically which groups represent a real pattern rather than isolated legacy code.

The result is auditable end to end: every line in the report traces to a record, every record traces to a shard, and re-running a single shard is cheap when someone disputes a finding.

When not to use it

Do not use map-reduce for questions about the whole. Architecture coherence, cross-cutting design decisions and anything requiring the reader to hold two distant files at once are not shardable, and a long-context single pass will beat a sharded one.

Do not use it when the reduce needs everything anyway. And do not use it for small inputs — if the whole thing fits in one context with room to reason, one call is cheaper, simpler and more accurate than any fan-out you could build.