Agent Caching Strategies: Prompt, Tool and Result Layers

An agent is a machine for doing the same work repeatedly. It resends the entire conversation on every turn because the model is stateless. It reads the same file in turn nine that it read in turn three. It runs the same test command four times against an unchanged tree.

Each of those repetitions is a separate caching opportunity, and they behave differently enough that treating them as one problem produces a system that helps less than it should. There are three layers worth building, in descending order of return on effort.

Layer one: prompt caching

This is the highest-value layer for almost every agent, because it targets the cost that grows fastest. On turn twenty you resend everything from turns one through nineteen, so a long session pays for the early transcript twenty times over.

Provider-side prompt caching removes most of that. The prefix of the request that has not changed is served from the provider's cache at a substantially reduced input rate — Kimi K3, for example, is priced at $3 per million input tokens against $0.30 per million for cached input, an order of magnitude difference on the part of the prompt that repeats.

The mechanic that matters is that caching works on prefixes. Anything that changes near the front of the prompt invalidates everything after it. So a timestamp in the system prompt, a randomly ordered tool list, or a per-request user detail placed above the static instructions will silently cost you the entire benefit.

Structure the prompt in strict order of volatility: system instructions, then tool definitions, then any large static context, then the conversation, then the current turn. Append only. Prompt caching explained covers the mechanism and the break-even math covers when the write cost is worth paying.

Layer two: tool result caching

Inside a single session, agents re-read files constantly. They read a file, do other work, lose track of what they saw, and read it again. Each re-read is tokens in and tokens out and a fresh copy in the transcript.

A session-scoped tool cache keyed on the normalised call — tool name plus canonicalised arguments — eliminates most of this. Canonicalisation matters: resolve relative paths, sort object keys, normalise whitespace, or you will miss hits that should have landed.

The hard part is invalidation, and the rule is that any write invalidates reads that could overlap it. A file write invalidates the cached read of that path. A command that could touch anything — a build, an install, a migration — invalidates the whole cache unless you can prove otherwise. Being conservative here is correct; serving a stale file read to an agent produces edits against content that no longer exists, and the resulting failure is genuinely hard to diagnose.

Mark tools as cacheable explicitly rather than inferring. A search is cacheable, a random-number tool is not, and a tool that talks to a live API is cacheable only for a short window if at all.

Layer three: cross-session caching

This is the layer with the best theoretical return and the worst practical risk. If ten sessions today all read the same configuration file, caching it across sessions saves nine reads.

It is also where staleness becomes dangerous, because the invalidating event happens outside the session that holds the cache. Someone merges a branch, the file changes, and your cached copy is now describing a codebase that no longer exists.

Key on content, not on identity. Use a commit hash, a file digest or an ETag as part of the key so a changed artefact simply misses rather than serving stale data. If you cannot key on content, use a short time-to-live measured in minutes and accept the reduced hit rate.

Restrict this layer to genuinely stable material — dependency documentation, language references, a frozen schema — and leave working files to session scope. The clean signal is whether anything outside your control can change the artefact without telling you. Caching strategies to cut cost covers the general patterns.

Semantic caching, and why to be careful

Embedding a request and serving the answer to a sufficiently similar earlier one is attractive and mostly a bad fit for agents. Agent tool calls are precise: reading line 40 to 80 of a file is not approximately the same as reading line 45 to 85, and a similarity threshold cannot tell you that reliably.

Where it does work is on natural-language front doors — a support surface where many users ask the same question in different words, and where a slightly generic answer is acceptable. Even then, set the threshold high and log every hit so you can audit what was served.

Do not use it for anything where the answer must be exactly right for the arguments given. The failure mode is not an error; it is a plausible answer to a question nobody asked, which is the worst kind of bug to find in a trace.

Measuring whether any of it works

Hit rate alone is a vanity metric. What you want is tokens saved and seconds saved per session, because a ninety percent hit rate on cheap calls is worth less than a thirty percent hit rate on the expensive ones.

Instrument three numbers: cached input tokens as a share of total input, tool calls served from cache as a share of total, and the cost per completed task before and after. The third is the only one that settles arguments. Agent token budgets covers wiring the accounting.

Watch for hit-rate cliffs. A prompt-cache hit rate that drops from eighty to twenty percent overnight almost always means something started varying near the front of the prompt — a new header, a reordered tool list, a version string. That regression is invisible in output quality and very visible in the invoice.

What to build, in order

Start with prompt caching, because it is mostly a matter of ordering your prompt correctly and it targets the dominant cost. Confirm the hit rate before doing anything else; teams frequently discover they were invalidating their own cache on every request.

Add session-scoped tool result caching next, with conservative invalidation on any write. Measure the token saving before considering the third layer.

Add cross-session caching only for artefacts you can key by content, and only after the first two are stable. Skip semantic caching unless you have a natural-language surface with genuine repetition. If you are also exploring speculative execution, route it through the same tool cache so a speculation and a real call cannot both run.