How much does prompt caching actually save?

For a typical agent loop with a 12,000-token stable prefix and short variable content, roughly a sixth of the uncached cost at Kimi K3 rates. The saving scales with how much of your input repeats.

Why is my cache hit rate lower than expected?

Usually a variable element near the top of the prompt — an injected timestamp or user identifier — which invalidates everything after it. Caching matches from the start forward and stops at the first difference.

Which workloads benefit least from caching?

One-shot processing of unrelated items, where nothing repeats beyond the instructions. Bursty traffic also loses out, because caches expire after minutes of inactivity and most requests then miss.

Prompt Caching Savings: The Arithmetic Nobody Runs

Providers price cached input dramatically below uncached input — Moonshot lists Kimi K3 at $3 per million input tokens against $0.30 cached, an order of magnitude. That discount is large enough to restructure a workload around, and most teams never calculate what it is worth to them.

What caching actually does

Processing a prompt means computing keys and values for every token, which is the expensive prefill phase. If two requests share a prefix, that work can be done once and reused.

The unit of reuse is a prefix, and this is the detail everything else depends on. Caching matches from the beginning of the prompt forward. It stops at the first difference. Anything after that point is recomputed.

So a prompt where the variable content sits at the top gets no benefit regardless of how much stable content follows. KV cache explained covers the mechanism.

The calculation

Take one representative request and split its input into two parts: the stable prefix that repeats across requests, and the variable remainder.

Uncached cost is the full input at the standard rate. Cached cost is the prefix at the cached rate plus the remainder at the standard rate.

Work a concrete case. A coding agent with a 12,000-token system prompt and tool definitions, plus around 800 tokens of varying user content, at K3 rates. Uncached, that is 12,800 tokens at $3 per million, roughly $0.038 per call. Cached, it is 12,000 at $0.30 plus 800 at $3 — about $0.006. Roughly a sixth of the cost.

Across a hundred thousand calls a month that is the difference between about $3,800 and about $600. The prompt did not change; only where the stable part sat.

Where the savings concentrate

Three workload shapes benefit most, and they are common.

Agent loops. Every turn resends the entire conversation. The prefix grows and is identical each time, so the cached fraction is enormous. This is the single biggest case.

Chat with a long system prompt. Same structure — stable instructions, short changing message.

Document question-answering. If several questions are asked against one document, the document is a cacheable prefix. Asking ten questions about a 200K-token document with caching costs a fraction of asking them without.

The workload that benefits least is one-shot processing of unrelated items, where nothing repeats. Batch classification of independent records has no shared prefix beyond the instructions.

The structural rule

Stable content first, variable content last. That single rule captures most of the available benefit, and violating it is common.

The usual offenders: a timestamp injected at the top of the system prompt, which invalidates the cache on every single request. User identifiers placed before instructions. Retrieved documents inserted above the tool definitions rather than below. Randomly ordered few-shot examples.

Each of these is a small piece of code that quietly costs six times the necessary spend. They are worth hunting for specifically, because none of them looks like a bug.

Cache lifetime is the other variable

Caches expire, typically after minutes of inactivity, and providers differ. That makes traffic pattern part of the calculation.

Steady traffic keeps caches warm and realises close to the theoretical saving. Bursty traffic with long gaps sees most requests miss. A workload of one request every twenty minutes may get almost nothing from caching regardless of prompt structure.

Some providers offer longer-lived caches at a higher write cost. Whether that is worth it depends on request spacing, which is a measurement rather than a guess — instrument your cache hit rate before paying for extended retention.

Measure the hit rate, not the theory

Most providers report cached token counts in the response. Log them and compute your actual cached fraction.

The number is frequently lower than expected, and the reasons are usually mundane: a variable element near the top, prompt versions differing between code paths, or traffic too sparse to keep caches warm. Each is fixable once visible.

Treat a low hit rate as a bug rather than a fact about your workload. Reducing token usage covers the complementary work of not sending the tokens at all, which is the only thing cheaper than caching them.

Caching changes what a long system prompt is worth

Without caching, a long system prompt is a tax on every request, and the standard advice is to trim it aggressively. With caching, that advice partly inverts.

A 12,000-token system prompt costing $0.30 per million rather than $3 is cheap enough that adding a few thousand tokens of genuinely useful context — coding conventions, an architectural map, worked examples — becomes economical where it previously was not. The marginal cost of stable context falls by an order of magnitude.

This is worth exploiting rather than ignoring. Teams that trimmed their prompts hard under uncached pricing are frequently leaving quality on the table now, because the constraint that justified the trimming no longer binds the same way.

The discipline that still applies is that the added content must be stable. Anything varying between requests belongs after the cacheable prefix, where it costs full rate and should still be minimal.