- Provider prefix caching reuses the model's internal key-value state for a stable leading prefix, so a cache hit skips prefill for those tokens: time-to-first-token drops and the cached input bills at a steep discount (the exact figure, TTL and minimum threshold vary by provider, so treat any number as illustrative).
- It only works if the prefix is byte-stable and ordered static-first (system prompt, tools, examples, stable context) with dynamic and per-user content last; a changing timestamp or a reordered tool block near the top silently busts the cache.
- Semantic response caching is a different thing entirely: it returns a whole stored answer for a semantically similar query via embedding lookup, which is a large win on repetitive FAQ traffic but carries a real risk of stale or wrong hits, so it needs a similarity threshold and a freshness policy.
The cheapest token is one you never recompute
Most production LLM traffic is far more repetitive than it looks. The same long system prompt goes out on every request. The same few-shot examples, the same tool definitions, the same policy text and the same retrieved context ride along in front of a user question that is often the only part that changes. On the naive path the model processes that entire preamble from scratch every single time, paying for tokens it has already read a thousand times that hour. That is pure waste, and at enterprise volume it is expensive waste.
Caching attacks the waste directly. The premise is simple: work you have already done for identical input does not need to be done again. Reuse it, and you cut both the bill and the wait. The prize has two faces. The cost face is fewer billed input tokens, because the reused portion is charged at a discount or not recomputed at all. The latency face is a faster first token, because the model skips the processing it would otherwise repeat before it can start generating. For interactive workloads the latency win is often the one users actually feel.
The trap is the word itself. Two genuinely different techniques both get called caching, and conflating them is one of the more common mistakes I see in architecture reviews. One is provider prefix caching, which reuses the model's internal compute for a stable prompt prefix. The other is application semantic caching, which reuses a whole finished answer for a similar-looking query. They live at different layers, they are owned by different parties, and they fail in completely different ways. The rest of this deep-dive treats them as the separate tools they are.
Two kinds of caching, often confused
The clearest way to separate the two is to ask three questions of each: what exactly is being reused, who owns and operates it, and what happens when it goes wrong. The answers pull the two techniques apart cleanly.
Provider prefix caching reuses low-level model state, the key-value tensors computed during prefill, for a run of leading tokens that are byte-for-byte identical to a previous request. It lives inside the inference provider or your own serving stack, below the API line. Its failure mode is benign: if the prefix does not match, you simply get a cache miss and pay the normal price. The answer is never affected, only the cost and the latency.
Semantic response caching reuses the finished output, the whole generated answer, for an incoming query that is judged similar enough to one you have served before, usually by comparing embeddings. It lives in your application, above the model. Its failure mode is dangerous: a false match returns a stored answer that does not actually fit the new question, so a bad hit produces a wrong or stale response rather than just a missed saving.
| Dimension | Provider prefix caching | Application semantic caching |
|---|---|---|
| What is reused | Model KV state for a stable prefix | The whole generated response |
| Match condition | Exact, byte-identical leading tokens | Approximate, embedding similarity |
| Who owns it | Provider or serving stack, below the API | Your application, above the model |
| What it saves | Prefill compute, input cost, first-token latency | An entire model call, end to end |
| Failure mode | Cache miss: pay full price, answer unchanged | False hit: stale or wrong answer returned |
Both are worth having, and mature platforms run both. But they are not substitutes and they are not interchangeable in a risk conversation. Prefix caching is close to free to adopt because the downside is only a missed discount. Semantic caching earns larger savings and demands real governance because the downside is a bad answer shipped to a user. Keep them in separate columns of your design.
How prefix caching works
To see why prefix caching works you have to remember what the model does before it generates anything. During prefill it reads the entire prompt and computes, for every token and every layer, a set of key and value tensors, the same KV cache covered in the inference optimization deep-dive. That prefill pass is a large share of the cost and almost all of the delay before the first output token appears. Prefix caching is the observation that if the leading tokens of a new request are identical to a previous one, their KV state is identical too, so it can be stored and reused rather than recomputed.
On a cache hit the provider loads the saved KV state for the matching prefix and begins real work only at the first token that differs. The prefill for the shared portion is skipped entirely. Two things follow directly: time-to-first-token falls, because the expensive read of the preamble no longer happens, and the input bill for those tokens drops, because cached tokens are charged at a reduced rate rather than the full input price.
Request A (cold): [ system + tools + examples ][ user Q1 ]
|------ prefill, full price -----| generate
Request B (warm): [ system + tools + examples ][ user Q2 ]
|==== cache hit, KV reused ====| generate
^ same bytes as A, prefill skipped
^ work starts here
Two constraints define the mechanism and both vary by provider, so treat the specifics as illustrative rather than fixed. First, caches carry a short time-to-live: the stored state is evicted after a period of inactivity measured in minutes, not hours, so the payoff comes from requests that cluster in time. Second, there is a minimum token threshold below which caching does not engage, because managing the cache is not worth it for a trivially short prefix. The practical reading is that prefix caching rewards long, stable, frequently reused preambles hit in bursts, and does little for short one-off prompts.
The economics of a cache hit
The economics are governed by one ratio: how many of your input tokens fall inside the reused prefix versus how many are fresh on each request. Cached input tokens bill at a steep discount to the normal input price. The exact discount varies by provider and changes over time, so treat any figure here as illustrative, but as a shape imagine cached tokens costing something like a tenth of the uncached input rate. Whatever the precise number, the direction is fixed: the more of your prompt that is stable and reused, the lower the effective input cost per request trends.
That is why the technique pays off most for exactly the prompts that look most wasteful. A long shared system prompt, a fat few-shot block, a large reused chunk of retrieved context, all of that is stable preamble that would otherwise be re-billed at full price on every call. Put those in the cacheable prefix and their cost collapses on every request after the first. A short prompt with a tiny stable header and a large variable body has little to reuse and sees little benefit. The lever scales with the size of the shared prefix relative to the whole.
Prompt = 4000 stable prefix tokens + 200 variable tokens Request 1 (cold): 4000 @ full + 200 @ full Request 2 (warm): 4000 @ ~1/10th + 200 @ full Request 3 (warm): 4000 @ ~1/10th + 200 @ full ... Effective input cost per request falls toward the variable tail as the warm streak lengthens. (discount rate illustrative; varies by provider)
Two caveats keep the model honest. The first request in a streak is a cold miss and pays full price to populate the cache, and some providers add a small write premium for that, so caching is a win over a run of requests, not on a single isolated call. And because the cache expires on a short TTL, sparse traffic that never clusters may keep paying cold prices. Model the saving against your real arrival pattern, not against a best case, and fold the result into the wider LLM economics picture.
Designing a prompt that actually caches
Prefix caching is not automatic in the sense that matters. The provider will cache for you, but only if you hand it a prefix worth caching, and that is an application design decision. The single rule that governs everything is ordering: put the static content first and the dynamic content last. System instructions, tool and function definitions, few-shot examples and stable background context belong at the top, in a fixed order. The user's message, the current timestamp, session identifiers and any per-request retrieval belong at the bottom.
The reason is that the cache matches on a leading run of byte-identical tokens and stops at the first difference. Everything before the first changed byte can be reused; everything from that byte onward must be recomputed. So a single variable character high in the prompt throws away the entire cacheable region below it. The discipline is to keep the prefix byte-stable: same text, same order, same serialization, request after request. Even cosmetic churn counts, because the match is on bytes, not meaning.
- A timestamp, request ID or "current date" injected near the top of the system prompt: it changes every call and invalidates everything after it.
- Tool or function definitions serialized in a non-deterministic order, so the same tools produce different bytes run to run.
- Per-user text (name, tenant, greeting) placed before the shared instructions instead of after them.
- Whitespace, key ordering or JSON formatting that drifts because the prompt is rebuilt by string concatenation rather than a stable template.
- A/B prompt tweaks or version strings edited mid-flight, resetting every warm cache to cold.
The build-time habit that prevents all of this is to assemble the prompt from a fixed template with a hard boundary: a frozen prefix block that is identical for every request in a class, then a clearly separated dynamic tail. Treat the prefix like a compiled artifact, changed deliberately and versioned, not edited casually. When you do need to update it, expect a cold window while the new prefix warms, and roll it out knowingly. This is the same layering discipline the context engineering playbook applies to what goes in the window in the first place.
Semantic response caching
Semantic caching operates at a completely different altitude. Instead of reusing model compute, it reuses a finished answer. When a query arrives, the application embeds it, searches a store of previously answered queries for a semantically close match, and if one is close enough returns the stored response without calling the model at all. The saving is total when it hits: no prefill, no generation, no token bill, just a lookup. For repetitive FAQ-style traffic where thousands of users ask the same handful of questions in slightly different words, that is a large and real win on both cost and latency.
The danger is equally real and lives entirely in the word "close enough." Embedding similarity measures surface resemblance, not equivalence of intent, and two questions can sit near each other in vector space while demanding different answers. "What is our refund policy?" and "What is our refund policy for enterprise contracts?" are neighbors that must not share a cached response. Set the similarity threshold too loose and you serve confident, wrong answers. Set it too tight and the hit rate collapses and the cache stops paying for itself. There is no universally correct threshold; it has to be tuned and monitored against real traffic.
The other hazard is staleness. A cached answer is a snapshot, and if the underlying facts change, a policy, a price, an inventory count, the cache keeps serving the old truth until something evicts it. So the safety of semantic caching is a property of the workload, not of the technique:
- Safe: high-volume, repetitive questions over stable, non-personalized knowledge, where an occasional near-miss is low-cost and answers change slowly (product FAQs, documentation lookups, definitional queries).
- Unsafe without strong guards: anything personalized to a user or account, anything transactional or fast-changing, anything where a wrong answer carries legal, financial or safety weight. Here a false hit is a real incident, not a rounding error.
When you do deploy it, deploy it with instrumentation: a tuned and monitored threshold, an explicit TTL and event-driven invalidation tied to the source of truth, and sampled auditing of served hits against fresh model answers. Semantic caching is worth having, but it is an application feature you own and must govern, not a free switch you flip.
The architect view
The mistake I most want CTOs to avoid is treating caching as a micro-optimization someone tunes at the end. It is a first-class cost and latency lever, and it belongs in the architecture from the start, because the shape of your prompt and the shape of your platform decide whether hits are the default or the exception. A prompt laid out static-first caches; the same prompt laid out carelessly does not, at identical functionality and identical model. That difference is free money left on the table by an ordering choice.
Make it measurable. Cache hit rate is the metric that turns caching from a hope into a managed number, and it should sit on the same dashboard as cost per request and time-to-first-token. A hit rate that is lower than your traffic's repetition suggests is a signal that something is busting the prefix, and it is usually one of the mistakes above. Watch it, alert on regressions, and treat a sudden drop the way you would treat a latency spike.
Caching also does not stand alone. It compounds with the other levers in this cluster. Route cheap, repetitive traffic to smaller models via model routing, serve them efficiently using the techniques in inference optimization, and let prefix caching drive the effective input cost of the shared preamble toward zero on top of both. The platform's job is to make all three the paved default so application teams get them without having to be experts.