advanced35 minby DevFox

Caching Across a Model Stack

Prompt caching and its layout rules, semantic caching and its correctness risk, and invalidation in a system where the same input may legitimately produce a different answer tomorrow.

AI
Performance

The docs team retired the old refund policy on Tuesday and published the replacement the same afternoon. On Thursday a customer quotes your assistant back at support, and it is still explaining the retired policy, accurately, in full. You check the corpus: the old document is gone. You check the trace: there is no trace, because the model was never called. The answer came out of a cache that has no idea any of this happened.

Caching is the largest single lever on cost and latency in a model stack, and the only one that can make the system wrong rather than merely slow. By the end of this lesson you will be able to name which of four caches you are actually building, what each one keys on, what each one can and cannot get wrong, and how to write a key that stops serving answers produced by a system you no longer run.

Four caches wearing one word

In a model stack, "cache" names at least four mechanisms that share nothing but the goal. They differ in what the key is, and therefore in what they are capable of getting wrong.

Prefix caching — keyed on leading bytes
Happens at the provider, and saves on the input you resend every time. The model still runs, so it cannot produce a wrong answer — only a more expensive one.
Exact-response caching — keyed on the whole request
Returns a stored answer without calling the model. Correct by construction as long as nothing about the system changed, and it almost never hits.
Semantic caching — keyed on an embedding
Returns a stored answer to a question that is not the one you asked, only a near neighbour of it. Hits far more often. The one that can be wrong.
Downstream caching — keyed on the inputs
Embeddings, retrieval results, reranker scores, tool results. Ordinary caches over ordinary functions, and frequently the biggest win in the stack.

Ordered by how much they can save and how badly they can be wrong — which run in opposite directions

The first and the last are engineering problems. The middle two are product decisions about when a previous answer counts as this answer, dressed up as infrastructure.

Prefix caching is a byte comparison

You already know the layout rule from the intermediate lesson Cost and Latency Engineering: stable content first, volatile content last. This section is what the rule is made of, because the mechanism predicts failures the rule alone does not.

The provider hashes your request from the first byte and matches the longest prefix it has already processed. Three mechanical details follow, and each one is a bug you will otherwise ship.

Matching happens in blocks, not per token. A difference at token 4,001 does not invalidate from token 4,001 — it invalidates from the start of the block containing it, so a single changed word can cost you more cached input than its length suggests, and a changed word near a block boundary and one in the middle of a block have different prices.

Entries have an idle time-to-live, refreshed on each hit and measured in minutes rather than hours. A route with steady traffic keeps its prefix warm for free. A route called eleven times a day pays the write cost — the first request that populates an entry costs more than plain input — and then lets it expire before the next caller arrives. That route is genuinely more expensive with caching than without, and no amount of prompt layout fixes it.

Entries are also evicted under memory pressure, which means a hit rate that was fine on Tuesday can sag on Wednesday because a different team's traffic arrived. Treat the hit rate as a measured quantity that moves on its own, not a property you configured once.

The third detail is the one that catches people: the prefix has to be byte-identical, and a surprising amount of your preamble is assembled by code that does not promise that.

Bad — the tool block is built from a per-user set, so every user gets a private prefix.

Good — one tool block per route, in a fixed order, identical for everyone on it.

flags.enabled_tools returns a set, so its iteration order is not stable, and its contents differ per user. With a thousand users the first version does not have a low hit rate — it has a thousand separate prefixes, each paying its own write cost and each idling out before its owner comes back. The dashboard reads as a caching problem; it is a fan-out problem. If a tool genuinely must be conditional on the user, it goes after the stable block, not inside it.

The same trap hides in JSON serialization that does not sort keys, dictionaries built in a loop, floats formatted by repr, a trailing newline that appears only when a section is non-empty, and a system prompt assembled by string concatenation whose middle section is optional.

Exact-response caching: safe, and quiet

The next cache skips the model entirely. The key is a hash of everything that determines the response: the model identifier, the sampling parameters, the full message list, the tool schemas. If all of that is identical, returning the stored response is defensible without any further argument.

It is also, on free-text traffic, close to useless. Real questions have a long tail, and anything per-request in the prompt — a timestamp, the user's name, a session identifier — makes every key unique on its own. Teams measure a two percent hit rate and conclude that response caching does not work.

It works where the input space is closed or repeated, and those cases are worth more than the hit rate implies:

Retries and duplicate submits. A user double-clicks, a client times out and retries, a queue redelivers. These are the same request within seconds, and they are the requests most likely to arrive while you are already overloaded.
Fan-out in a multi-agent run. Three sub-agents independently deciding they need the same sub-question answered is common, and the deduplication is free.
Evaluation runs. Re-running a suite after changing one prompt should not re-pay for the cases that did not change.
Anything driven by a fixed set of controls — a "summarise this" button, a canned set of starter questions — where the input space is a list you wrote.

One property to state out loud: with a non-zero temperature, the same request has many valid answers, and caching pins one of them forever. For a factual assistant that is a feature. For anything where the variety is the product — draft alternatives, brainstorming, rewrite suggestions — a cache hit turns "give me another one" into "here is the same one", and users read that as the feature being broken.

Semantic caching and the nearly right answer

Semantic caching embeds the incoming question, finds the nearest stored question, and if the similarity clears a threshold, returns that question's answer. Hit rates jump from two percent to something that makes a real difference to the bill, which is why it is tempting, and why it is the most dangerous thing in this lesson.

The reason is a mismatch between what an embedding measures and what you need it to measure. Embedding distance measures topical similarity. You are using it as a proxy for answer equivalence, and the two come apart precisely where it costs you most:

Negation and direction. "Can I downgrade mid-cycle?" and "Can I upgrade mid-cycle?" differ by one token and have opposite answers.
Quantities and bands. "…on the 200-seat plan" and "…on the 20-seat plan" are near-identical vectors and different rows in the pricing table.
Entities. Two region names, two product names, two document titles. Swapping one changes the answer completely and the vector barely.
Time. "How much did I spend this month" is the same question every day and a different answer every day.

An ordinary cache that is wrong gives you a stale value. This one gives you a fluent, confident, well-formatted answer to a question nobody asked, in thirty milliseconds, with no model call to trace and no error to alert on. It is the failure profile of a bug that survives for months.

The first defence is not the threshold. It is refusing to let similarity be the whole key.

Bad — one namespace, so any two questions in the corpus can match each other.

Good — similarity only chooses between candidates that already agree on the facts that decide the answer.

The facets come from the entity extraction your retrieval filters already run, so this is usually plumbing you own rather than new machinery. Without it, "can I downgrade from the 200-seat plan mid-cycle" and the same sentence with "20-seat" sit within a hair of each other, and the first version answers one with the other — billing advice, given confidently, to the wrong customer size.

Only once the namespace is bounded does the threshold become a dial worth tuning, and it is a genuine trade-off with no setting that removes the risk. Lower it and you serve more wrong answers. Raise it and you pay for paraphrases you could have reused. You choose the point by measuring, on a set built for this purpose: paraphrase pairs that must share an answer, and near-miss pairs that must not.

Run it across a sweep and the shape of the decision appears:

There is no free column. At 0.93 you are choosing to answer roughly five percent of near-miss questions with the wrong answer in exchange for reusing two thirds of paraphrases, and the only responsible version of that sentence is one you have said out loud to whoever owns the product.

The caches downstream of the model

While the answer caches get the attention, the pieces the answer is assembled from are usually the better investment. They key exactly, they never invent, and in a retrieval-heavy feature they cover more of the latency than the model call does.

Document embeddings are the easiest cache you will ever write. Key on the content hash plus the embedding model identifier, and the hit rate on a re-ingest is however much of the corpus did not change — typically almost all of it.

Query embeddings key on the normalised query plus the model identifier, and pay off through exactly the repeated traffic that exact-response caching catches.

Retrieval results key on the query, k, the filter set, the corpus version, and the caller's visibility fingerprint. Skip the last two and you have built the bug from the cold open, with an access-control problem stapled to it.

Reranker scores key on the query plus the document identifier plus that document's version. A reranker is a model call wearing a different name, and it is often the single slowest span in the request.

Tool results are where correctness returns, because a tool result is a read of a live system and caching it changes what the agent believes about the world. Sort every tool into three buckets and give each a different rule. Pure reads — a document by version, a historical exchange rate, a static catalogue entry — can be cached for as long as you like. Slow-moving reads, like a plan catalogue or a feature matrix, take a short time-to-live and an explicit refresh path. Live reads — current balance, current usage, order status — are not cacheable in any way the user would forgive, because the reason they asked is that it changes.

Invalidating when the system moves

An ordinary cache goes stale when its data changes. Here, four independent things change what the correct answer is, and only one of them is data.

The documents changed: something was published, edited or retired. The prompt changed: a new instruction, a different output contract, a reordered set of rules. The model changed: a version bump, or a route now pointing at a different one. And time passed, which is enough on its own for any answer about a live quantity.

Miss any of these and you do not have a stale cache. You have a cache serving answers produced by software you no longer run, presented as current, at a latency that makes them look healthier than the ones you compute honestly.

Bad — the key describes the question and nothing about the system that answered it.

Good — the key describes the whole computation, so a change to any input misses.

The first version keeps quoting the retired refund policy for as long as its time-to-live lasts, and it does so at cache speed, so nothing on your latency or error dashboards moves while it happens. The second one misses instead of lying.

That difference — version the key, do not flush the entries — is worth stating on its own. Bumping a version rolls the whole namespace over atomically at deploy time. The old namespace is unreferenced and evaporates on its own time-to- live. Nothing is deleted, so a rollback lands back on a warm namespace instead of a cold one, and you never write the flush-everything script that turns a routine deploy into a thundering herd against your model provider.

Time-sensitivity is the piece a version key cannot express, and it needs a per-class time-to-live rather than a global one. The text of a published policy can sit for hours. A plan catalogue gets minutes. This month's usage figure gets zero. Store generated_at with every entry, and if you ever display a cached answer in a context where freshness matters, surface it — an answer with a visible timestamp is a different, and honest, product to one that implies it was computed just now.

Whose answer is this

Everything above assumed one population of users. The moment you have more than one customer, a cache key is an access control decision, and a bad one is a data breach with an excellent response time.

The rule is short and stricter than it first sounds: the key must include everything that authorised the answer, not only everything that produced it. Two users who send byte-identical questions are not entitled to the same answer if they can see different documents. The retrieval step correctly filtered by permission on the way in; the cache hands that work back if the key does not carry it.

The visibility fingerprint is a hash of the effective permission set the retrieval step would apply. Hashing it, rather than listing it, keeps the key short and means a permission revocation changes the fingerprint and therefore misses — which is the behaviour you want on the day someone loses access to a folder.

Share caches globally only where the input has no tenant component at all: document embeddings for a public corpus, the text of published policy, the query embedding itself. Every cache holding a generated answer is per-principal until you can prove otherwise.

This is also where prefix caching and tenancy meet, and the interaction is counterintuitive. Provider prefix caches are scoped to your account, so they do not leak between your customers — but if you move a tenant's configuration up into the stable block to get it cached, you have built one prefix per tenant, and each of them now has to carry enough traffic on its own to stay warm. Tenant data stays after the shared block. You lose the caching on that fragment and keep it on the large preamble in front, which is the trade that pays.

A cheat sheet to keep

Where to go next

You can now tell the four caches apart, which is most of the work: prefix caching is a byte comparison you protect with stable serialization and measure as a ratio; exact-response caching is safe and pays off on repeats rather than volume; semantic caching trades correctness for hit rate on a dial you must measure rather than guess; and the downstream caches over embeddings, retrieval and read-only tools are usually the largest, dullest, safest win available. Every generated answer carries the system version and the principal in its key, or it is quietly speaking for software you retired.

The natural next lesson is Routing and Model Portfolios, which answers the question this one deliberately did not: when the cache misses, which model gets the request, and how do you know whether choosing between them saved anything. Caching and routing pull on the same budget from opposite ends — one avoids the call, the other prices it.

The thing to go and try is a single deploy. Add the corpus, prompt and model versions to your existing answer cache key, ship it, and watch the hit rate fall to zero and climb back over the next hour. That gap is not a regression. It is the exact volume of answers you were serving, until this morning, from a system that no longer exists.