Most production LLM applications resend the same enormous block of text on every single request: the system prompt, the tool definitions, the few-shot examples, the retrieved reference documents. Only a small tail of that request — the user’s actual message — changes call to call. Prompt caching exists because reprocessing that stable, repeated block from scratch on every request is pure waste, and it is one of the few optimizations available to application developers that requires no model changes and no infrastructure of your own, just discipline about how you structure prompts.

What Actually Gets Cached

It helps to be precise about what “prompt caching” means at the mechanical level, because the marketing framing (“cache your prompt”) undersells what’s really happening. A large language model processes an input prompt by running it through the network and, for every token, computing key and value vectors used by attention — the same key/value pairs that are stored in the KV cache during generation. Processing a long prompt for the first time (the “prefill” phase) is exactly this: computing and storing KV cache entries for every prompt token before generation even starts.

The insight behind prompt caching is that if two requests share an identical prefix — the exact same sequence of tokens, in the exact same order, with the exact same tokenization — the KV cache entries for that shared prefix are also identical, because they’re a deterministic function of the model’s weights and the input tokens up to that point. There is no need to recompute them; they can be computed once and reused by every subsequent request that starts with that same prefix.

Request 1: [System prompt + tools][Few-shot examples][User message A]
                    ↑ computed and cached
Request 2: [System prompt + tools][Few-shot examples][User message B]
                    ↑ cache hit, reused          ↑ only this part is freshly processed

This is why prompt caching is sometimes called prefix caching in serving engine documentation — the term describes the mechanism more accurately than “prompt caching” does. What’s cached is a shared prefix, not “the prompt” as a whole, and the moment two requests diverge at some token position, everything after that divergence point has to be recomputed regardless of how much they still have in common further along.

What Makes a Prompt Cacheable

Given that mechanism, cacheability comes down to one property: does this request share an exact, ordered token prefix with other requests you’re making? A few consequences follow directly from that:

  • Order matters more than content. Two prompts with the same components in a different order do not share a prefix, even if every individual piece is identical. [system][tools][user_msg] and [tools][system][user_msg] are two entirely different token sequences from the cache’s point of view.
  • Stability matters more than size. A short prompt that’s byte-identical across a thousand requests caches perfectly. A huge prompt that has even one token of variation near the beginning caches nothing for anything after that variation.
  • The variable part belongs at the end. Anything that changes per-request — the user’s message, a timestamp, a session ID injected into the system prompt — should be placed after all the stable content, never interleaved with it or placed before it.
  • Tokenization must match exactly, not just the rendered text. Whitespace differences, differing JSON key ordering in a serialized tool schema, or a templating engine that inserts variable indentation can all silently change the token sequence even when the text looks identical to a human reading it.

Structuring Prompts for Cache Hits

The practical implication is that prompt construction should treat “stable” and “variable” content as genuinely different categories, laid out in a fixed order:

def build_prompt(system_prompt, tool_definitions, few_shot_examples, user_message, session_context):
    # Stable, identical across many requests — put first, in a fixed order.
    stable_prefix = [
        {"role": "system", "content": system_prompt},
        {"role": "system", "content": render_tools(tool_definitions)},
        *few_shot_examples,  # fixed set, fixed order
    ]

    # Variable, unique per request — always last, never interleaved above.
    variable_suffix = [
        {"role": "user", "content": user_message},
    ]

    return stable_prefix + variable_suffix

A subtler mistake is burying something variable inside what looks like a stable block — for example, a system prompt template that includes “Current date: {today}” at the top. That single interpolated value invalidates the cache for the entire block that follows it, every single day, even though 99% of the block’s content never changed. If you need per-request or per-day context like a timestamp, push it down into the variable suffix rather than the stable prefix, even if it feels more natural to describe the model’s context at the top of the system prompt.

# Costly: date interpolated into the stable prefix invalidates the whole cache daily
system_prompt = f"You are a support assistant. Today's date is {today}. ..."

# Better: keep the prefix static, move the variable fact into the per-request suffix
system_prompt = "You are a support assistant. ..."
user_message_with_context = f"[Context: today is {today}]\n{user_message}"

Retrieval and Tool Results Complicate This

Retrieval-augmented prompts are the trickiest case in practice, because retrieved documents are, by definition, different per query — that’s the whole point of retrieval. A naive RAG prompt structure looks like:

[system prompt][retrieved chunk 1][retrieved chunk 2][user question]

Here, the retrieved chunks sit between the stable system prompt and the variable question, and since retrieval usually returns a different set of chunks per query, this middle section rarely repeats across requests — which means only the system prompt at the very front benefits from caching, and everything after it, including chunks that might occasionally repeat across similar queries, gets recomputed regardless.

A few patterns help without changing your retrieval logic:

  • Cache what you can up to the point retrieval results are inserted — even caching just the system prompt and tool definitions is worthwhile if that block is large relative to the retrieved content.
  • If your retrieval corpus is small and queries cluster around a limited set of frequently-retrieved documents (a common FAQ, a small policy document set), consider caching those documents as a semi-stable block, appended in a canonical, deterministic order, with genuinely per-query content strictly after them.
  • Avoid re-ordering retrieved chunks non-deterministically (for example, by a tie-breaking sort that isn’t stable) — identical retrieval results returned in a different order defeat the cache for no benefit.

A Mental Model, Not a Vendor Feature

It’s worth internalizing prompt caching as a structural property of how you build prompts, rather than a checkbox you flip on with a provider. Providers differ in exact mechanics (minimum cacheable prefix length, how long a cache entry persists before eviction, whether you must explicitly mark a cache breakpoint versus it being detected automatically) and in exactly how much of a discount a cache hit gets versus a cache miss, and those specifics shift often enough that they’re best confirmed against current provider documentation rather than assumed. But the underlying principle — stable content first, in a fixed order, variable content last — holds across providers and even across self-hosted serving engines that implement prefix caching internally (the same mechanism discussed for KV cache reuse in serving stacks like vLLM), because it follows directly from how attention and the KV cache work, not from any one vendor’s pricing model.

A Simple Audit for an Existing Application

If you already have an LLM application in production and haven’t thought about this, a quick audit usually finds low-hanging fruit:

def audit_prompt_stability(recent_requests):
    """
    Given a sample of recent request payloads, find the longest common
    prefix (by token, not by character) shared across all of them.
    A large gap between "shared prefix length" and "total prompt length"
    means there's caching opportunity being left on the table.
    """
    tokenized = [tokenize(req["prompt"]) for req in recent_requests]
    shortest = min(len(t) for t in tokenized)
    common_prefix_len = 0
    for i in range(shortest):
        if len(set(t[i] for t in tokenized)) == 1:
            common_prefix_len += 1
        else:
            break
    avg_total_len = sum(len(t) for t in tokenized) / len(tokenized)
    print(f"Common prefix: {common_prefix_len} tokens out of ~{avg_total_len:.0f} average total")
    return common_prefix_len

Running this over a sample of real traffic and finding, say, that only the first handful of tokens are shared even though the system prompt and tools are supposedly identical, is usually a sign that something (an interpolated timestamp, a non-deterministic ordering, an inconsistent templating whitespace) is silently breaking the cache further up than expected.

Key Takeaways

  • Prompt caching works by reusing KV cache entries computed for a shared token prefix across requests; it is more accurately described as prefix caching.
  • Cacheability requires an exact, identically-ordered, identically-tokenized prefix — content overlap alone is not enough if the order or surrounding tokens differ.
  • Structure prompts as stable content first (system prompt, tool definitions, fixed few-shot examples) and variable content last (the user’s message, session-specific values), never interleaved.
  • Small interpolated values placed early, like a timestamp inside the system prompt, silently invalidate the entire cached block; push per-request facts into the variable suffix instead.
  • Retrieval-augmented prompts are the hardest case because retrieved content sits between the stable and variable parts and rarely repeats across requests; cache what precedes retrieval and consider a semi-stable block for frequently-retrieved documents.
  • The exact mechanics and discount depend on the provider or serving engine, but the underlying principle is provider-agnostic and follows directly from how the KV cache works.
  • Auditing real production traffic for the longest common token prefix across requests is a fast way to find caching opportunity you’re currently leaving on the table.