Context Caching

What is Context Caching?

Context caching, also called prompt caching, stores the pre-computed attention states of a prompt prefix at the provider or infrastructure level so that later API calls sharing that prefix can skip recomputing it. When a subsequent request shares an identical prefix with a prior one, the large language model (LLM) bypasses the computationally expensive prefill phase for those matched tokens. This reduces latency, specifically Time-To-First-Token (TTFT), and lowers token costs for developers.

In a stateless LLM inference architecture, every incoming request is processed in isolation. If a user uploads a 100,000-token legal contract and asks a simple question, the model has to read, tokenize, and encode the entire document. If the user asks a second, follow-up question about that same document seconds later, the model repeats the entire encoding process from scratch. As context windows have expanded to millions of tokens, this redundant processing wastes GPU compute and adds latency for end users.

Context caching addresses this by recognizing that large portions of a prompt, such as system instructions, few-shot examples, large codebases, or reference documents, remain static across multiple interactions. Providers can temporarily cache the mathematical representations of these tokens, making long-context applications more practical for production use.

This page covers context caching at the provider or KV-cache level. For the related application-layer technique, which reuses a stored response when a new query means the same thing as a past one rather than matching a token prefix exactly, see semantic caching.

How Context Caching Works (KV Attention State Persistence & GPU Memory Pinning)

To understand context caching, it helps to look at the mechanics of transformer-based LLMs. Text generation happens in two distinct phases: the prefill phase and the decode phase.

During the prefill phase, the model processes the entire input prompt at once. The attention mechanism calculates the Key and Value (KV) vectors for every single token in the input sequence. This process is highly parallelizable but intensely compute-bound. Once the prefill is complete, the model moves to the decode phase, generating one token at a time sequentially. The decode phase is memory-bound, as it requires constantly reading the KV vectors (the "KV Cache") from memory to compute attention for the next token.

When context caching is active, the provider's infrastructure takes the KV vectors generated during the prefill phase of the static prompt prefix and persists them. Depending on the provider's architecture, these states are either pinned directly in the GPU's high-bandwidth memory (HBM) for immediate access, or temporarily offloaded to a fast tiered storage system (such as CPU RAM or NVMe SSDs) to be quickly paged back in when needed.

When a subsequent request arrives, the provider's routing layer hashes the initial tokens of the prompt and checks if they match a stored sequence. If a cache hit occurs, the system retrieves the pre-computed KV states and loads them directly into the inference engine. The model skips the compute-heavy prefill phase entirely for the matched tokens, instantly beginning the decode phase for the new, dynamic portion of the prompt. This state persistence is maintained for a specific Time-To-Live (TTL) window, ensuring resources are freed when no longer needed.

Provider Implementation Comparison: Anthropic vs. Google Gemini vs. OpenAI

The major LLM providers take different architectural approaches to context caching. All three now bill cache reads at 10% of the standard input price, so the differences that matter are control, minimums, and retention. Figures below reflect provider documentation as of August 2026.

Anthropic uses an explicit caching model: developers place up to four cache_control breakpoints within their API requests to mark which parts of a multi-part prompt (system messages, tools, reference documents) should be cached. This gives granular control over what gets cached and when. The default cache has a sliding 5-minute TTL, refreshed on every hit, with a 1-hour TTL available for bursty traffic. Reads cost 10% of the standard input price; writes carry a premium of 1.25x for the 5-minute TTL or 2x for the 1-hour TTL. That premium sets the break-even point: a 5-minute cache pays for itself on the second request (1.25x + 0.1x versus 2x uncached), while the 1-hour cache needs at least three. The minimum cacheable prefix varies by model, from 1,024 tokens on Sonnet-class models to 4,096 on Haiku 4.5, and shorter prefixes silently fail to cache rather than erroring.

Google Gemini enables implicit caching by default on all Gemini 2.5 and later models, including the Gemini 3 family. Cached tokens bill at 10% of the input price with no write fee, and the minimum prefix is 4,096 tokens on Gemini 3 models (2,048 on Gemini 2.5). Explicit caching remains available for guaranteed hits and longer retention: developers create a cached-content object with a chosen TTL and pay an hourly storage fee per token held (Google's pricing page lists $0.50 per million tokens per hour for Gemini 3 Flash-class models through the end of 2026). Explicit caches suit large static corpora queried steadily over hours; the storage fee means an idle explicit cache loses money.

OpenAI offers a fully implicit, zero-configuration caching system. If a prompt prefix exceeds 1,024 tokens and matches a recently processed sequence, it is automatically cached and retrieved, with no API changes and no write fee; matching extends in 128-token increments. Cached input on the GPT-5 family bills at 10% of the standard rate (for example $0.125 versus $1.25 per million tokens on GPT-5). Entries typically evict after 5 to 10 minutes of inactivity, and the API offers an extended-retention option for prefixes that must survive longer idle gaps.

Best Practices for Structuring Prompts to Maximize Cache Hit Rates

Because caching relies on exact prefix matching, the structure of your prompts dictates your cache hit rate. A single character difference at the beginning of a prompt will invalidate the cache for the entire remainder of the sequence.

1. Static Content First: Always place immutable content at the absolute beginning of your prompt array. This includes system instructions, tool definitions, few-shot examples, and large reference documents.
2. Dynamic Content Last: Place user queries, session-specific variables, timestamps, and previous conversation history at the very end of the prompt. As the conversation grows, only the newest messages require prefilling.
3. Consistent Ordering: If you are passing multiple documents or database records, always pass them in the exact same order. Sorting by an immutable identifier (like a UUID or creation date) ensures consistent prefixes.
4. Strategic Breakpoints (Anthropic): If using Anthropic, place your cache_control breakpoints at the end of the largest static chunks. Do not place breakpoints on rapidly changing content, as writing to the cache incurs a slight premium over standard inputs.
5. Avoid Unnecessary Whitespace Variations: Ensure your application does not inject random spaces, newlines, or dynamic timestamps into the system prompt, as this will instantly cause a cache miss.

In a multi-provider setup, an AI gateway can help apply these prefix-ordering rules consistently across providers whose caching mechanics differ.

Latency and Cost Calculations in Production Document & Code Workflows

Consider a production application where users can ask questions about a 1,000,000-token codebase. A user typically asks 5 questions in a 10-minute session.

Without Context Caching:

• Request 1: 1M tokens prefilled. TTFT: ~10 seconds. Cost: 1M input tokens.
• Request 2-5: 1M tokens prefilled each time. TTFT: ~10 seconds each.
• Total Cost: 5,000,000 standard input tokens.
• User Experience: Sluggish, with a 10-second wait before every single response.

With Context Caching:

• Request 1: 1M tokens written to cache. TTFT: ~10 seconds. Cost: 1M cache-write tokens (standard price, or a 1.25x premium on Anthropic).
• Request 2-5: Cache hit. TTFT: ~150 milliseconds. Cost: 4M cache-read tokens billed at 10% of the input rate.
• Total Cost: the equivalent of roughly 1.65M standard input tokens instead of 5M, a cut of about two thirds. The savings grow with every additional question in the session.
• User Experience: The first request takes time, but all subsequent interactions feel instant, which supports real-time conversational agents over large datasets.

Frequently Asked Questions

Does context caching affect model output quality or accuracy? No. Context caching is a mathematically exact optimization. The KV states retrieved from the cache are identical to the states that would have been computed if the prefill phase had run normally. There is zero degradation in reasoning or output quality.

How is my cached data secured and isolated? All major providers ensure strict tenant isolation. Cached KV states are securely segregated and keyed to your specific organization and API key. A prompt cached by your application cannot be accessed, read, or utilized by another customer, even if they send the exact same prompt string.

Can multiple users share the same cache? Yes, as long as the requests originate from the same organizational API key. If User A uploads a company handbook and User B asks a question about it 30 seconds later, User B will benefit from the cache hit, provided the prompt prefixes are identical.

Why does my cache show zero hits even though my prompts look identical? The usual culprit is a silent invalidator: a timestamp or request ID rendered into the system prompt, JSON serialized with unstable key ordering, or a tool list that changes between requests. Because matching is byte-exact on the prefix, any of these breaks the cache for everything after the change. Check the provider's usage fields (Anthropic reports cache_read_input_tokens per response); if they stay at zero, diff the raw rendered prompts of two consecutive requests. A prefix below the model's minimum (up to 4,096 tokens on some models) also fails silently.

Do parallel requests share a cache write? No. A cache entry only becomes readable after the first response begins. If you fire N identical requests simultaneously, all N pay the full uncached price. For fan-out workloads, send one request first, wait for its first streamed token, then dispatch the rest so they read the cache the first one wrote.

What happens if a cache expires before the next request? If the TTL expires, the cache is simply evicted from memory to free up resources. The next time a request is sent with that prompt, it is treated as a standard, uncached request. The model will perform a full prefill, incur standard latency, charge standard input token rates, and then re-cache the KV states, restarting the TTL timer.