Modern LLM serving workloads spend a disproportionate share of computational budget and time-to-first-token (TTFT) latency on prompt prefill. In agentic loops, retrieval-augmented generation (RAG), and multi-turn chat applications, repeated prompts often share 80% to 95% of their token sequences across requests. Without caching, inference engines recompute key-value (KV) attention tensors across every input token on every turn, driving quadratic compute overhead and memory bandwidth saturation.
Prompt and context caching techniques eliminate redundant prefill by persisting precomputed KV activation tensors across requests. In production architectures, caching implementations fall into two distinct paradigms: managed provider-side caching with vendor-enforced time-to-live (TTL) mechanics, and self-hosted radix-tree memory management.
The Mechanics of KV Cache Reuse
In standard Transformer self-attention, generating the first token requires computing Key () and Value () projections for every token in the prompt:
Because attention operates over all preceding tokens, any token sequence that matches a previously evaluated prefix exactly will produce identical KV tensors. Recomputing these projections across identical system prompts, schema definitions, and retrieved chunks wastes GPU compute and memory bandwidth.
Prefix caching maps token sequences to their precomputed KV state blocks. When an incoming request arrives:
- The engine computes a cryptographic hash or walks a prefix tree for the token sequence starting at index zero.
- If a matching prefix exists in memory, the engine loads the pre-computed KV tensors directly into the attention layer, bypassing the prefill compute phase for those tokens.
- Generation begins immediately after the cached prefix, reducing TTFT from seconds to milliseconds.
The fundamental constraint across all caching engines is strict prefix invariance: KV cache reuse requires an exact, contiguous token match starting from the very first token (). A single modified byte, inserted timestamp, or reordered tool definition at the beginning of a prompt invalidates the entire subsequent cache chain.
Provider-Native Architectures
Major foundation model providers have integrated prefix caching directly into their API layers, though their retention policies, activation triggers, and pricing models differ significantly.
Anthropic Claude: Explicit Breakpoint Caching
Anthropic's prompt caching uses explicit developer-defined breakpoints. Engineers place a cache_control: {"type": "ephemeral"} parameter on up to four content blocks within tools, system prompts, or conversation turns.
- Minimum threshold: 1,024 tokens for Claude 3.5 Sonnet and Claude Opus; 2,048 tokens for Claude Haiku.
- Time-to-live (TTL): 5 minutes by default, refreshed on every cache hit. Anthropic also supports an optional 1-hour extended TTL for longer workflows.
- Pricing mechanics: Cache writes incur a 25% premium (1.25x the standard input token rate), while cache reads receive a 90% discount (0.10x standard input rate).
- Prefix sequence order: Anthropic hashes blocks in a strict order (tools first, then system prompts, then message turns). Modifying a tool definition invalidates all downstream system prompt and message caches.
OpenAI: Implicit Prefix Matching
OpenAI's prompt caching operates implicitly across GPT-4o, GPT-4o mini, o1, and newer model families without manual code markers.
- Minimum threshold: 1,024 tokens.
- Chunk granularity: Cache matches occur in 128-token increments beyond the initial 1,024-token boundary.
- Time-to-live (TTL): Dynamic LRU retention between 5 and 10 minutes (persisting up to 1 hour during off-peak periods).
- Pricing mechanics: Cache writes carry no surcharge (billed at standard input rates), while cache hits receive a 50% discount on input token costs.
- Operational behavior: The gateway automatically matches the longest shared prefix in memory.
DeepSeek: Multi-Tier Disk and Memory Offloading
DeepSeek's context caching is enabled automatically by default for all API requests, utilizing a multi-tiered storage architecture.
- Storage tiering: Hot KV blocks reside in GPU HBM/SRAM and host DRAM, while colder prefixes are offloaded asynchronously to distributed NVMe disk arrays.
- Minimum threshold: Implicit from token zero, matching complete contiguous prefix blocks.
- Pricing mechanics: No cache write fees; cache hits receive up to a 90% discount (charging $0.014 per million tokens on hits versus $0.14 per million on misses for base models).
- Operational behavior: The disk-backed tier enables deep cache persistence across long periods without requiring active keep-alive traffic.
Google Gemini: Context Caching and Explicit Object Management
Google Cloud Vertex AI and Gemini Developer API offer both explicit context cache objects and implicit prefix caching on Gemini 2.5 and newer models.
- Explicit cache objects: Developers create an explicit cache resource via
client.caches.create, assigning custom TTLs (e.g., 1 hour, 2 hours, or multi-day durations). - Storage billing: Explicit caches incur an hourly storage charge (typically $0.50 to $1.00 per million tokens per hour) alongside discounted input query fees (75% to 90% discount on cached input tokens).
- Minimum threshold: 1,024 to 32,768 tokens depending on the model tier and API endpoint.
- Operational behavior: Ideal for static enterprise corpora, legal bundles, or large codebases queried repeatedly over fixed operational windows.
Self-Hosted Infrastructure: RadixAttention and Prefix Routing
In self-hosted deployments using open-weight models, managing KV cache memory across dynamic requests requires algorithmic tree management.
RadixAttention in SGLang
Traditional PagedAttention in vLLM maps non-contiguous physical memory blocks to logical request sequences. However, standard paging schemes discard KV blocks once a request finishes, missing cross-request sharing opportunities.
SGLang introduced RadixAttention, which retains and indexes KV cache blocks in a radix tree (compressed trie) data structure:
- Tree-structured sharing: Each node in the radix tree represents a sequence of tokens. Multiple concurrent requests sharing a common system prompt or RAG context branch off the same root and intermediate nodes.
- Branching and exploration: In agent workflows that generate multiple speculative paths, Monte Carlo tree searches, or parallel tool calls, RadixAttention forks execution at zero memory duplication cost.
- Eviction policies: When GPU VRAM reaches capacity, SGLang uses a tree-aware Least Recently Used (LRU) policy that recursively evicts leaf nodes while preserving high-frequency root prefixes.
Prefix-Aware Gateway Routing
In distributed multi-GPU clusters running vLLM Automatic Prefix Caching (APC) or SGLang, round-robin load balancers degrade cache hit rates by scattering identical requests across separate worker nodes.
Production gateways implement prefix-aware hash routing:
- Locality-sensitive dispatch: The proxy hashes the static prompt prefix (system prompt and tool schemas) and routes requests with identical prefix hashes to the same GPU worker replica.
- Distributed KV sharing: Frameworks like LMCache enable cross-replica KV transfer over high-speed interconnects (InfiniBand/RoCE) and host DRAM, decoupling cache residency from single-node memory constraints.
Production Architectural Best Practices
Maximizing cache hit rates requires strict adherence to prompt construction patterns:
- Deterministic prompt layering: Always place immutable tokens first and volatile tokens last. The optimal ordering is:
- Fixed tool definitions and function calling schemas.
- Core system prompt and instruction guidelines.
- Static domain knowledge documents or reference few-shot examples.
- Historical multi-turn dialogue context.
- Dynamic user query, ephemeral timestamps, and per-request metadata.
- Eliminating prefix jitter:
- Never inject dynamic dates, request UUIDs, or user session IDs at the start of the system prompt.
- Normalize JSON serialization for tool parameters (ensure dictionary keys are sorted deterministically before tokenization).
- Standardize whitespace and newline conventions across prompt templates.
- Managing TTL boundaries:
- For provider APIs with rolling 5-minute TTLs (such as Anthropic), batch background jobs within active 300-second windows to avoid paying repeated cache creation surcharges.
- For workloads with predictable periodic queries, utilize explicit context caches with fixed hourly expiration or implement lightweight keep-alive pings.
