Frontier large language models cost between $0.15 and $15 per million tokens depending on model class. Semantic caching can remove the majority of these calls from the critical path when production workloads repeat similar queries. Yet despite five years of tooling (GPTCache, SemCache, Redis Vector Similarity, Weaviate, Pinecone), most engineering teams never deploy it. The barrier is not the availability of vector databases. It is that semantic caching moves the failure mode from "slow" to "subtly wrong": a poorly chosen similarity threshold silently degrades user experience at scale.
This post walks through the production architecture of a semantic cache, the three threshold decisions that determine whether it helps or hurts, and the operational patterns that make it viable in multi-tenant deployments.
What Semantic Caching Actually Replaces
A traditional HTTP or prompt cache keys responses on an exact string match. Any difference in punctuation, whitespace order, or parameter name produces a cache miss. Semantic caching replaces that key with an embedding: the incoming query is encoded into a vector and compared against a vector index of previously seen (query, response) pairs using cosine or Euclidean distance. A hit returns the cached response; a miss routes the query to the target LLM and stores the pair for the next request.
The value proposition is specificity. Two queries that differ in surface form but share intent ("What is the refund policy?" and "How do I get a refund?") map to nearby vectors and can share a response. The cost is latency budget: every lookup now pays for an embedding call plus a vector search.
The Architecture: Three Layers in Production
A production semantic cache has three layers, each with a distinct latency and cost profile.
L1 — In-memory exact cache. The fastest layer is a standard dictionary or LRU cache keyed on the raw query string. It handles true duplicates and near-instant returns. Hit rates are typically 5 to 15 percent of all traffic but at sub-millisecond latency.
L2 — In-memory vector cache. A FAISS or hnswlib index held in the same process or a co-located Redis instance. This layer catches semantically similar queries from the last few hours. Memory-constrained, so it is capped at a few thousand to a few tens of thousands of entries.
L3 — Persistent vector store. Pinecone, Weaviate, Qdrant, or Postgres with pgvector. This is the long-tail cache covering days or weeks of traffic. It is where the embedding model's quality is most visible, because the index is large enough that irrelevant neighbors pollute results.
This paper (Zhang et al., 2024) empirically validates the two-tier approach: SCALM, a semantic caching system evaluated on real chat-service logs, reported a 63 percent relative improvement in cache hit ratio over GPTCache baselines by layering semantic analysis across multiple cache tiers.
The embedding model choice for the L3 store is consequential. OpenAI's text-embedding-3-small, Cohere's embed-v3, and open models like BGE and MiniLM differ in both retrieval quality and per-call cost. GPTCache documents the trade-off: the embedding step must itself be faster and cheaper than the model it replaces.
The Three Threshold Decisions
The architecture is sound. The mistakes live in the thresholds.
Decision 1: Similarity Threshold (Precision vs. Recall)
The cosine similarity cutoff determines whether a neighbor is "close enough" to count as a hit. GPTCache defaults to 0.74; empirical tuning typically lands between 0.65 and 0.85 depending on query volume and model temperature. Too low and the cache returns wrong answers. Too high and the hit rate collapses, leaving the cache as pure overhead.
There is no universal setting. The threshold must be calibrated per use case using a labeled sample of real query pairs and their similarity scores. This is the step most teams skip.
Decision 2: Query Complexity Filter
Not every query should be eligible for caching. High-token, low-repetition queries (unique creative writing, specific code generation with rare variables) will never hit and only inflate index size. A production system typically gates on a complexity score: queries above a token count or entropy threshold bypass the cache entirely and go straight to the LLM.
Decision 3: Tenant Isolation
In multi-tenant systems, the vector index must be partitioned so that one customer's queries cannot be answered from another's cached responses. This is a correctness requirement, not a performance one. Partitioning is typically enforced through metadata filters on the vector index, but the filter itself adds latency and cost to every lookup.
The Latency Math
Every cache lookup has a fixed tax. With text-embedding-3-small at roughly $0.00001 per call and Pinecone serverless at roughly $0.0002 per query, a cache lookup costs about 2x to 4x more than Redis. That means the cache is only worth it if it achieves a hit rate above roughly 50 percent, assuming the cached LLM call would have cost at least 10x more than the embedding-and-query combination.
The breakeven is more favorable for fast, cheap models. An ultra-fast 8B model running at $0.06 per million tokens narrows the margin, which is why semantic caching is most common in stacks routed through frontier models — where the per-call differential is largest.
When Caching Fails: The Research View
Recent academic work has begun measuring the gap between theoretical caching gains and real-world outcomes. GPTCache, evaluated on real agent benchmarks, achieved only 37.9 percent accuracy on intent classification — not because the model was wrong, but because the cache key was wrong. The cache returned responses that were semantically plausible but not correct for the specific user context.
The root cause identified by the authors: caching effectiveness requires key consistency (the same intent maps to the same key) and key precision (different intents map to different keys), not classification accuracy of the embedding model alone. Their proposed remedy — structured intent canonicalization using a lightweight SetFit model — achieved 91.1 percent accuracy at roughly 2 milliseconds, versus 3,447 milliseconds for a 20B-parameter LLM.
The practical takeaway: a semantic cache is only as good as its key construction, and key construction is a clustering problem, not a similarity problem.
Operational Checklist
Before deploying a semantic cache, teams should be able to answer four questions:
- Can you label 100 real query pairs and compute their similarity to set a threshold?
- Can you measure the latency contribution of embedding + vector search relative to your target LLM?
- Can you partition the index by tenant without breaking correctness?
- Can you evict stale entries when the underlying source data changes?
Cache invalidation is the second hard problem in computer science. In semantic caching, the issue is not just stale responses but stale similarity boundaries: a query that was once novel can become common as user behavior shifts, and the threshold that was safe yesterday may return false positives tomorrow.
Conclusion
Semantic caching is not a drop-in optimization. It is a separate service with its own data store, its own threshold configuration, and its own failure modes. It pays off when query repetition is high and per-call LLM cost is measured in cents. It breaks silently when the similarity threshold drifts and the cache returns plausible but wrong answers. The teams that deploy it successfully treat the cache as infrastructure that must be calibrated, not a feature to be flipped on.
Sources
- Zhang, Y., et al. (2024). SCALM: Towards Semantic Caching for Automated Chat Services with Large Language Models. arXiv:2406.00025. <https://arxiv.org/abs/2406.00025>
- Wang, B., et al. (2025). Why Agent Caching Fails and How to Fix It: Structured Intent Canonicalization with Few-Shot Learning. arXiv:2602.18922. <https://arxiv.org/abs/2602.18922>
- GPTCache Documentation. <https://github.com/Supervisor-GPT/GPTCache>
- OpenAI Embeddings API. <https://platform.openai.com/docs/guides/embeddings>
- Pinecone Serverless Documentation. <https://docs.pinecone.io/>
- Qdrant Vector Similarity Search Documentation. <https://qdrant.tech/documentation/>
- PostgreSQL pgvector. <https://github.com/pgvector/pgvector>
- Redis Vector Similarity Documentation. <https://redis.io/docs/latest/develop/use/vector-search/>
