Semantic caching reduces token consumption, lowers latency and improves user experience by reusing responses based on meaning rather than exact wording, making large language model (LLM) applications more efficient and cost-effective.
As organisations increasingly deploy applications powered by large language models (LLMs), managing operational costs has become a growing challenge. Every user request consumes tokens and computational resources, and as applications scale, inference costs and response times increase. While each query may appear unique, many users ask similar questions using different words. Semantic caching addresses this problem by identifying queries that have the same meaning and reusing previously generated responses, reducing both latency and LLM usage.
Unlike conventional caching, which relies on exact keyword matches, semantic caching compares the meaning of user queries using vector embeddings. This enables AI applications to respond more quickly to repeated requests while significantly lowering operational costs.
The growing cost of LLM applications
A common AI workflow begins when a user submits a query, which is then processed by an LLM to generate a response. Although effective, every request requires a complete inference cycle that typically takes between five and ten seconds. As applications move into production and user traffic grows, token consumption increases rapidly, leading to substantially higher operating costs.
Many of these requests, however, are not entirely unique. Approximately 31 per cent of user queries are duplicates, as users often ask the same question using different words. Processing each request independently results in unnecessary token usage while increasing latency and operational expenses.
Customer support systems illustrate this challenge clearly. Questions such as “How can I get a refund?”, “Can I get my money back?” and “What is your refund policy?” all express the same intent. Treating each as a separate request forces the application to repeatedly invoke the LLM, increasing costs while offering no additional value to the user.
Why traditional caching falls short
A conventional cache operates by matching exact text. If an incoming request precisely matches a previously stored query, the cached response is returned. Otherwise, the application generates a completely new response.
While this approach works well for conventional software applications, it is less effective for generative AI systems because users rarely phrase identical questions in the same way. Two requests may contain entirely different words while seeking the same information. As a result, traditional caching produces very few cache hits in conversational AI applications.
How semantic caching works
A semantic caching workflow begins when a user’s query is received. The application first converts the query into a vector embedding using an embedding model. The embedding is then compared with existing entries stored in the semantic cache through vector similarity search.
If a sufficiently similar query already exists, the cached response is returned directly to the user, eliminating the need for another LLM call. If no suitable match is found, the request proceeds through the complete retrieval-augmented generation (RAG) pipeline, where relevant documents are retrieved before the LLM generates a response. New response is then stored alongside its embedding, improving future cache performance.
Because responses are increasingly served from the cache rather than generated repeatedly, both inference latency and token consumption decrease over time.
Demonstrating semantic caching
A demonstration using a streaming service support assistant illustrates this behaviour.
In the first interaction, a user asks how to download a movie for offline viewing. The request passes through an RAG pipeline that retrieves relevant policy documents before the LLM generates a response. This process consumes 701 tokens and takes approximately 5.3 seconds.
A second user later asks how to watch a movie without an internet connection. Although the wording is completely different, both questions have the same meaning. Rather than invoking the LLM again, the application retrieves the response directly from the semantic cache.
As a result, no additional tokens are consumed, and response latency falls dramatically to around 700 milliseconds. Since the request is served directly from Redis instead of passing through the LLM, inference delays and network overheads are largely eliminated.
This highlights how semantic caching simultaneously reduces operating costs and improves the user experience by delivering faster responses without sacrificing accuracy.
When semantic caching works bestSemantic caching delivers the greatest benefits when users repeatedly ask similar questions. Applications such as customer support systems, chatbots and virtual assistants often receive recurring requests with different wording but identical intent. In these environments, semantic caching can significantly improve response times while lowering inference costs. It is particularly effective when:
As more requests are cached, the cache hit ratio increases, allowing a greater proportion of responses to be served without repeatedly invoking an LLM. When semantic caching is not suitableDespite its advantages, semantic caching is not appropriate for every application. Rapidly changing information, such as live sports scores, financial market prices or continuously updated sensor readings, is generally unsuitable because cached responses become outdated quickly. Similarly, highly personalised information should not be stored in a semantic cache. Queries such as checking an individual’s bank balance or retrieving private account information require user-specific responses and cannot be safely reused across different users. Applications that receive mostly unique requests with very little repetition are also unlikely to benefit significantly from semantic caching, since cache hit rates remain low. Selecting the appropriate use cases is essential to achieving meaningful improvements in performance and cost. |
Redis: Beyond traditional caching
Although Redis is widely recognised as an in-memory caching platform, it also supports vector data structures, allowing it to function as a vector database for AI applications. This enables developers to combine semantic caching, vector search and real-time data processing within a single platform.
By storing vector embeddings alongside cached responses, Redis supports semantic similarity searches while maintaining the low-latency performance required for production AI applications.
Real-world applications of semantic caching
Semantic caching is already being used in production environments to reduce LLM costs while improving application performance.
One example is Asurion, a device insurance provider serving approximately 300 million customers. By combining Redis Vector Database, semantic caching and a semantic router, the company achieved a 70 per cent reduction in LLM costs. Frequently asked questions were served directly from the semantic cache, while the semantic router determined the most appropriate processing path for incoming requests.
Another example is Mangos AI, a voice AI healthcare application. The company also reported a 70 per cent cache hit ratio and a corresponding 70 per cent reduction in LLM costs by using LangCache, Redis’ managed semantic caching service. LangCache manages embedding generation and cache operations automatically, allowing developers to integrate semantic caching through a REST API.
Introducing the semantic router
Many AI applications use an LLM to determine a user’s intent before deciding which tool to invoke or which knowledge source to query. Although effective, this additional LLM call increases both latency and cost.
A semantic router provides an alternative approach by performing intent classification using vector similarity instead of an LLM. Incoming queries are converted into embeddings and compared against predefined intent vectors, allowing the application to route requests without invoking a language model.
By relying on vector search rather than generative inference, semantic routing produces deterministic results while reducing response time and token consumption.
Design principle 1: Choose the right embedding model
The effectiveness of semantic caching depends heavily on the quality of the embedding model used to represent user queries (Figure 1).
Rather than selecting a model based on general recommendations, developers should evaluate multiple embedding models using their own datasets. Benchmarking against real application data provides a more accurate indication of retrieval quality than relying solely on published benchmark results.
An example is the LangCache Embed V2 model, which has been fine-tuned specifically for semantic caching. Unlike general-purpose embedding models, it is designed to distinguish between queries that genuinely share the same intent and those that are merely topically related.
For example, questions such as “What is the refund policy?” and “How do I get my money back?” should generate similar embeddings because they require the same answer. By contrast, questions such as “What is the capital of Australia?” and “What is the largest city in Australia?” discuss the same country but require different responses. A generic embedding model may place these questions close together because they share geographical context, whereas an embedding model optimised for semantic caching is more likely to distinguish between them correctly.
Selecting an embedding model should be based on benchmark results obtained from the target application’s own dataset rather than adopting a model simply because it performs well elsewhere.
Contributing to open sourceOpen source projects provide an effective way to gain practical experience with emerging technologies. Redis maintains a collection of open issues within its RedisVL project, offering contributors an opportunity to work on semantic caching and other AI-related features. For developers interested in learning semantic search, vector databases and AI infrastructure, contributing to these projects provides valuable hands-on experience while helping improve the wider open source ecosystem. |
Design principle 2: Thresholds and safety checks
Unlike traditional caching, semantic caching does not operate as a simple match-or-no-match system. Instead, responses are retrieved based on similarity scores, making the choice of distance threshold a critical design decision (Figure 2).
A threshold that is set too low may reject valid matches, reducing the cache hit ratio through false negatives. Conversely, a threshold that is too high increases the likelihood of returning incorrect responses by treating unrelated queries as equivalent.
A practical approach is to benchmark against representative datasets. Tools such as the Retrieval Optimiser can evaluate different threshold values, helping identify settings that maximise retrieval accuracy while reducing incorrect cache hits.
Another recommended practice is to apply an intent classification filter before accessing the semantic cache. This filter determines whether an incoming request is suitable for caching, preventing personalised or frequently changing queries from being stored unnecessarily.
Redis also provides an open source semantic router, enabling intent classification through vector search without requiring an additional LLM call.
Design principle 3: Cache design and eviction
Building a semantic cache involves more than storing responses. Since Redis is an in-memory database, memory must be managed carefully to ensure the cache remains efficient as applications scale (Figure 3).
One of the first considerations is selecting an appropriate eviction policy. When memory usage becomes high, older or less frequently accessed entries should be removed to make room for new ones. Common approaches include Least Recently Used (LRU) and Least Frequently Used (LFU) eviction policies, both of which help maintain cache performance without requiring excessive memory resources.
Setting an appropriate Time to Live (TTL) is equally important. The TTL should reflect how frequently the underlying information changes. Responses based on policies or documentation may remain valid for weeks or months, whereas information that changes more frequently requires a shorter lifetime to prevent outdated responses from being served.
For multi-tenant applications, the cache should also be partitioned appropriately to ensure data belonging to different users or organisations remains isolated.
Design principle 4: Observability and feedback
Semantic caching should not be treated as a one-time implementation. As applications evolve, the cache must be monitored continuously to ensure it continues delivering the expected benefits (Figure 4).
Developers should track metrics such as cache hit ratio, latency, token consumption and memory usage. These measurements help determine whether the cache is reducing LLM usage effectively and whether further optimisation is required.
Tracing cache behaviour is equally important. Reviewing cache traces can reveal whether the similarity threshold should be adjusted or whether new categories of user queries have emerged that are suitable for caching.
Monitoring platforms can also provide accuracy metrics, including true positive and true negative rates, alongside usage statistics such as memory consumption and token savings. Together, these measurements help maintain both retrieval quality and application performance over time.
Implementing semantic caching
Several implementation approaches are available depending on an application’s architecture and the level of management required.
Developers can build a custom implementation using RedisVL, an open source library that provides semantic caching together with additional AI capabilities. Developers can configure cache entries, update responses, define similarity thresholds, manage expiration policies and customise cache behaviour according to application requirements.
RedisVL also supports embedding caching. Since generating vector embeddings requires computational resources, repeatedly calculating embeddings for identical queries introduces unnecessary overhead. By caching embeddings alongside responses, applications can reduce both compute requirements and response times.
Applications built with LangChain can integrate semantic caching through the Redis middleware, which supports both semantic response caching and tool caching. Tool caching stores the results of previous tool invocations, preventing repeated tool selection and execution when identical requests are received.
For organisations seeking a fully managed solution, LangCache provides semantic caching through a simple API. Embedding generation, cache management, scaling and availability are handled automatically, allowing developers to focus on application development rather than infrastructure management.
Practical considerations for production deployments
There are several practical considerations when deploying semantic caching in production systems.
One concern is ensuring that cached responses do not replace workflows that require retrieval-augmented generation (RAG). A common implementation pattern is to check the semantic cache first. If a matching query already exists, the cached response is returned immediately. Otherwise, the application proceeds through the complete RAG pipeline, generates a fresh response using the LLM and stores the new result in the semantic cache for future requests.
Another important consideration is hallucination. Because semantic caching retrieves previously generated responses from a vector database rather than generating new text through an LLM, responses remain deterministic. The semantic router performs intent classification using vector search instead of language model inference, further reducing variability.
Many production AI systems minimise unnecessary LLM usage wherever possible by using deterministic components such as vector databases for routing, retrieval and intent classification.
Managing stale information is another important aspect of deployment. Information that changes frequently should not remain in the cache indefinitely. Appropriate TTL values and eviction policies ensure that outdated responses are removed and replaced as new information becomes available.
Finally, semantic caching is generally less suitable for multi-turn conversations. As conversations progress, later questions often depend on the context established by previous exchanges. Since semantic caching primarily compares individual queries, policy documents, FAQs, and other relatively stable knowledge sources are typically better candidates for caching than long conversational sessions.
When combined with vector databases, appropriate embedding models, carefully selected similarity thresholds and continuous monitoring, semantic caching can substantially reduce token usage while improving response times. Successful adoption, however, depends on selecting suitable use cases, designing effective cache policies, and continuously evaluating application behaviour. Rather than replacing LLMs, semantic caching complements them by ensuring that language models are used only when genuinely necessary.
This article is based on the talk ‘Reduce LLM Costs with Semantic Caching: Real World Lessons’ by Bhavana Anant Giri, developer advocate, Redis, at AI DevCon India 2026. It has been transcribed by Ananthu Ashok of the EFY Group.
