LLM Response Caching Strategies for RAG and Agentic Applications

Caching in a conventional application usually means storing the result of a database query or API request. In an LLM application, there are many more opportunities: repeated questions, equivalent paraphrases, identical embeddings, repeated retrieval and reranking, repeated tool calls, stable system prompts, and recurring agent planning steps.

The strongest architecture therefore does not implement one “LLM cache.” It implements a hierarchy of specialised caches across the RAG and agent pipeline.

1. The most important distinction​

There are two fundamentally different forms of LLM caching.

Provider-side prompt or prefix caching​

Prompt caching stores the model’s internal computation for an identical prompt prefix. The model still generates a new response, but it does not need to recompute the repeated portion of the input.

This is useful when requests repeatedly contain:

• A long system prompt
• Tool definitions
• Few-shot examples
• A large reference document
• An expanding conversation history
• Stable compliance or formatting instructions

Prompt caching normally requires the repeated content to appear at the beginning of the prompt. Variable content should be placed afterwards. OpenAI, Anthropic, Amazon Bedrock, Google Gemini and self-hosted systems such as vLLM all support variations of prefix or context caching. (OpenAI Developers)

Prompt caching reduces input processing and time to first token, but it normally does not:

• Skip vector retrieval
• Skip tool execution
• Skip output generation
• Return a previously generated answer
• Protect you from repeated agent loops

Application-level response caching​

Response caching stores a complete answer or an intermediate pipeline result.

On a cache hit, the system may skip:

• Embedding generation
• Vector retrieval
• Reranking
• Prompt construction
• LLM generation
• Agent planning
• Tool execution

This can deliver much larger savings than prompt caching, but it creates a greater correctness risk because the application is reusing an old result rather than merely reusing model computation.

The two techniques should normally be combined:

2. Cache every reusable layer, not just the final response​

A typical RAG request might follow this path:

Each stage has different reuse opportunities, freshness requirements and security risks.

Recommended cache hierarchy​

3. Exact-match response caching​

Exact caching is the safest application-level caching strategy.

The system creates a deterministic key from all inputs that could affect the answer.

For example:

cache_key = hash(
    tenant_id
    + user_permission_fingerprint
    + canonical_query
    + conversation_scope
    + knowledge_base_version
    + prompt_template_version
    + model_version
    + tool_schema_version
    + retrieval_configuration
    + language
    + response_format
    + safety_policy_version
)

The cache returns a hit only when the complete key matches.

Example​

These two inputs could produce different exact-cache keys:

What is the refund policy?
what is the refund policy?

You can improve exact-hit rates through conservative normalisation:

def canonicalise_query(query: str) -> str:
    return " ".join(query.strip().split())

However, avoid aggressive transformations that remove meaningful information.

For example, these are not equivalent:

Show transactions above $1,000.
Show transactions below $1,000.

Similarly, dates, IDs, account numbers, product names, negation, currencies and quantities must be preserved.

Best use cases​

Exact response caching works especially well for:

• FAQ questions
• Deterministic document summaries
• Classification
• Structured extraction
• Policy questions
• Product documentation
• Code explanation for identical input
• Repeated automated workflows
• Machine-to-machine LLM requests

Include generation settings in the key​

The following parameters can affect the output and should normally be represented:

• Model and model version
• Temperature
• Top-p
• Maximum output tokens
• Structured output schema
• System and developer prompt versions
• Tool definitions
• Response language
• Safety configuration
• Retrieval strategy
• Knowledge-base revision

Without these fields, an answer generated under an old configuration may be returned after the application has changed.

4. Semantic response caching​

Exact caching only works when the query is identical after normalisation. Semantic caching tries to reuse answers for queries that mean the same thing.

For example:

What is your refund policy?

How can I get my money back?

Can customers request a refund?

A semantic cache normally stores:

{
  "original_query": "What is your refund policy?",
  "query_embedding": [...],
  "response": "Customers may request...",
  "tenant_id": "tenant-123",
  "knowledge_base_version": "kb-2026-07-28",
  "prompt_version": "support-v7",
  "locale": "en-GB",
  "created_at": "2026-07-28T10:00:00Z",
  "expires_at": "2026-07-29T10:00:00Z"
}

When a new query arrives:

1. Generate its embedding.
2. Search cached query embeddings.
3. Apply mandatory metadata filters.
4. Find the nearest cached queries.
5. Compare the similarity score with a threshold.
6. Return the answer if the candidate is considered safe.
7. Otherwise, execute the complete RAG pipeline.

Redis describes this pattern as storing the prompt, embedding, response and metadata together, then using vector search with hard filters such as tenant, locale, model version and safety status. (Redis)

Semantic cache flow​

Semantic caching is approximate​

Embedding similarity does not guarantee answer equivalence.

Consider:

How do I reset my personal account password?

How do administrators reset another user’s password?

These questions may have high semantic similarity but require materially different instructions and permissions.

Research on verified semantic caching notes that fixed similarity thresholds create a trade-off:

• A low threshold increases cache hits but also incorrect matches.
• A high threshold reduces false matches but misses useful reuse opportunities.
• Except for exact matching, semantic caches remain inherently approximate. (arXiv)

Therefore, cache hit rate should never be the main semantic-cache KPI. The primary metrics should be:

• Semantic-cache precision
• False-hit rate
• Grounding accuracy
• Permission correctness
• Stale-answer rate
• User correction rate

5. Use a two-threshold semantic policy​

Do not begin with a single arbitrary similarity threshold such as 0.80.

Similarity distributions vary by:

• Embedding model
• Distance metric
• Domain
• Query length
• Language
• Intent
• Cache content
• Vector normalisation

A safer design uses two thresholds.

Similarity ≥ 0.95
    Return automatically, subject to hard filters.

Similarity between 0.85 and 0.95
    Perform secondary validation.

Similarity < 0.85
    Treat as a miss.

These numbers are illustrations, not universal recommendations.

The secondary validator could be:

• A cross-encoder
• A lightweight classifier
• Intent-and-slot comparison
• A small LLM judge
• Rule-based entity comparison
• Verification against current retrieved documents

Intent-and-slot validation​

Convert questions into a structured signature:

{
  "intent": "refund_eligibility",
  "product": "premium_subscription",
  "country": "United Kingdom",
  "purchase_date": "2026-07-20",
  "customer_type": "individual"
}

A semantic hit should require:

Intent matches
AND product matches
AND country matches
AND customer type matches
AND date falls under compatible policy version

This is more reliable than relying only on cosine similarity.

Shadow deployment​

Before returning semantic hits to users:

1. Run the cache in shadow mode.
2. Record which candidate it would have returned.
3. Generate the normal RAG answer.
4. Compare the two.
5. Label correct and incorrect cache decisions.
6. Calibrate thresholds by intent and domain.
7. Enable automatic reuse only for high-confidence categories.

A published semantic-cache experiment reported substantial API-call reductions, but such results are workload-specific and should not be interpreted as a universal expected hit rate. (arXiv)

6. Embedding caching​

RAG systems repeatedly embed identical text.

This happens during:

• Repeated user queries
• Query rewriting
• Document reindexing
• Chunking pipeline retries
• Duplicate document ingestion
• Evaluation runs
• Multi-agent workflows

Create the key from:

embedding_cache_key = hash(
    normalised_text
    + embedding_model
    + embedding_model_version
    + dimensions
    + preprocessing_version
)

Example:

embedding_key = (
    f"embedding:{embedding_model}:"
    f"{preprocessing_version}:"
    f"{sha256(text.encode()).hexdigest()}"
)

Document embedding cache​

During ingestion, compute a content hash for every chunk:

chunk_hash = SHA256(cleaned_chunk_content)

When a document is reprocessed:

• If the chunk hash is unchanged, reuse its existing embedding.
• If the chunk changed, generate a new embedding.
• If only metadata changed, update metadata without regenerating the vector where possible.

This is particularly valuable when reindexing large document repositories after minor updates.

Important version fields​

Never reuse embeddings across incompatible:

• Embedding models
• Vector dimensions
• Text preprocessing logic
• Chunking transformations
• Language-specific transformations

Changing from one embedding model to another should produce a new namespace rather than silently mixing vectors.

7. Retrieval-result caching​

A retrieval cache stores the documents returned for a query.

Example:

{
  "query": "What is the parental leave policy?",
  "document_ids": [
    "hr-policy-2026:chunk-14",
    "hr-policy-2026:chunk-15"
  ],
  "scores": [0.91, 0.87],
  "retrieval_config": {
    "method": "hybrid",
    "top_k": 20,
    "filters": {
      "country": "UK"
    }
  },
  "knowledge_base_version": "2026-07-28"
}

A retrieval key should include:

canonical query or query embedding fingerprint
knowledge-base version
retrieval algorithm
index name
namespace
filters
top-k
hybrid weights
minimum score
query expansion version
user permission fingerprint

Why retrieval caching matters​

Even when a final answer cannot be reused, the same relevant documents may be useful for many similar requests.

For example:

What does the parental leave policy cover?

How many weeks of parental leave are available?

Who qualifies for parental leave?

The final answers differ, but the same policy chunks may be retrieved.

Approximate retrieval caching can reuse previously retrieved document sets for semantically similar queries, although similarity thresholds must be evaluated against retrieval recall. Research prototypes have demonstrated meaningful retrieval-latency reductions, but results depend heavily on the dataset and retriever. (arXiv)

Exact versus approximate retrieval caching​

Use exact retrieval caching when:

• Correctness is critical.
• Query filters are complex.
• The corpus changes frequently.
• User permissions differ.
• A small query change could alter the relevant documents.

Use semantic retrieval caching when:

• The corpus is relatively stable.
• Questions are heavily repetitive.
• Documents are broad FAQ or support material.
• You have measured recall under the proposed threshold.
• A fallback retrieval can run when confidence is low.

8. Reranker caching​

Cross-encoder and LLM-based rerankers can become an expensive part of RAG.

Cache individual query-document scores:

rerank_key = hash(
    query
    + document_chunk_hash
    + reranker_model
    + reranker_prompt_version
)

Example value:

{
  "relevance_score": 0.92,
  "reason": "The document directly explains refund eligibility.",
  "model": "reranker-v3"
}

Caching at the query-document-pair level is often better than caching the entire ranked list because individual scores can be reused when:

• top_k changes
• Additional documents are introduced
• A second-stage retriever returns overlapping documents
• Multiple agents investigate the same subject

Invalidate the result when either the query interpretation, chunk content or reranker version changes.

9. Context-construction caching​

After retrieval and reranking, many RAG systems perform additional processing:

• Deduplication
• Chunk merging
• Context compression
• Citation assignment
• Section ordering
• Token-budget allocation
• XML or JSON formatting
• Prompt-injection filtering

These outputs can also be cached.

A context cache key might include:

ordered document hashes
query
context-builder version
token budget
compression model
citation format
security policy version

Caching the context package can be especially useful where multiple models consume the same grounded evidence:

10. Final-answer caching in RAG​

Final-answer caching can produce the greatest cost and latency improvement because it bypasses the full pipeline.

However, the response must be treated as a derived artifact, not as the source of truth.

Every cached answer should retain provenance:

{
  "response": "Employees receive...",
  "source_documents": [
    {
      "document_id": "leave-policy",
      "version": "2026-07-01",
      "chunk_id": "chunk-42"
    }
  ],
  "knowledge_base_version": "kb-174",
  "generated_at": "2026-07-28T14:30:00Z",
  "model": "model-version",
  "prompt_version": "hr-assistant-v8",
  "validation_status": "grounded",
  "safety_status": "approved"
}

Before returning the answer, verify that:

• The underlying source documents still exist.
• Their versions remain current.
• The user still has access.
• The answer has not expired.
• The application policy version has not changed.
• Any time-dependent values remain valid.

Cache only validated answers​

A useful production rule is:

Do not cache every model response.
Cache only responses that pass quality gates.

Possible gates include:

• Citation coverage above threshold
• Groundedness score above threshold
• No unsupported numerical claims
• No prompt-injection indicators
• No safety-policy violation
• No unresolved tool errors
• Correct structured-output schema
• No personal data unless the cache is user-isolated

11. Caching in agentic applications​

Agentic applications are harder because their output depends on state, tools, observations and previous decisions.

A typical agent loop looks like:

Caching the final answer alone misses most of the reuse potential.

11.1 Tool-result caching​

Cache read-only tool calls such as:

• Search queries
• Product-catalog lookups
• Weather lookups for a short period
• Document retrieval
• Database reads
• Configuration fetches
• API metadata
• Schema discovery
• Repository file reads
• Identity-provider validation for a short period

Key:

tool_result_key = hash(
    tool_name
    + canonical_arguments
    + caller_scope
    + source_version
    + authorisation_context
)

Example:

{
  "tool": "get_product_documentation",
  "arguments": {
    "product": "A",
    "version": "4.2"
  },
  "result": {...},
  "expires_at": "2026-07-29T12:00:00Z"
}

Never blindly cache side-effecting tools​

These calls should not be treated as ordinary cacheable reads:

• Send email
• Transfer money
• Place an order
• Delete a record
• Update customer data
• Submit an application
• Book an appointment

For these operations, use idempotency keys, not response caching.

idempotency_key = hash(
    user
    + action
    + validated_parameters
    + transaction_scope
)

If the agent retries the same action, the system returns the original transaction result instead of executing the side effect again.

11.2 Plan caching​

Agents frequently create structurally similar plans.

Example:

1. Find the customer.
2. Retrieve the order.
3. Check delivery status.
4. Explain the result.

Rather than caching a fully instantiated plan, cache a reusable plan template:

{
  "intent": "check_order_status",
  "required_slots": [
    "customer_id",
    "order_id"
  ],
  "steps": [
    "authorise_customer",
    "retrieve_order",
    "retrieve_delivery",
    "compose_response"
  ],
  "tool_schema_version": "v12"
}

At runtime, the plan is instantiated with the current user, order and authorisation state.

Do not reuse a plan if:

• The available tools changed.
• Tool permissions changed.
• The policy changed.
• Required entities differ.
• The environment state is materially different.

11.3 Agent-node caching​

In a graph-based agent, cache deterministic node outputs independently:

Query classifier
Entity extractor
Policy router
Document retriever
Tool-schema selector
Context compressor
Output formatter

A graph node should declare:

CachePolicy(
    cacheable=True,
    key_fields=[
        "query",
        "tenant",
        "policy_version"
    ],
    ttl_seconds=3600
)

LangGraph’s server caching supports basic get/set caching as well as stale-while-revalidate behaviour for graph workloads. (Docs by LangChain)

11.4 Agent state is not the same as cache​

Agent checkpoints and memory preserve state so a workflow can continue.

A cache stores a reusable computation.

For example:

Checkpoint:
“The agent already contacted the customer and is waiting for approval.”

Cache:
“The product-price lookup for product X returned $100.”

Deleting a cache should reduce performance but not destroy the workflow. Deleting a checkpoint may make the agent unable to resume correctly.

12. Provider prompt caching for agent loops​

Long-running agents repeatedly send:

• System instructions
• Tool definitions
• Previous messages
• Planning rules
• Safety policies
• Output schemas

This makes them strong candidates for prefix caching.

Prompt ordering​

Use:

1. Stable system instructions
2. Stable safety and compliance policies
3. Stable tool definitions
4. Stable examples
5. Relatively stable conversation history
6. Dynamic tool outputs
7. Current user request

Do not put request-specific values near the beginning unless necessary.

A single dynamic timestamp, request ID or user-specific string near the start can break exact-prefix matching.

Tool schemas​

Large tool definitions can consume thousands of tokens. Keep:

• Tool order stable
• JSON property order stable
• Tool descriptions stable
• Schema formatting stable

OpenAI’s current documentation states that messages, images, tools and structured-output schemas can participate in prompt caching, provided the relevant prefix remains identical. (OpenAI Developers)

Anthropic supports automatic or explicit cache controls and provides short-lived caching for stable prompt sections. (Claude Platform)

A 2026 preprint evaluating long-horizon agentic workloads reported that prompt caching materially reduced cost and time to first token, while also finding that dynamic tool results and poorly structured context could reduce cache effectiveness. These results are promising but remain workload- and provider-dependent. (arXiv)

13. Cache invalidation strategies​

Cache invalidation is more important than cache storage.

13.1 TTL-based expiration​

Assign TTL according to volatility.

These are starting points. The correct TTL comes from the underlying source’s freshness requirement.

13.2 Version-based invalidation​

Instead of deleting every cache record when the knowledge base changes, include the corpus revision:

kb_version = 2026-07-28T15:00Z

Then:

rag:v8:tenant-1:kb-20260728-1500:...

When the corpus updates, new queries automatically use a new namespace. Old entries can expire naturally.

This is one of the safest RAG invalidation strategies.

13.3 Event-driven invalidation​

When a source document changes:

Maintain a reverse dependency index:

Every response cache entry records the source document IDs used to create it.

13.4 Stale-while-revalidate​

Stale-while-revalidate provides:

Fresh period:
Return cache immediately.

Stale-but-allowed period:
Return cached value and refresh asynchronously.

Expired period:
Block and fetch a fresh value.

This is useful for configuration, low-risk metadata and moderately changing content. It should not be used without careful analysis for critical financial, medical, legal or permission-sensitive information.

LangGraph Agent Server exposes this pattern through fresh and maximum-age windows. (Docs by LangChain)

13.5 Manual and tag-based invalidation​

Assign tags:

{
  "tags": [
    "product:A",
    "country:UK",
    "policy:refund",
    "document:refund-policy-v7"
  ]
}

Then operational teams can invalidate:

all responses based on refund-policy-v7

without clearing the entire cache.

14. Knowledge-base versioning is essential​

A common mistake is using only the user query as the cache key:

hash("What is the refund policy?")

Suppose the policy changes. The same query now requires a different answer.

Use:

hash(
    query
    + knowledge_base_version
    + policy_version
)

Microsoft’s RAG guidance recommends incremental, partial, trigger-based and versioned update strategies to keep retrieval indexes aligned with changing organisational content. (Microsoft Learn)

A corpus revision can be generated from:

• Index deployment ID
• Maximum document update timestamp
• Source-control commit
• Content manifest hash
• Database snapshot ID
• Policy-release version

For highly dynamic systems, use separate versions by domain:

{
  "hr_policy_version": "2026.07.2",
  "product_catalog_version": "58112",
  "support_articles_version": "2026-07-28T14:00Z"
}

This avoids invalidating unrelated cache entries.

15. Multi-tenant and permission-aware caching​

The most serious caching failure is returning information from one security scope to another.

Authorise before using the cache​

The flow must be:

Not:

Permission fingerprint​

Represent the user’s effective data access:

acl_fingerprint = hash(
    tenant_id
    + roles
    + departments
    + document_groups
    + region
    + clearance
)

Use this in exact keys and semantic-cache metadata filters.

However, avoid creating unbounded cache fragmentation from every individual user where role-level sharing is safe. Define stable access groups where possible.

Default isolation policy​

Use:

Shared cache within the same:
tenant
+ permission scope
+ locale
+ knowledge-base version

Avoid cross-tenant semantic caching unless the content is explicitly public and isolated from private context.

Self-hosted vLLM supports cache salts to isolate prefix-cache reuse across trust groups and reduce timing-based information leakage. (vLLM)

16. Prevent cache poisoning​

A malicious or incorrect response can become more damaging if cached and repeatedly served.

Potential attack:

1. Attacker submits a query containing prompt injection.
2. The model produces a compromised answer.
3. The answer is stored in the semantic cache.
4. Similar questions receive the compromised answer.

Mitigations:

1. Run prompt-injection detection before retrieval and generation.
2. Mark untrusted retrieved content.
3. Require grounding validation before cache insertion.
4. Do not cache responses containing tool errors.
5. Do not cache low-confidence answers.
6. Store safety and validation status with each entry.
7. Re-run output policy checks on cache hits.
8. Restrict who can populate shared caches.
9. Maintain provenance for every answer.
10. Support immediate tag-based invalidation.

A cached response should be treated as untrusted derived data, even if it passed validation when initially created.

17. Cache stampede protection​

When a popular cache entry expires, hundreds of requests may simultaneously trigger the same expensive RAG pipeline.

This is called a cache stampede.

Use request coalescing or a single-flight pattern:

Pseudo-implementation:

async with distributed_lock(cache_key):
    value = await cache.get(cache_key)

    if value is not None:
        return value

    value = await run_rag_pipeline()
    await cache.set(cache_key, value, ttl=3600)

    return value

Other strategies include:

• Soft expiration
• Stale-while-revalidate
• TTL jitter
• Pre-warming popular entries
• Background refresh before expiration
• Per-key rate limiting

Add random TTL jitter so thousands of related entries do not expire at the same instant.

18. Negative caching​

Negative caching stores “not found” or empty results briefly.

Example:

No document exists for product version X.

Without negative caching, agents may repeatedly search the vector store, database and external APIs for something known to be absent.

Use short TTLs because absence can change:

product lookup not found: 30 seconds
document not found: 5 minutes
external API unavailable: 5–30 seconds

Do not cache all errors in the same way.

Distinguish:

• Valid empty result
• Temporary timeout
• Permission denied
• Rate limit
• Invalid request
• Internal server error

A permission-denied result must be scoped to the relevant user or permission context.

19. Cache admission policies​

Not every response should enter the cache.

A cache-admission function should consider:

def should_cache(result) -> bool:
    return (
        result.status == "success"
        and result.groundedness >= 0.90
        and result.safety_passed
        and not result.contains_secret
        and not result.contains_unscoped_personal_data
        and not result.used_unstable_live_data
        and result.expected_reuse_probability > 0.10
    )

Avoid final-response caching when​

• The user requests creative diversity.
• Temperature is deliberately high.
• The answer depends on current prices or availability.
• The answer is highly personalised.
• The task performs side effects.
• The query contains secrets.
• The response contains short-lived credentials.
• The source returned uncertain or incomplete data.
• The user asks for “latest,” “current,” “today” or “right now.”
• The output is legally or financially consequential and freshness cannot be assured.

You may still cache safer intermediate stages such as embeddings, tool schemas or document parsing.

20. A production reference architecture​

21. Example cache-key design​

from __future__ import annotations

import hashlib
import json
from dataclasses import asdict, dataclass
from typing import Any


@dataclass(frozen=True)
class CacheContext:
    tenant_id: str
    acl_fingerprint: str
    locale: str
    knowledge_base_version: str
    prompt_version: str
    model_version: str
    retrieval_version: str
    safety_policy_version: str
    response_format: str


def stable_hash(value: Any) -> str:
    serialized = json.dumps(
        value,
        sort_keys=True,
        separators=(",", ":"),
        ensure_ascii=False,
        default=str,
    )
    return hashlib.sha256(serialized.encode("utf-8")).hexdigest()


def canonicalise_query(query: str) -> str:
    # Conservative normalisation: preserve case-sensitive entities,
    # numbers, dates, punctuation and negation.
    return " ".join(query.strip().split())


def build_response_cache_key(
    query: str,
    context: CacheContext,
    retrieval_filters: dict[str, Any],
) -> str:
    fingerprint = {
        "query": canonicalise_query(query),
        "context": asdict(context),
        "retrieval_filters": retrieval_filters,
    }

    return f"rag-response:v1:{stable_hash(fingerprint)}"

22. Simplified cache-aside implementation​

from __future__ import annotations

from typing import Any, Protocol


class ExactCache(Protocol):
    async def get(self, key: str) -> dict[str, Any] | None:
        ...

    async def set(
        self,
        key: str,
        value: dict[str, Any],
        ttl_seconds: int,
    ) -> None:
        ...


class SemanticCache(Protocol):
    async def search(
        self,
        query: str,
        filters: dict[str, Any],
        limit: int,
    ) -> list[dict[str, Any]]:
        ...

    async def store(
        self,
        query: str,
        response: dict[str, Any],
        metadata: dict[str, Any],
        ttl_seconds: int,
    ) -> None:
        ...


async def answer_question(
    *,
    query: str,
    context: CacheContext,
    retrieval_filters: dict[str, Any],
    exact_cache: ExactCache,
    semantic_cache: SemanticCache,
) -> dict[str, Any]:

    exact_key = build_response_cache_key(
        query=query,
        context=context,
        retrieval_filters=retrieval_filters,
    )

    # 1. Safe exact lookup.
    exact_result = await exact_cache.get(exact_key)
    if exact_result and cache_entry_is_valid(exact_result, context):
        return {
            **exact_result,
            "cache": {
                "hit": True,
                "layer": "exact_response",
            },
        }

    # 2. Approximate semantic lookup with hard boundaries.
    semantic_filters = {
        "tenant_id": context.tenant_id,
        "acl_fingerprint": context.acl_fingerprint,
        "locale": context.locale,
        "knowledge_base_version": context.knowledge_base_version,
        "prompt_version": context.prompt_version,
        "safety_policy_version": context.safety_policy_version,
        "response_format": context.response_format,
    }

    candidates = await semantic_cache.search(
        query=query,
        filters=semantic_filters,
        limit=3,
    )

    candidate = choose_safe_semantic_candidate(query, candidates)

    if candidate is not None:
        return {
            **candidate["response"],
            "cache": {
                "hit": True,
                "layer": "semantic_response",
                "similarity": candidate["similarity"],
            },
        }

    # 3. Execute full RAG or agent pipeline.
    result = await run_rag_agent_pipeline(
        query=query,
        context=context,
        retrieval_filters=retrieval_filters,
    )

    # 4. Cache only trusted responses.
    if should_cache(result):
        await exact_cache.set(
            exact_key,
            result,
            ttl_seconds=3600,
        )

        await semantic_cache.store(
            query=query,
            response=result,
            metadata=semantic_filters
            | {
                "source_document_versions": result.get(
                    "source_document_versions",
                    [],
                ),
                "validation_status": "approved",
            },
            ttl_seconds=3600,
        )

    return {
        **result,
        "cache": {
            "hit": False,
            "layer": None,
        },
    }

The important elements are not the specific database or framework. They are:

• Authorisation-aware keys
• Versioned knowledge
• Separate exact and semantic stages
• Hard metadata boundaries
• Candidate validation
• Quality-controlled cache insertion
• Provenance storage

23. Provider prompt caching configuration​

Provider caching works best when the request is stable at the beginning:

messages = [
    {
        "role": "system",
        "content": STABLE_SYSTEM_PROMPT,
    },
    {
        "role": "system",
        "content": STABLE_COMPLIANCE_POLICY,
    },
    {
        "role": "system",
        "content": STABLE_FEW_SHOT_EXAMPLES,
    },
    {
        "role": "user",
        "content": dynamic_user_question,
    },
]

Avoid:

messages = [
    {
        "role": "system",
        "content": (
            f"Request ID: {request_id}\n"
            f"Current timestamp: {current_timestamp}\n"
            f"{STABLE_SYSTEM_PROMPT}"
        ),
    }
]

The dynamic values at the beginning can destroy prefix reuse.

Current OpenAI documentation describes automatic caching for sufficiently long prompts and recommends keeping stable content first, using consistent cache keys and monitoring cached-token usage. (OpenAI Developers)

Google Gemini offers implicit and explicit context caching, while Anthropic provides automatic or explicit cache controls with configurable short-lived cache durations. (Claude Platform)

For self-hosted inference, vLLM’s automatic prefix caching hashes token blocks and their preceding prefixes so matching KV-cache blocks can be reused. (vLLM)

24. Advanced self-hosted RAG caching​

Application caches store embeddings, retrieval results or final responses. Self-hosted inference systems can go deeper and cache internal model states for repeatedly retrieved documents.

Suppose document chunk A appears in thousands of RAG requests:

A standard system repeatedly computes model attention states for Chunk A.

Research systems such as RAGCache propose caching the internal KV states of frequently reused knowledge and sharing them across RAG requests. Its experimental prototype reported improvements in time to first token and throughput relative to baseline systems, although this is primarily relevant to teams operating their own model-serving infrastructure. (arXiv)

Chunk-level caching is more difficult than prefix caching because retrieved chunks can appear:

• In different orders
• At different positions
• With different preceding context
• Alongside different documents

Cache-Craft is another research approach that attempts to reuse document-chunk KV caches while selectively recomputing enough state to preserve quality. (arXiv)

For most API-based enterprise applications, begin with application-level caching and provider prompt caching. Internal KV-state reuse becomes relevant when operating high-volume self-hosted models.

25. Observability and metrics​

Measure every cache layer separately.

Performance metrics​

Exact-response hit rate
Semantic-response hit rate
Embedding-cache hit rate
Retrieval-cache hit rate
Reranker-cache hit rate
Tool-cache hit rate
Provider cached-input-token ratio
P50 and P95 latency
Time to first token
Cache lookup latency
Cache write latency

Quality metrics​

Semantic-cache precision
False-positive cache-hit rate
False-negative rate
Groundedness
Citation correctness
Stale-response rate
Permission mismatch rate
User correction rate
Escalation rate after cached responses

Cost metrics​

LLM calls avoided
Embedding calls avoided
Vector searches avoided
Reranker calls avoided
Tool/API calls avoided
Input tokens read from provider cache
Cache storage cost
Cache embedding cost
Total cost saved

A useful formula is:

Net cache saving
=
avoided pipeline cost
− cache lookup cost
− cache-write cost
− embedding cost
− cache infrastructure cost
− quality incident cost

For provider prompt caching:

Saving
≈
cached input-token discount
+ reduced input-processing latency

For application response caching:

Saving
≈
embedding cost
+ retrieval cost
+ reranking cost
+ input-token cost
+ output-token cost
+ tool-call cost
− cache overhead

Application response caching usually has greater potential savings because it can skip output generation as well as the preceding pipeline.

26. Recommended implementation sequence​

Phase 1: Instrument the pipeline​

Before caching, capture:

• Normalised query
• Intent
• Retrieval results
• Model and prompt versions
• Tool calls
• Response latency
• Token usage
• Repeated-query frequency
• Source-document versions

Identify which stages actually repeat.

Phase 2: Add low-risk exact caches​

Begin with:

• Document embeddings
• Query embeddings
• Exact response cache
• Static configuration
• Tool schemas
• Deterministic classification
• Read-only API calls

Phase 3: Optimise provider prompt caching​

Reorganise prompts:

• Static instructions first
• Dynamic content last
• Stable tool order
• Stable schemas
• Stable formatting
• No early request IDs or timestamps

Measure provider-reported cached tokens.

Phase 4: Add retrieval and reranker caches​

Version keys against:

• Corpus revision
• Retrieval configuration
• ACL scope
• Model version

Phase 5: Introduce semantic caching in shadow mode​

Evaluate:

• Correct hit rate
• Threshold by intent
• Threshold by language
• High-risk exclusions
• Entity and slot matching

Phase 6: Add event-driven invalidation​

Connect document and policy updates to:

• Index updates
• Cache-tag invalidation
• Namespace version changes
• Reverse dependency cleanup

Phase 7: Cache agent nodes and tools​

Classify every tool as:

Read-only and stable
Read-only but volatile
Side-effecting and idempotent
Side-effecting and non-idempotent
Non-cacheable

Phase 8: Pre-warm frequent workloads​

Precompute:

• Popular FAQ responses
• Common retrieval results
• Stable document embeddings
• Common plan templates
• Frequently used reference-context prefixes

27. Recommended default strategy​

For a production enterprise RAG or agentic application, use this ordering:

1. Authenticate and resolve permissions.

2. Determine whether the request is cacheable.

3. Check an in-process exact cache.

4. Check a distributed exact cache.

5. Check a semantic response cache using:
   tenant filters,
   ACL filters,
   corpus version,
   prompt version,
   intent,
   entity/slot validation,
   and a calibrated confidence threshold.

6. On a miss, run the pipeline with:
   embedding caching,
   retrieval caching,
   reranker caching,
   context caching,
   and read-only tool caching.

7. Send a prefix-cache-friendly prompt to the model.

8. Validate grounding, citations, safety and freshness.

9. Cache only approved responses.

10. Store document provenance and dependency metadata.

11. Invalidate through document events, versions and TTLs.

12. Monitor false hits, not just hit rate.

The governing principle is:

Exact caches optimise safely. Semantic caches optimise approximately. RAG and agent caches must therefore be version-aware, permission-aware, provenance-aware and quality-gated.

A cache should make the application faster and cheaper without becoming an uncontrolled alternative source of truth.

Key takeaways​

Sources and further reading​

• Prompt caching — OpenAI
• Prompt caching — Anthropic Claude Platform
• Redis semantic cache — Redis
• Automatic Prefix Caching — vLLM
• Use server-side caching — LangChain / LangGraph
• Build Advanced RAG Systems — Microsoft Learn
• GPT Semantic Cache — arXiv
• Verified semantic caching trade-offs — arXiv
• Approximate caching for faster RAG — arXiv
• Don't Break the Cache — prompt caching for long-horizon agents
• RAGCache — knowledge KV caching for RAG
• Cache-Craft — chunk-cache management for RAG
• Related playbook: LLM Caching Ultimate Guide, Performance Engineering and AI FinOps, Retrieval-Augmented Generation

What to do next​

1. Instrument one production RAG/agent path for repeated queries, embeddings, retrieval sets and tool calls before adding caches.
2. Ship exact response + embedding caches with ACL and KB version in the key; enable provider prefix caching with static content first.
3. Shadow-deploy semantic caching; calibrate thresholds by intent; promote only categories with measured precision.