LLM inference is expensive because the model repeatedly recomputes attention state over the same tokens. Caching attacks that redundancy at every layer of the stack: inside one decode loop (KV cache), across concurrent requests on a serving GPU (PagedAttention / prefix caching), across API calls for stable prompt prefixes (prompt caching), and finally at the application layer when the answer itself can be reused (exact / semantic response caching).

This guide synthesizes the mechanics from Sebastian Raschka’s KV-cache and architecture work, Sankalp’s PagedAttention deep dive, provider prompt-caching docs, AWS caching patterns, and application-layer guides from Machine Learning Mastery, IBM, Latitude, ngrok, and related sources into one engineering playbook.

0. The caching stack at a glance​

These layers are complementary, not alternatives (Machine Learning Mastery). KV caching always runs. Prefix caching is usually the first production lever. Semantic caching pays off when query volume and similarity are high enough to justify embedding + vector search overhead (AWS Database Blog, Latitude).

Part I — Foundations: why inference needs a cache

1. Prefill vs decode​

Modern serving splits inference into two phases (Sankalp, NVIDIA NIM metrics):

1. Prefill — process the full prompt; compute Q/K/V for every input token; emit the first output token. Highly parallel; compute / FLOPs bound. Dominates time to first token (TTFT) on long prompts.
2. Decode — generate one token at a time. Each step needs the new token’s query against all prior keys/values. Memory bound: the engine must load a large KV cache from GPU memory for little arithmetic.

Agents and coding copilots often have a huge prefill:decode ratio (large context, short answers). That is exactly when prefix/prompt caching pays off: you skip most of the prefill bill when the shared prefix is already cached.

Figure: Shareable vs non-shareable prompt segments across document Q&A, multi-turn chat, and few-shot patterns. Source: How prompt caching works (via SGLang-style diagrams).

2. What attention actually needs each step​

For each token, attention forms three projections (ngrok, Raschka):

• Q (query) — “what am I looking for?”
• K (key) — “what do I offer for matching?”
• V (value) — “what content should be mixed in?”

Scores ≈ Q · Kᵀ, masked for causality, softmax → weights, then weights · V.

Critical caching insight: during autoregressive decode you only need the current token’s Q. Older queries are discarded. Older K and V must remain, because the new token still attends over the whole prefix (Raschka FAQ).

Temperature / top_p / top_k affect only the final sampling step after attention, so changing them does not invalidate a KV/prompt cache (ngrok).

Part II — Layer 1: the KV cache

3. The redundancy problem​

Without a cache, generating “Time flies fast” reprocesses “Time flies” on every step:

Figure: Token-by-token generation. Source: Understanding and Coding the KV Cache (Sebastian Raschka).

Figure: Redundant recomputation of the same prefix. Source: Raschka.

When a new token arrives, prior keys/values have not changed—recomputing them is pure waste:

Figure: Same K/V for “Time” and “flies” after adding “fast”. Source: Raschka.

4. How the KV cache works​

1. Prefill the prompt; store K and V for every layer.
2. For each new token, compute K/V only for that token.
3. Append to the cache.
4. Attend with the new Q over cached K/V.

Figure: Side-by-side generation with and without KV cache. Source: Raschka / FAQ.

Complexity intuition (Raschka):

• Without cache: cumulative attention work grows roughly O(n²) over a sequence of length n.
• With cache: each token’s K/V computed once → roughly O(n) incremental work, at the cost of O(n) memory growth for the cache.

Educational GPT (~124M) generating 200 tokens: ~5× speedup on Apple M4 CPU in Raschka’s readable torch.cat implementation.

5. Minimal implementation pattern​

Conceptually (PyTorch-style, from Raschka’s tutorial):

# Inside MultiHeadAttention
self.register_buffer("cache_k", None)
self.register_buffer("cache_v", None)

def forward(self, x, use_cache=False):
    keys_new = self.W_key(x)
    values_new = self.W_value(x)
    queries = self.W_query(x)

    if use_cache:
        if self.cache_k is None:
            self.cache_k, self.cache_v = keys_new, values_new
        else:
            self.cache_k = torch.cat([self.cache_k, keys_new], dim=1)
            self.cache_v = torch.cat([self.cache_v, values_new], dim=1)
        keys, values = self.cache_k, self.cache_v
    else:
        keys, values = keys_new, values_new
    # ... attention with queries, keys, values

Operational rules:

• Reset caches between independent generations (stale K/V → nonsense).
• Track position indices so new tokens align after the cached prefix (current_pos).
• In generation with cache: prefill full prompt once, then feed only the new token each step.

Figure: Cache buffers in attention code. Source: Raschka.

6. Production hardening of Layer 1​

Memory formula (float16 example for a 7B-class layout, from Sankalp):

kv_bytes ≈ 2 × layers × kv_heads × head_dim × seq_len × bytes_per_elem

Roughly ~0.5 MB/token → ~512 MB at 1K context per request before concurrency. At 100 concurrent long contexts, KV alone can dominate GPU RAM.

Part III — Shrinking the KV cache (architecture)

Modern open-weight models treat KV-cache size as a first-class design constraint for long-context agents and reasoning (Recent Developments in LLM Architectures, Big LLM Architecture Comparison).

7. Attention variants that reduce cache footprint​

Figure: GQA reduces KV heads vs full MHA. Source: Raschka architecture articles.

Figure: Multi-Head Latent Attention compresses what is cached. Source: Raschka / DeepSeek discussion.

Figure: Cross-layer KV sharing (Gemma 4 E2B/E4B). Source: Recent Developments….

Takeaway for solution engineers: when choosing models for long-context agents, compare not only quality benchmarks but KV bytes per token and attention pattern (GQA vs MLA vs hybrid window). That number drives GPU concurrency and cost more than raw parameter count.

Part IV — Layer 2: PagedAttention and automatic prefix caching

Layer 1 caches within one request. Production engines must also:

• avoid GPU fragmentation from per-request contiguous KV allocations;
• share identical prefixes across concurrent users;
• schedule prefill vs decode under load.

That is the story of PagedAttention (vLLM) and automatic prefix caching (Sankalp, vLLM paper).

8. Why naive KV allocation fails at scale​

Figure: One big contiguous KV slab per request. Source: Sankalp.

Classic OS problems appear on the GPU:

• Internal fragmentation — allocate for max length; actual length much shorter.
• External fragmentation — finished requests leave holes; a new long request cannot find a contiguous slab even if free bytes exist.
• Redundancy — 100 users with the same system prompt → 100 copies of the same KV.

Figure: Fragmentation in KV cache memory. Source: Sankalp / vLLM paper diagrams.

9. PagedAttention: OS paging for KV tensors​

vLLM pre-allocates a pool of fixed-size blocks (often 16 tokens each). Logical token positions map to blocks; a block table maps logical → physical GPU blocks—like a page table.

Figure: Paging inspiration. Source: Sankalp.

Each KVCacheBlock tracks roughly: block_id, ref_cnt (how many live requests share it), and a content block_hash for prefix lookup.

Figure: Block reuse across requests. Source: Sankalp.

10. Block hashing and longest prefix hit​

Hashes are parent-chained:

hash(block 0) = H(NONE, tokens[0:16], extras)
hash(block 1) = H(hash(block 0), tokens[16:32], extras)
...

Why chain? Causal attention: token 32’s K/V depend on tokens 0–31. Matching block 2’s hash implies blocks 0–1 match. Independent per-block hashes would be unsafe.

Scheduler flow:

1. Hash full blocks of the new request.
2. find_longest_cache_hit() walks block 0 → N until first miss.
3. Prefill only the miss suffix; attach hit blocks via the block table (ref_cnt++).

Figure: Cross-request prefix reuse while request 1 is still decoding. Source: Sankalp.

Mental model correction: prompt caching is content-addressed, not “per conversation.” User B can hit blocks written by User A if the token prefix is identical. Any change early in the prefix breaks the entire hash chain from that point onward.

SGLang’s RadixAttention solves the same problem with a radix tree over prefixes instead of vLLM’s block hash map—same product outcome, different data structure.

11. Provider “prompt caching” = selling Layer 2​

When OpenAI / Anthropic / Bedrock advertise prompt or context caching, they are productizing cross-request KV reuse for matching prefixes—not caching the final chat reply (ngrok, IBM, MLM).

Evidence: identical prompts with caching still return different samples (temperature), while usage shows cached input tokens.

Typical economics (order-of-magnitude; verify current rate cards):

Default retention is often minutes in VRAM; some models offer hours-long retention by spilling KV to GPU-local SSD (Sankalp).

Figure: Three complementary inference caching types. Source: Machine Learning Mastery.

Figure: Prefix / prompt caching across requests. Source: MLM.

Part V — How to hit the prefix cache (application design)

12. Golden rule: longest stable prefix, byte-identical​

Prefix caching requires exact token identity from position 0 until the first difference. One trailing space, reordered JSON key, or moved timestamp kills everything after the divergence (OpenAI / Anthropic / Manus guidance summarized by Sankalp, IBM).

Recommended message order:

1. System instructions (stable)
2. Tool / function definitions (stable; avoid mid-session mutation)
3. Few-shot examples / policy docs (stable)
4. Large reference documents (stable when possible)
5. Conversation history (append-only)
6. Latest user turn / dynamic fields LAST

Figure: OpenAI-style prompt structuring for cache hits. Source: Sankalp (via OpenAI docs imagery).

13. Practical checklist (agents & tools)​

Figure: Context-engineering tips that improve cache hit rate. Source: Sankalp / Manus blog.

14. What prompt caching is not​

Part VI — Layers 3–4: request/response caching

When the business answer can be reused, skip the model.

15. Exact vs semantic response caching​

Figure: Embed → vector search → return or call LLM. Source: MLM.

Figure: Semantic cache-augmented query workflow. Source: AWS Database Blog.

Storage options (AWS, Latitude)​

• In-memory / MemoryDB / Redis / Valkey — sub-ms; exact or vector features.
• SQLite / local disk — simple single-node apps.
• DynamoDB / OpenSearch / pgvector / Pinecone — distributed exact or semantic stores.
• LangChain / GPTCache-style adapters — wire exact + semantic backends quickly for prototypes.

Multilayer example​

1. Exact Redis for identical FAQ within minutes.
2. Regional Valkey for common policy answers.
3. OpenSearch / MemoryDB semantic layer for paraphrases.
4. Bedrock / OpenAI / Anthropic prompt cache for the long system+RAG prefix underneath.

16. Invalidation and safety​

Caching without invalidation creates confident wrong answers.

Heuristic from AWS: if you cannot apply caching to a meaningful share of calls (they cite ~60% as a rule of thumb), prefer prompt design / streaming / smaller models over complex cache ops.

Version the cache key with: model_id, prompt_template_version, tool_schema_version, kb_version, and sampling params that affect the stored answer (for response caches).

Part VII — Decision framework and metrics

17. Choose the right lever​

18. Metrics that matter​

Instrument provider usage.cache_read_input_tokens / cache_creation_* fields (names vary) and your own Redis hit counters.

19. Negative cases (do not ignore)​

• Dynamic dates / user IDs near the start of the prompt → permanent misses.
• Truncating or rewriting history mid-conversation → breaks prefix chain.
• Semantic cache threshold too loose → confident wrong answers.
• Caching personalized or regulated content without tenancy → data leakage across users (also a multi-tenant KV-sharing research concern).
• Caching before guardrails → poison the cache with toxic or PII-laden outputs.
• Expecting prompt cache to make outputs identical → it does not.
• Paying Anthropic-style write premiums for prefixes that never get re-read.

Part VIII — End-to-end reference architecture

Implementation order for most enterprise apps:

1. Restructure prompts for prefix stability; enable provider prompt caching / Bedrock prompt cache / vLLM automatic prefix caching.
2. Add exact response cache for truly identical calls.
3. Add semantic cache only if paraphrases dominate and you can measure wrong-hit rate.
4. Pick models with efficient KV designs for long-context workloads.
5. Wire FinOps dashboards to hit rates and cost per task.

Key takeaways​

Sources and further reading​

• Understanding and Coding the KV Cache in LLMs from Scratch — Sebastian Raschka
• What is a KV cache? — Raschka FAQ
• The Big LLM Architecture Comparison — GQA, MLA, MoE
• Recent Developments in LLM Architectures — KV sharing, CCA, compressed attention
• How prompt caching works — PagedAttention and Automatic Prefix Caching — Sankalp
• Prompt caching: 10x cheaper LLM tokens, but how? — ngrok / Sam Rose
• The Complete Guide to Inference Caching in LLMs — MLM
• Optimize LLM response costs and latency with effective caching — AWS
• What is Prompt Caching? — IBM
• Ultimate Guide to LLM Caching for Low-Latency AI — Latitude
• Efficient Memory Management for LLM Serving with PagedAttention — vLLM paper
• Related playbook: Build a Large Language Model from Scratch, LLM Response Caching for RAG and Agents, Performance Engineering and AI FinOps

What to do next​

1. Audit one production prompt: move all dynamic fields to the end; enable provider prefix caching; measure cache_read tokens for a week.
2. Add an exact-match FAQ cache with TTL + tenant namespace before investing in semantic caching.
3. For self-hosted serving, verify automatic prefix caching is on in vLLM/SGLang and watch block hit metrics under concurrent load.