Retrieval-Augmented Generation

Executive summary

Retrieval-Augmented Generation (RAG) grounds LLM outputs in authorised evidence instead of parametric memory. Enterprise RAG spans the full pipeline: ingest and parse (including OCR limits), chunking and metadata, hybrid search with rerank and query rewrite, optional GraphRAG for relational questions, citations and ACL-enforced retrieval, then evaluation of faithfulness, precision, recall, latency and cost. A RAG system that retrieves well but cites poorly—or leaks restricted documents—fails audit and customer trust alike.

When to use this page

• When answers must reference internal policies, contracts, product specs, clinical pathways or regulatory text—not model training data.
• When hallucination risk is material and traceable citations are required for conduct, clinical or legal defensibility.
• When knowledge changes frequently and retraining a model is impractical.
• Before scaling a copilot from pilot to thousands of users with heterogeneous access rights.
• When search alone fails because users need synthesis across multiple documents.
• Pair with Prompt and Context Engineering for context packing and AI Evaluation and Quality Assurance for release gates.

Executive view

Ask for **citation correctness**, **access enforcement** and **eval thresholds** before scale—not demo answers that “sound right.” Challenge any RAG rollout without named owner for corpus freshness and ACL parity with source systems.

Decision required: Can restricted users retrieve zero forbidden chunks in penetration test—and does faithfulness meet the risk-class bar on golden queries?

Technical view

Design ingest idempotently; chunk for retrieval not for storage aesthetics; enforce identity filters at query time; measure precision/recall on labeled sets; monitor cost per successful answer.

Treat GraphRAG and hybrid search as tools for specific query classes—not default complexity. See [RAG Architecture](/learning-map/13-1-rag-architecture) and [Graph RAG Architecture](/learning-map/13-2-graph-rag-architecture) before adding hops or graphs. Fail closed when citations cannot be validated.

Why this matters

Generic LLMs answer confidently from weights learned months ago. In regulated enterprises, confident wrong is worse than I don’t know. RAG shifts the contract: the model may reason and summarise, but facts must come from retrieved chunks the user is allowed to see, with source identifiers auditors can replay.

RAG failures dominate production AI incidents: junior staff retrieving credit memos above their grade because vector search ignored ACL; answers citing superseded policy because ingest missed deprecation metadata; OCR garbage from scanned endorsements indexed as truth; chunk boundaries splitting “Section 4: exclusion” from the exclusion list; hybrid search returning keyword hits without semantic relevance; rerankers adding 800ms that destroy contact-centre SLAs; faithfulness at 71% on golden set while leadership expects 90% for customer-facing servicing.

The AI Solution Engineer designs RAG as a system, not a notebook demo: pipeline diagrams, chunking rationale tied to corpus anatomy, retrieval eval harness, attribution design, cost model and operational runbooks for reindex. Weak RAG produces pretty answers until legal asks “show the source”—then the programme stalls. Strong RAG produces measurable retrieval precision, citation correctness gates and ACL proofs that CISO can map to existing data classification.

This topic connects to RAG playbook for delivery patterns, Prompt and context engineering for pack order, and Evaluation and observability for offline and online metrics.

Learn

Ingest and parse — from source systems to normalised documents

Ingest moves authoritative content into a processing pipeline with provenance preserved.

Source types and considerations:

Source	Parse approach	Risk
PDF (digital)	Text extraction (pdfminer, etc.)	Columns, headers misordered
PDF (scanned)	OCR (Tesseract, cloud OCR)	Character errors, table breakage
Word / HTML	Structured parsers	Hidden revision text
SharePoint / Confluence	API export + HTML clean	Embedded macros, stale links
Email / tickets	Thread reconstruction	PII, access class drift
Database exports	ETL to documents	Loss of relational context

Ingest principles:

1. Idempotent jobs — document source_id + content_hash; skip unchanged
2. Provenance chain — store original URI, ingest timestamp, parser version
3. Classification at ingest — sensitivity label, ACL groups, effective/expiry dates
4. Deletion propagation — tombstone index when source retracted (GDPR, policy withdrawal)
5. Format normalisation — markdown or canonical JSON blocks before chunking

Parse quality gates: Sample 1% of new docs for human spot-check; block index if OCR confidence below threshold on critical corpora (e.g. policy numbers).

OCR limits — when scanned content misleads retrieval

OCR introduces character substitutions (0/O, 1/l), table collapse, and missing strikethrough on superseded clauses. Indexing OCR output without quality metadata causes RAG to retrieve nonsense that still embeds “similarly” to user queries.

Mitigations:

• OCR confidence scores per page; exclude pages below cutoff from embed index
• Dual index: OCR text + human-approved structured fields where high stakes
• Regex validation on known patterns (policy numbers, clause IDs)
• Flag answers drawing from low-confidence pages for mandatory human review
• Prefer digital source re-acquisition over re-OCR when legally available

When not to RAG scanned docs: If error rate > agreed threshold and no human QA budget, keep scanned content browse-only with warning—not generative source.

Chunking — split for retrieval, not for aesthetics

Chunking divides documents into retrieval units. Bad chunking is the silent killer of RAG precision.

Strategies:

Strategy	When to use	Risk
Fixed token window	Uniform prose	Splits mid-clause
Structure-aware	Headings, numbered sections	Needs reliable parse tree
Semantic	Long heterogeneous docs	Higher compute at index
Parent-child	Need context + precise cite	Two-stage retrieval
Sentence/paragraph	Short articles	Many tiny chunks, dilution

Metadata per chunk (minimum):

• source_id, document_title, section_path (e.g. "4.2.1 Exclusions")
• effective_date, expiry_date, jurisdiction, product_line
• acl_groups[], classification (public/internal/confidential/restricted)
• chunk_index, parent_doc_id, parser_version

Overlap: 10–20% token overlap reduces boundary cuts through bullet lists; measure recall impact vs index size.

Size tuning: Start 256–512 tokens for policy prose; evaluate on labeled questions—adjust before reranker tuning.

Metadata and filters — pre-filter before vector math

Metadata enables pre-filters shrinking candidate sets: jurisdiction=UK, product=motor, effective_date ≤ today.

Design rules:

• Align filter fields with how users actually ask (“UK motor” not only doc_type)
• Index fields used in filters; avoid post-filter dropping 90% after expensive vector search
• Stale metadata worse than none—automate sync from source CMDB or policy registry

Faceted browse for eval: Golden queries should declare expected filter path.

Hybrid search — lexical + semantic recall

Pure vector search misses exact identifiers (policy numbers, SKU, statute references). Pure BM25 misses paraphrase. Hybrid search combines scores.

Common fusion:

• Reciprocal Rank Fusion (RRF) across BM25 and dense retriever
• Weighted linear combo with weights tuned on dev set
• Two-stage: BM25 narrow to 500 → vector rerank top 50

When hybrid helps: Regulated IDs, tables, codes, internal acronyms with sparse embedding signal.

When hybrid hurts: Very small corpus where brute force is cheap; adds ops complexity without recall gain—measure before shipping.

Rerank — precision at the top of the pack

Cross-encoder rerankers score (query, chunk) pairs; improve top-k precision at latency cost.

Practices:

• Retrieve 50–100, rerank to 8–12 for LLM pack
• Cache rerank for repeated FAQ queries
• Track p95 latency budget—contact centre may cap at 300ms rerank
• Evaluate reranker on domain; generic rerankers underperform on legal tables

Fallback: If rerank SLA missed, degrade to hybrid top-8 with metric alert—not silent quality drop.

Query rewrite — bridging user language to corpus vocabulary

Users ask “cooling off period”; corpus says “statutory cancellation rights.” Query rewrite expands or normalises queries before retrieval.

Techniques:

• LLM rewrite with schema: {original, rewritten, filters_suggested}
• Synonym glossary from domain ontology
• HyDE (hypothetical document embedding)—use cautiously; can hallucinate off-domain terms
• Multi-query: retrieve for original + rewrite, merge with RRF

Guardrails: Rewrite must not inject factual claims; log rewrites for audit; A/B test faithfulness impact—rewrite sometimes hurts precision.

13.1 RAG architecture — when the baseline topology is not enough

Topic 13 is the pipeline contract. When you need index topologies, intent routing, iterative multi-hop retrieve-then-read, latency budgets and assembly contracts, use companion 13.1:

→ RAG Architecture

Escalate from simple hybrid RAG only when a labeled intent class shows a measurable gap—not because a framework tutorial enabled every pattern.

13.2 GraphRAG — when relationships matter

GraphRAG (broadly: knowledge graph + retrieval) helps when answers require multi-hop relations: “Which suppliers feed this BOM component under contract X?” or “Which policies reference regulator notice Y?”

When to adopt: Eval shows vector-only (and deep iterative RAG) fails on “connection” questions with a measurable recall gap—and a curator budget exists.

When to defer: Flat FAQ corpora; graph build cost unjustified; ontology unmaintained—graph rots faster than text index.

Architecture depth in companion 13.2 (schema, hybrid expand + chunks, relationship eval, adopt/defer memo):

→ Graph RAG Architecture

Citations and attribution design

Citation contract:

• Every factual sentence maps to ≥1 source_id + locator (section, page, span)
• UI shows clickable source; API returns structured citations array
• Citation correctness eval: cited chunk supports claim (automated + human)

Attribution anti-patterns:

• Citing document title only without section
• Citing chunk that doesn’t contain claimed number
• “Citation stuffing” from irrelevant retrieved set

Post-generation validation: NLI or LLM judge checks claim ⊆ union(retrieved chunks); fail closed or strip unsupported sentences.

ACL retrieval — identity at query time, not hope

Access control must enforce same rules as source system. Vector index containing restricted embeddings still leaks if query returns them to unauthorised users.

Patterns:

Pattern	Mechanism
Filter at query	Pass user groups[] / ABAC claims; Elasticsearch/OpenSearch filter
Partitioned index	Separate index per sensitivity (coarser but simple)
Document-level ACL field	Each chunk stores allowed groups; mandatory filter
Pre-retrieval scope	User selects corpus (jurisdiction/product) narrowing surface

Testing: Synthetic users in each role; forbidden query suite must return zero restricted chunks; log near-misses.

Fail closed: If identity missing, return no results—not public corpus default.

Incident pattern: Junior analyst retrieves executive compensation policy—root cause ACL field not populated at ingest from SharePoint permissions.

Evaluation — faithfulness, precision, recall, latency, cost

Offline eval dimensions:

Metric	Definition	Typical gate (medium risk)
Retrieval precision@k	Relevant chunks in top-k	≥0.85 @8
Retrieval recall@k	Gold chunk retrieved	≥0.90 @20
Faithfulness	Answer supported by retrieved	≥0.90
Citation correctness	Cited sources support claims	≥0.90
Answer relevance	Addresses question	≥0.85
Refusal appropriateness	Says unknown when no evidence	≥0.95 on negative set
Latency p95	End-to-end	SLA per channel
Cost per query	Embed + rerank + LLM tokens	FinOps budget

Golden set construction:

• 100–300 queries stratified: factual lookup, synthesis, negative (unanswerable), adversarial ACL
• Label expected chunks and acceptable answer patterns
• Refresh quarterly or on corpus major version

Online metrics: Thumbs down, escalation rate, “source not found” clicks, zero-citation rate, ACL deny rate.

Regression gates: Block release if faithfulness drops >2 points or citation correctness below threshold.

Frameworks and methods

RAG pipeline reference architecture

Document each box with owner, SLA and failure mode.

CHUNK-FILTER-RETRIEVE-VERIFY (CFRV)

Step	Check
Chunk	Structure-aware? Metadata complete?
Filter	Jurisdiction/product/date applied?
Retrieve	Hybrid + k sized for rerank?
Verify	Citations + faithfulness before user sees answer

Use CFRV in design reviews and incident retros.

Corpus readiness assessment

Before RAG build, score corpus 0–2 each (≥14/20 to proceed):

• Authoritative source identified
• Digital quality adequate or OCR plan funded
• ACL mappable to index fields
• Effective/expiry dates available
• Owner for updates named
• Volume and growth estimated
• PII handling agreed
• Duplicate/near-duplicate strategy
• Legal sign-off for generative use
• Eval query sample exists

Cost model template

Component	Unit cost driver	Mitigation
Embed at ingest	Corpus size × reindex frequency	Delta ingest
Embed at query	Query volume	Cache frequent queries
Rerank	Candidates × QPS	Reduce k, cache
LLM generation	Output tokens	Schema max length, summarisation
Storage	Vectors + text	Compression, tiered storage

Publish £/1k queries at expected load for CFO.

Incident triage for RAG quality drops

1. Corpus drift? (recent ingest job)
2. ACL regression? (identity provider change)
3. Embedding model change?
4. Chunk boundary change?
5. Prompt/schema regression?
6. Query volume spike on hard class?

Real-world scenarios

Scenario A — UK retail bank: ACL failure on credit memos

Context: Internal credit policy assistant; SharePoint corpus including restricted credit committee memos. Pilot used single service account ingest—no ACL on chunks. Junior credit analysts retrieved SVP-only limit guidance in UAT.

Intervention:

1. Re-ingest with per-document ACL from SharePoint permissions graph.
2. Mandatory allowed_groups filter on every query using analyst Entra ID groups.
3. Forbidden query suite: 40 cases; CI fails build if any restricted chunk in top-20.
4. UI banner when results filtered by ACL (“Some sources not visible at your access level”).

Measurable outcomes:

• Forbidden retrieval 12/40 → 0/40 in automated suite within two sprints
• Retrieval recall@20 on authorised golden set 88% → 91% (some restricted chunks removed from pool—expected)
• CISO sign-off for 3,200-user rollout
• Zero ACL incidents in nine-month production monitoring

Lesson: Ingest identity must mirror source permissions; service-account indexing is a liability.

Scenario B — Global insurer: 10k-page policy corpus chunking redesign

Context: Commercial lines policy wording; fixed 512-token chunks split exclusions from headings; faithfulness 67%; adjusters lost trust.

Intervention:

1. Structure-aware chunking on numbered clauses; parent-child index (summary chunk + detail child).
2. Metadata: product, endorsement_id, effective_date.
3. Hybrid search (BM25 + vector) for clause numbers.
4. Rerank top 50 → 10; query rewrite for synonym glossary (40 terms).

Measurable outcomes:

• Faithfulness on 180-query golden set 67% → 93%
• Retrieval recall@20 76% → 94%
• Median latency 4.8s → 5.4s (rerank cost accepted)
• Adjuster weekly active use 34% → 71% (eight-week adoption study)
• Estimated handle-time savings −11 minutes/complex claim (sample n=90)

Lesson: Chunk for legal structure, not token convenience.

Scenario C — NHS trust: OCR-heavy clinical admin pathways

Context: Scanned PDF pathways for referral criteria; OCR errors caused wrong eligibility summaries; clinical governance blocked rollout.

Intervention:

1. OCR confidence threshold 0.92; low-confidence pages human-verified queue (prioritised list).
2. Dual publication: verified structured YAML for top 50 pathways; RAG only on verified set initially.
3. Citation UI shows page scan thumbnail + text highlight for nurse verification.
4. Refusal schema when only low-confidence OCR available.

Measurable outcomes:

• Faithfulness on clinical golden set 58% → 89% post verified subset
• Nurse verification time 12 min/pathway average for backlog clearance
• Phase-1 rollout 14 wards with zero patient harm incidents attributed to assistant (governance audit)
• Full corpus coverage projected 11 months with funded QA FTE

Lesson: OCR limits are safety issues—quality metadata and refusal beat silent error.

Scenario D — Energy retailer: GraphRAG for contract–tariff dependencies

Context: B2B energy; answers needed linking master supply agreements, amendments and time-of-use tariff schedules. Vector-only RAG missed “which contracts still reference deprecated Schedule 7?”

Intervention:

1. Entity graph: Contract, Amendment, Schedule, TariffBand with supersedes / references edges.
2. Hybrid: graph traversal for relationship queries + chunk retrieval for text evidence.
3. Query router: classify relationship vs factual lookup (small classifier + rules).

Measurable outcomes:

• Relationship-query recall 41% → 87% on 60-query eval slice
• Legal review time for contract audit −38% (hours per audit pack)
• Graph maintenance 4 hours/week ontology curator—costed in runbook
• Incorrect tariff citation rate 9% → 2%

Lesson: GraphRAG earns its keep on multi-hop enterprise questions—with maintenance budget. Full design pattern: Graph RAG Architecture.

Practice exercises

Primary exercise — Chunking and eval for policy corpus (2 hours)

Brief: 10,000-page UK employment policy library for HR Q&A. Medium risk; SSO identity; mixed Word and scanned PDF.

Tasks:

1. Draw ingest pipeline with idempotency, ACL and tombstone handling.
2. Propose chunking strategy with example section and metadata schema (≥10 fields).
3. Specify hybrid + rerank parameters (k values, fusion method).
4. Write 20 golden queries with expected chunk IDs (5 negative, 5 ACL, 10 factual).
5. Define release gates: faithfulness, citation correctness, ACL suite, p95 latency.
6. Estimate cost per 1k queries with assumptions table.

Acceptance criteria:

• Chunk example preserves complete exclusion list in one retrieval unit or parent-child
• ACL test cases name role and forbidden document
• Negative queries expect structured refusal
• Gates numerically specified per risk class

Stretch exercise — End-to-end RAG architecture review (full day)

Brief: Customer-facing mobile tariff FAQ for European telecom—GDPR, marketing conduct, 200ms retrieval SLA budget for FAQ tier.

Tasks:

1. Complete CFRV worksheet for architecture.
2. Design query rewrite with guardrails and logging.
3. Plan online vs offline eval (what runs in CI vs shadow traffic).
4. Write incident runbook for “faithfulness dropped 5 points after ingest job.”
5. GraphRAG decision memo: adopt or defer with evidence criteria (use the template in Graph RAG Architecture).
6. Optional: draft a RAG topology ADR (flat vs multi-index vs parent-child) per RAG Architecture.

Acceptance criteria:

• SLA decomposition: retrieve + rerank + LLM within budget
• Customer-facing citation UX wireframe described in prose
• FinOps line item for embed reindex on tariff change day
• GraphRAG memo states curator hours or explicit defer

Questions you should be able to answer

1. What is authoritative source for this corpus and who owns updates?
2. How are documents parsed, and what OCR quality gates apply?
3. What chunking strategy matches document structure, and how was it validated?
4. Which metadata fields drive filters and ACL?
5. How is hybrid search fused and tuned?
6. What k values precede and follow rerank, and what is p95 latency?
7. When does query rewrite run, and how are failures logged?
8. Is GraphRAG in scope—what eval gap justifies it?
9. How are citations validated post-generation?
10. How is ACL enforced at retrieval, and how often is suite run?
11. What faithfulness and citation correctness bars gate release?
12. What is cost per successful query at target QPS?
13. How are deletions and expiries propagated to the index?
14. What happens when no relevant chunks retrieved?
15. How do online metrics trigger rollback or reindex?

Negative cases — when RAG fails

Failure mode	Symptom	Prevention
ACL vacuum	Restricted content in answers	Identity filter mandatory
Stale index	Superseded policy cited	effective_date + reindex SLA
OCR blindness	Wrong numbers in answers	Confidence gates + refusal
Chunk split	Missing exclusions	Structure-aware chunking
Over-trust rerank	Latency breach	SLA-aware degrade path
Rewrite drift	Off-domain retrieval	Rewrite logging + A/B
Citation theatre	Irrelevant sources cited	Post-gen verification
No negative eval	Confident hallucination on empty retrieve	Refusal golden set
Graph without curator	Wrong edges	Maintenance budget or defer
Cost surprise	Reindex bill 3× forecast	Cost model + delta ingest

Case study: the shared service account. A wealth manager indexed client suitability letters with backup service account. All chunks ACL=“internal.” Every adviser retrieved every client letter until trade press incident. Re-ingest + per-user ACL took 14 weeks and £420k rework.

Case study: faithfulness without citation check. Servicing bot hit 88% faithfulness on LLM-judge eval but 61% citation correctness—judge scored paraphrase without verifying source span. Customer complaints rose. Added citation verifier; effective launch faithfulness 91% with 94% citations.

Case study: hybrid ignored. Product code “AX-9912” retrieved poorly with vectors alone; BM25 not deployed. Support escalations +22% until hybrid added—recall@8 54% → 91% for SKU class queries.

Operating-model notes

Role	Responsibility	KPI
Corpus owner (business)	Authoritative content, expiry	Time-to-update index after policy change
Data engineer	Ingest pipeline SLA	Failed job rate <1%
Search platform	Index health, ACL mapping	p95 retrieve latency
ASE	Chunking, eval gates	Faithfulness on golden
Security	ACL suite, penetration test	Zero forbidden retrieval
FinOps	Cost per query	Within budget ±10%

Reindex runbook triggers: Policy major version; ACL schema change; embedding model upgrade; OCR backlog completion; regulatory event requiring corpus freeze.

SLA examples:

• Standard policy update → searchable within 24 hours
• Emergency regulatory withdrawal → tombstone within 2 hours
• Full re-embed → scheduled maintenance window with comms

Index lifecycle and embedding model changes

Changing embedding model requires full re-embed unless vendor guarantees compatible vector space (rare). Plan:

1. Shadow index build alongside production
2. Dual-query during cutover window OR hard cut with rollback index snapshot
3. Re-tune hybrid weights—BM25 unaffected but fusion weights may shift
4. Re-run full retrieval eval; expect 1–2 week stabilisation

Store embedding_model_version on each chunk; reject queries against mixed-version index without explicit routing.

Duplicate and near-duplicate handling

Policy corpora accumulate versioned PDFs with 90% overlap. Indexing all versions pollutes retrieval with superseded clauses.

Strategies:

• Canonical document ID — only latest effective version in primary index; archive index for audit queries
• Near-dedup at chunk — simhash/minhash collapse identical paragraphs
• Metadata precedence — rerank boost is_current=true

Eval must include “superseded clause trap” queries expecting current version only.

Personal data in corpora

HR, clinical and customer correspondence RAG requires DPIA alignment:

• Minimise indexed fields; redact at ingest where possible
• Query logging retention limits
• Right-to-erasure propagation to vector index (hard delete vs tombstone)
• Purpose limitation in UI (“internal HR guidance only”)

Fail closed if erasure request pending on document ID in query scope.

Serving architecture and caching

Query path optimisations:

• Cache embed(query) for FAQ top 500 queries (TTL 1 hour)
• Cache full retrieval+answer for exact query hash on public FAQ tier only—never for personalised ACL queries
• Precompute hot chunks for product launch windows

Capacity planning: Retrieve QPS × rerank cost dominates at scale; load-test with ACL filter enabled—filters change query plan performance.

Scenario E — Asset management: latency SLO breach drives architecture simplify

Context: Fund factsheet Q&A for advisers; p95 latency target 3s; initial stack: query rewrite + hybrid + cross-encoder + GPT-class generation averaged 6.8s p95.

Intervention:

1. Removed rewrite for factual lookup class (router by query length and pattern)
2. Reduced rerank 80→30 candidates; distilled reranker model
3. Smaller LLM for answer with schema; escalate to large only on low confidence
4. Aggressive embed cache on top 200 monthly queries

Measurable outcomes:

• p95 latency 6.8s → 2.9s
• Faithfulness 90% → 88% (accepted with steering sign-off)
• Adviser satisfaction 3.6 → 4.2 / 5
• Infra cost −41% at 12k QPS peak

Lesson: Eval gates include latency; simplify stack when quality margin allows.

Extended eval methodology — human labeling guide

For faithfulness labeling, train raters:

• Supported: claim appears in retrieved chunk text or direct logical entailment
• Unsupported: claim absent or contradicted
• Partial: core correct but number wrong—score as fail for regulated numerics

Double-label 10% of cases; adjudicate disagreements; track labeler drift monthly.

Automated NLI pre-label for triage only—human remains gold for release gate on Tier High.

Query understanding and intent routing

Before retrieval, classify query intent to tune path:

Intent class	Retrieval path	Generation style
Lookup	Narrow k, hybrid boost IDs	Short extractive answer
Compare	Multi-doc retrieve, GraphRAG optional	Table or structured JSON
Procedural	Boost how-to chunks	Numbered steps with citations
Out of scope	Skip retrieve or empty	Refusal template
Personal account	Route to API tool, not RAG	No corpus generation

Router may be small classifier, rules, or LLM with constrained enum output—eval router accuracy separately from answer faithfulness.

Attribution UX and audit replay

Users and auditors must replay why an answer appeared:

• Show chunk excerpts with highlight spans
• Link to source system viewer (SharePoint page, clause viewer)
• Log retrieval set hash with answer ID for later replay even if index updated
• “Report wrong answer” feeds golden set pipeline

Regulatory exams ask: “Show the evidence for this answer on date X.” Logging must support point-in-time retrieval snapshot or stored chunk text with version.

FinOps deep dive — when RAG becomes material spend

At 1M queries/month illustrative:

Component	£/1k queries (illustrative)	Notes
Query embed	£0.02	Cache reduces 40%
Rerank	£0.08	Dominant variable
LLM 2k out	£0.45	Schema short answers help
Ingest amortised	£0.01	Spikes on reindex

Optimisation order: reduce rerank k → shorten generation → cache → smaller embed model → last resort quality trade.

Review monthly with product owner—faithfulness gates may forbid some cuts.

Workshop facilitation — RAG design review agenda (half day)

1. Corpus readiness score (30 min)
2. ACL walkthrough with forbidden queries (45 min)
3. Chunking whiteboard on 3 representative docs (60 min)
4. Golden query brainstorm stratified by bucket (45 min)
5. CFRV sign-off and owners (30 min)

Output: architecture one-pager with gates and SLA—not open-ended “more research.”

Cross-reference to prompt engineering

RAG quality ceiling is set by Prompt and Context Engineering:

• How retrieved chunks are delimited in the prompt (<retrieved> tags)
• Whether schema requires citations matching retrieval IDs
• Context pack order placing top chunks at start/end

When retrieval eval passes but answer faithfulness fails, debug prompt pack and schema before re-embedding entire corpus.

Multi-tenant and BU isolation

Group enterprises may require tenant_id or bu_id filters alongside ACL:

• Shared platform index with mandatory tenant filter on every query
• Or separate indices per BU with federated search only for group roles

Golden set must include cross-tenant leakage attempts (User A @ BU1 must never retrieve BU2 chunk).

Failure to design tenant isolation early forces costly reindex when second BU onboarded.

Read-aloud checklist before RAG go-live

Use in release review meeting—each item owner says “done” or “blocked”:

1. Corpus owner named and refresh SLA signed
2. ACL forbidden suite 100% last run attached
3. Faithfulness and citation gates pass on current tag
4. OCR/low-confidence path documented if applicable
5. Incident runbook linked in ops wiki
6. FinOps cost per 1k within approved envelope
7. User-facing citation UX demonstrated live
8. Rollback to prior index snapshot tested

No “silent” go-live without recorded checklist in change ticket.

Summary matrix — RAG component decisions

Component	Default enterprise choice	Escalate when
Chunking	Structure-aware + metadata	Eval shows boundary failures
Search	Hybrid BM25 + vector	SKU/ID lookup dominates
Rerank	Yes, top 50→10	Latency budget allows
Query rewrite	Glossary + cautious LLM	Vocabulary mismatch in eval
GraphRAG	Defer (13.2)	Multi-hop recall gap >15 pts + curator funded
ACL	Filter at query, fail closed	Always—non-negotiable

Use matrix in architecture decision records to avoid re-debating defaults on every project.

Related playbook content

• RAG — delivery patterns, architecture workshops and checklists
• RAG Architecture — index topologies, advanced retrieval paths, assembly and ops
• Graph RAG Architecture — hybrid graph-plus-vector design, curator ops, adopt/defer
• Evaluation and observability — metrics, gates and monitoring
• Prompt and Context Engineering — context packing and schemas
• AI Evaluation and Quality Assurance — golden sets and regression
• Data Engineering and Architecture — ingest pipelines and quality
• Privacy, Legal and Compliance — PII and lawful basis for indexing
• Responsible AI and Governance — risk class and human oversight
• MLOps, LLMOps and Observability — deployment and reindex ops
• Performance and FinOps — latency and cost budgets
• LLM Response Caching for RAG and Agents — embedding, retrieval, reranker and answer caches
• How to use this Learning Map — reference depth standards
• 8D Framework — Develop stage RAG specifications

Output

Expected outputs for this topic:

• RAG architecture diagram — ingest through generation with owners and SLAs
• Chunking strategy memo — rationale, metadata schema, parent-child if used
• ACL design — filter mechanism, forbidden query suite, fail-closed behaviour
• Retrieval eval set — labeled queries with expected chunks and gates
• Attribution design — citation schema, UI/API behaviour, verification method
• Cost and latency model — per-query estimate and capacity assumptions
• Corpus readiness score — CFRV assessment before build sign-off

Practice checklist

• I documented ingest with idempotency, provenance and tombstone handling
• I chose chunking for document structure—not default token split only
• I specified hybrid retrieval, rerank k values and latency budget
• I designed ACL enforcement with forbidden-query tests
• I defined faithfulness, citation correctness and refusal metrics with thresholds
• I estimated cost per 1k queries with stated assumptions
• I completed ≥20 golden queries including negative and ACL cases
• I linked design to RAG playbook deliverables

Key takeaways

• RAG is a pipeline—ingest quality and ACL matter as much as embedding model choice.
• Chunking and metadata determine whether the right evidence is retrievable at all.
• Hybrid search, rerank and query rewrite are tuned on labeled data—not enabled by default.
• GraphRAG solves relationship queries when maintained; otherwise defer complexity—see Graph RAG Architecture.
• RAG architecture topologies earn complexity on labeled intent-class gaps—see RAG Architecture.
• Citations must be validated; faithfulness without citation correctness fails audit.
• Eval gates on faithfulness, precision/recall, latency and cost block launches that demos hide.

What to do next

1. Complete the primary chunking and eval exercise for the HR policy brief.
2. Deepen system design with RAG Architecture; use Graph RAG Architecture when relationship eval demands it.
3. Continue to Agentic AI and Workflow Automation when retrieval feeds tool-using agents.
4. Open RAG playbook for workshop templates and architecture reviews.
5. Pair with AI Evaluation and Quality Assurance before production cutover.