RAG Architecture

Executive summary

RAG architecture covers the full ladder—Naive → Advanced → Modular → Agentic (and hand-off to Graph RAG)—plus the two-tempo system design: index topologies, query paths, assembly contracts, ACL in the path, and offline vs online ops. Default enterprise delivery is Advanced/Modular hybrid RAG. Agentic RAG (single-agent, multi-agent, corrective, adaptive) is a bounded control loop around retrieve and tools—not a slogan for “smarter RAG.” Choose each type with eval evidence, owners and SLAs.

When to use this page

• After you understand the baseline pipeline in Retrieval-Augmented Generation and need architecture depth for a design review.
• When pilots “work on 50 docs” but production spans multiple domains, versions, tenants or latency tiers.
• When eval shows recall gaps that chunking alone cannot fix—multi-query, hierarchical index or iterative retrieve-then-read may be justified.
• When product asks for multi-hop answers without jumping straight to GraphRAG (see Graph RAG Architecture).
• Before locking cloud spend on re-embed, multi-index or always-on rerank.
• Pair with AI Evaluation and Quality Assurance for gates and Performance and FinOps for cost/latency envelopes.

Executive view

Fund architecture that names **owners and SLAs per stage**, not a single “AI platform” line. Challenge always-on complexity: multi-index, multi-hop and rewrite layers must earn their keep on labeled query classes.

Decision required: Which topology is the production default, what eval gap would force an upgrade, and what is the cost envelope per 1k successful answers?

Technical view

Draw the reference path with fusion, rerank k, assembly rules and fail-closed ACL. Separate offline ingest from online query path. Measure precision/recall and p95 by intent class before adding hops or indexes.

Prefer parent-child and query routing over premature GraphRAG; escalate to graph only when relationship recall stays broken after deep vector patterns.

Why this matters

Topic 13 teaches the RAG contract: authorised evidence, citations, ACL, eval. Deep architecture answers the partner question: how is the system put together so that contract holds at scale? Teams that skip this step ship a flat vector index, enable every retrieval trick in a framework tutorial, and discover in UAT that p95 is 4.2s, cross-BU leakage exists, and multi-hop questions still fail.

Architecture debt in RAG shows up as: one index for every corpus so product SKUs drown policy text; no parent-child so citations point at orphan sentences; rewrite without logging so faithfulness drifts after a prompt change; rerank on every FAQ hit so contact-centre SLAs break; ACL as a UI filter after generation; full re-embed every content tweak because delta ingest was never designed.

The AI Solution Engineer produces a reference architecture with stage owners, an index topology ADR, a query-path matrix by intent class, a context assembly contract, an online/offline eval plan, and a decision matrix that stops re-litigating “should we add HyDE?” every sprint. Weak designs look clever in notebooks. Strong designs survive FinOps, CISO ACL suites and on-call reindex incidents.

Connect to Retrieval-Augmented Generation for the baseline pipeline, Graph RAG Architecture when relationships dominate, RAG playbook for workshops, and Architecture Layer 6 for platform placement.

13.1 RAG architecture flowchart

Preview fits the page width — use the expand icon for the full flowchart.

Learn

RAG architecture types — where Agentic RAG sits

Architecture ladder (choose with evidence, not slogans):

Naive → Advanced → Modular → Agentic → Graph (13.2)

RAG architecture type ladder

Preview fits the page width — use the expand icon for the full flowchart.

Do not jump to agents. Each step adds cost and failure surface.

Type	How it works	Use when	Avoid when
Naive RAG	Embed query → top-k vector chunks → generate	Tiny FAQ pilots	Enterprise ACL, IDs, multi-hop
Advanced RAG	Query rewrite, hybrid BM25+vector, rerank, metadata filters, parent-child	Default enterprise grounded QA	You have not measured a naive baseline
Modular RAG	Composable modules + intent router (lookup / compare / refuse / tool)	Multi-domain corpora, mixed intents	Single homogeneous FAQ
Agentic RAG	Agent decides when/how to retrieve, may loop, call tools, critique	Multi-step research, mixed tools + corpus	Simple lookup with tight p95 SLA
Graph RAG	Relationship retrieve via graph (+ chunks)	Multi-hop / connection questions	Flat FAQ; no curator — see 13.2

Decision rule: Stay on Advanced/Modular until eval proves a gap. Agentic RAG is not “better RAG”; it is a control loop around retrieval and tools (Agentic AI).

Agentic RAG types

Flow: classify complexity → pick path (FAQ / Advanced RAG / single-agent / multi-agent) → plan → retrieve or tool → critique → budgets/HITL → cited answer or refuse.

Agentic pattern	Behaviour	Enterprise fit
Single-agent	One agent with retrieve + optional tools; ReAct-style loop	Default when agentic is justified
Multi-agent	Planner / retriever / critic (sometimes writer)	Only if single-agent eval fails; high ops cost
Corrective / self-RAG	Agent checks evidence; re-queries or refuses if weak	High hallucination risk domains
Adaptive	Classifier picks no-retrieve / one-shot / multi-step	Cost control at scale
Agentic + Graph	Agent chooses vector vs graph expand vs tool	Relationship + tool workloads — see 13.2

Hard bounds (non-negotiable):

• Max retrieve rounds and tool calls (e.g. 2–3)
• Token / £ budget per request
• Tool allow-list; mutating tools behind HITL
• Same ACL fail-closed as non-agentic RAG — agents must not bypass filters
• Offline golden set for agentic paths (task completion, citation, loop/timeout cases)

When not agentic: FAQ, SKU lookup, policy extract with stable Advanced RAG gates. Agents add latency and opaque failure modes.

Reference architecture — stages, owners and SLAs

RAG architecture is a pipeline with two tempos: offline (ingest → index) and online (query → answer). Every stage needs an owner and a measurable SLA. Document each stage with owner, SLA and failure mode before sign-off.

Stage	What “done” means	Typical owner	Example SLA
Ingest / parse	Idempotent job; provenance; OCR gates	Data engineering	Policy change searchable ≤24h
Chunk / metadata	Structure preserved; ACL fields present	ASE + corpus owner	Spot-check pass rate ≥98%
Index (dense + sparse)	Versioned snapshot; rollback tested	Search platform	Index health green; rebuild window agreed
Query transform	Logged; no factual injection	ASE	Rewrite A/B shows faithfulness Δ ≤1 pt
Retrieve + fuse	Hybrid tuned on labeled set	Search + ASE	Recall@k gate by intent class
Rerank	Optional; latency budgeted	ASE	p95 within tier budget
Context assembly	Dedup, pack order, refuse threshold	ASE	Empty/low evidence → structured refuse
Generate + cite	Citation IDs match retrieved set	ASE + product	Citation correctness gate
Evaluate	Offline CI + online shadow	Eval owner	Release blocked on gate fail

Worked example: A UK insurer servicing assistant targets p95 2.5s end-to-end. Budget: retrieve 200ms, fuse 20ms, rerank 400ms (top 40→8), assembly 30ms, LLM 1.6s, margin 250ms. Always-on cross-encoder on top-100 breaks the budget—architecture chooses top-40 or FAQ-tier skip-rerank.

Index topologies — flat, hierarchical, multi-index, hot/cold

Topology is an architecture decision, not a library default.

Topology	Shape	Use when	Avoid when
Flat corpus	One chunk index	Homogeneous FAQ or single product line	Multi-domain enterprises
Parent-child	Small child for retrieve; parent for generate	Need precise cite + surrounding section	Corpus already short cards
Hierarchical / summary index	Summaries retrieve → expand to children	Long reports, annual filings	Latency-critical FAQ
Multi-index by domain	Separate indexes (policy, product, ops) + router	Vocabulary clash across domains	Tiny pilots (<5k docs)
Versioned / effective-dated	Current primary + archive	Regulated superseded text	Static marketing copy
Hot / cold	Hot SSD/low-latency; cold cheaper	High QPS head corpus	Uniform low traffic

Multi-index routing: Classify intent (rules + small classifier) → query one or two indexes → merge with score normalisation. Mis-routing is a first-class failure mode—golden set must include “wrong index” traps.

Versioning contract: Primary index holds is_current=true only. Archive index serves audit queries with explicit as_of parameter. Rerank boosts current. Eval includes “superseded clause trap” queries.

Pitfall: Creating five indexes without a router and unioning top-k from all—latency and noise explode. Prefer router precision over naive federation.

Index topology router

Preview fits the page width — use the expand icon for the full flowchart.

This diagram covers vector / hybrid index topologies only. Relationship-heavy queries that need a graph path belong in Graph RAG Architecture—not as another index branch here.

Router mistakes are a first-class failure mode—golden sets must include wrong-index traps.

Advanced retrieval patterns — earn each hop

Start from hybrid BM25 + dense + metadata filters (topic 13 default). Add patterns only when a labeled intent class shows a measurable gap.

Hybrid fusion

• Score normalisation (min-max or RRF — reciprocal rank fusion)
• Tunable weights per corpus (SKU-heavy → BM25 weight up)
• Metadata pre-filter before expensive vector k (tenant, product, jurisdiction)

Multi-query and decomposition

• Expand one user question into N sub-queries (glossary + cautious LLM)
• Retrieve per sub-query; fuse; dedup by chunk_id
• Guardrail: sub-queries must not invent entity IDs or policy numbers

HyDE-style rewrite

• Generate a hypothetical answer passage, embed that for dense retrieve
• High risk of drift; log every rewrite; A/B faithfulness
• Prefer controlled glossary rewrite before open HyDE in regulated domains

Iterative / multi-hop retrieve-then-read

• Round 1: retrieve seed evidence
• Model extracts missing entities or follow-up questions
• Round 2: retrieve with constraints; hard cap rounds (e.g. 2) and token budget
• Stop on evidence sufficiency score or refuse

Tool-routed retrieval vs pure RAG

• Account balances, live prices, ticket state → API tools, not corpus RAG
• Architecture must route: RAG for documents; tools for systems of record
• Mixing live facts into the vector index creates stale-truth incidents

Online RAG query sequence

Preview fits the page width — use the expand icon for the full flowchart.

When not to deepen: If hybrid + structure-aware chunking already hits recall/faithfulness gates, stop. Deep patterns are cost and failure surface.

Context assembly — packing, dedup, refuse

Retrieval quality is wasted if assembly dumps noise into the prompt.

Assembly contract (minimum):

1. Dedup near-identical chunks (simhash / same parent section)
2. Pack order — highest rerank first; optional lost-in-middle mitigation (critical chunks at start/end)
3. Token budget — hard cap; drop lowest scores before truncating mid-chunk
4. Citation locators — every included chunk carries source_id, section, page/span, version
5. Refuse threshold — if top score < τ or zero candidates after ACL → structured refusal, no speculative generation
6. Delimiter schema — consistent tags so the model cannot confuse instructions with evidence (Prompt and Context Engineering)

Pitfall: Passing top-20 “just in case” when the model only uses three—raises cost and hallucination from contradictory clauses. Prefer smaller, higher-precision packs after rerank.

Security and tenancy in the path

Deep architecture treats ACL as a retrieve-time invariant, not a display filter.

Control	Requirement
Identity	SSO groups / roles on every query
Filter	Mandatory allowed_groups / tenant / BU predicates in search
Fail closed	Identity service down → no retrieve
Forbidden suite	CI fails if any restricted chunk appears in top-k
Provenance	Original URI + ingest hash for audit replay
Tombstones	Retractions remove or mark unsearchable within SLA
PII	Classification at ingest; high-sensitivity corpora may be browse-only

Multi-tenant: Shared index with mandatory tenant_id or per-tenant indexes. Golden set includes cross-tenant leakage attempts. Second-BU onboarding without this design forces expensive reindex.

Worked example: Wealth platform shared a service account at ingest → all chunks ACL=internal. Architecture fix: per-document ACL from source permissions graph, forbidden suite of 40 cases, UI banner when results filtered. See negative cases in topic 13 for incident cost.

Runtime vs offline — delta ingest, re-embed, eval gates

Concern	Offline	Online
Content change	Delta ingest by content_hash	Serve current snapshot only
Embedding model upgrade	Dual-write or blue/green index	Cutover after offline eval pass
Eval	Golden set in CI on PRs	Shadow traffic + faithfulness sampling
Cost	Amortise reindex	Cost per successful answer
Incident	Rollback to prior index tag	Kill-switch / degrade (skip rerank, FAQ cache)

Re-embed strategy: Never silent full re-embed in production path. Stage new vectors; run offline gates; switch alias; keep previous snapshot for rollback.

Online gates: Latency burn, citation correctness sample, ACL canary queries, cost per 1k. Tie to MLOps, LLMOps and Observability runbooks.

Architecture decision matrix — simple vs deep

Signal	Stay on Advanced / Modular RAG	Escalate
Corpus	Single domain, stable	Multi-domain → modular router
Queries	Lookup / FAQ	Compare / multi-hop → iterative or Graph (13.2)
Eval gap	Gates met	Recall gap ≥10–15 pts on a class
Latency	Budget holds with rerank	Tiered paths; FAQ skip rerank
Tenancy	Single BU	Multi-tenant / BU isolation
Autonomy need	Fixed pipeline enough	Agentic RAG with round/tool budgets
Relationships	Rare	Graph RAG
Ops maturity	Small team	Named corpus, search, eval owners

Use this matrix in ADRs. Default enterprise choice: Advanced/Modular hybrid + ACL fail closed. Escalate to Agentic or Graph only with evidence.

Caching in the deep RAG path

Caching is an architecture lever, not only a FinOps afterthought. Place caches deliberately so ACL and freshness cannot be bypassed.

Cache layer	What to store	Invalidation	Risk if wrong
Embedding	Query / chunk vectors	Model version change	Stale similarity
Retrieval	Top-k chunk IDs per normalised query + identity	Corpus version / ACL change	ACL leak if identity omitted from key
Rerank	Reordered IDs	Same as retrieval	Serving reordered restricted set
Answer	Final grounded response	Corpus tag + prompt version	Hallucinated “cached truth”

Rules: Cache keys must include tenant/BU + principal groups (or a hash of effective ACL). Prefer caching retrieval IDs over full answers for high-risk classes. See LLM Response Caching for RAG and Agents.

Observability and tracing for architecture reviews

RAG architecture without traces cannot be debugged in partner review. Minimum span set:

1. route.intent — class + indexes selected
2. query.transform — rewrite text + model/version
3. retrieve.hybrid — k, filters, fusion weights
4. rerank — in/out k, model, latency
5. assemble — token count, dropped chunks, refuse flag
6. generate — model, citation IDs emitted
7. verify.citations — pass/fail

Correlate with corpus_snapshot_id and index_alias. Online faithfulness sampling without these spans becomes unactionable opinion.

Anti-patterns catalogue (architecture)

Anti-pattern	Why it fails	Replace with
One giant index for all BUs	Vocabulary clash + ACL complexity	Multi-index or mandatory tenant filter
Rerank everything	p95 death	Tiered paths
Prompt-only “multi-hop”	Model invents links	Bounded retrieve rounds or GraphRAG
Embed live CRM fields	Stale operational truth	Tool route
UI-only security trim	Pen-test failures	Query-time predicates
Framework default chunk 512	Boundary splits on exclusions	Structure-aware + eval
“We’ll tune in prod”	No baseline	Offline golden gates first

Engagement deliverable pack (what good looks like)

When ASE exits design phase, the pack should include:

1. One-page reference architecture (offline/online)
2. Topology ADR signed by search + security
3. Intent-class matrix with owners
4. Latency and cost worksheets
5. Assembly + citation schema
6. Forbidden ACL suite (sample attached)
7. Offline golden set size and gates
8. Online degrade/rollback runbooks
9. Explicit GraphRAG adopt/defer (link or memo)

If any item is missing, the programme is still in prototype mode regardless of UI polish.

Frameworks and methods

CFRV for deep architecture

Extend corpus readiness into architecture readiness:

Letter	Ask
C Corpus	Domains, versions, OCR risk, owners
F Filters	ACL, tenant, jurisdiction fields on every chunk
R Retrieval	Topology, fusion, k, rewrite policy
V Verification	Citation check, refuse, offline/online gates

Intent-class query matrix

Intent class	Retrieval path	Generation style
Lookup	Narrow k, hybrid boost IDs	Short extractive
Compare	Multi-doc, optional second hop	Table / structured JSON
Procedural	Boost how-to chunks	Numbered steps + cites
Multi-hop document	Iterative retrieve-then-read	Cap rounds; cite each hop
Relationship-heavy	Escalate to GraphRAG eval	See graph companion
Out of scope / empty	Skip or empty retrieve	Refusal template
Live account data	Tool route, not RAG	No corpus generation

Latency budget worksheet

Decompose p95 by stage; assign degrade path (skip rewrite → skip rerank → cached FAQ). Never treat average latency as the design target.

Real-world scenarios

Scenario A — Global bank: multi-index policy and product corpus

Context: 180k policy pages + 40k product sheets. Flat index: product SKU queries polluted by policy boilerplate; recall@8 for SKU class 48%. Contact-centre p95 target 3s.

Intervention:

1. Split policy and product indexes; intent router (rules on SKU patterns + classifier fallback).
2. Parent-child on policy sections; product sheets chunked by tariff table with SKU metadata.
3. Hybrid fusion with higher BM25 weight on product index; rerank top 30→8 only on policy path; product FAQ tier skips rerank.
4. Forbidden ACL suite + tenant filter for ring-fenced private-bank corpus.

Measurable outcomes:

• SKU recall@8 48% → 93%
• Policy faithfulness on golden 76% → 91%
• p95 4.1s → 2.7s (FAQ tier 1.4s)
• Cross-tenant leakage attempts 0/120 in pen test

Lesson: Topology and routing beat “bigger embedding model” for multi-domain clash.

Scenario B — Telecom: iterative retrieve for plan comparison

Context: Consumer plan comparison (“which plan is cheaper if I roam 12 days in EU?”). Single-shot hybrid retrieved one plan page; missed fair-usage clause in separate annex. Complaint rate on AI answers +18% QoQ.

Intervention:

1. Intent class compare → force multi-query decomposition (price, fair usage, roaming).
2. Max two retrieve rounds; assembly requires ≥1 chunk per sub-topic or refuse “incomplete evidence.”
3. Citation verifier rejects answers that assert price without a locator on the price chunk.
4. Offline golden set of 80 compare queries; CI gate on unsupported-claim rate.

Measurable outcomes:

• Unsupported claim rate 14% → 3%
• Compare-class faithfulness 69% → 90%
• Average LLM tokens +22% but complaint-driven contacts −31%
• Decision: defer GraphRAG—document multi-hop closed the gap

Lesson: Iterative retrieve-then-read can postpone graph complexity when relations are still textual annexes.

Scenario C — Public sector: hot/cold indexes for statutory guidance

Context: Central government department indexes current statutory guidance (hot) and 15 years of archived circulars (cold). Analysts occasionally need historical interpretation; citizens never should see withdrawn text in the primary assistant. Flat index mixed eras; superseded guidance cited in 11% of citizen-facing answers during soft launch.

Intervention:

1. Hot index — is_current=true only; citizen and most staff paths.
2. Cold archive index — opt-in as_of / research mode with watermarked UI (“historical — not in force”).
3. Router blocks cold index for public channel; staff research role may federate with explicit flag.
4. Rerank boost + metadata filter on effective_to null/open.
5. Golden set: 40 “superseded trap” queries expecting current-only; 20 historical queries expecting archive + watermark.

Measurable outcomes:

• Citizen-facing superseded citation rate 11% → 0.4%
• Research-mode historical recall@10 72% → 90%
• Hot index size −63% vs flat; p95 retrieve −35%
• Comms pack for policy owners: “withdrawn ⇒ tombstone ≤2 hours”

Lesson: Versioned/hot-cold topology is a product and safety control, not only a storage optimisation.

Practice exercises

Primary exercise — RAG architecture one-pager (3 hours)

Brief: Mid-size insurer wants an adviser-facing assistant over underwriting guidelines (digital PDF) and product wordings (mixed Word + scanned schedules). Two BUs sharing a platform. Risk class: medium-high.

Tasks:

1. Draw reference architecture (offline + online) with owners and SLAs.
2. Choose index topology and justify in ≤1 page ADR.
3. Fill intent-class matrix for lookup, compare, procedural, refuse, live-data.
4. Specify hybrid + rerank parameters and a latency budget with degrade path.
5. Define assembly contract including refuse threshold and citation fields.
6. List 10 golden queries that would force multi-index or multi-hop if topology is wrong.

Acceptance criteria:

• ACL and bu_id appear on the retrieve arrow, not after generation
• At least one degrade path preserves SLA without silent quality loss
• ADR states what evidence would trigger GraphRAG evaluation

Stretch exercise — Architecture review pack (full day)

Brief: Same insurer; leadership asks why not “agentic multi-hop GraphRAG platform.”

Tasks:

1. Run decision matrix: simple hybrid vs deep patterns vs GraphRAG.
2. Design blue/green re-embed cutover with offline gates.
3. Write incident runbook: “p95 > SLA for 15 minutes” and “faithfulness −5 pts after ingest.”
4. Produce FinOps model per 1k queries at two QPS levels.
5. Facilitate a half-day review agenda (corpus, ACL, topology, golden set, CFRV).

Acceptance criteria:

• GraphRAG adopt/defer is evidence-based, not preference
• Rollback to prior index snapshot is testable
• Review output is a signed one-pager, not “more research”

Questions you should be able to answer

1. What is the default topology, and what eval gap upgrades it?
2. Who owns each pipeline stage and what is the SLA?
3. How are dense and sparse scores fused, and on which labeled set?
4. When is rerank skipped, and what quality risk does that accept?
5. How does the intent router choose indexes or tools?
6. What is the context assembly refuse threshold?
7. How is ACL/tenancy enforced at retrieve time?
8. What is the delta ingest key (source_id + hash), and tombstone SLA?
9. How do you cut over an embedding model upgrade safely?
10. What online metrics trigger degrade vs rollback?
11. Which query classes use iterative multi-hop, and what is the round cap?
12. Why is this not GraphRAG yet—or why is it?
13. Which RAG type is production default (Naive / Advanced / Modular / Agentic)—and why?
14. If Agentic: which pattern (single / multi / corrective / adaptive), and what are the round and tool caps?
15. What is cost per successful answer at target QPS?
16. How do superseded documents stay out of primary retrieve?
17. Where does RAG hand off to Agentic AI tool loops?

Negative cases — when RAG architecture fails

Failure mode	Symptom	Prevention
Complexity theatre	Every pattern on; gates flat	Decision matrix + ADR
Agent by default	Slow FAQ, opaque failures	Adaptive classifier; agent only on hard class
Unbounded agent loop	Cost/latency explosion	Round, token, £ caps + kill-switch
Router blindness	Wrong index every time	Router golden set
Unbounded hops	Latency/cost explosion	Hard round and token caps
Rewrite drift	Faithfulness drop after prompt tweak	Log + A/B + CI
Assembly stuffing	Contradictory chunks in prompt	Dedup + smaller pack
ACL after generate	Leakage in pen test	Filter in search query
Silent re-embed	Weekend outage, no rollback	Blue/green + snapshot
Tool facts in index	Stale balances in answers	Route live data to APIs
Average-latency design	p95 breaches SLA	Budget worksheet on p95
Graph leap	Unmaintained edges	Exhaust deep vector patterns first

Case study: federation without routing. A manufacturer unioned top-20 from six indexes per query. p95 6.8s; faithfulness 62%. Router to two indexes + RRF restored p95 2.9s and faithfulness 88%.

Case study: HyDE in claims. Hypothetical-answer embeddings retrieved plausible but wrong clause families; citation correctness 71%. Replaced with glossary rewrite; correctness 93% on same set.

Operating-model notes

Role	Deep-RAG focus	KPI
ASE	Topology ADR, assembly contract, intent matrix	Gates green on golden
Search platform	Index aliases, fusion infra, p95 retrieve	Rollback drill quarterly
Data engineering	Delta ingest, tombstones	Freshness SLA
Security	Forbidden suite, tenant tests	Zero leakage
Eval owner	Offline CI + online sample	Gate breach → block
FinOps	Cost per 1k / reindex	Within envelope

Workshop agenda (half day): topology options → latency budget → ACL path walkthrough → intent matrix → golden traps → ADR sign-off.

Capacity planning sketch

Architecture reviews ask whether the design holds at launch QPS. Answer with a transparent model:

Component	Drivers	Notes
Embed (query)	QPS × model latency	Cache normalised queries
Sparse retrieve	Index size, filter selectivity	Tenant filters reduce fan-out
Dense retrieve	k, ANN parameters	Trade recall vs latency
Rerank	Candidates × cross-encoder	Often dominates after LLM
LLM	Packed input + output tokens	Assembly discipline caps input
Ingest	Docs/day × parse + embed	Separate pool from online path

State assumptions (concurrency, cache hit rate, rewrite rate). Revisit monthly with FinOps. Under-provisioned rerank is a common silent quality cut when autoscaling is LLM-only.

Migration path from pilot to deep topology

1. Pilot — flat hybrid, one domain, SSO ACL, ≥50 golden queries
2. Prove gap — intent classes failing recall or latency
3. ADR — choose parent-child, multi-index, and/or iterative hops
4. Shadow — dual-run new path on a traffic percentage; compare gates
5. Cutover — index alias switch; keep prior snapshot for rollback
6. Reassess GraphRAG — only if the relationship class still fails

Skipping step 2 produces complexity theatre. Skipping step 4 produces weekend incidents.

Hand-off to agents and tools

RAG architecture remains a retrieve–assemble–generate system. When the user needs multi-step side effects (create ticket, update CRM), hand off to Agentic AI and Workflow Automation with RAG as a read tool inside a bounded workflow—not an open agent with raw index access. Agents that bypass assembly contracts and ACL filters recreate every failure mode in this chapter.

Worked example — writing the topology ADR

Title: Multi-index routing for Policy vs Product corpora
Context: Single flat index; SKU recall@8 = 48%; p95 = 4.1s.
Decision: Split indexes; rules+classifier router; policy path keeps rerank; product FAQ skips rerank.
Rejected: Larger embedding model only (does not fix vocabulary clash); naive six-index federation (latency).
Upgrade trigger: Relationship-class recall gap >15 pts after this topology → open GraphRAG spike.
Rollback: Previous flat alias retained 14 days.
Owners: ASE (router), Search (aliases), Security (ACL suite).

Copy this shape on every topology change so debates stay evidence-based.

Checklist before calling the architecture “deep”

• Offline and online paths drawn with owners
• At least one labeled class justifies each non-default pattern
• p95 budget with degrade path documented
• ACL/tenancy on retrieve predicates tested
• Assembly refuse threshold numeric
• Corpus snapshot IDs in traces
• GraphRAG explicitly adopted, deferred, or spiked—with date

If three or more boxes are unchecked, you still have a pilot with optional complexity—not a deep RAG architecture.

Related playbook content

• Retrieval-Augmented Generation — baseline pipeline, chunking, ACL, eval
• Graph RAG Architecture — when relationships need a maintained graph
• Prompt and Context Engineering — pack order and citation schemas
• Agentic AI and Workflow Automation — tool-routed paths beyond retrieve
• AI Evaluation and Quality Assurance — golden sets and regression
• MLOps, LLMOps and Observability — deploy, reindex, rollback
• Performance and FinOps — latency and cost budgets
• Data Engineering and Architecture — ingest and quality
• RAG — engagement playbook and checklists
• Architecture — Layer 6 knowledge and retrieval
• Evaluation and observability — metrics and gates
• LLM Response Caching for RAG and Agents — cache layers in the path
• How to use this Learning Map — reference depth standards

Output

Expected outputs for this topic:

• Reference architecture diagram — offline and online stages with owners and SLAs
• Index topology ADR — chosen layout, reject alternatives, upgrade triggers
• Intent-class query matrix — path per class including tool vs RAG
• Latency budget worksheet — p95 by stage plus degrade path
• Context assembly contract — pack, dedup, refuse, citation fields
• Online/offline eval plan — CI gates and shadow metrics
• Cost model — per 1k successful answers and reindex events

Practice checklist

• I separated offline ingest from online query path with owners
• I justified topology with an ADR, not a framework default
• I defined fusion, k values and when rerank is skipped
• I documented assembly refuse threshold and citation locators
• I enforced ACL/tenancy on the retrieve path with forbidden tests
• I specified delta ingest, tombstone and re-embed cutover
• I filled the simple vs deep vs GraphRAG decision matrix with evidence
• I linked design to Retrieval-Augmented Generation and RAG playbook

Key takeaways

• Climb the ladder deliberately: Naive → Advanced → Modular → Agentic → Graph.
• Default enterprise path is Advanced/Modular hybrid + ACL fail closed—not agents.
• Agentic RAG types (single, multi, corrective, adaptive) need round/tool budgets and HITL on mutating tools.
• ACL and tenancy belong in retrieve predicates; agents must not bypass them.
• Exhaust modular/iterative patterns before GraphRAG (13.2).
• Ship with ADRs, gates and rollback—demos without snapshots are not production architecture.

What to do next

1. Complete the primary architecture one-pager and name the RAG type you are proposing.
2. If multi-step autonomy is required, specify an Agentic RAG pattern with caps; else stay modular.
3. If relationship queries still fail, study Graph RAG Architecture.
4. Return to Retrieval-Augmented Generation for chunking and citation fundamentals.
5. Align release gates with AI Evaluation and Quality Assurance.