Data Engineering and Data Architecture

Executive summary

Every AI failure you will see in enterprise delivery traces back to data long before it traces back to model architecture. Data engineering and architecture capability lets an AI Solution Engineer inventory sources, score readiness, design ingestion and storage patterns, enforce quality and lineage, and gate pilots when corpora are stale, ownerless, or legally unusable. Without this skill, teams ship impressive demos on curated samples and collapse when production corpora, ACLs, and freshness SLAs appear.

When to use this page

• A sponsor asks for RAG or a copilot before anyone has audited the knowledge corpus.
• Discovery mentions "we have data in the lake" but no one can name freshness, owner, or quality metrics.
• ML or GenAI pilots succeed in notebooks but fail when connected to operational stores.
• Compliance or privacy asks where training and inference data came from—and the team cannot answer.
• Integration discussions start with APIs while document PDFs, email threads, and tickets remain un-ingested.
• You need to compare lake, warehouse, lakehouse, and vector store roles for a target use case.
• Feature stores or batch pipelines are proposed without lineage to model inputs or retrieval chunks.
• Stage 2 feasibility review is due and data is the weakest link on the RAID log.

Executive view

Focus on **readiness scores**, **go/no-go gates**, and scenario outcomes with £ and risk impact. Ask: who owns the data, how fresh is it, and what happens if we answer from the wrong version?

Technical view

Build inventories, lineage sketches, pipeline designs, and quality rules before model selection. Treat unstructured ingest and ACL-aware retrieval as first-class architecture—not a "later" workstream.

Why this matters

AI Solution Engineers are accountable for feasibility, not only capability. Sponsors hear "we have years of data in Snowflake" and assume GenAI will work. In practice, availability ≠ readiness: tables may exist without join keys, PDFs may lag product launches by quarters, and SharePoint folders may contain three conflicting policy versions with no authoritative owner. Data engineering literacy is how you translate storage existence into decision-grade evidence—freshness SLAs, quality dimensions, lineage, consent, residency—and refuse pilots that would automate confusion at scale.

For retrieval use cases, the corpus is the product. Chunking and embeddings cannot fix deprecated content, missing ACL enforcement, or ingestion that runs annually when policies change weekly. For prediction use cases, feature pipelines that leak future information or silently drop nulls destroy trust faster than a wrong chatbot answer. Data architecture choices—warehouse vs lake vs lakehouse, batch vs streaming, CDC vs nightly dump—determine whether monitoring and retraining are even possible.

Data work is also the long pole in enterprise AI. Legal review for new processing, MDM alignment, DLP on exports, and data-owner prioritisation routinely take 8–16 weeks while model teams idle. Surfacing readiness gaps in week one prevents £200K–£500K spend on pilots that cannot reach production. This topic pairs with the Data & Knowledge Guide for delivery patterns and with topic 13 (RAG) for retrieval-specific design.

Learn

Data stores and their roles

Definition. Enterprise AI sits on a portfolio of stores, each optimised for different access patterns: operational RDBMS (transactional truth), data warehouse (curated analytics), data lake (raw and semi-structured at scale), lakehouse (ACID tables over object storage), NoSQL/document (flexible schemas), graph (relationship traversal), vector (similarity search), and specialised feature stores (ML serving consistency).

Engagement use. Map each source in the inventory to a store type and intended consumer: batch training, real-time inference, RAG retrieval, or human audit. Do not assume one lake holds everything ready for AI—lakes often hold land-only data without quality contracts. Warehouses hold conformed dimensions suitable for tabular ML; lakes hold call transcripts and PDFs needing parse pipelines. Vector stores hold embeddings but not authoritative text unless paired with source-of-truth versioning.

Pitfalls.

• Treating the data lake as production-ready without bronze/silver/gold discipline.
• Duplicating corpora into vector DB without sync strategy—embeddings drift from source.
• Using warehouse star schemas directly for unstructured RAG without document layer.
• Ignoring graph stores when relationship queries ("who approved this policy version?") dominate.

Worked example. Insurance claims assistant needs: (1) warehouse tables for claim status and SLA metrics—structured features; (2) lake PDF folder for policy wording—unstructured RAG after OCR and section parsing; (3) vector index over approved policy sections only—gold layer; (4) operational claims DB not copied to lake for training—access via governed API with row-level security. Each store has owner, refresh cadence, and PII classification.

Pipelines: ETL, ELT, streaming, and CDC

Definition. Pipelines move and transform data from source to consumer. ETL transforms before load; ELT loads raw then transforms in the target; streaming processes events continuously; CDC (change data capture) propagates inserts/updates/deletes incrementally rather than full reloads.

Engagement use. Choose pipeline pattern from freshness requirement and source capability. Nightly batch suffices for annual policy manuals; CDC required when RAG must reflect same-day product changes. Document orchestration (Airflow, Dagster, cloud-native), idempotency, backfill strategy, and failure alerting. For AI, pipelines must emit lineage metadata (run ID, source version, row counts, quality check results).

Pitfalls.

• Full-table nightly dumps on billion-row tables—cost and latency explode.
• No dead-letter queue for failed parse jobs—silent corpus gaps.
• Streaming without ordering guarantees for event-sourced features.
• Pipeline success = job green while quality checks failed (empty extract).

Worked example. Retail bank product FAQ assistant: CMS publishes HTML articles via webhook → streaming ingest to bronze blob → normalize to markdown in silver → chunk + embed to vector gold on publish event. Legacy PDFs: weekly batch OCR pipeline with manual QA sample (2% pages human-reviewed). CDC from CMS is source of truth; PDF batch tagged legacy with lower retrieval weight.

Data quality dimensions

Definition. Data quality is fitness for purpose across dimensions: accuracy (correct values), completeness (required fields present), consistency (same meaning across systems), timeliness (fresh enough for decision), validity (conforms to rules), uniqueness (no unintended duplicates), and accessibility (legal and technical use permitted).

Engagement use. Define quality rules per consumer. RAG on compliance policies requires accuracy and timeliness above all; fraud ML may prioritise completeness of transaction joins. Implement checks at pipeline boundaries: reject vs quarantine vs flag. Publish quality scorecards to data owners monthly—AI teams inherit scores in readiness model.

Pitfalls.

• Single "quality %" without dimension breakdown—masks stale-but-accurate corpus.
• Rules defined by IT without SME validation—legal text "valid" by regex but wrong meaning.
• Cleaning training data differently from serving data—train/serve skew.
• Ignoring label quality in supervised sets—garbage labels beat garbage features.

Worked example. Knowledge corpus quality dashboard: completeness (% docs with required metadata: owner, effective date, product), timeliness (median lag from publish to index), consistency (duplicate title detection), accuracy (SME sample 50 chunks/quarter, target ≥98% faithful to source). Pilot gate: timeliness ≤7 days and accuracy ≥95% on sample.

Lineage and metadata

Definition. Lineage documents data origin, transformations, and downstream consumers—enabling impact analysis ("policy v3.2 changed—what answers break?") and regulatory explanation. Metadata catalogues schemas, classifications, owners, and SLAs.

Engagement use. Minimum viable lineage for AI pilot: diagram from source system → ingest job → store → chunk/embed → model/retrieval → user-facing answer. Integrate with catalog tools (Collibra, Alation, Purview) where they exist; do not wait for enterprise perfection—maintain engagement-level lineage doc. For each retrieval answer, log source document ID and version.

Pitfalls.

• Lineage only at table level—not document or chunk level for RAG.
• Embeddings regenerated without version bump—cannot reproduce answers.
• No owner field in catalog—escalations go nowhere.
• Impact analysis skipped when upstream schema changes.

Worked example. FNOL assistant answer cites POL-UK-HOME-2024-03 §4.2. Lineage: SharePoint Policy/Home/2024-03.pdf → OCR job ocr-8841 → chunk ch_9921 → index v2024.03.1 → retrieval log req_77102. When §4.2 amended, re-index triggers and regression suite runs 40 golden questions.

Master data, classification, retention, and consent

Definition. MDM ensures golden records for customers, products, policies. Classification tags PII, PCI, health, confidential. Retention policies define lawful hold and deletion. Consent and lawful basis govern processing for training and inference under GDPR and sector rules.

Engagement use. Before ingest, classify fields and documents; apply minimisation (mask account numbers in chunks), purpose limitation (inference-only vs training), and residency (EU data stays EU). Map retention—do not embed chat logs with indefinite PII storage. Consent gaps block training more often than inference; document legal sign-off.

Pitfalls.

• Scraping all SharePoint without DLP scan—PCI in retrieval context.
• Using production PII in dev vector index.
• Retention deletion not propagated to embeddings copy.
• Assuming public LLM API is fine because "we anonymised"—re-identification risk remains.

Worked example. Telco customer assistant: call transcripts classified PII-high; only PII-redacted silver copies enter RAG index; raw retained 90 days per policy; training on transcripts prohibited without explicit consent programme—pilot inference-only with DPA addendum to hoster.

Lake, warehouse, and lakehouse for AI workloads

Definition. Warehouse (Snowflake, BigQuery, Redshift): SQL analytics on structured data, BI, feature aggregation. Lake (S3, ADLS, GCS): cheap object storage for raw files, logs, media. Lakehouse (Delta, Iceberg, Hudi): table formats adding ACID and schema evolution on lake storage—unifies batch ML and analytics.

Engagement use. Pattern: lake bronze (raw) → silver (cleaned) → gold (business-ready) → vector index or feature store for AI. Warehouse feeds tabular ML and metrics; lake holds documents and logs; lakehouse reduces dual-hop ETL when teams mature. Size cost: warehouse compute for exploratory SQL; lake storage cheap but egress and listing costs bite at scale.

Pitfalls.

• Running embedding jobs inside warehouse warehouse—wrong economics and tool fit.
• No partition strategy on lake—full scans for daily delta.
• Lakehouse without vacuum/compaction discipline—small file problem.
• Treating warehouse dim tables as NLP corpus without text extraction.

Worked example. Healthcare prior-auth assistant: warehouse holds structured auth rules and codes (SQL features for routing); lake holds clinical PDF attachments (parse pipeline); gold table links auth_id to doc_id; vector search only on de-identified guideline sections approved by clinical governance—not raw notes.

Feature engineering and feature stores for AI

Definition. Features are model inputs derived from raw data—aggregations, encodings, embeddings of structured fields. A feature store serves consistent online/offline features with point-in-time correctness for training and inference.

Engagement use. For classical ML (topic 10), define feature catalog: name, source table, transformation, freshness, owner. Offline store for training snapshots; online store for low-latency inference. Prevent training/serving skew by shared transformation code. For GenAI, "features" may include metadata filters (product line, region) applied at retrieval time—not only numeric columns.

Pitfalls.

• Ad hoc notebook features not deployed to serving path.
• Point-in-time violations—future data in training labels.
• Feature store as dumping ground without ownership or deprecation.
• Neglecting categorical drift when new product codes appear.

Worked example. Credit propensity model: features avg_balance_90d, delinq_count_12m, product_holdings from warehouse gold; feature store serves same definitions in batch training and API inference; weekly drift monitor on distribution; new feature requires data owner sign-off and backfill job.

Unstructured data ingest: documents, audio, images, and web

Definition. Unstructured ingest converts PDFs, HTML, slides, scans, call recordings, and tickets into searchable, chunkable text (or multimodal representations) with layout awareness, language detection, and deduplication.

Engagement use. Pipeline stages: acquire → virus/DLP scan → format detect → extract text (OCR if needed) → structure (headings, tables) → normalise encoding → dedupe near-duplicates → chunk with overlap policy → optional enrich (entities, summaries) → index. Measure parse success rate and character error rate on OCR samples. Web ingest requires robots.txt, rate limits, and copyright review.

Pitfalls.

• Plain text dump losing table structure—wrong premium amounts retrieved.
• OCR on low-DPI scans without human QA sample.
• Chunking fixed 512 tokens splitting mid-sentence legal clauses.
• Ingesting drafts and published versions with equal weight.

Worked example. Reinsurance contract assistant: PDF ingest preserves clause numbering; tables extracted to markdown tables; chunks bounded by clause IDs; draft contracts in /drafts/ path excluded by metadata rule; ingest logs 1.2% OCR failure pages quarantined for manual review.

Access control, tenancy, and data mesh boundaries

Definition. Access control ensures users and models retrieve only authorised data—row-level, column-level, document ACL, ABAC attributes. Data mesh assigns domain ownership with federated governance—AI consumers must respect domain APIs and contracts, not bypass via bulk export.

Engagement use. RAG must enforce same ACL as source—security trim at query time, not post-hoc filter on all results. Multi-tenant SaaS requires tenant ID on every chunk metadata field. Document break-glass access for support. For mesh, consume via domain-published data products with SLAs—not raw lake paths.

Pitfalls.

• Indexing entire drive then filtering in app—leak via embedding neighbours.
• Service account with excessive read scope "to simplify."
• Cross-region copy breaking residency commitment.
• Ignoring contractor/partner data separation in index.

Worked example. Global bank policy assistant: employee JWT carries country, entity, role; retrieval query applies metadata filter + source ACL sync nightly; audit log records chunk IDs returned; penetration test verifies user A never receives entity B chunks in top-20 results.

Orchestration, observability, and cost of data platforms

Definition. Orchestration schedules and monitors pipeline DAGs. Observability tracks job duration, row counts, failure rates, cost per TB processed. FinOps for data assigns spend to domains and use cases.

Engagement use. Define SLAs: ingest completion by 06:00 local, index freshness ≤24h for tier-1 docs. Alert on quality check failure, not only job failure. Tag cloud resources with use_case, cost_centre. Right-size: avoid perpetual cluster for weekly batch.

Pitfalls.

• Silent retry loops doubling cloud bill.
• No cost attribution—AI programme perceived as "free data."
• Observability without runbook owner—alerts ignored.
• Over-engineering real-time when batch meets business SLA.

Worked example. Document ingest platform: £18K/month storage, £4K compute; 62% spend from re-embedding unchanged docs—optimisation adds content-hash skip saving £1.1K/month; SLA dashboard shared with steering committee.

Frameworks and methods

Data readiness scoring model

Score each source 1–5 on: availability (can we access it?), quality (fit dimensions), ownership (named owner responds), timeliness (meets SLA), sensitivity (legal use clear). Weight timeliness and ownership heavily for RAG; quality and consistency for ML.

Score band	Meaning	Typical action
4.0–5.0	Production-ready with monitoring	Proceed to pilot
3.0–3.9	Gaps with remediation plan	Time-boxed fix before user-facing
2.0–2.9	Material risk	Limited internal pilot only
<2.0	Not viable	Stop or change use case

Medallion architecture (bronze / silver / gold)

Bronze: raw ingest, immutable. Silver: cleaned, conformed, PII tagged. Gold: business aggregates and AI-ready exports. AI consumers should read gold or governed silver—not bronze except for reprocessing.

DAMA-DMBOK quality dimensions

Use DAMA vocabulary with executives already familiar with data governance programmes—reduces reinventing wheels.

Data contract pattern

Domain teams publish contracts: schema, SLAs, breaking-change notice period. AI teams consume contracts; escalations when corpus misses SLA become formal vendor/internal breach conversations.

RACI for data owners

Activity	Data owner	AI team	Legal	IT
Source prioritisation	A	R	C	C
Classification	A	C	R	C
Pipeline build	C	R	C	A
Quality sign-off	A	C	C	I
Pilot go/no-go	A	R	C	C

A = accountable, R = responsible, C = consulted, I = informed.

8D alignment

Define/Discover: inventory and readiness. Design: pipeline and store architecture. Develop: implement ingest and quality gates. Deliver: production SLAs and lineage in runbooks. See 8D Framework.

Data readiness workshop facilitation

Run a 90-minute data readiness workshop with data owners, IT, legal, and AI delivery—not a slide read-through.

Agenda pattern:

Block	Duration	Output
Use case and decision recap	10 min	Confirmed consumer of data
Source walk-through	25 min	Inventory rows updated live
Quality and freshness debate	20 min	Dimension scores with owners
Sensitivity and lawful basis	15 min	Red/amber/green classification
Go/no-go and remediation	20 min	Dated action list

Facilitation rules: Ban model discussion until scores exist. Capture dissent in minutes—silent disagreement resurfaces at go-live. End with signed or email-confirmed owner actions, not "IT will look into it."

Chunking and metadata design for RAG-bound corpora

Even before topic 13 deep dive, data architecture must define chunk boundaries and metadata schema at ingest:

• Chunk keys: doc_id, version, effective_date, product, region, language, classification, section_id.
• Overlap policy: 10–15% token overlap for narrative text; zero overlap for structured clause IDs.
• Parent-child: store small retrieval chunks with pointer to parent section for display context.
• Hash-based skip: content SHA-256 to avoid re-embed when body unchanged.

Document in lineage sketch so retrieval engineers inherit contracts, not reinvent metadata in the vector DB alone.

Disaster recovery and reproducibility for AI data paths

AI systems fail audits when teams cannot reproduce an answer from six months ago. Minimum DR requirements:

• Versioned indexes aligned to corpus version tags—not only latest.
• Backup of gold layer and embedding index metadata weekly; test restore quarterly.
• Runbook for full re-index from gold (time estimate documented—often 4–24 hours).
• Cross-region: if DR region exists, residency rules may forbid automatic failover—document manual approval path.

Operating model: who does what in data for AI

Role	Responsibility	AI engagement touchpoint
Data owner (business)	Accuracy, timeliness, SME sign-off	Readiness score accountability
Data steward	Catalog, classification, quality rules	Metadata schema approval
Platform engineering	Pipelines, stores, SLAs	Ingest jobs, observability
Security	ACL, DLP, encryption	Retrieval trim validation
Legal/privacy	Lawful basis, DPIA	Training vs inference boundary
AI Solution Engineer	Readiness gate, lineage doc	Feasibility artefact pack

Escalate when owner missing more than two weeks—do not proceed to customer-facing pilot on "best effort" access.

Cost and capacity planning template

Rough-order monthly cost drivers for data platform supporting one GenAI use case:

Component	Example range	Driver
Object storage (lake)	£500–£5K	Corpus size, retention
Warehouse compute	£1K–£15K	Feature SQL, dashboards
Ingest compute (OCR/parse)	£2K–£20K	Page volume, OCR %
Embedding batch jobs	£500–£8K	Chunk count, re-embed frequency
Vector DB	£1K–£10K	Dimensions, QPS, HA
Egress	£200–£3K	Cross-cloud mistakes hurt

Present sensitivity: 2× document volume or weekly → daily re-embed changes embedding line item 3–7×.

Real-world scenarios

Scenario A: Stale policy corpus (UK retail banking)

A bank prioritises GenAI for Tier-1 servicing lookups after Stage 1 value case shows £2.1M/year opportunity from handle-time reduction. IT reports "all policies in SharePoint." Audit finds 340 GB, 12,000 files, median last modified 14 months ago; product launch in Q2 updated 23 customer-facing policies but only 8 uploaded to shared library; no owner for 40% of folders.

Readiness scores: availability 4, quality 2, ownership 2, timeliness 1, sensitivity 3 → weighted 2.1. Gate: no customer-facing RAG until CMS feed with owner RACI and ≤7-day freshness SLA live. Interim: internal-only retrieval on gold subset (120 approved articles) with watermark "verify in core system."

Outcome: 10-week corpus programme £180K; pilot delayed but compliance signed off; avoided estimated £400K mis-selling exposure from wrong APR text. Lineage doc becomes template for other BUs.

Scenario B: Claims FNOL document chaos (general insurance)

Insurer wants vision+LLM to read damage photos and FNOL forms. Lake holds 8M images; labels sparse. Structured FNOL data in warehouse complete for 91% of digital submissions; paper channel OCR quality 76% CER on sample. Data science proposes end-to-end multimodal model.

Analysis: bottleneck is structured field extraction from low-quality scans—not photo understanding. Recommendation: silver layer OCR with human-in-loop for confidence <0.85; tabular ML on extracted fields for triage (topic 10); GenAI summarisation only on gold text with citations. Ingest pipeline quarantines 9% failed parses.

Metrics: digital FNOL auto-routing accuracy 88% vs 62% on notebook using leaked labels; cycle time −18% on pilot region. Lesson: fix ingest and labels before model complexity.

Scenario C: Healthcare prior auth (regulated clinical text)

Regional health system wants assistant over clinical guidelines and payer rules. Epic holds patient data—cannot enter general RAG index. Guidelines in PDF on intranet; BAAs restrict cloud processing. US residency required.

Architecture: on-prem silver parse; de-identified guideline chunks only; vector store in approved enclave; no patient queries mixed with guideline retrieval—separate interfaces. Consent for any quality-improvement logging. Freshness: guideline committee publishes quarterly; CDC from CMS when HTML updates.

Readiness: sensitivity score drove private deployment; public API rejected. Pilot internal clinicians only; eval on 200 guideline Q&A pairs with pharmacist review. Outcome: 34% reduction in "wrong guideline version" support tickets internally—patient-facing deferred pending legal review.

Scenario D: Telco network fault tickets (streaming + unstructured)

Telco NOC wants copilot over 10K daily tickets and runbooks. Runbooks in Confluence—decent gold. Tickets streaming to Kafka; PII in free text. Engineers need sub-minute freshness for known outage patterns.

Design: streaming silver ticket anonymisation; retrieval hybrid: structured filter on region, product + vector on runbook chunks; ticket text summarised not indexed raw. Feature store holds outage counters for ranking context. ACL: NOC role only.

Numbers: index lag p95 45 seconds; retrieval precision@5 0.78 on golden set; false escalation rate unchanged. Cost £9K/month ingest+index vs £120K quoted for re-platforming data lake—reuse existing stream.

Scenario E: Pharma regulatory submissions (GxP-adjacent document control)

Pharma client wants RAG over IND/CTD submission sections for medical writers. Sources: validated document management system (VDS), not SharePoint. 21 CFR Part 11 considerations: audit trail on read access, electronic signatures on gold promotion, no draft in production index.

Readiness: availability 5 (VDS API), quality 4 (validated metadata), ownership 5 (Regulatory Affairs), timeliness 4 (24h publish SLA), sensitivity 5 (trade secret + patient narratives in some modules). Weighted 4.6—proceed with strict change control.

Architecture: silver parse preserves section numbering matching CTD template; gold index only after approved workflow state; embeddings tagged submission_id, version, status=approved. Lineage logged to audit system—not only app logs. Re-index triggers validation script (chunk count ± tolerance vs prior version).

Outcome: Medical writer time on cross-reference lookup −22% on pilot (n=18 writers); zero unapproved draft retrieval in 12-week penetration test. Programme extended to CMC sections with separate index namespace—isolation prevents cross-contamination of submission types.

Lesson: In GxP-adjacent contexts, workflow state in metadata is as important as OCR quality—data architecture is compliance architecture.

Practice exercises

Primary exercise: Data readiness assessment (3–4 hours)

Take a fictional or real use case: internal HR policy assistant, invoice exception copilot, or warranty claims knowledge base.

Deliver:

1. Data-source inventory — minimum eight sources; columns: system, owner, format, volume, refresh, classification, consumer.
2. Quality assessment — score five DAMA dimensions per critical source with evidence notes.
3. Readiness score — weighted 1–5 with explicit weights justified for use case.
4. Go/no-go recommendation — proceed, proceed with conditions, or stop—with timeline for remediation.
5. Lineage sketch — ASCII or diagram from source to answer/log.

Acceptance criteria: Reviewer can identify one stop condition that would block production; lineage includes document/chunk granularity for at least one unstructured path.

Stretch exercise: Pipeline and store design memo (3 hours)

For commercial lending credit memo assistant (PDFs, warehouse financials, CRM notes):

1. Choose lake/warehouse/lakehouse split with rationale.
2. Define bronze/silver/gold artefacts and SLAs.
3. Specify unstructured ingest for PDFs (OCR, clause chunking, table handling).
4. ACL model for relationship managers vs analysts vs audit.
5. Cost rough-order estimate (storage, compute, embedding monthly).

Acceptance criteria: Design explains train vs inference data boundaries; includes failure/quarantine path for OCR.

Reflection exercise: Negative case post-mortem (45 minutes)

Read scenario A above. Write one-page post-mortem as if the bank ignored readiness and launched customer-facing RAG. Include: incident timeline, regulatory trigger, remediation cost, what artefact would have prevented it. File in pattern library.

Questions you should be able to answer

1. What sources feed this use case—and who owns each?
2. What is the freshness SLA required for safe answers—and what is actual lag today?
3. Which quality dimensions fail first for this corpus—and how do you measure them?
4. Where does authoritative content live versus drafts and duplicates?
5. What PII or regulated data could appear in chunks—and how is it minimised?
6. Can you draw lineage from source to cited answer in retrieval?
7. Why warehouse vs lake vs vector store for each data type in scope?
8. What pipeline pattern (batch, CDC, stream) matches the business event cadence?
9. How are ACLs enforced at retrieval—not only at source?
10. What happens when upstream schema or policy version changes—impact analysis path?
11. What is quarantined vs rejected vs flagged in ingest—and who reviews queues?
12. What is the cost driver of the data platform for this use case?
13. Is training data permitted under consent and contract—or inference-only?
14. What readiness score gates the pilot—and who signed exceptions?
15. What monitoring proves corpus drift before users complain?

Negative cases

SharePoint-as-RAG. Entire drives indexed; deprecated PDFs rank highly; no owner. Fix: gold corpus curation + freshness + citation to version ID.

Lake without silver. Raw dumps ingested; OCR garbage in index. Fix: medallion discipline and quality gates.

Embedding-only security. Vector DB holds text attacker can extract. Fix: trim at query, encrypt at rest, minimise stored text.

Service account god mode. Pipeline reads all tables; leak in logs. Fix: least privilege, column masking, audit.

Notebook features never served. Training metrics fantasy. Fix: feature store shared code path.

Ignoring residency. EU data embedded in US region. Fix: region pinning in architecture review gate.

Lineage theatre. Table-level diagram only; cannot debug wrong answer. Fix: chunk-level logging.

Quality = job green. Empty extracts succeed. Fix: row count and checksum assertions.

Consent ambiguity. Fine-tune on customer emails without basis. Fix: legal sign-off matrix.

Cost blind spot. Re-embed unchanged docs monthly. Fix: content-hash incremental processing.

MDM bypass. Duplicate customer records pollute features. Fix: golden record join required.

Draft equals published. RAG returns draft rates. Fix: metadata workflow states in index filter.

Integration with Data & Knowledge Guide

Topic 09 capability	Guide chapter
Corpus design, knowledge layers	Data & Knowledge
RAG-specific ingest	Topic 13 + Guide retrieval sections
Governance hooks	8D Framework Design gate

Study patterns in the Guide; produce engagement artefacts from this topic.

Crosswalk to RAG and feature pipelines

Data architecture artefact	Downstream consumer
Gold corpus + metadata schema	Topic 13 chunking and ACL filters
Feature store definitions	Topic 10 training and serving
Lineage + version IDs	Topic 21 eval reproducibility
Quality scorecards	Steering committee go/no-go
Ingest SLA	Freshness SLO in operating model

When data readiness score <3.0 on timeliness, do not promise "real-time answers" in topic 11 model selection—align expectations in Stage 1 exec summary rewrite if needed.

Facilitation script: challenging "we already have a lake"

When sponsors say data is ready because a lake exists, ask:

1. "Show me the last successful pipeline run for the documents this assistant needs."
2. "Who is accountable if an answer cites wrong policy version?"
3. "What percentage of top-20 enquiry types have gold-layer artefacts today?"
4. "Walk me one lineage path from source to chunk ID."
5. "What happens to embeddings when source deletes a document?"

Record answers in readiness workbook—gaps become remediation backlog with dates, not footnotes.

Output

Expected outputs for this topic:

• Data-source inventory (systems, owners, classification, refresh)
• Data-quality assessment (dimension scores with evidence)
• Data readiness score (weighted) and go/no-go rationale
• Lineage sketch (source → pipeline → store → AI consumer)
• Optional: pipeline SLA and monitoring checklist for pilot

Practice checklist

• I can explain lake vs warehouse vs vector roles without vendor slides
• I scored readiness on a real or fictional corpus and enforced a gate
• Lineage includes chunk/document level for unstructured path
• ACL and residency constraints documented for pilot data
• I linked remediation owners and dates—not only technical gaps
• Practice artefacts filed in pattern library with sector tag

Related playbook content

• Data & Knowledge Guide — corpus, ingest, and knowledge architecture delivery
• AI Patterns — when retrieval and features combine
• 8D Framework — Design and Develop gates
• How to use this Learning Map — study method and artefact standards
• AI and data foundations — where this topic sits in the curriculum
• Retrieval-Augmented Generation — retrieval layer atop ready data

Supplemental scenario — insurance claims document corpus readiness

Context: UK insurer scales RAG on 840k claim notes + 12k policy PDFs; steering asks "why 6-week slip?"

Readiness scorecard (weighted):

Dimension	Score (1–5)	Evidence	Remediation
Completeness	3.2	19% PDFs missing OCR text layer	4-week ingest sprint
Timeliness	2.8	Nightly job failed 11/30 nights	SLA owner named
Lineage	2.5	Chunk ID → source broken on legacy archive	2 engineers × 3 weeks
ACL	4.1	Entra groups mapped	Monitor only
Quality (dedupe)	3.0	8% near-duplicate chunks	Dedupe pipeline v2

Weighted readiness: 3.04 — gate requires ≥3.5 for scale; go/no-go: NO until timeliness ≥3.5.

Executive conversation (BLUF): "Defer £720k scale 6 weeks; fix pipeline SLA and lineage—not embedding model swap. Cost of premature scale: wrong-policy citations estimated £180k rework/quarter (Assumption from compliance desk)."

Negative case avoided: Team nearly procured larger embedding model; readiness gate redirected £90k spend to ingest—faithfulness on pilot golden set +11 points without model change.

Key takeaways

• AI quality ceilings are set by data readiness, not embedding model choice.
• Medallion layers, quality dimensions, and lineage are non-optional for enterprise RAG and ML.
• Unstructured ingest is a product engineering workstream—OCR, structure, chunking, dedupe, ACL.
• Readiness scores are executive negotiation tools—use them to gate spend early.
• Feature stores and governed gold layers prevent train/serve skew and leakage.
• Pair this topic with the Data & Knowledge Guide for delivery depth.

What to do next

1. Complete the primary readiness exercise before opening model selection.
2. Continue with Machine Learning Foundations for problem-type and metrics discipline.
3. If RAG is in scope, preview Retrieval-Augmented Generation after readiness passes.
4. Open the interactive playbook for workshop templates.