Generative AI and LLM Fundamentals
Executive view
Technical view
Why this matters
Large language models changed the default interface for knowledge work: drafting, summarisation, classification via instructions, dialogue, and tool orchestration. They did not remove physics—context limits, hallucination, non-determinism, latency, cost per token, and data processing agreements still govern enterprise viability. AI Solution Engineers who only know buzzwords approve architectures that send PII to public APIs, fine-tune on 200 examples when RAG would generalise, or provision frontier models for tasks a 10× smaller regional model handles at 1/30th cost.
Understanding internals at practitioner depth builds credibility. When a CISO asks whether attention "remembers" training data, you explain weights versus context, retention policies, and logging. When a network architect asks about GPU sizing, you connect parameter count, quantisation, and tokens per second to instance types. When legal asks about EU residency, you distinguish inference region, support access, and training opt-out clauses.
Model selection is a multi-criteria decision, not a leaderboard exercise. Public benchmarks rarely reflect your corpus, languages, tool schemas, or conduct rules. This topic links to AI Patterns for solution shapes, Model landscape for catalogued options, topic 12 (prompt/context), topic 13 (RAG), topic 14 (agents), and topic 21 (evaluation).
Learn
Tokens and tokenisation
Definition. Tokens are the units models read and generate—often subword pieces, not whole words. Tokenisation splits text via algorithms (BPE, SentencePiece); different models tokenise the same text differently, affecting cost, context consumption, and language performance.
Engagement use. Estimate prompt + completion token budget for each use case. English averages ~4 characters per token; code and JSON denser; some languages inflate token count vs English. Price API quotes in $/1M tokens—multiply by expected daily volume for FinOps. Log token counts in observability from day one.
Pitfalls.
- Assuming 1 token = 1 word—budget errors up to 40%.
- Ignoring tokenizer differences when switching models—prompts break or cost spikes.
- stuffing JSON schemas that consume thousands of tokens repeatedly.
- Not reserving completion budget—truncated answers mid-JSON.
Worked example. Policy assistant: average user question 120 tokens; retrieved context 2,800 tokens; system prompt 600 tokens; expected answer 400 tokens → ~3,920 tokens/request. At 50K requests/month → 196M tokens/month—dominates model SKU choice.
Embeddings and semantic representation
Definition. Embeddings are dense vectors representing text meaning—used for semantic search, clustering, deduplication, and routing. Embedding models differ from generative LLMs; smaller, cheaper, fixed-dimension outputs optimised for similarity—not fluent generation.
Engagement use. Choose embedding model with same language coverage as corpus; benchmark recall@k on golden questions (topic 13). Dimension size affects vector DB memory: 1M chunks × 1536 dims × 4 bytes ≈ 6 GB vectors alone. Re-embed corpus when switching embedding model—version indexes.
Pitfalls.
- Using generative LLM hidden states ad hoc instead of dedicated embedder.
- Mixing embedding models in one index without full re-embed.
- English-only embedder on multilingual policy corpus.
- Ignoring normalisation and distance metric (cosine vs dot product).
Worked example. Bank FAQ: embedder A recall@5 0.71 on 200-question set; embedder B 0.79 at 2× latency—choose B for customer-facing; A for internal low-risk search with cost cap.
Context windows and context engineering
Definition. Context window is maximum tokens the model can process in one forward pass—prompt + retrieved material + conversation history + completion reserve. Context engineering allocates that budget deliberately: system instructions, tools, RAG chunks, examples, user message.
Engagement use. Build context budget table per request type. Prioritise authoritative retrieved chunks over long chat history; summarise history when needed. Long-context models (128K+) reduce retrieval need but do not replace RAG for freshness and citation—they increase cost and latency and still hallucinate without grounding.
Pitfalls.
- "We have 128K context" used to skip corpus governance.
- Lost-in-the-middle effect—critical facts buried mid-prompt ignored by models.
- Unbounded conversation history filling window with stale turns.
- Same context for all intents—wasteful token burn.
Worked example. Underwriting assistant: system 500 tokens; top-6 chunks 4,200 tokens; structured submission JSON 1,800 tokens; user question 200 tokens; reserve 800 completion → fits 8K model with margin; 128K model unnecessary—save 40% latency with 8K + good retrieval.
Attention and the transformer architecture (practitioner view)
Definition. Transformers use self-attention to relate tokens in a sequence—capturing long-range dependencies without recurrent loops. Layers, heads, and parameters scale capability and compute. You do not implement attention manually; you use it to explain why context length costs compute quadratically in naive forms and why KV cache matters for inference optimisation.
Engagement use. When vendors claim "new architecture," ask: parameter count band, context length, tool/function calling support, multimodal modalities, licence. Explain to stakeholders: model does not "look up database" in weights—it predicts next tokens conditioned on context—including retrieved text you provide.
Pitfalls.
- Believing model "knows" your proprietary data without RAG or fine-tune.
- Confusing training (expensive, infrequent) with inference (per request).
- Ignoring batch vs streaming inference for UX latency.
- Assuming all transformer checkpoints are equal—instruction-tuning quality varies hugely.
Worked example. Executive briefing: "The model reads your question plus the policy excerpts we retrieve—like an open-book exam—not memorised secrets from training. Wrong book on desk → wrong answer."
Pretraining, instruction tuning, and alignment
Definition. Pretraining learns general language patterns from massive corpora (unsupervised/next-token). Instruction tuning fine-tunes on curated prompt-response pairs to follow instructions. Alignment (RLHF, DPO, etc.) shapes behaviour toward helpfulness and safety preferences.
Engagement use. Default to instruction-tuned chat models for enterprise assistants—not base pretrained checkpoints. Evaluate refusal behaviour, instruction following, and JSON adherence on your tasks—not only knowledge QA. Alignment reduces but does not eliminate hallucination or policy violations.
Pitfalls.
- Using base models expecting chat behaviour.
- Over-trusting alignment for regulated advice without retrieval and rules.
- Assuming alignment generalises to your domain jargon without eval.
- Ignoring model knowledge cutoff date for time-sensitive facts.
Worked example. Internal HR bot: instruction-tuned model still invents leave policies—must RAG over signed handbook; alignment helps tone and refusals on medical/legal advice.
Fine-tuning, PEFT, and when not to fine-tune
Definition. Fine-tuning updates some or all model weights on domain tasks. PEFT (parameter-efficient fine-tuning)—LoRA, adapters—trains small additive layers, cheaper and storable per use case. Full fine-tune demands GPU budget and MLOps maturity.
Engagement use. Prefer prompt + RAG first for knowledge tasks. Consider PEFT when: stable task format (extraction schema), large labelled set (>1–5K quality examples), proprietary style/terminology, or latency need to shorten prompts. Avoid fine-tune to substitute for bad corpus—model learns stale facts.
Pitfalls.
- Fine-tuning 200 examples—overfit and hallucination style lock-in.
- Fine-tune on PII without contractual basis.
- Multiple LoRA adapters without routing governance.
- Fine-tune cost exceeds years of RAG API spend for low volume.
Worked example. Legal clause extraction: 8K labelled contracts; LoRA on 7B open model in private VPC; F1 91% vs GPT-4 prompt 86% at 1/8 inference cost; RAG still supplies clause library for rare types.
Quantisation and efficient inference
Definition. Quantisation reduces numeric precision of weights (FP16, INT8, INT4) to cut memory and increase throughput—often small quality loss. Distillation trains smaller student models from larger teachers. Speculative decoding drafts tokens faster for latency wins.
Engagement use. On-prem or VPC self-host: pick quantisation level based on eval regression suite—not hardware convenience alone. GPU sizing: 7B INT4 may run on single L4; 70B needs multi-GPU or specialised serving stack. Compare tokens/sec/$ across SKUs.
Pitfalls.
- Aggressive quantisation crushes JSON/tool reliability.
- Serving framework mismatch (vLLM, TGI, TensorRT-LLM) causing ops pain.
- No A/B on quantised vs full for compliance-critical tasks.
- Underestimating cold start and autoscaling lag.
Worked example. Customer email summarisation: INT8 13B model meets p95 latency 1.1s vs FP16 1.9s; JSON field error rate rises 0.4% → 1.1%—acceptable for human review queue; not acceptable for straight-through processing without fix.
Sampling, temperature, and output control
Definition. Autoregressive generation samples next token from probability distribution. Temperature scales randomness (low = deterministic, high = creative). Top-p, top-k, seed modify sampling. Structured outputs (JSON mode, grammar constraints) reduce format errors.
Engagement use. Production assistants: low temperature (0–0.3) for factual/policy tasks; higher only for brainstorming. Enforce JSON schema or tool calls for downstream automation. Log sampling params per request for reproducibility. Deterministic regression tests use temperature 0 where supported.
Pitfalls.
- High temperature on compliance answers—inconsistent citations.
- Parsing free-text when JSON mode available.
- Same params for creative marketing and credit explanations.
- Ignoring stop sequences—runaway token cost.
Worked example. Claims summarisation: temperature 0.1, max tokens 800, JSON schema {summary, risks[], citations[]}; failure rate 2.3% vs 11% free-text parsing path.
Model selection criteria: quality, latency, cost
Definition. Quality on your eval set (accuracy, citation faithfulness, tool success). Latency p50/p95/p99 end-to-end including retrieval. Cost $/request at expected volume including embedding, reranking, logging, GPU amortisation.
Engagement use. Build weighted scorecard—weights from sponsor priorities. Measure on held-out internal eval, not vendor demos. Include human review cost when model requires mandatory review. Scenario-plan volume growth 3× and 10×.
Pitfalls.
- Single global model for all intents—overpay on easy queries.
- Latency measured model-only excluding RAG and network.
- Ignoring caching (semantic cache) for repeat questions.
- Cost model excludes engineering ops headcount.
Worked example. Scorecard weights: quality 40%, latency 25%, cost 20%, privacy/residency 15%. Model A wins quality; Model B wins composite for PII-heavy tier-1 intents—route by intent classifier.
Privacy, residency, and data processing
Definition. Privacy covers PII handling, logging, retention, re-identification risk. Residency restricts geography of processing and storage. Training opt-out and zero retention API modes vary by vendor and contract tier.
Engagement use. Data classification drives deployment pattern: public API with redaction, private hosted, customer VPC, air-gapped on-prem. Document what is logged (prompts, completions, embeddings), retention days, subprocessors, support access jurisdictions. DPIA inputs for GDPR; sector add-ons (HIPAA BAA, PCI scope minimisation).
Pitfalls.
- "Enterprise tier" assumed without reading DPA schedule.
- Prompts logged with account numbers—PCI incident.
- EU inference but US-based support access without SCCs.
- Fine-tune data retained by vendor indefinitely.
Worked example. UK bank retail chat: customer PII → regional private deployment; prompts masked; logs 30-day retention; training on customer data prohibited contractually; fallback to smaller local model if frontier route unavailable—no silent failover to US public API.
Tool use, function calling, and agents (fundamentals)
Definition. Tool use lets models emit structured calls to APIs (search, calculator, CRM update). Agents loop: plan → act → observe—bounded by guardrails (topic 14). Reliability depends on schema design, idempotency, and human approval for irreversible actions.
Engagement use. Start with single-tool or fixed workflow before open-ended agents. Validate tool call success rate on eval set. Idempotent read tools first; write tools behind approval. Timeout and circuit-break external APIs.
Pitfalls.
- Agent with write access to production without sandbox.
- Tool schemas too loose—argument injection.
- Unbounded loop burning tokens and API quotas.
- Confusing tool success with task success—downstream validation required.
Worked example. Service desk agent: tools = {search_kb, create_ticket, fetch_account_status}; create_ticket read-only draft until human click; eval: 94% correct tool choice on 300 scenarios; 0 auto ticket closes in pilot.
Multimodal models (vision, audio, documents)
Definition. Multimodal LLMs accept images, audio, or document layouts—not only text—for captioning, VQA, OCR-assisted reasoning, meeting transcription summarisation.
Engagement use. Use when unstructured visual evidence matters—damage photos, scanned forms, charts. Combine with dedicated OCR/pipeline (topic 09) when layout fidelity critical. Higher cost and latency; smaller context for images as tokens.
Pitfalls.
- Vision model on poor-resolution scans without quality gate.
- Multimodal for pure tabular PDFs better served by extraction pipeline + ML.
- Audio transcription PII in logs.
- Assuming vision removes need for authoritative text source.
Worked example. FNOL photos: multimodal model describes damage for adjuster note draft; pricing and coverage from structured policy DB only—vision output not sole authority.
Vendor lock-in, portability, and model routing
Definition. Lock-in arises from proprietary APIs, fine-tuned weights on vendor-only runtime, prompt features unavailable elsewhere, and eval sets tied to one tokenizer. Routing sends requests to different models by intent, sensitivity, or cost. Portability uses open weights and standard serving where strategic.
Engagement use. Abstract inference interface internally; store prompts versioned; avoid vendor-specific hacks without fallback. Maintain model card per route. Negotiate exit: export adapters, index ownership, log formats.
Pitfalls.
- 200 hard-coded Azure-only features in app layer.
- Single vendor without DR model path.
- Routing rules untested—wrong model on PII route.
- Open-weight choice without security patch process.
Worked example. Three-tier routing: Tier A sensitive → on-prem 7B; Tier B standard internal → hosted EU medium model; Tier C complex reasoning → frontier with masking; monthly review of route distribution and cost.
Frameworks and methods
Model selection scorecard template
| Criterion | Weight | Model A | Model B | Notes |
|---|---|---|---|---|
| Task quality (eval set) | 30% | Include citation accuracy | ||
| p95 latency | 20% | End-to-end | ||
| $/1M tokens at forecast volume | 20% | Include embed + rerank | ||
| Privacy/residency fit | 15% | Legal sign-off | ||
| Tool/multimodal fit | 10% | |||
| Ops maturity / lock-in | 5% |
Normalise scores 1–5; document sensitivities if weights ±10% flip winner.
Adaptation decision tree
- Can RAG + prompt solve with eval pass? → Stop here.
- Need consistent structured output? → JSON mode / grammar.
- Large labelled set + fixed task? → PEFT.
- Need behaviour on proprietary style only? → LoRA + RAG.
- Still failing with domain shift? → Consider larger model or data fix—not endless prompt stretch.
Deployment pattern catalogue
| Pattern | When | Risk profile |
|---|---|---|
| Public API + redaction | Low sensitivity, fast start | Highest data egress concern |
| Vendor private instance | Medium, need isolation | Contract-dependent |
| Customer VPC managed | Regulated, cloud OK | Ops shared responsibility |
| On-prem open weight | Strict residency/air-gap | CapEx, patching burden |
Detail in AI Patterns.
Token economics quick estimate
monthly_cost ≈ (prompt_tokens + completion_tokens) × requests × price_per_token
+ embedding_tokens × index_updates
+ GPU_hours (if self-host)
Add 30–50% buffer for retries, eval, and non-prod.
Eval dimensions beyond accuracy
Faithfulness to sources, refusal appropriateness, toxicity, PII leakage in output, tool call validity, latency SLO, cost SLO. Topic 21 expands.
Reference: Model landscape
Use Model landscape for capability bands, licence classes, and comparative positioning—refresh selection quarterly.
KV cache, batching, and streaming inference
KV cache stores attention keys/values during generation—repeated prompt prefixes (system + RAG) should be cached to cut latency and cost. Continuous batching improves GPU utilisation in serving frameworks. Streaming tokens to UI improves perceived latency even when total time unchanged.
Engagement use: Ask platform team: is prefix caching enabled for fixed system prompts? Measure time-to-first-token separately from total generation—executives feel latency at first token.
Pitfalls: Cache invalidation when prompt template changes without version bump. Streaming without cancellation—user navigates away, tokens still generated.
Prompt caching and semantic cache economics
Prompt caching (vendor feature) discounts repeated long prefixes. Semantic cache stores prior Q→A pairs for near-duplicate questions—hits can cut cost 20–40% on repetitive internal helpdesks. Risk: stale answers if source changed—tie cache TTL to corpus freshness SLA.
Open-weight vs proprietary models
| Factor | Open weight (Llama, Mistral, etc.) | Proprietary API |
|---|---|---|
| Control | Full weights, air-gap possible | Vendor dependency |
| Licence | Review commercial use, attribution | Contractual |
| Safety stack | Self-build guardrails | Vendor filters + your policies |
| Upgrade path | Manual merge of new releases | Vendor roadmap |
| Cost at scale | GPU ops labour | Token billing |
Many enterprises use both: open weight for sensitive bulk; proprietary for cutting-edge reasoning on redacted subsets.
Guardrails, content filters, and policy layers
Models include safety training; enterprises add input/output filters (PII detect, blocked topics, regex allow-lists), prompt templates with refusal instructions, and downstream validation (schema, rules engine). No single layer sufficient—defence in depth.
Document which layer blocked which request in logs—for regulator and tuning false positives.
Long-context models: when they help and hurt
128K–1M context models help: multi-doc merge summaries, long codebases, few-shot with many examples. They hurt: cost, latency, lost-in-the-middle recall, temptation to skip RAG governance. Always benchmark against retrieve-then-read baseline on your eval set before paying long-context premium.
Contract and procurement checklist for LLM vendors
- Inference and support data regions
- Training opt-out and zero-retention API modes
- Subprocessor list and change notification
- SLA latency and availability; credits
- Rate limits and burst behaviour at forecast QPS
- Audit rights and SOC2/ISO artefacts
- Exit: export of indexes, adapters, logs format
- Liability caps on model errors—often low; internal review mandatory
Legal review parallel to technical matrix—not sequential after code complete.
Model card and system card for deployment
Per deployed route document: model ID/version, intended use, prohibited use, training data summary (vendor), eval scores, known failure modes, monitoring owner, retrain/eval cadence, dependency on RAG corpus version.
FinOps governance for GenAI programmes
Establish monthly token budget by use case; alert at 80%; require sponsor approval to exceed. Attribute spend tags in cloud/API billing. Review cache hit rate, routing distribution, and average tokens per session in steering—volume growth often linear with adoption while cost super-linear if frontier-heavy.
Emerging patterns: small language models (SLMs)
SLMs (1–8B) on device or edge for PII-sensitive or offline scenarios improve latency and cost; quality gap narrows for narrow tasks. Pattern: SLM first pass; escalate to larger model on low confidence—cascade routing saves 30–50% cost in some call-centre pilots when confidence calibrator well tuned.
Real-world scenarios
Scenario A: PII-heavy customer servicing (retail bank)
Context: 1.2M monthly chat sessions; average 3.2 turns; strict UK/EU residency; conduct risk if wrong APR or fee quoted.
Options compared:
| Option | p95 latency | Quality (golden 250) | Est. monthly cost |
|---|---|---|---|
| Frontier public API (masked) | 2.8s | 88% citation match | £185K (at volume) |
| EU hosted 20B class | 1.6s | 84% | £62K |
| On-prem 7B quantised | 1.1s | 79% | £28K infra + ops |
Decision: Route 82% of intents to on-prem after intent classifier; 15% to EU hosted for complex disputes; 3% to frontier with strict masking and no retention API mode. Weighted quality 83% at £41K/month vs single frontier £185K.
Governance: DPIA updated; model cards per route; regression weekly on 250 golden + 50 red-team prompts.
Scenario B: Engineering knowledge assistant (global manufacturer)
Context: 400K Confluence pages; engineers in US/EU/APAC; English + German docs; need citation to page version.
Architecture: RAG (topic 13) + 8K context model sufficient after chunking; embedding multilingual model; no fine-tune—corpus updates weekly. Instruction-tuned 13B in each region for residency. Quality metric: answer correctness 86%, citation accuracy 91% on 400-question eval.
Anti-pattern avoided: 128K model ingesting whole pages—cost 4× with no quality gain vs good retrieval. Fine-tune rejected—only 400 labelled Q&A; RAG generalises better.
Outcome: p95 1.4s; adoption 62% MAU engineers; escalations to level-3 down 18%.
Scenario C: Insurance policy wording copilot (stretch)
Context: Underwriters draft endorsements; long clauses; high hallucination cost if wrong exclusion inserted.
Approach: Low temperature; RAG over approved clause library; JSON clause blocks output; human mandatory sign-off. PEFT considered on 6K historical endorsements—improves formatting consistency 7% but not faithfulness—RAG + templates win.
Numbers: Hallucinated exclusion rate 0.4% on eval vs 3.1% prompt-only GPT-4 class; each false exclusion estimated £50K–£2M tail risk—faithfulness weighted 50% in scorecard.
Scenario D: Multimodal FNOL intake (general insurance)
Context: 35% digital FNOL includes photos; adjusters want draft damage descriptions.
Design: Vision-capable model describes images; coverage decision from rules engine + structured policy DB—not vision output. Transcription for voice notes optional. Photos moderate resolution gate—reject <480p.
Metrics: Description helpfulness 4.1/5 adjuster survey; zero auto-approval of total loss from vision-only path. Cost £0.08 per FNOL vs £4.50 manual note time average saved 6 minutes.
Scenario E: Legal contract review platform (professional services)
Global law firm pilots clause deviation detection on MSAs. Corpus: 14K historical MSAs in document management; labels from lawyer mark-ups on 900 deals.
Comparison:
| Approach | Deviation F1 | Cost/deal review | Latency |
|---|---|---|---|
| GPT-4 class + long context | 0.81 | £12 | 45s |
| 7B + RAG over clause library | 0.77 | £0.90 | 8s |
| Fine-tuned 13B LoRA + RAG | 0.86 | £1.10 | 9s |
Decision: LoRA + RAG in private tenant; lawyer remains decision-maker; model highlights deviations with citation to playbook clause ID. No fully automated redlines—professional responsibility retained.
Governance: Client matter ACL on index; ethical wall metadata prevents cross-client chunk retrieval; eval includes confidentiality red-team (firm attempts to retrieve Client A clauses while authenticated as Client B matter—must fail).
Outcome: Associate review time −28% on standard MSAs; partner sign-off unchanged; malpractice insurer briefed on human-in-loop design.
Lesson: Model selection matrix must include professional liability and confidentiality architecture—not only F1 and token cost.
Scenario F: Public sector benefits eligibility (transparency requirements)
Government agency explores LLM to help citizens understand benefit rules. Algorithmic transparency policy requires plain-language explanation of deterministic rules outcome—not probabilistic model guess.
Architecture: Rules engine remains system of record for eligibility; LLM generates explanation layer grounded in published statutory text via RAG; every output includes rule ID citations and link to official source. Model cannot override rules engine score.
Selection: Smaller regional language model in sovereign cloud; temperature 0.1; eval includes citizen comprehension user testing (n=120) and misleading rate review by legal—target misleading <1%.
Outcome: Call centre handle time on explanation calls −15%; zero deployment of eligibility decision to LLM; transparency board approves architecture because decision and explanation are separated—pattern reusable for other public services.
Practice exercises
Primary exercise: Weighted model selection matrix (2–3 hours)
Use case: internal IT helpdesk assistant, HR policy bot, or sales proposal drafter (pick one).
Deliver:
- Five criteria with weights summing to 100%.
- Two or three model/deployment options scored 1–5 per criterion with evidence notes.
- Routing recommendation if composite winner differs by intent or sensitivity.
- Token economics rough monthly estimate at stated volume.
- Privacy/residency paragraph legal can review.
- Adaptation choice: prompt-only, RAG, PEFT, or full fine-tune—with rationale.
Acceptance criteria: Winner sensitive to weight changes documented; includes at least one non-frontier option; no vendor name as sole justification.
Stretch exercise: Inference architecture one-pager (2 hours)
For regulated wealth management client Q&A:
- Deployment pattern diagram (API vs VPC vs on-prem).
- Context budget table for typical request.
- Quantisation vs quality trade-off note with eval impact hypothesis.
- DR/failover model route—no silent cross-border failover.
- Link eval dimensions to topic 21 thresholds.
Acceptance criteria: CISO question "what is logged?" answered explicitly.
Reflection exercise: Fine-tune vs RAG debate (45 minutes)
Write pro/con memo (one page each side) for fine-tuning on 1,500 support tickets vs RAG over KB. Conclude with decision and kill criteria. File in pattern library.
Questions you should be able to answer
- How many tokens does a typical request consume—and what drives cost at 10× volume?
- Why use embeddings vs generative model for search?
- What is in the context window for your design—and what was excluded?
- How does the model differ from the retrieval corpus on freshness and authority?
- What is instruction tuning—and why not use a base pretrained checkpoint?
- When would PEFT beat prompt + RAG for this use case?
- What quantisation level is acceptable given JSON/tool reliability requirements?
- What temperature and sampling settings apply in production—and why?
- Where is inference physically processed—and what crosses borders?
- Are prompts/completions logged, retained, or used for training—per contract?
- What is the failover model if primary route is down—residency preserved?
- Which tasks require multimodal vs text-only models?
- How do tool calls get validated before side effects?
- What eval score gates promotion from pilot to wider rollout?
- Why this model family versus smaller local or larger frontier—on your metrics?
Negative cases
Context window as corpus strategy. Stale policies stuffed into prompt. Fix: RAG + freshness SLAs.
Public API on raw PII. Regulatory breach. Fix: masking, private deploy, contract review.
Fine-tune on tiny set. Overfit tone; wrong facts. Fix: RAG first; more labels or PEFT with regularisation.
Leaderboard shopping. MMLU score ≠ your German HR corpus. Fix: internal eval set.
Single temperature everywhere. Creative settings on compliance tasks. Fix: param profiles per intent.
Tool agent without guardrails. Deletes records. Fix: read-only tools; human approval; topic 14 bounds.
Ignoring token economics. Pilot £2K/month → production £180K/month surprise. Fix: FinOps model day one.
Quantisation without regression. JSON errors spike. Fix: eval gate on quantised artifact.
No routing. Frontier on every "hello." Fix: intent classifier and tiered models.
Multimodal overreach. Vision decides coverage. Fix: human/rules on decisions; vision descriptive only.
Vendor lock-in surprise. Cannot export LoRA at contract end. Fix: portability clause and open interface.
Alignment trust. Model refuses then jailbroken in red-team. Fix: layered controls, not model alone.
Integration with AI Patterns and Model landscape
| Topic 11 capability | Resource |
|---|---|
| Solution shapes (RAG, agents, routing) | AI Patterns |
| Model catalogues and benchmarks context | Model landscape |
| Prompt design | Topic 12 |
| Retrieval layer | Topic 13 |
| Evaluation gates | Topic 21 |
Crosswalk to architecture and ops stages
Model choices in topic 11 constrain Stage 3 (hosting, GPU, network) and Stage 4 (logging, DLP, residency controls). Capture non-functional requirements emergent from selection matrix:
- GPU type and count if self-host quantised model
- Egress paths if API-based with retrieval
- Log retention days aligned to privacy policy
- Fallback route documented in runbook—not improvised at outage
Revisit matrix when Model landscape updates quarterly—new SKUs change cost/latency frontier.
Study drill: explain transformers in 90 seconds
Practice executive explanation without whiteboard: "The model predicts the next word based on everything in the prompt—including retrieved policy text. It does not browse your database like a search engine; we must put the right excerpts in the prompt and cite sources. Longer context costs time and money; retrieval keeps answers fresh when policies change." Record and trim filler words—target ≤90 seconds.
Practice checklist
- I can explain tokens, context, and attention at executive level without jargon overload
- Selection matrix includes quality, latency, cost, privacy—not quality alone
- I compared at least one smaller/regional model to frontier on internal criteria
- Data processing and logging documented for legal review
- Fine-tune vs RAG decision explicit with kill criteria
- Practice artefacts filed in pattern library
Related playbook content
- AI Patterns — RAG, agents, routing, and deployment patterns
- Model landscape — model families, licences, and capability bands
- Data & Knowledge Guide — corpus readiness for grounded GenAI
- 8D Framework — Design and Develop model decisions
- How to use this Learning Map — study method
- AI and data foundations — curriculum context
Supplemental scenario — legal firm model selection with cost envelope
Context: 120-partner law firm; contract review assist; €180k annual inference cap; documents 80% English, 20% German.
Selection matrix (excerpt):
| Candidate | Quality on golden (F1) | P95 latency | €/1k successful reviews | EU residency |
|---|---|---|---|---|
| Frontier A | 0.89 | 8.1s | €4.20 | Yes (region) |
| Mid B | 0.84 | 3.4s | €1.10 | Yes |
| Small C | 0.79 | 2.1s | €0.55 | No — reject |
Routing table: Tier 1 clauses (liability, indemnity) → Frontier A with mandatory human sign-off; Tier 2 boilerplate → Mid B; batch OCR cleanup → rules pipeline, not LLM.
Token economics: Average pack 18k tokens; at Mid B pricing, €0.019/review vs €0.076 Frontier-only path—projected €142k annual at 40k reviews vs cap breach in month 9 on Frontier-only.
Adaptation decision: RAG + prompt first; PEFT deferred—labelled clause set only 2,400 examples; kill criterion "PEFT if human-edit rate >25% after RAG v2."
Discussion
Comments
Share feedback or questions about this page. No account required.
Loading comments…