MLOps, LLMOps and Observability

Executive summary

MLOps and LLMOps make AI systems operable in production: versioned models, prompts and indexes; deployment pipelines with eval gates; canaries and rollbacks; structured tracing and logging; drift detection on quality and data; SLI/SLO/SLA targets; and incident response tailored to LLM failures (retrieval regression, prompt drift, tool loops, cost spikes). Classical MLOps emphasised training-serving skew and model registries; LLMOps adds prompt registries, RAG index versions, token/cost telemetry, retrieval-source attribution and LLM-specific runbooks. An AI Solution Engineer defines observability so when faithfulness drops from 92% to 78% after a corpus update, engineers see which chunker version, index build and model changed—rollback completes in <15 minutes, post-incident case enters golden set (topic 21). Operating without traces is blind flight; operating without rollback is hope deployment.

When to use this page

• Before declaring pilot "production" or scaling traffic beyond informal monitoring.
• When building CI/CD for models, prompts, embeddings or agent configurations.
• When users report quality degradation and team cannot reproduce retrieval path.
• When FinOps flags cost spike or latency SLO breaches.
• When connecting observability to governance kill-switches (topic 19) and security monitors (topic 17).
• After silent deploy of chunker, index or prompt without eval gate caused incident.
• Pair with AI Evaluation for offline gates and Evaluation & observability playbook for implementation.

Executive view

Ask for **SLO dashboard**, **last rollback drill date**, **MTTR** on last Sev2 incident, and **cost per successful task**. Challenge scale when team cannot answer "what changed?" within one trace ID.

Decision required: Are production deployments gated on eval—and can we revert model, prompt and index versions independently within agreed minutes?

Technical view

Implement OpenTelemetry traces with `trace_id`, model/prompt/index versions, retrieval IDs, tool calls, token counts. Wire CI/CD: eval regression → staging canary → prod rollout. Automate drift alerts on faithfulness proxies and zero-citation rate.

Maintain rollback runbooks tested quarterly; integrate incident loop to golden set (topic 21).

Why this matters

LLM applications fail gradually then suddenly: citation quality erodes over weeks after corpus drift; a prompt edit breaks JSON parsing; a new model version loops tools until budget exhaustion. Without tracing, teams debate whether the model "got worse." Without versioning, rollback means git archaeology. Without SLOs, leadership discovers problems from Twitter, not dashboards.

MLOps maturity separates demos from products. A RAG assistant serving 12,000 queries/day without retrieval attribution cannot explain wrong refund advice. An agent without step-level traces cannot debug why it called payment API twice. MTTR measured in days converts to regulatory interest, customer churn and engineer burnout.

The AI Solution Engineer owns observability strategy, SLO definition, deployment pipeline requirements, drift detection design and incident runbooks—partnering with SRE for implementation. You align online signals with offline eval (topic 21) so production degradation triggers the same metrics used at release gates.

Weak programmes log "request finished" and watch CPU. Strong programmes trace full orchestration, alert on citation missing rate, block deploys when golden set regresses, run canary index builds, and document rollback with owners and timers. When incident commander asks "what changed at 14:32 UTC?" you answer with deployment timeline + trace exemplar—not shrugs.

Connect to Evaluation & observability playbook, RAG, Agents, and Performance and FinOps.

Learn

MLOps vs LLMOps — shared and distinct

Concern	Classical MLOps	LLMOps extension
Artefact versioning	Model weights	+ Prompts, tools schema, index builds
Training pipeline	Feature pipeline, retrain	+ Fine-tune jobs; often inference-only
Serving	Model endpoint	+ RAG orchestrator, agent runtime
Monitoring	Prediction drift, latency	+ Faithfulness proxies, retrieval quality, tokens
Rollback	Previous model version	+ Prompt version, index snapshot
Eval	Holdout AUC	+ Golden conversational eval (topic 21)

Many enterprises run inference-only LLMOps—still needs registries and gates.

Deployment pipelines — promote with evidence

Typical stages:

Pipeline inputs versioned together or compat matrix documented:

• Application code
• model_version (API deployment name)
• prompt_version (system + task templates)
• index_version / embedding_model_version
• tool_schema_version
• Feature flags

Promotion checklist:

• CI eval gates green (topic 21)
• Model card / inventory updated (topic 19)
• Security scans pass (topic 17)
• Changelog entry with risk tier note
• Rollback steps verified in staging

Registries — models, prompts, indexes

Model registry: deployment names, provider, region, cost tier, approval date, eval report hash.

Prompt registry: semantic versioning (sys-v3.2), author, change reason, linked golden eval run, diff review for Tier High.

Index registry: corpus snapshot ID, chunker version, chunk count, build timestamp, source document hash manifest.

Tool registry: JSON schema, authZ scope, owner—sync with security abuse catalogue.

Treat registries as source of truth—deploy pulls by ID, not "latest file on disk."

Canary, shadow and blue-green for LLM

Pattern	Use
Canary	5% traffic on new prompt/index; compare SLOs
Shadow	New stack runs parallel; no user impact; compare offline metrics
Blue-green	Switch traffic at gateway; instant rollback pointer
A/B	Product experiment with statistical plan (topic 21)

LLM-specific canary metrics: faithfulness proxy, citation rate, refusal rate, latency p95, cost/query, tool error rate.

Auto-rollback if canary breaches guardrails for N minutes.

Drift detection — data, model and quality

Drift type	Signal	Response
Data/corpus	New doc ratio, embedding distribution shift	Re-index eval gate
Query	Intent distribution change	Review golden coverage
Model provider	Silent model update notification	Mandatory re-eval
Quality	Zero-citation rate ↑, human fail sample ↑	Rollback prompt/index
Cost	Tokens/query ↑ 20%	Prompt or model investigation
Latency	p95 ↑	Scale or retrieval optimisation

Drift alerts should link to trace samples for triage—not only aggregate charts.

Tracing — OpenTelemetry for LLM apps

Minimum span hierarchy:

Correlate with logs via trace_id; expose to support portal (role-restricted).

Privacy: align redaction with topic 18—attributes over raw content in default export.

Logging — structured and actionable

Log fields (JSON):

• timestamp, trace_id, session_id_hash, user_id_hash
• intent, outcome (answered, refused, escalated, error)
• model_version, prompt_version, index_version
• latency_ms, input_tokens, output_tokens, cost_estimate
• retrieval_count, citation_count, tool_calls_count
• error_code, security_flag (DLP block, injection pattern)

Avoid: full prompt/response in default logs.

Retention: per topic 18 schedule; break-glass encrypted store if needed.

SLI, SLO, SLA and error budgets

Example SLIs for RAG assistant:

SLI	Measurement
Availability	Successful responses / total requests
Latency	p95 end-to-end < 5.5s
Quality proxy	Zero-citation rate < 8%
Correct escalation	Refusal rate within band on negative set proxy
Cost	Mean cost per successful answer < £0.04

SLO: 99.5% availability monthly; quality proxy within band 28-day rolling.

Error budget: 0.5% unavailability/month—when exhausted, freeze feature releases until stability restored.

SLA: external customer contract—stricter; ties to support credits.

Publish SLO dashboard to steering quarterly.

Incident response for LLM applications

Severity matrix (illustrative):

Sev	Example	Response target
1	Mass wrong financial advice live	Kill-switch <15 min; exec comms
2	Faithfulness proxy ↓25% sustained 2h	Rollback <60 min
3	Single-tool errors ↑	Fix next business day
4	Cosmetic formatting	Backlog

IR phases:

1. Detect — alert or user report with trace_id
2. Triage — classify Sev; assign incident commander
3. Contain — kill-switch, disable tool, rate limit
4. Diagnose — compare versions, retrieval samples, recent deploys
5. Recover — rollback or hotfix with expedited eval
6. Post-incident — golden case, abuse catalogue update, governance notify

Runbook sections: contacts, flag locations, rollback commands, comms templates, regulatory notify criteria (link topic 18).

Rollback runbook — independent levers

Lever	Action	Typical time
Feature flag	Off assistant	<5 min
Prompt version	Pin previous in registry	<10 min
Index version	Route to prior build	<15 min
Model deployment	API alias swap	<15 min
Tool disable	Config off single tool	<5 min

Test quarterly in staging; record drill in governance pack (topic 19).

FinOps observability

Track:

• Tokens in/out by model, feature, tenant
• Cost per successful task (denominator from outcome flag)
• Cache hit rate on embeddings/LLM
• Budget burn rate vs forecast

Alert 3× normal daily spend; integrate with security DoW (topic 17).

See Performance and FinOps.

Continuous evaluation in production

Bridge topic 21:

• Sample production queries for human review (1–5% stratified)
• Nightly prod-shadow against golden subset
• Automated checks on sampled outputs (citation validator)
• Feedback loop — thumbs feed triage queue, not automatic training

Online eval validates offline gates; divergence triggers golden set refresh.

Agent-specific observability

Additional spans/metrics:

• Steps per task, loop detection
• Tool success/failure by name
• Token budget consumption per episode
• HITL wait time and override rate
• Task completion flag vs abandonment

Alert on runaway loops (>10 LLM calls without terminal tool).

ML training path observability (when applicable)

If fine-tuning:

• Experiment tracking (MLflow etc.)
• Dataset version hash
• Eval on holdout after train
• Promotion same gate discipline as prompts

Most regulated enterprises limit training frequency—still document pipeline.

Frameworks and methods

OPS-LLM — observability design checklist

Step	Question
Owners	SRE + ASE on-call rotation defined?
Pipeline	Eval gates on promote path?
SLOs	3 SLIs with numeric targets?
Logs/traces	trace_id + versions on every request?
L drift	Alerts on quality proxies?
M rollback	Runbook tested this quarter?
Incident	Sev matrix + kill-switch linked?
Review	Weekly triage of top failures?

Deployment gate diagram

Dashboard panels — first screen in incident

1. Availability and latency p95/p99 (28d)
2. Error rate by class (4xx, 5xx, model error)
3. Zero-citation rate (24h rolling)
4. Token cost per hour vs baseline
5. Recent deploy timeline (model, prompt, index)
6. Top 5 tool errors
7. Human review fail rate (sampled)

Online/offline alignment (from topic 21)

Offline metric	Online proxy	Alert
Faithfulness ≥90%	Human sample fail ≤12% weekly	>12%
Citation ≥90%	Zero-citation rate	>8% daily
Refusal on negative	Escalation rate on unanswerable class	+5 pts vs baseline

Incident commander checklist (first 30 minutes)

• Confirm Sev and open incident channel
• Capture exemplar trace_ids from reports/alerts
• List deploys last 24h (all version dimensions)
• Decide contain: flag off vs rollback vs tool disable
• Assign scribe for timeline
• Notify governance if Sev1/2 Tier High
• Preserve logs/traces before TTL expiry if needed

Real-world scenarios

Scenario A — UK insurer: faithfulness drop after corpus update

Context: Customer policy RAG; 12,000 queries/day; faithfulness human sample 6.2% fail (within SLO). After quarterly corpus ingest, fail rate jumps to 19% over 72 hours.

Tracing analysis:

• index_version changed 2026-Q2-build-04 → 2026-Q2-build-05
• Chunker config changed overlap 10% → 25% (misconfigured YAML)
• Retrieval IDs show split clauses breaking citation spans

Response:

• Rollback index pointer to build-04 in 11 minutes
• Fail rate returns to 6.8% within 24h
• Post-incident: chunker change requires PR eval gate + canary index build
• Golden set +15 chunk-boundary cases

Lesson: Index versioning and rollback must be independent of app deploy.

Scenario B — Retail bank: model upgrade latency and cost

Context: Team upgrades planner model for internal agent; p95 latency 3.2s → 8.7s; cost/query £0.02 → £0.06.

Observability catch:

• Canary deployment 5% traffic; alert fires at 2 hours
• Traces show duplicate retrieval calls after model change (prompt regression)

Fix: Prompt v2.4 restores single retrieval; partial rollback keeps new model for generation only.

Numbers:

• Full rollout avoided; saved ~£40K/month at projected volume
• Final config: p95 4.1s, cost £0.025—accepted

Lesson: Canary + cost/latency SLOs catch upgrades that pass offline task accuracy.

Scenario C — SaaS support bot: tool loop incident

Context: Agent with update_ticket, search_kb, close_ticket; provider silent update increases tool-calling eagerness.

Incident: 847 tickets receive duplicate updates in 3 hours; tool loop average 14 LLM rounds per session.

Containment:

• Disable update_ticket tool via config 6 minutes
• Kill-switch not required—surgical tool off

Diagnosis: Traces show loop pattern; eval harness lacked max_steps gate.

Remediation:

• Hard cap 8 LLM steps; circuit breaker on repeated identical tool args
• CI agent scenario AGENT-LOOP-09 added
• MTTR 45 minutes to stable; customer apology campaign

Lesson: Agent traces and step caps mandatory for write tools.

Scenario D — Regulator request: reproduce answer from 90 days ago

Context: FCA asks how answer to specific policy question was produced 90 days prior.

Evidence retrieved via observability:

• trace_id from audit log metadata (content not stored)
• Archived index manifest 2026-Q1-build-02 recreated in read-only sandbox
• prompt_version sys-v3.1, model_version gpt-4.1-2026-01
• Reproduction match on retrieval IDs and output hash

Outcome: Satisfactory evidence without retaining full prompt text in logs—metadata-rich design validated (topic 18 alignment).

Lesson: Reproducibility requires version manifests—not eternal full-text logs.

Practice exercises

Primary exercise — SLO and dashboard design (60 minutes)

Brief: Internal HR policy Q&A RAG; 5,000 employees; SSO; Tier Medium governance.

Tasks:

1. Define 3 SLIs and numeric SLOs (availability, latency, quality proxy).
2. Specify trace attributes minimum set.
3. Design 7 dashboard panels for incident commander first screen.
4. Define 2 rollback levers with time targets.
5. Map one offline metric to online proxy and alert threshold.

Acceptance criteria:

• Quality proxy is measurable without storing full prompts in logs
• SLOs numeric with measurement window stated
• Rollback targets align with topic 19 kill-switch expectations for tier

Stretch exercise — Full LLMOps pack (full day)

Brief: Customer-facing telecom tariff FAQ RAG; 200k queries/month; multi-region; Tier High.

Tasks:

1. Deployment pipeline diagram with CI eval stages (integrate topic 21 gates).
2. Registry schema for model, prompt, index (fields and owners).
3. Canary plan — traffic %, duration, guardrails, auto-rollback rules.
4. Drift detection — 5 signals with thresholds and owners.
5. Incident runbook — Sev1-3, contacts, kill-switch, comms.
6. Rollback drill script for staging execution.
7. FinOps dashboard — cost drivers and alerts.
8. Post-incident loop to golden set documented.

Acceptance criteria:

• Pipeline blocks promote on golden fail—not prod-only discovery
• Trace schema includes retrieval_ids and version triple
• IR links to governance notification for Sev1/2

Questions you should be able to answer

1. What SLOs apply and current error budget status?
2. What trace_id fields prove retrieval and model path?
3. How are model, prompt and index versions recorded per request?
4. What deployment pipeline gates block promote?
5. How does canary rollout work and what triggers rollback?
6. What drift signals fire alerts and who responds?
7. What is MTTR on last significant incident?
8. Can you rollback each version dimension independently—in what time?
9. What appears on the incident dashboard first?
10. How is cost per successful query measured?
11. What logs exist by default vs break-glass?
12. How does online sampling connect to offline golden eval?
13. What agent-specific metrics apply (loops, tools, HITL)?
14. When was rollback drill last executed successfully?
15. What post-incident updates feed eval and governance?

Negative cases — when MLOps/LLMOps fails

Failure mode	Symptom	Prevention
Deploy without eval	Quality cliff	CI golden gates
No trace_id	Cannot debug	OTel mandatory
Version soup	Unknown prod state	Registries
Log prompts only	Privacy + no metrics	Structured metadata
Manual rollback	Hours MTTR	Runbook + drills
Single lever rollback	Index bad, prompt rolled only	Independent pointers
Vanity uptime SLO	Available but wrong	Quality proxies
Ignore cost alerts	Invoice shock	FinOps dashboards
Silent vendor update	Mystery regression	Provider change trigger
Incident amnesia	Repeat failure	Golden case loop

Case study: silent chunker deploy. Engineer changes chunker locally; CI skips index rebuild test; prod index rebuild Sunday night. Monday +40% escalations. No index versioning—18-hour rebuild rollback. Cost £220K estimated handling surge.

Case study: the missing canary. Full prompt swap to prod 100%. JSON parse errors 30%. Rollback 45 min—should have been 5% canary with schema gate.

Case study: observability vendor lock. Traces in SaaS without export. Contract ends; no historical MTTR data for audit. Now contract requires OTLP export to enterprise backend.

Operating-model notes

Role	Responsibility
ASE	SLO definition, eval gate spec, runbook authorship
SRE / platform	Pipeline, dashboards, on-call
ML engineer	Registry, canary automation
Product	Online feedback, A/B design
Governance	Sev1/2 notification
Security	DoW alerts integration

Cadence: Weekly failure triage (top traces); monthly SLO review; quarterly rollback drill.

Integration across Stage 4

Topic	Ops integration
17 Security	Injection/DLP flags in logs
18 Privacy	Redacted trace schema, TTL jobs
19 Governance	Kill-switch in runbook; deploy ↔ model card
21 Eval	CI gates, prod sampling, golden loop

On-call and escalation for LLM incidents

L1 (support): Collect trace_id, user impact description; link known issues.

L2 (platform/SRE): Dashboard triage; execute runbook rollback levers within authority.

L3 (ASE/ML): Root cause on retrieval/prompt/model; expedited eval for hotfix.

L4 (incident commander): Sev1/2; governance notify; external comms with corporate affairs.

Escalation triggers: SLO burn rate >2× normal; Sev1 security/privacy overlap; media mention.

Rotate on-call with runbook training—not only generic infra engineers.

Feature store vs prompt store operational distinction

Feature store (classical ML): Batch features for training/serving parity.

Prompt store (LLMOps): Git or dedicated registry with PR review, semver tags, diff visualisation, linked eval runs.

Anti-pattern: Prompts edited in prod config UI without version bump—audit nightmare.

Enforce: prod pulls only tagged prompt versions; emergency hotfix still tags hotfix-YYYYMMDD-HHMM.

Index rebuild orchestration

Large corpus re-index may take hours:

1. Build new index version offline (build-NN+1)
2. Run eval against new index in staging—compare to build-NN
3. Canary route 5% queries to new index
4. Promote pointer or rollback
5. Retain build-NN 7 days minimum for rollback

Monitor build job failures—partial index worse than old complete index.

Synthetic monitoring and probes

Hourly canary queries with known expected citations (synthetic accounts):

• Probe fails → page on-call before user flood
• Separate probes per region and intent class
• Store probe results 90 days for trend

Cheap insurance vs waiting for CSAT collapse.

Chaos engineering for LLM apps (controlled)

Quarterly exercises:

• Kill model API endpoint in staging—verify graceful degradation message
• Simulate retrieval timeout—verify partial answer or honest failure
• Spike latency 5×—verify queue backpressure not unbounded retry storm
• Disable single tool—verify agent recovery path

Document surprises in runbook updates.

Scenario E — Black Friday traffic surge

Context: E-commerce product Q&A; normal 2k q/day; Black Friday 48k q/day; p95 latency SLO 5s breached at 12s; error rate 4%.

Ops response:

• Auto-scale orchestrator pods +300%
• Route 20% traffic to smaller model with disclaimer banner
• Enable aggressive retrieval cache (TTL 15 min)
• Pause non-critical eval batch jobs

Outcome: p95 back to 5.8s within 4 hours; SLO miss for 6 hours—within monthly error budget if pre-approved seasonal plan; post-mortem adds seasonal capacity playbook.

Lesson: Capacity observability and pre-approved degradation modes are ops requirements.

Scenario F — Observability gap in multi-vendor chain

Context: App → orchestration SaaS → Azure OpenAI; incident in middle tier; only outer logs available; 6-hour blind spot.

Fix: Require OTLP export from all tiers; contractual SLA on trace delivery; end-to-end trace_id propagation contract.

Cost: £15K/year additional observability spend; MTTR improved 4.2h → 55min on subsequent incident.

Lesson: Multi-vendor chains need trace propagation in contracts—not assumptions.

Production config drift detection

Alert when prod config differs from last approved deploy artefact:

• prompt_version in runtime ≠ registry expected
• Feature flag enabled without ticket
• Model deployment alias drift

Weekly config reconciliation job compares live vs Git/registry manifest.

Handoff to support and CX

Support macros with:

• How to collect trace_id
• Known limitations from model card excerpt
• When to escalate to AI ops vs generic IT
• No asking customer to paste full chat if policy prohibits

Train support before launch—not week after.

MLOps/LLMOps tooling landscape (vendor-neutral)

Categories to evaluate:

• Experiment tracking — MLflow, W&B (fine-tune path)
• Prompt management — git, LangSmith, Humanloop, custom registry
• Observability — Langfuse, Arize, Datadog LLM, OpenTelemetry backends
• Eval harness — custom CI, promptfoo, DeepEval (topic 21)

Enterprise standard: one approved stack per concern; avoid每个 team different SaaS with no export.

Weekly ops review agenda (30 min)

1. SLO status and error budget
2. Top 3 trace exemplars of user-visible failures
3. Deploy changelog since last week
4. Cost vs forecast variance
5. Open incidents and runbook gaps
6. One golden case candidate from prod

Output: action owners; feeds topic 21 golden maintenance.

Performance regression vs quality regression

Distinguish in triage:

Symptom	Likely layer	First action
Slow but correct	Scale/retrieval	Profile spans
Fast but wrong	Prompt/model/index	Compare versions
Tool errors only	AuthZ/tool backend	Tool registry diff
Sporadic both	Dependency flake	Check provider status + retries

Avoid rolling back model when retrieval index is root cause—use independent levers.

SLI specification template (copy for engagements)

sli_id: rag-citation-rate
description: Percentage of answers with at least one valid citation ID
measurement_window: 28d rolling
data_source: structured logs + citation validator
good_event: citation_count >= 1 AND validator_pass = true
valid_event: successful_response = true
objective: 0.92  # 92% SLO target
alert_burn_rate: 14.4x multi-window per Google SRE practice
owner: ase-lead@company.com

Document denominator explicitly—avoid SLO gaming by excluding error responses silently.

Deployment rollback post-mortem template

1. Incident timeline (UTC)
2. Versions before/after (model, prompt, index, code)
3. Detection method (alert vs user report)
4. Rollback levers used and actual times
5. User impact estimate (queries affected, error rate)
6. Root cause category
7. Golden cases added (IDs)
8. Pipeline changes to prevent recurrence
9. Governance notification record

File within 5 business days of Sev2+; feeds board incident review.

Observability cost optimisation

Technique	Saving	Trade-off
Sample traces 10% at steady state	~90% trace storage	Harder debug rare issues—raise sample on alert
Aggregate metrics pre-compute	Lower query cost	Less ad-hoc exploration
Cold storage logs >30d	Cheaper retention	Slower DSAR queries—legal alignment
Drop verbose debug spans in prod	CPU + storage	Enable temporary debug flag per trace_id

Balance with privacy minimisation—cheaper must not mean more PII.

Integration with enterprise ITSM

• Auto-create incident ticket from Sev1/2 alert with trace exemplars attached
• Link deploy tickets to eval report artefacts
• Change management CAB references inventory ID for AI prod changes
• Problem records for recurring failure clusters (not only incidents)

ITSM integration reduces tribal knowledge when on-call rotates.

LLM provider outage playbooks

When Azure OpenAI / Bedrock / Vertex regional outage:

1. Confirm provider status page
2. Failover to secondary region if residency allows
3. Else degrade: cached FAQ responses, static message, queue requests
4. Communicate ETA on status page
5. Post-incident: test failover quarterly

Document residency constraints preventing failover—steering pre-accepts downtime vs transfer risk.

Runbook excerpt — faithfulness proxy alert

Alert: zero_citation_rate > 8% for 15 minutes

Steps:

1. Acknowledge pager; open dashboard panel 3
2. Check deploy timeline last 2 hours—all version dimensions
3. Pull 5 sample trace_ids with zero citations
4. Compare retrieval_ids—empty retrieval vs model omission?
5. If index deploy correlates → rollback index pointer (Runbook RB-INDEX)
6. If model/prompt → rollback respective version
7. If inconclusive → increase human sample rate to 10% temporarily
8. Update incident channel every 30 min until resolved
9. Post-incident golden case within 48h

Target MTTR <60 min for Sev2.

Building eval into deployment pipeline (YAML sketch)

stages:
  - smoke_eval:
      cases: 50
      block_on_fail: true
  - staging_full_eval:
      cases: 240
      metrics: [faithfulness, citation, acl]
      block_on_fail: true
  - canary_prod:
      traffic_percent: 5
      duration_hours: 2
      guardrails: [p95_latency, zero_citation_rate]
      auto_rollback: true

Adapt to CI platform; link report URLs to inventory entry on success.

Capacity planning for LLM inference

Estimate monthly cost:

queries × (avg_input_tokens + avg_output_tokens) × price_per_1k_tokens
+ embedding_reindex_cost + observability_overhead

Plan headroom 40% for launch week; negotiate provider quota increases 2 weeks before marketing campaigns.

Capacity review tied to FinOps dashboard and seasonal playbooks (Scenario E Black Friday).

Multi-environment promotion discipline

Environment	Eval depth	Data
Dev	Smoke 20 cases	Synthetic only
Staging	Full golden	Anonymised or synthetic
Pre-prod	Full + red team subset	Prod-like masked
Prod	Canary + online monitors	Real

Anti-pattern: prod-only debugging because staging lacks parity—invest in staging corpus representativeness.

Long-term observability maturity roadmap

Quarter 1: Traces + basic SLOs + manual rollback runbook

Quarter 2: CI eval gates + automated canary + drift alerts

Quarter 3: Synthetic probes + chaos exercises + ITSM integration

Quarter 4: Error budget policy in change management + cross-system trace contracts

Maturity roadmap in steering pack sets expectations—avoid "full observability day one" fiction.

Alert noise reduction for LLM metrics

• Use multi-window burn rates for SLO alerts—not single spike pages
• Suppress known provider outage windows after status confirmation
• Group related alerts (latency + error rate) into one incident
• Weekly alert tuning review with on-call feedback

Alert fatigue causes ignored faithfulness drops—tune as carefully as thresholds.

Post-deployment monitoring calendar

Period	Activity
Day 1–7	Enhanced alerting (lower thresholds); daily standup
Week 2–4	Weekly SLO review; human sample at 5–10% if Tier High
Month 2–3	Fortnightly ops review; drift baseline stabilises
Quarterly+	Standard on-call; rollback drill; golden refresh trigger

Document in observability plan—prevents "launch and forget" when hypercare ends abruptly.

Trace-driven debugging worked example

User report: "Wrong cooling-off period quoted."

Investigation:

1. Support collects trace_id from admin panel
2. Trace shows retrieval_ids: [POL-2024-033#chunk-12]
3. Manifest shows chunk from superseded policy version 2024-03 vs current 2025-01
4. Index pointer still on 2025-Q1-build-02 but source doc not re-ingested after CMS update
5. Fix: CMS webhook triggers re-ingest; add freshness eval gate

Time to root cause: 22 minutes with traces vs estimated days without.

Publish anonymised exemplars in team wiki for training.

Version compatibility matrix

Document which combinations are tested in CI:

App version	Prompt	Index	Model	Status
2.1.0	sys-v3.2	build-05	gpt-4.1	✅ golden pass
2.1.0	sys-v3.1	build-05	gpt-4.1	⚠️ deprecated
2.0.x	sys-v3.2	build-04	gpt-4.1	❌ unsupported

Prevents "works on my laptop" deploy combos; registry enforces allowed tuples for prod promote.

On-call handoff template

Outgoing: Active incidents, recent deploys, known drift alerts, error budget status, scheduled index rebuilds.

Incoming: Acknowledge pager duty; confirm runbook access; verify kill-switch credentials in break-glass vault test monthly.

Poor handoff caused 2-hour delay in Scenario C tool loop—mandatory written handoff in ITSM ticket.

Reliability targets by tier (illustrative)

Tier	Availability SLO	MTTR target	Rollback drill
Low internal	99.0%	4 business hours	Annual
Medium	99.5%	2 hours	Semi-annual
High customer	99.9%	60 minutes	Quarterly
Critical	99.95%	15 minutes contain	Monthly

Align on-call staffing with tier—High customer-facing requires 24×7 coverage or explicit downtime acceptance in SLA.

Observability acceptance criteria for go-live

• Dashboard live with all seven first-screen panels
• Alerts routed to on-call with runbook links
• Trace sampling verified on 10 prod requests
• Rollback drill completed in staging within target time
• Log retention TTL jobs deployed and verified
• Cost dashboard baseline captured for week 1 comparison

Gate Stage 4 exit topic 20 on checklist completion—not "we'll add monitors later."

Ops acceptance is part of Demonstrate in the 8D framework—treat observability as launch criteria, not backlog.

Related playbook content

• Evaluation & observability — tracing, dashboards, pipeline patterns
• Governance — kill-switch and inventory linkage
• Security & privacy — security monitoring integration
• Retrieval-Augmented Generation — index lifecycle
• Agentic AI and Workflow Automation — agent metrics
• Security Engineering
• Privacy, Legal and Compliance
• Responsible AI and Governance
• AI Evaluation and Quality Assurance
• Performance and FinOps
• AI Data Scientist roadmap · Full guide
• How to use this Learning Map
• 8D Framework

Output

Expected outputs for this topic:

• Observability plan — traces, logs, metrics, retention alignment
• SLO/SLI definition — numeric targets, error budgets, measurement method
• Deployment pipeline spec — stages, eval gates, canary rules
• Registry design — model, prompt, index versioning fields
• Drift detection catalogue — signals, thresholds, owners
• Dashboard wireframe — incident first-screen panels
• LLM incident runbook — Sev matrix, containment, rollback, comms
• Rollback drill record — dates, levers tested, time achieved

Practice checklist

• I defined ≥3 SLIs with numeric SLOs and measurement windows
• I specified trace attributes including model, prompt, index versions
• I designed deployment pipeline with eval gates before prod
• I documented canary guardrails and auto-rollback triggers
• I listed drift signals beyond uptime and latency
• I wrote incident runbook with Sev levels and kill-switch linkage
• I mapped offline metrics to online proxies with alert thresholds
• I filed observability pack with rollback drill plan in pattern library

Key takeaways

• LLMOps extends MLOps with prompt, index and tool versioning plus conversational quality monitoring.
• Deployment pipelines must gate on eval regression—not deploy-then-hope.
• Tracing with retrieval attribution is minimum viable debuggability for RAG and agents.
• Drift detection on quality proxies catches corpus and model changes before escalations spike.
• Rollback runbooks with tested levers and time targets reduce MTTR from days to minutes.
• Incident response closes loop to golden set and governance—ops is not separate from quality.

What to do next

1. Complete the SLO and dashboard exercise for the HR policy assistant brief.
2. Continue to AI Evaluation and Quality Assurance — align CI gates with observability proxies.
3. Open Evaluation & observability playbook to implement pipelines and dashboards.
4. When Stage 4 exit criteria pass, continue to Commercialisation and delivery.