Operations and Production Support

Executive summary

Operations for AI is not standard ITIL with a chatbot skin. Production support must cover runbooks for retrieval failure, faithfulness degradation, prompt regression, model version drift, cost spikes, provider outage, safety incidents and misuse—alongside classical incident, problem, change and release management. You define SLOs that matter to users, on-call rotations with AI-specific first-response checklists, operational readiness reviews before launch, and model version rollback paths tested under pressure. An AI Solution Engineer who hands off "to BAU" without ORR artefacts leaves SRE teams flying blind when the copilot starts citing withdrawn policy at 2 a.m.

When to use this page

• When transitioning from pilot to production ("move to run").
• When defining or reviewing on-call, runbooks and support tiers for an AI service.
• After an incident: harmful output, data leak via prompt, runaway agent, or cost anomaly.
• When setting SLOs/SLAs that reflect quality—not only API uptime.
• Before major release: model swap, corpus bulk update, new tool integration.
• When DR/BCP must cover vector indexes, embedding pipelines and model failover.

Executive view

Ask whether the service has operational readiness—named on-call, tested rollback, customer comms templates, and SLOs for quality degradation—not a launch date alone.

Decision required: Can we disable or roll back the AI feature in minutes without taking down the whole platform?

Technical view

Build runbooks for AI failure modes; wire observability to alerts; practise rollback; align change management with eval re-runs.

Pair with [Delivery and Operating Model](/ai-solution-engineering/delivery) for release trains and environment promotion.

Why this matters

Traditional application operations optimises for availability and error rate. AI services fail in probabilistic and semantic ways:

• API green but answers wrong after corpus update
• Retrieval returns empty silently; model hallucinates confidently
• Prompt change in production regresses safety refusals
• Token usage spikes 8x from loop or abuse
• Provider deprecates model with 30-day notice
• Agent calls write tool with wrong customer ID

Without AI-aware operations, teams either over-react (disable everything at first alert) or under-react (treat harmful output as "user error"). Strong ops produces:

• Predictable incident response with severity matrix
• Runbooks that on-call can execute without calling the original build team
• SLOs on faithfulness, latency and cost—not only uptime
• Change discipline linking releases to eval gates
• DR that includes search indexes and model routing failover

Weak ops produces reputational incidents, regulatory notification, and consulting permanence because nobody else knows how the system behaves under failure.

Learn

ITIL-aligned ops concepts for AI services

Process	Purpose	AI adaptation
Incident	Restore service quickly	Include harmful output, quality collapse
Problem	Root cause	Prompt regression, corpus pollution, model drift
Change	Controlled release	Model version, index rebuild, tool addition
Release	Deploy packages	Blue/green model routes; canary on eval slice
Capacity	Right-size resources	Token budgets; GPU; index size growth
Knowledge	Runbooks, known errors	Eval baselines; rollback steps

You need not certify in ITIL—but speak the language of enterprise ops teams.

Operational Readiness Review (ORR)

Before production launch, complete ORR checklist:

• On-call roster and escalation matrix published
• Runbooks for top 10 failure modes drafted and walkthrough completed
• Observability: traces, logs, metrics, eval sampling in prod
• SLOs defined and dashboarded
• Rollback tested (model + prompt + index version)
• Kill-switch or feature flag documented
• Support tiers: L1 script, L2 engineering, L3 vendor
• Comms templates: internal, customer, regulator if applicable
• DR/BCP: RTO/RPO for index and config
• FinOps alerts on spend anomaly

ORR sign-off: service owner, SRE lead, risk/compliance for tier-high.

On-call — first response for AI incidents

On-call must not require ML PhD. First-hour checklist:

1. Acknowledge alert; assign incident commander
2. Classify severity (see matrix below)
3. Contain — disable feature flag, switch to degraded model, block tool writes
4. Preserve evidence — trace IDs, prompt version, model ID, retrieval snapshot
5. Communicate — status page / internal template
6. Escalate — L2 if not contained in SLA window

Rotate on-call with shadow week before solo shift.

SLOs and SLIs for LLM applications

SLI	Example SLO	Measurement
Availability	99.5% monthly	Orchestration health checks
Latency p95	≤4s	End-user request
Faithfulness	≥90% on weekly sample	Automated eval + human audit
Harmful output rate	<0.05%	Policy classifier + human review
Empty retrieval rate	<2%	Logged with fallback behaviour
Cost per 1k queries	Within ±15% of budget	FinOps dashboard

Error budget: When faithfulness SLO burns, freeze feature changes; prioritise eval fix.

Incident management — severity matrix

Severity	Example	Response target	Comms
Sev 1	Harmful advice; data leak; autonomous wrong financial action	15 min contain	Executive + legal
Sev 2	Widespread wrong answers; major latency	30 min	Service owner
Sev 3	Partial degradation; single region	2 hours	Internal status
Sev 4	Minor bug; workaround exists	Next business day	Team channel

AI-specific: wrong but fluent output is not Sev 4 if use case is regulated—tier determines severity.

Runbooks — structure and AI-specific entries

Each runbook: symptoms → detect → contain → diagnose → fix → verify → post-incident.

Mandatory runbook titles:

1. Faithfulness / quality drop after corpus update
2. Empty or stale retrieval
3. Prompt injection / agent tool abuse
4. Provider outage — failover to secondary model
5. Cost spike / token loop
6. Harmful or policy-violating output
7. Model version rollback
8. Index corruption or rebuild
9. PII in logs alert
10. Rate limit / abuse traffic

Example contain step for faithfulness drop: enable human-review-all mode via feature flag; reduce corpus to last known-good snapshot ID.

Model version rollback

Rollback must be practised, not theoretical:

Detect regression (eval alert or user reports)
→ Identify last known-good: model_id + prompt_version + index_version
→ Route traffic 10% canary to rollback stack
→ Compare eval slice ≥ baseline
→ Full rollback or forward fix decision
→ Post-incident: change freeze until root cause

Maintain N-1 model deployment minimum 30 days after upgrade.

Change and release management

Change type	Eval required	Approval
Prompt tweak (safety)	Full safety suite	Tech lead + risk if tier-high
Prompt tweak (UX)	Smoke + sample eval	Tech lead
Model version bump	Full regression	Model risk / architecture
Corpus >10% delta	Retrieval + faithfulness	Product owner
New agent tool	Security review + integration test	CISO delegate

Link to release train calendar—no Friday deploy for tier-high without exec approval.

DR/BCP for AI components

Asset	RTO target example	RPO example	Notes
Vector index	4 hours	1 hour	Snapshot frequency
Corpus source	1 hour	15 min	Source of truth in object store
Config/prompts	30 min	0	Git-backed
Model endpoint	15 min	N/A	Multi-region or secondary provider

Test restore drill quarterly; document actual minutes, not aspirational.

AI-specific failure modes — catalogue

Failure	Signal	Common cause
Drift	Eval scores trend down	Data shift; seasonal queries
Retrieval fail	Low similarity scores	Index stale; chunking bug
Prompt regression	Safety refusals drop	Bad merge; untested prompt
Cost spike	Token metric 5x	Loop; verbose model route
Provider outage	5xx from API	Regional outage
Misuse	Abuse pattern	Prompt bombing; scraping
Silent wrong	High user thumbs-down	No retrieval guard

Support model — tiers and knowledge base

Tier	Handles	Escalation trigger
L1	Known errors, password, "how to"	Wrong answer on regulated topic
L2	Config, index refresh, prompt rollback	Sev 2+ incident
L3	Engineering, vendor TAM	Architecture change

Knowledge base articles: expected behaviour, known limitations, when to escalate.

Frameworks and methods

Operate loop for AI services

Monitor (SLOs, eval samples, cost)
→ Alert (severity-tagged)
→ Respond (runbook)
→ Learn (post-incident, problem record)
→ Improve (eval, prompt, architecture)

Post-incident review (PIR) template

• Timeline (detect → contain → resolve)
• Customer impact (quantified)
• Root cause (technical + process)
• What worked in runbook
• Action items with owners (eval gap, new alert, training)
• Blameless tone; focus on system fixes

Canary release with eval gate

1. Deploy new model to 5% traffic
2. Compare 100-query canary eval hourly
3. Auto-rollback if faithfulness < baseline −2% or harmful rate up
4. Full promote after 24h clean

Requires automation—in manual-only orgs, document manual canary steps in runbook.

FinOps in operations

Ops owns daily cost review for tier-high:

• Anomaly detection on tokens/query
• Top ten expensive sessions investigated
• Model routing misconfiguration alerts

Pair with topic 22 and Delivery Guide for environment cost attribution.

Handoff from project to BAU

Artefact	Project delivers	BAU owns
Runbooks	Draft + walkthrough	Update after incidents
Dashboards	Initial panels	Tune alerts
On-call	Shadow schedule	Rotation
Eval suites	Baseline scores	Weekly/monthly re-run
Vendor contacts	Escalation list	Relationship

Hypercare period: 2–4 weeks post-launch with build team on-call backup.

Real-world scenarios

Scenario A — Insurer: provider outage failover

Context: UK insurer claims copilot; 6,200 daily users; primary Azure OpenAI region outage 47 minutes; secondary region configured but never load-tested.

Incident: Sev 2; p95 latency 12s; faithfulness drops on secondary model (86% vs 92% baseline).

Runbook execution:

1. T+8 min — incident commander; status internal
2. T+12 min — route 100% to secondary; enable banner "Reduced capability mode"
3. T+25 min — tighten retrieval threshold; human review for payout-related queries
4. T+47 min — primary restored; canary 10% for 20 min; full restore

Outcome: No regulatory report required; 34 incorrect answers flagged in review queue (0.8% of session volume during outage).

Numbers: Secondary model cost +22% per query during failover; one-time runbook gap closed—load test now quarterly.

Scenario B — Bank: faithfulness drop after corpus update

Context: Internal markets research assistant; weekly corpus sync from SharePoint; Monday faithfulness eval 91% → 78%; 1,400 users.

Detection: Automated Sunday night eval alert; on-call L2 initially dismisses as "eval flake."

Root cause: Duplicate conflicting policy PDFs ingested; retrieval returns superseded doc 63% of time.

Response:

• Contain: Roll index to Sunday 00:00 snapshot (RPO 1h)
• Fix: Dedup pipeline; source metadata effective_date required
• Verify: Faithfulness 90.5%; problem record opened

Outcome: Sev 2 shortened to 3.2 hours user impact; post-incident adds corpus delta eval gate before index promote.

Numbers: ~210 bad answers prevented by rollback within 45 min of confirm; engineering 40 hours dedup fix.

Scenario C — E-commerce: cost spike from agent loop

Context: Merchant seller support agent; tool calls to inventory API; Saturday token spend $18k vs daily budget $2.1k.

Detection: FinOps alert 08:14; on-call finds agent loop on out-of-stock SKU edge case.

Contain: Disable agent tool update_inventory; switch to read-only FAQ mode.

Fix: Max tool iterations 5; circuit breaker on repeated identical calls; eval case added.

Outcome: Sev 2 financial impact $16k overage absorbed; vendor credits $4k after support ticket.

Numbers: 412 sessions in loop pattern; average 890k tokens/session before break—now capped at 120k.

Scenario D — Healthcare: harmful output incident

Context: Patient FAQ bot (no diagnosis claimed); user receives inappropriate medication suggestion from hallucination; tier-high use case.

Severity: Sev 1; legal notified within 2 hours; feature killed globally 23 min after report.

Runbook: Kill-switch → preserve trace → clinical safety review → regulator template prepared (not submitted after root cause isolated to prompt regression).

Root cause: Friday prompt merge removed refusal clause for medical dosing questions.

Outcome: Change policy—no prompt deploy Friday; mandatory safety eval on Medical-Refusal-50 set; 2 FTE weeks process fix.

Numbers: 1 confirmed harmful session; ~12k sessions/day** during exposure window 6 hours—estimated exposure 0.008% session risk; insurance notified per policy.

ITIL incident record — AI fields extension

Extend incident ticket template:

Field	Example
model_version	gpt-4o-2024-08-06
prompt_version	pr-442
index_snapshot	idx-2026-07-28-01
eval_faithfulness	81% (baseline 92%)
harmful_output	Y/N
trace_ids	link

Without fields, post-incident analysis impossible—problem management fails.

Change record — AI release template

Field	Required
Change type	Standard / Normal / Emergency
Eval suite run ID	pass/fail
Rollback plan	one-click route
Feature flags	list
Risk tier	low/med/high
CAB approval	ticket #

Emergency change still requires retrospective eval within 24h.

War room roles during Sev 1 AI incident

Role	Person	Responsibility
Incident commander	On-call lead	Coordination
Technical lead	Senior engineer	Diagnose/fix
Comms	Service owner	Internal/external
Legal/risk	On-call delegate	Harm assessment
Scribe	Anyone	Timeline log

Rotate commander to avoid single hero dependency.

Observability alert catalogue — starter set

Alert	Threshold	Severity
Faithfulness sample	< SLO −3% × 2 runs	Sev 2
Harmful classifier	>0.1% hourly	Sev 1
p95 latency	>2× SLO 15 min	Sev 2
Token rate	>3× daily avg	Sev 2
Empty retrieval	>5% 1 hour	Sev 3
Provider 5xx	>1% 5 min	Sev 2

Tune per environment—staging alerts must not page same roster without label.

BCP tabletop exercise — AI scenario script

Inject: Primary region down + faithfulness drop on secondary during failover.

Participants: SRE, AI eng, comms, service owner.

Success: Contain <30 min simulated; runbook gaps logged ≤3.

Run annually for tier-high; biannually tier-medium.

Handover checklist — build team to BAU

• All runbooks walkthrough recorded (video ≤90 min)
• Alert runbook linked in PagerDuty/Opsgenie
• Eval baseline scores documented
• Vendor escalation numbers verified
• FinOps dashboard access for ops
• Known errors KB ≥5 entries
• Open problem records assigned

Sign-off: build lead + BAU lead + service owner.

Practice exercises

Primary exercise — Faithfulness drop runbook (60 minutes)

Brief: Legal contract review copilot; 800 lawyers; faithfulness SLO ≥88%; alert fires Sunday 02:00 showing 81% on weekly sample.

Tasks:

1. Write runbook sections: detect → contain → diagnose → fix → verify → communicate.
2. Define Sev level and response targets.
3. List five diagnostic checks (index version, corpus delta, model ID, prompt hash, retrieval empty rate).
4. Document rollback decision tree: index vs prompt vs model.
5. Draft internal comms template (100 words max).

Acceptance criteria:

• Contain step executable by L2 without original developer
• Evidence preservation listed
• Post-incident action to prevent recurrence

Stretch exercise — ORR pack for production launch (half day)

Brief: Public sector benefits eligibility assistant; 40k queries/week; tier-high; multi-region requirement.

Tasks:

1. Full ORR checklist with owners and dates.
2. SLO table with measurement method.
3. On-call rota and escalation matrix (names as placeholders).
4. Top 10 runbook titles with one-paragraph scope each.
5. DR drill plan: index restore + model failover.
6. Hypercare plan 3 weeks post-launch.
7. Link to Delivery and Operating Model release process.

Acceptance criteria:

• Kill-switch owner named
• Model N-1 rollback tested requirement documented
• FinOps alert thresholds specified

Scenario G — Multi-region active-active ops

Context: Global bank; EU + US active-active RAG; corpus diverges 47 min during network partition; users see conflicting answers.

Ops response:

• Split-brain playbook: read-only both regions until index version reconciled
• Authoritative region per document source system
• Post-incident: CRDT or single-writer index strategy

Numbers: 12k affected queries; 0 Sev1 harmful; £95k engineering investment in partition tolerance.

Scenario H — LLM ops team organisational design

Model	Pros	Cons
Central AI SRE	Standard runbooks	Far from domain
Embedded in product squad	Context	Inconsistent ops
Hybrid	Platform + embedded	Coordination cost

Recommend hybrid for enterprise: central golden paths, squad on-call for tier-high with central backup.

Document in ORR operating model section.

Ops metrics targets — example tier-medium internal copilot

Metric	Target month 1	Target month 6
Availability	99.0%	99.5%
Faithfulness sample	88%	90%
MTTR Sev2	4h	2h
Runbook coverage incidents	70%	95%
Cost/query vs budget	+10%	±5%

Improvement trajectory proves operational maturity to steering.

FAQ — operations and production support

Q: Who owns on-call for AI—ML team or SRE?
A: Hybrid: SRE owns platform alerts; ML/App own quality eval alerts; single roster with escalation matrix in ORR.

Q: Do we need 24×7 for internal copilot?
A: Depends on tier and SLA—tier-high customer-facing yes; tier-low internal may be business-hours + best-effort nights.

Q: How often rollback drill?
A: Quarterly tier-high; biannual tier-medium; document actual minutes not theory.

Q: What is first action on harmful output?
A: Contain—kill-switch or human-review-all—then preserve trace; notify per severity matrix before root cause.

Q: Is API uptime enough for SLA?
A: No—add quality sample SLO where contract permits; ops tracks internally even if contract lags.

Runbook quality rubric

Score	Criteria
5	Executed successfully in drill; on-call solo capable
3	Complete but untested
1	Outline only—blocks ORR sign-off

Target all tier-high runbooks at 4+ before launch.

Monthly ops review agenda (60 minutes)

1. SLO dashboard vs target (10 min)
2. Incidents and open PIRs (15 min)
3. Change success / rollback rate (10 min)
4. Cost vs FinOps budget (10 min)
5. Runbook drill status (10 min)
6. Vendor ops issues (5 min)

Standing meeting—same roster as incident commanders plus service owner.

Production support handoff email template

Subject: [Service] BAU handoff complete — ORR signed

BLUF: [Service] entered steady-state [DATE]. On-call: [roster link].

SLOs: [link dashboard]
Runbooks: [link KB]
Known issues: [KE-xxx list]
Open actions: [PIR tickets]
Hypercare ends: [DATE]
Build team contact (office hours only): [name]

Send to steering distribution—closes loop visibly.

ORR gate — executive one-pager for steering

When ORR incomplete, steerco sees:

BLUF: Production not recommended until ORR red items closed.

Item	Status	Owner	ETA
Runbooks tested	Red	SRE	2026-08-05
Rollback drill	Green	Platform	Done
On-call roster	Amber	Service owner	2026-08-02

Decision requested: Approve delayed launch or accept risk with written acceptance by [buyer name].

Prevents silent launch pressure without ops readiness visible to executives.

Incident communication templates — internal (Sev 2 quality drop)

Subject: [INC-1234] Quality degradation — [Service Name]

Status: INVESTIGATING / CONTAINED / RESOLVED
Impact: [user count], [duration], [faithfulness delta]
Action: [human review mode enabled / rollback to index vX]
Next update: [time UTC]
Incident commander: [name]

Template reduces ad hoc panic emails during incidents.

Production readiness — joint sign-off meeting agenda (90 min)

1. ORR checklist walk (30 min)
2. Runbook tabletop scenario (20 min)
3. On-call roster confirmation (10 min)
4. SLO dashboard tour (10 min)
5. Hypercare plan (10 min)
6. Go/no-go poll—service owner, SRE, AI lead, risk delegate (10 min)

Minutes filed within 24h—audit trail for regulators asking "how did you launch?"

Questions you should be able to answer

1. What is the ORR status—what items are red before launch?
2. Who is on-call primary and secondary, and what is the escalation path?
3. What are the SLOs—and which measure quality vs availability?
4. What is the Sev 1 scenario and contain-within target?
5. How do you roll back model, prompt and index independently?
6. Where is the kill-switch and who can authorise activation?
7. What runbook covers faithfulness drop vs provider outage?
8. How is corpus change gated before production index promote?
9. What are RTO/RPO for vector index and corpus source?
10. How do FinOps anomalies escalate to on-call?
11. What eval runs automatically in production—and how often?
12. What is the hypercare model post-launch?
13. How are post-incident actions tracked to closure?
14. What customer/regulator comms templates exist for Sev 1?
15. How does change management link to eval regression suites?

Runbook deep dive — faithfulness drop (full outline)

Symptoms: User reports "wrong answers"; eval sample below SLO; thumbs-down spike; retrieval similarity scores drop.

Detect: Automated weekly eval; real-time sample 50/hour; alert if faithfulness < baseline −3% for 2 consecutive runs.

Contain (target <15 min):

1. Enable human-review-all feature flag OR reduce traffic to 10% canary
2. If corpus updated in last 72h, roll index to last known-good snapshot ID
3. Post internal status; notify service owner

Diagnose:

• Compare model_id, prompt_hash, index_version vs last green deploy
• Check empty retrieval rate and top-k similarity distribution
• Review recent CRs and prompt merges
• Sample 20 bad traces—classify: retrieval miss vs model hallucination vs prompt

Fix:

• Retrieval miss → reindex, fix chunking, dedup
• Model regression → rollback model or prompt
• Corpus pollution → source fix + re-ingest gate

Verify: Eval ≥ SLO on two consecutive runs; 24h canary at 10%→50%→100%.

Communicate: All-clear to stakeholders; customer comms if external impact.

Post-incident: Problem record; add eval case; change freeze 48h.

Runbook deep dive — provider outage failover

Symptoms: 5xx from model API; latency timeout; provider status page incident.

Contain: Route to secondary region/model per routing table; enable degraded mode banner; disable agent write tools if secondary lacks full capability.

Diagnose: Provider incident ID; check quota vs outage; verify network path.

Fix: Failover complete; monitor faithfulness on secondary; tune retrieval if quality drop.

Verify: Error rate <1%; p95 latency within 2× normal SLO.

Communicate: Status page update every 30 min during Sev 2+.

Post-incident: Failover drill gap logged; update runbook with actual timings.

On-call rotation design for AI services

Role	Responsibility	Hours
Primary L2	First technical response	24×7 tier-high
Secondary L2	Backup if primary no ack in 15 min	24×7
L3 vendor	Provider incident	Business hours + Sev1 page
Service owner	Comms, business decisions	Business hours; Sev1 anytime

Shadow week mandatory; runbook quiz before solo.

Handoff: Slack channel + pager; runbook link in alert payload; no "check dashboard" without URL.

SLO error budget policy

When monthly error budget consumed (>50% failures on quality SLO):

• Freeze non-critical releases
• Daily eval review standup 15 min
• Root cause work prioritized over features
• Steering informed if budget exhausted before month end

Pair with FinOps—cost SLO burn triggers similar review.

Problem management — recurring incidents

Third similar incident in 90 days → problem record mandatory:

• Root cause category (people/process/tech)
• Permanent fix vs workaround
• Eval gap closed?
• Runbook updated?

Prevents permanent firefighting culture.

Knowledge management for ops teams

Known error database entries:

• KE-001: Empty retrieval after SharePoint sync — fix: reset delta token
• KE-002: Prompt merge dropped refusal clause — fix: safety eval gate

Link from alerts where pattern match confidence high.

Capacity planning for AI workloads

Driver	Planning metric
Token growth	30% headroom over forecast
Index size	Reindex duration vs maintenance window
GPU/CPU	Batch vs realtime peaks
Embedding refresh	DE pipeline duration

Review quarterly with FinOps—cost spike often capacity misconfiguration.

Operational metrics dashboard — minimum panels

1. Availability and latency p50/p95
2. Faithfulness / eval sample (rolling 7d)
3. Harmful output rate (if classifier)
4. Empty retrieval rate
5. Token cost per 1k queries
6. Incident count by severity
7. Change success rate (rollback %)
8. On-call MTTA/MTTR

Dashboard without quality panel is ops theatre.

Hypercare to steady-state transition

Week post-launch	Hypercare activity	BAU handoff
1	Build team primary on-call backup	Runbook walkthrough recorded
2	Joint on-call; daily eval review	L2 owns alerts
3	Build team office hours only	Steering reports ops metrics
4	Exit hypercare sign-off	ORR lessons in backlog

Sign-off requires zero Sev1 open; ≤2 Sev2 with closed PIR.

Linkage to Delivery guide — release coordination

From Delivery and Operating Model:

• Release calendar shared with ops 4 weeks ahead
• Change advisory board for tier-high includes ops rep
• Rollback tested in staging same week as prod release
• Feature flags default off; ops documents flag map

Ops is not surprised by Tuesday 6pm deploys.

Scenario E — Index rebuild during business hours

Context: Retail RAG; Monday 10:00 DE triggers full reindex without change ticket; 45 min degraded retrieval; £12k estimated sales assist impact.

Gap: No change window; no ops notification.

Fix: CAB rule—index rebuild Class B change; ops approval; eval gate on 1% sample before full swap.

Numbers: One process change cheaper than repeat incidents—ops engagement saves ~4h revenue impact per quarter modelled at £48k/quarter.

Scenario F — Model version auto-upgrade surprise

Context: Vendor auto-upgraded model; safety refusals dropped 12%; 3 days undetected—no deprecation notice honoured.

Response: Contract CR with vendor; freeze auto-upgrade; manual eval gate; N-1 retained 60 days.

Numbers: 847 sessions before detect; 11 flagged harmful in audit sample 0.4%—Sev 2 narrowly avoided Sev 1.

30/60/90 day post-launch review template

Day	Review focus
30	SLO actuals; alert tuning; runbook gaps from incidents
60	Eval drift; cost vs forecast; support ticket themes
90	ORR retrospective; problem records closed; handoff complete

Feed results to pattern library (topic 35).

Extended Learn — production support operating model

Tier 1 (Service desk) → scripted triage, KB, escalate
Tier 2 (App/AI ops)   → runbooks, config, rollback
Tier 3 (Engineering)  → code, model, architecture
Tier 4 (Vendor)       → platform defects, API outages

AI adds eval analyst rotation for quality incidents—L2 or L3 depending on org.

Extended Learn — capacity incident playbook

Symptoms: Latency spike; queue depth; autoscale maxed; cost alert.

Contain: Rate limit non-critical clients; shed batch jobs; enable aggressive cache.

Fix: Scale horizontally; optimise retrieval; model downgrade route temporarily.

Verify: p95 restored; cost within 2× daily budget.

Link FinOps—capacity and cost incidents often same root cause.

Extended Learn — seasonal and campaign traffic

Retail and tax-season spikes require pre-campaign runbook:

• Load test 2× forecast
• Freeze non-critical changes 2 weeks before
• War room roster pre-published
• Pre-written degraded mode comms

Ops plans for predictable peaks—not only surprises.

Extended Learn — logging and PII in ops triage

On-call must access redacted traces by default; break-glass for full prompt with audit log when Sev 1.

Runbook step: verify DPO notification if full prompt access used.

Ops without privacy discipline creates second incident during first.

Extended Learn — vendor management in run

Quarterly vendor ops review:

• Incident history and RCA from vendor
• Roadmap deprecations
• Support ticket SLA compliance
• Cost vs commit

Feed into renewal evaluation (topic 31)—ops data is evidence.

Extended Learn — runbook maintenance lifecycle

Trigger	Action
Post-incident	Update within 5 days
Architecture change	Review affected runbooks
Quarterly	Dry-run top 3 scenarios
Team change	New on-call certification

Stale runbooks worse than none—false confidence.

Negative cases — when operations fails

Launch without ORR

Symptom: Go-live on deadline; no runbooks; build team phone rings at night forever.

Impact: Burnout; slow incidents; executive loss of confidence.

Fix: ORR gate; no production without on-call and rollback test.

Uptime-only SLO

Symptom: Dashboard green; users flooded with wrong answers after index bug.

Impact: Reputational damage; "AI doesn't work" narrative.

Fix: Faithfulness and retrieval SLIs; eval alerts.

Rollback never tested

Symptom: Incident occurs; rollback script fails; 4-hour outage.

Impact: Extended Sev 1; regulatory scrutiny.

Fix: Quarterly rollback drill; automate N-1 route.

Prompt change without eval

Symptom: Friday deploy; safety regression Monday.

Impact: Harmful output; legal exposure.

Fix: Change calendar; mandatory safety eval; no Friday tier-high deploys.

Orphaned service

Symptom: Original consultants leave; no knowledge transfer; ops refuses ticket.

Impact: Silent degradation; cost creep.

Fix: Handoff pack; hypercare; runbook walkthrough recorded.

Alert fatigue

Symptom: 200 daily alerts; on-call ignores faithfulness warning.

Impact: Missed real incident.

Fix: Tune thresholds; severity tagging; weekly alert review.

Output

Expected outputs for this topic:

• ORR checklist — Signed readiness with owners
• SLO/SLI table — Quality, latency, availability, cost
• Incident severity matrix — With response and comms rules
• Runbook set — Top 10 AI failure modes
• On-call roster and escalation — Primary, secondary, L3
• Rollback procedure — Model, prompt, index; tested date
• DR/BCP summary — RTO/RPO per component
• Support tier model — L1/L2/L3 and KB articles
• Hypercare plan — Post-launch coverage period
• PIR template — Blameless post-incident format

Practice checklist

• I can execute first-hour on-call steps for a quality degradation incident
• I wrote a faithfulness drop runbook with contain step
• I defined SLOs that include quality, not only uptime
• I documented model rollback with N-1 retention policy
• I completed the primary runbook exercise
• I linked ops handoff to Delivery guide release process
• I identified three negative cases from real or practice context

Related playbook content

• Delivery and Operating Model — Release trains, environments and handoff
• MLOps, LLMOps and Observability — Tracing, eval in prod, drift detection
• AI Evaluation and Quality Assurance — Eval gates for change management
• Performance Engineering and AI FinOps — Cost anomaly detection
• Security Engineering — Incident types for injection and tool abuse
• Responsible AI and Governance — Kill-switch and tier-high requirements
• Delivery and Programme Management — Programme to BAU transition
• RFP, Procurement and Contracting — SLA alignment with SLOs
• How to use this Learning Map — Reference-depth study method
• 8D Framework — Deploy and operate phases

Key takeaways

• AI operations needs quality SLOs and eval alerts, not only API uptime dashboards.
• Runbooks for semantic failures—faithfulness drop, empty retrieval, prompt regression—are as critical as outage failover.
• Model version rollback must be tested; maintain N-1 and canary with eval gates.
• ORR is a launch gate—on-call, kill-switch, DR and hypercare before production traffic.
• Link handoff to Delivery and Operating Model so release discipline survives project end.

What to do next

1. Complete the faithfulness drop runbook exercise and add it to your pattern library.
2. Continue to Leadership for people and ethics topics—or review MLOps, LLMOps and Observability for observability depth.
3. Open Delivery and Operating Model to align ORR with your client's release process.
4. Cross-check AI Evaluation and Quality Assurance for production eval sampling design.