Software Engineering

Executive summary

Software engineering for AI solutions is production discipline applied to non-deterministic components. An AI Solution Engineer must design versioned APIs, enforce authentication and authorisation, structure code for testability, run CI/CD pipelines that gate merges on unit, integration, security, load, and AI-specific regression tests (prompt changes, retrieval corpus updates, model version bumps)—not ship prompt edits from a chat window on Friday afternoon. Without this capability, teams demo flawlessly in notebooks and collapse under duplicate integrations, silent eval drift, undeployable hotfixes, and outages when retrieval latency doubles after a config change nobody tested.

When to use this page

• Moving from notebook or low-code prototype to deployable services with SLAs.
• Multiple engineers touch prompts, tools, and retrieval config—need version control and review gates.
• Security asks how secrets, PII, and service accounts are managed per environment.
• Operations needs rollback when model or prompt regression hits production metrics.
• Integration partners require OpenAPI contracts, idempotency, and error semantics.
• Load tests have not been run—marketing promises 10K concurrent users on a single pod.
• Compliance requires audit trail of what software version + prompt version + corpus version answered each query.
• Stage 3 exit requires test strategy and CI outline—this topic produces them.

Executive view

Focus on **release risk**, **mean time to recover**, and scenario cost of untested prompt deploys. Ask: can we roll back in minutes, and does CI block bad releases?

Technical view

Implement APIs, test pyramids, and pipelines before scaling users. Treat prompts and retrieval configs as **code**—reviewed, versioned, evaluated.

Why this matters

AI programmes treat model behaviour as magic and software as an afterthought—until the first production incident. A regional insurer shipped a RAG assistant with strong offline demos; production had no contract tests on the CRM adapter, no golden-set eval in CI, and prompts edited live in a admin UI without audit. A prompt tweak broke citation formatting required for regulatory quotes; 23% of golden-set cases failed; manual rollback took 9 hours; compliance logged a severity-2 incident. Software engineering discipline would have blocked the merge and enabled one-click rollback to prompt version v2.4.1.

Reliability for AI systems adds layers beyond traditional apps: non-deterministic outputs, retrieval dependency, token limits, streaming UX, tool-call side effects, and eval flakiness. You must test orchestration, retrieval, and generation separately and together. CI must run fast enough for developer flow (~10–15 min) yet deep enough to catch regressions (nightly full eval + load).

This topic pairs with Architecture Guide for service boundaries and Delivery Guide for release patterns. Topic 21 covers eval design; topic 15 covers embedding eval in engineering workflow.

Learn

API design for AI orchestration services

Definition. APIs expose stable contracts for clients (web, mobile, Teams bot, partner BFF). AI orchestration APIs typically include: POST /v1/sessions, POST /v1/sessions/{id}/messages (stream or sync), GET /v1/sessions/{id}/citations, admin endpoints for config (restricted). Use OpenAPI specs; version in URL (/v1) or header with explicit deprecation policy.

Engagement use. Separate read (chat, retrieve) from write (tools, ticket create) endpoints for auth granularity. Document error model: 429 rate limit, 503 model unavailable, 422 validation, problem+json body with correlation_id. Support SSE or WebSocket for streaming with heartbeat. Idempotency-Key header on side-effect routes (topic 24).

Pitfalls.

• Fat API returning entire RAG context to client—leak and payload bloat.
• No request size limits—prompt injection via huge payloads.
• Breaking changes without version bump—partner integrations break silently.
• Synchronous long-running agent loops in HTTP request—timeouts and duplicate side effects.

Worked example. Servicing API: POST /v1/chat max body 32 KB; returns message_id, streamed tokens, citations[] with doc_id, version; POST /v1/actions/create-ticket requires Idempotency-Key; OpenAPI published to corporate gateway; SLA p95 2.5s to first token documented.

Authentication, authorisation, and service identity

Definition. Authentication (authn) proves identity (OIDC from Entra ID, Okta). Authorisation (authz) enforces what identity can do (RBAC, ABAC on LOB, country). Service accounts or workload identity for service-to-service calls—no long-lived passwords in config.

Engagement use. Human users via OIDC with short-lived tokens; BFF exchanges token for session. Backend services use managed identity to vector DB, secret vault, model API. On-behalf-of flow when calling ERP as user. Tool endpoints check scoped permissions—not every user triggers write tools. Rotate secrets via vault; never in repo.

Pitfalls.

• Shared API key for all clients—no revocation granularity.
• Service account with admin on CRM—blast radius.
• Authz only at edge—internal services trust all callers.
• JWT without audience validation—token replay across services.

Worked example. Bank assistant: Entra ID groups map to roles; servicing.read vs servicing.write_ticket; orchestrator validates JWT and passes sub, entity, country to retrieval filter; CRM adapter uses OBO flow with scoped OAuth consent.

Clean architecture and module boundaries for AI services

Definition. Clean architecture separates domain logic from infrastructure: entities (session, message, citation), use cases (answerQuestion, escalateCase), adapters (LLM client, vector store, CRM). AI-specific: prompt templates and tool registry are infrastructure adapters configured from versioned files—not hardcoded strings scattered.

Engagement use. Structure repo: domain/, application/, infrastructure/, api/. Swap model provider without rewriting use cases. Dependency rule: inner layers know nothing of FastAPI or Azure SDK. Unit-test use cases with fake LLM returning fixed outputs.

Pitfalls.

• God class ChatService with 2,000 lines—untestable.
• Business rules in Jinja prompt only—no executable validation.
• Direct SDK calls from HTTP handlers—no test seam.
• Copy-paste retrieval logic across endpoints.

Worked example. AnswerQuestionUseCase accepts Question, UserContext; calls RetrievalPort, GenerationPort, PolicyPort (PII mask); returns Answer with citations; tests inject FakeGenerationPort for golden tests without API cost.

Testing pyramid for AI systems

Definition. Test pyramid: many unit, fewer integration, selective end-to-end, plus AI-specific layers: prompt regression, retrieval recall@k, tool mock tests, red-team suites (nightly). Balance cost vs confidence.

Engagement use.

Layer	What it catches	Typical count/runtime
Unit	Masking, routing, parsers, schema	Hundreds, seconds
Integration	DB, vector, model stub, CRM sandbox	Dozens, minutes
Contract	OpenAPI consumer/provider	Per partner
Prompt/RAG golden	Answer quality regressions	50–500 cases, minutes–hours
E2E	Critical user journeys	5–15, slow
Load	Latency, rate limits, pool exhaustion	Pre-release
Security	OWASP, injection, auth bypass	Per release

Pitfalls.

• Only manual "vibe checks" on ChatGPT before release.
• Golden set never updated—false confidence.
• E2E only—slow feedback, flaky CI.
• Eval in prod only—users are testers.

Worked example. RAG service: 142 unit tests; 38 integration with Testcontainers for vector DB; 120 golden questions in CI (fail if pass rate <92%); 8 Playwright E2E; nightly 400 extended eval + 50 red-team injections.

Prompt, model, and retrieval regression testing

Definition. Prompt regression detects behaviour change when templates, model version, or decoding params change. Retrieval regression detects when corpus or embedding model changes hurt recall@k on labelled queries. Pin versions in test config: prompt_version=2.4.1, model= gpt-4o-2024-08, index=v2024.03.

Engagement use. Store golden inputs + expected properties (not always exact string match): JSON schema match, citation doc_id set, forbidden phrases absent, numeric tolerance. Use LLM-as-judge only with human-audited calibration subset. Block merge on regression beyond threshold; allow waivers with ADR. Re-run full suite on corpus re-index.

Pitfalls.

• Exact string match on LLM output—flaky CI.
• No seed/temperature 0 in regression—non-determinism noise.
• Judge model same as SUT—blind spots.
• Corpus hash change without triggering eval—silent quality drop.

Worked example. CI job eval-regression: loads golden.jsonl; runs retrieval+generation; asserts citation_format regex, grounded flag via metadata match, latency <3s; fails at 91.2% pass when threshold 92%—blocks PR until fix or waiver.

CI/CD pipelines and release gates

Definition. CI/CD automates build, test, security scan, deploy to environments. Gates: unit+integration pass, eval pass, SAST clean, container scan, manual approval to prod. Environments: dev → test → staging (prod-like) → prod with promotion, not rebuild from different branch mystery.

Engagement use. Pipeline stages: lint → unit → integration → eval (subset on PR, full on main nightly) → build image → deploy staging → smoke + e2e → approval → prod canary (5% traffic) → full. Feature flags for prompt version rollout. Rollback: redeploy previous image + prompt config tag in <15 min target.

Pitfalls.

• Deploy on Friday without automated rollback tested.
• Staging missing private link—prod-only failures.
• Skipping eval on "doc-only" PR that changed retrieval chunking.
• Same database for staging and dev—schema drift surprises.

Worked example. GitHub Actions + Argo CD: PR runs fast eval (30 cases, 6 min); merge to main runs full eval (120 cases, 22 min); staging deploy auto; prod requires change ticket + canary 30 min on error rate <0.5% and eval smoke 100%.

Code quality, review, and static analysis

Definition. Code quality practices: linting (ruff, eslint), formatting (black, prettier), type checking (mypy, tsc strict), Sonar or equivalent for complexity and security smells, mandatory peer review with checklist for AI routes (PII log?, tool auth?).

Engagement use. Define quality gate in CI: no new critical Sonar issues; coverage ≥80% on domain and application layers (not chasing 100% on boilerplate). Review checklist includes: secrets scan, dependency CVE, prompt injection handling, rate limits.

Pitfalls.

• # noqa everywhere—gate theatre.
• Coverage on generated code only—false metric.
• Review skips when "only prompt change"—highest risk change.
• Unpinned dependencies—supply chain surprise.

Worked example. Pre-commit: black, ruff, mypy strict on src/; CI fails on High CVE in pip-audit; PR template requires eval impact checkbox and link to golden diff if prompts changed.

Containers, configuration, and twelve-factor alignment

Definition. Containers package services immutably; twelve-factor app principles: config in environment, logs as streams, disposability, dev/prod parity. AI services need GPU node pools separately from CPU orchestration in many setups.

Engagement use. One process per container (or sidecar pattern for observability). Config: model deployment name, retrieval index URL, feature flags—env vars from vault at runtime, not baked in image. Health checks: /health/live, /health/ready (includes vector ping). Resource limits prevent OOM killing node pools.

Pitfalls.

• 2 GB model loaded per worker on autoscaled CPU pods—bankruptcy.
• Config drift between regions—manual console edits.
• Missing readiness—traffic to pod before index warm.
• Giant images with notebook junk—slow deploy.

Worked example. Orchestrator container 512Mi–1Gi CPU; GPU embedding job separate CronJob; env MODEL_DEPLOYMENT, INDEX_VERSION, PROMPT_BUNDLE=v2.4.1 from Key Vault; readiness fails if vector p99 >500ms on probe query.

Observability, SLOs, and incident response for AI services

Definition. Observability: structured logs (JSON), metrics (latency, tokens, retrieval hit rate, error codes), distributed traces (OpenTelemetry) across orchestration → retrieval → model. SLOs define error budget; runbooks for incident types.

Engagement use. Log correlation_id, session_id, prompt_version, model, index_version, token counts, retrieval ids—not raw PII prompts in central log (ADR). Metrics: time_to_first_token, retrieval_latency, tool_success_rate, eval_pass_rate in staging. Alerts on SLO burn rate; runbook links from pager.

Pitfalls.

• Logging full prompts to Splunk—compliance incident.
• No trace across async tool calls—debugging nightmare.
• Alert on every 500—noise; no SLO framing.
• Missing business metric tie-in (escalation rate spike).

Worked example. Datadog dashboards: p95 latency by route; alert if grounded_answer_rate (sampled eval) drops 5 pts week-over-week; runbook RB-014 Model 429 surge covers failover to secondary deployment.

Reliability patterns: retries, circuit breakers, bulkheads, graceful degradation

Definition. Retries with exponential backoff on transient errors; circuit breakers stop hammering failing ERP; bulkheads isolate thread pools for chat vs batch embed; graceful degradation serves reduced functionality when model or retrieval down.

Engagement use. Retry idempotent reads only; writes use idempotency keys (topic 24). Breaker opens after 5 failures in 30s; half-open probe. Degradation ladder: full RAG → cached FAQ → static message with ticket link. Timeout budget per request segment documented.

Pitfalls.

• Retry POST create-ticket—duplicates.
• Infinite retry on 401—auth misconfig melts system.
• No timeout on LLM stream—hung connections exhaust pool.
• Degradation message exposes internal errors.

Worked example. Orchestrator: retrieval timeout 800ms, one retry; model timeout 30s stream; CRM breaker; if retrieval fails, return FAQ fallback + log degraded_mode=true; SLO allows 2% degraded responses monthly.

Security testing and supply chain for AI codebases

Definition. Security testing: SAST, dependency scan, container scan, DAST on staging, prompt injection test cases, tool privilege escalation tests. Supply chain: pin dependencies, verify signatures, minimal base images.

Engagement use. Red-team suite includes: indirect injection via retrieved doc, tool call to unauthorized endpoint, SSRF via URL tool, JWT tampering. Block release on critical findings. Secrets scanning in git history on CI.

Pitfalls.

• curl | bash in Dockerfile.
• LLM tool with unrestricted HTTP—SSRF to metadata service.
• Dev dependency in prod image.
• Ignoring OWASP LLM Top 10 in test plan.

Worked example. CI: Trivy scan; 42 injection cases in red-team job; tool allow-list test fails if agent calls file:// or internal IP—PR blocked.

Performance and load testing AI endpoints

Definition. Load testing validates throughput, latency distribution, and cost at target concurrency. AI-specific: measure tokens/sec, queue depth on model rate limits, vector QPS, connection pool saturation.

Engagement use. k6 or Locust scripts: ramp to 500 concurrent sessions; assert p95 <3s first token at target; monitor 429 rate from model API. Load test retrieval separately from generation. Cost projection from load test token counts.

Pitfalls.

• Load test against prod model endpoint—budget fire.
• Single-turn only—ignore multi-turn context growth.
• No warm-up—cold start skews results.
• Ignore embedding batch job impact on shared cluster.

Worked example. Pre-release: k6 300 VUs, 10 min; p95 first token 2.1s; error rate 0.3%; projected £19K/month inference at full rollout—within NFR ceiling £28K.

Frameworks and methods

Test strategy template (AI-adjusted)

Area	Scope	Owner	Frequency	Pass criteria
Unit	Domain, masks, routers	Dev	Every PR	100% pass
Integration	Vector, CRM stub	Dev	Every PR	100% pass
Golden eval	RAG quality	ML/QA	PR subset, main full	≥92% pass
E2E	Top journeys	QA	Staging deploy	100% pass
Load	Peak concurrency	Perf	Pre-prod	NFR table
Red team	Injection, tools	Security	Monthly	No critical

CI/CD stage diagram

PR → lint/type → unit → integration → eval-fast → security scan
  → merge → eval-full → build image → deploy staging → e2e → smoke eval
  → approval → canary prod → monitor SLO → full promote

API review checklist

• OpenAPI published and versioned
• Authn/z on every route including health admin
• Rate limits and body size caps
• Idempotency on writes
• Error model with correlation_id
• Streaming timeout and cancel documented

Release rollback runbook outline

1. Identify last good image tag + prompt_bundle + index_version
2. Argo rollback or redeploy manifest
3. Verify smoke eval 100% on canary subset
4. Notify stakeholders; post-incident if user impact
5. Root cause: prompt, model, retrieval, or integration?

8D alignment

Develop: implement services with test pyramid. Deploy: CI/CD gates and rollback. See 8D Framework and Delivery Guide.

Real-world scenarios

Scenario A: RAG service prompt regression (UK financial services)

Wealth management firm edits system prompt to "be more conversational." Developer merges without eval gate. Citation format [Policy §x.y] breaks to free text; compliance sampling finds 23% non-compliant citations on 80 case sample. Manual rollback 9 hours; £40K estimated rework and review.

Fix implemented: Golden set 120 cases in CI; pass threshold 92%; prompt changes require 2 reviewers; rollback automation to prompt_bundle v2.4.1 in 12 minutes. Subsequent quarter: zero citation-format incidents in production sampling.

Scenario B: CRM integration without contract tests (telecom)

Servicing assistant returns account summary from CRM adapter. CRM team ships schema change removing nested billingCycle field. Assistant throws 500 for 18% of accounts; 6 days to diagnose; £180K handle-time loss estimate.

Fix: Pact contract tests in CI; CRM sandbox consumer-driven contract; adapter returns partial data with explicit unknown_fields flag; versioned CRM API v3 pinned. Incident repeat risk low after 2 release cycles.

Scenario C: Load test prevents launch disaster (e-commerce)

Retail peak season planned 8K concurrent chat sessions. Pre-load test on staging shows p95 11s first token at 2K users—single orchestrator pod, model 429 at 40% quota. Architecture adds HPA 3–20 pods, request queue, secondary model route for FAQ-only path.

Numbers: After fix, p95 2.4s at 8K simulated; 429 rate <0.2%; infra cost +£6K/month—within NFR. Launch proceeds; peak day zero Sev-1.

Scenario D: Agent tool auth bypass (insurance)

Internal agent tool chain includes updateClaimStatus. Pen test finds any authenticated user can invoke tool via crafted API—not only adjusters. Critical finding; launch delayed 6 weeks.

Fix: Tool-level ABAC; integration tests assert 403 for wrong role; red-team 15 cases in CI; security sign-off gate on tool registry changes. Lesson: AI software engineering includes tool surface as API security.

Scenario E (stretch): Multi-region active-active orchestration

Global SaaS vendor deploys orchestration active-active EU + US. Sticky sessions wrong region cause retrieval index lag cross-region; 5% answers cite stale corpus. CI adds region affinity tests; config enforces geo DNS + session region pin; eval per region nightly.

Outcome: Stale citation rate <0.5%; RTO for region failure 15 min via traffic shift.

Session state, databases, and conversation persistence

Definition. Production assistants persist sessions, messages, tool results, and citation metadata in databases (PostgreSQL, Cosmos DB, DynamoDB)—not in-memory only. Schema design affects GDPR deletion, audit, and replay for eval.

Engagement use. Tables: sessions(user_id, region, created_at, retention_class), messages(session_id, role, content_redacted, prompt_version, model, tokens), citations(message_id, doc_id, version). Retention job purges per policy. Encrypt at rest; row-level tenant isolation for multi-tenant SaaS.

Pitfalls.

• Storing full prompts with PAN/SSN—compliance breach.
• No migration strategy—schema change breaks prod.
• Session stickiness without region pin—wrong index replica.
• Conversation log used as training without consent.

Worked example. PostgreSQL with monthly partition on messages; PII columns encrypted with KMS; delete-user API cascades within 72 hr; eval replay uses exported redacted JSONL only.

Feature flags, config bundles, and safe rollout

Definition. Feature flags toggle prompt bundles, model routes, tool enablement without redeploy. Config bundles version prompts, retrieval params, and tool registry as immutable artifacts (bundle-2024.03.2).

Engagement use. Progressive rollout: 5% users on new bundle; compare eval metrics and error rates; auto-rollback flag if grounded rate drops >2 pts. Separate flags for read tools vs write tools—enable write only after integration tests pass.

Pitfalls.

• Flags edited in prod UI without audit.
• 47 boolean flags—combinatorial untest.
• Flag offboard forgotten—dead code paths.
• Same bundle in staging and prod names—confusion.

Worked example. LaunchDarkly (or open-source equivalent): prompt_bundle_v3 canary; Datadog compares eval_sample_pass_rate canary vs control; rollback automatic on SLO burn.

Dependency management and reproducible builds

Definition. Pin direct and transitive dependencies; scan CVE; use lockfiles (poetry.lock, package-lock.json); SBOM export for customer security questionnaires.

Engagement use. Dependabot/Renovate with auto-merge only for patch after CI green. Major upgrades require eval regression. Container digest pin not :latest.

Pitfalls.

• Unpinned openai SDK—silent API behaviour change.
• Vulnerable requests ignored—audit finding.
• Different lockfile dev vs CI—"works on my machine."

Worked example. CI fails on High CVE; SBOM attached to release artifact; model SDK upgrades run full golden eval before promote.

Database migrations and zero-downtime deploy

Definition. Schema migrations (Flyway, Alembic, Prisma migrate) run in CI/CD with backward-compatible steps: expand → migrate → contract.

Engagement use. Never drop column same release as code stop using it. Expand add nullable column; dual write if needed; contract drop later. AI-specific: adding index_version to messages—backfill job async.

Pitfalls.

• Long migration lock table—chat outage.
• Breaking migration Friday deploy.
• No rollback migration tested.

Worked example. Alembic migration adds citation_format_version nullable; backfill overnight; app reads with default 1; week 2 enforce NOT NULL after backfill complete.

Chaos and resilience testing

Definition. Chaos experiments inject failure: kill retrieval pod, model 429 storm, CRM 503, network partition to vector DB—validate degradation and no duplicate writes.

Engagement use. Game day quarterly: run in staging with production-like load; measure error budget burn; fix runbooks. Include idempotency verification under retry storm.

Pitfalls.

• Chaos in prod without guardrails—career-limiting.
• Only kill pods—ignore dependency failures.
• No success criteria— theatre.

Worked example. Staging game day: CRM 503 for 10 min; circuit opens; degraded mode serves FAQ; zero duplicate tickets in idempotency audit table; runbook gaps filed as 3 Jira epics.

Monorepo vs polyrepo for AI programmes

Definition. Monorepo houses orchestration, adapters, prompts, eval fixtures—atomic changes across layers. Polyrepo splits services per team—clear ownership, harder coordinated releases.

Engagement use. Prefer monorepo early pilot (<15 engineers) for prompt+code+eval co-versioning. Move to polyrepo when platform/product split solidifies—publish prompt bundles and eval SDK as versioned packages.

Pitfalls.

• Polyrepo without semver on shared prompt package—version hell.
• Monrepo without CODEOWNERS—everyone edits prompts.
• Eval fixtures orphaned in separate repo—never run.

Worked example. Monorepo servicing-assistant/: services/orchestrator, prompts/, eval/golden.jsonl, .github/workflows/eval.yml; tag release-1.8.4 pins all three.

Documentation, runbooks, and operability

Definition. Production readiness includes: README for onboarding, runbooks for incidents, API docs (OpenAPI), architecture decision links, on-call playbooks for model 429, retrieval empty, CRM down.

Engagement use. Definition of Done includes runbook for new integration and new tool. Link runbooks from pager alerts. Architecture diagram in repo root updated on container change.

Pitfalls.

• Runbook "call Sudip"—bus factor.
• OpenAPI generated but never published to gateway catalog.
• Incident fixed without runbook update—repeat incident.

Worked example. Runbook catalog: RB-001 Model 429, RB-002 Empty retrieval, RB-003 CRM circuit open; each with symptoms, checks, mitigation, escalation; reviewed quarterly.

Technical debt register for AI services

Definition. Tech debt register tracks shortcuts: missing contract test, temporary public endpoint, manual prompt edit path, single-region index—each with risk, owner, paydown date.

Engagement use. Review in sprint planning; block scale phase if critical debt unpaid (e.g., no idempotency on write). Link debt items to ADR waivers.

Pitfalls.

• Debt list invisible to sponsor—surprise at scale gate.
• Everything marked "low"—prioritisation useless.
• Paydown never scheduled—permanent waiver.

Worked example. Debt item TD-014: "ServiceNow write sync only"—risk duplicate on timeout; priority high; paydown sprint 12 async+idempotency; linked to Stage 3 exit criterion.

OpenAPI-first design and consumer SDK generation

Definition. OpenAPI-first writes API spec before implementation; generates server stubs, client SDKs, and contract tests from same source.

Engagement use. Publish OpenAPI to corporate API catalog; frontend and partner teams generate typed clients—reduce integration bugs. Include examples for chat request/response, error cases, streaming event schema.

Pitfalls.

• Spec drifts from implementation—CI diff check required.
• Generated server stubs edited manually—regen overwrite.
• Streaming documented as REST JSON—clients implement wrong.

Worked example. openapi.yaml in repo; CI openapi-diff fails breaking changes without version bump; TypeScript client generated for Teams bot; Pact uses same examples.

Performance budgets per request segment

Definition. Performance budget allocates milliseconds per hop: auth 50, retrieval 800, model TTFT 1200, CRM 400, serialization 50—total must meet NFR.

Engagement use. Instrument each segment in traces; fail CI perf test if budget exceeded on golden path. Optimise largest segment first—often model or ERP, not JSON parsing.

Pitfalls.

• Budget only total latency—cannot prioritise fixes.
• Budget ignores parallelism where safe.
• No budget for worst-case ERP—only happy path.

Worked example. Budget table in ADR-015; trace shows CRM 620ms p95—adapter team adds composite API; p95 380ms—orchestration back within budget.

Accessibility and API usability for internal developers

Definition. Internal developer experience affects adoption: clear errors, sandbox access, Postman collection, rate limit headers, request ID in responses, documented quota for model calls.

Engagement use. Treat internal integrators as customers—onboarding doc 30 min to first successful call. SDK handles auth refresh and idempotency key generation.

Pitfalls.

• Cryptic 500 "internal error"—integration delays.
• No sandbox—teams test in prod.
• Missing Retry-After on 429—retry storm.

Worked example. Developer portal page: quickstart, OpenAPI, sandbox keys via self-service (non-prod); error catalog E1001–E1040 documented; integration COE satisfaction 4.2/5 survey.

Release versioning and artifact promotion

Definition. Semantic versioning for services (MAJOR.MINOR.PATCH); prompt bundles and eval sets versioned in lockstep; container images tagged immutable digest; config promoted via GitOps not SSH.

Engagement use. Release notes list: app version, prompt bundle, min index version, model deployment, migrations applied. Rollback = redeploy previous tuple—tested in game day.

Pitfalls.

• Prompt change without bundle version bump—audit gap.
• :latest in prod manifest—rollback guesswork.
• DB migration forward-only without expand-contract—rollback impossible.

Worked example. Release v1.9.0 = image sha256:abc… + prompt_bundle=1.9.0 + index_min=2024.04.2 + Alembic revision 8841; rollback script restores v1.8.4 tuple in 12 min drill.

Pair programming and review norms for AI code paths

Definition. Review norms for AI services: two reviewers for tool registry changes; security review for new write tool; ML engineer on eval threshold changes.

Engagement use. PR template sections: eval impact, integration impact, secrets impact, migration impact. Pair on idempotency implementation first time per team—pattern reuse after.

Pitfalls.

• Rubber-stamp review on "prompt only" PRs.
• No security on new tool—privilege escalation.
• Eval threshold lowered to green CI—quality debt.

Worked example. Team charter: tool changes require security + product review; eval threshold change requires ML lead sign-off; violations caught in retrospective after near-miss duplicate ticket incident.

Local development and test doubles

Definition. Local dev uses docker-compose or devcontainers with test doubles: fake LLM returning fixtures, local vector DB, CRM wiremock—enables offline work without cloud spend.

Engagement use. make test runs unit+integration locally in <5 min; .env.example documents required vars; no prod secrets on laptops—vault dev paths only. Recorded VCR cassettes optional for adapter demos—not sole CI strategy.

Pitfalls.

• Dev connects prod CRM—data breach.
• Fake LLM too optimistic—prod surprise.
• Compose stack diverges from prod Helm chart—parity drift.

Worked example. Devcontainer: Postgres + Qdrant + WireMock Salesforce; FAKE_LLM=1 returns golden responses; developers run 142 unit tests before push; integration nightly only.

Production readiness review (PRR) gate

Before first production traffic, complete PRR checklist:

• CI/CD green on release candidate
• Full eval pass on staging
• Load test meets NFR
• Runbooks linked in pager
• Secrets rotation tested
• Rollback drill <15 min achieved
• On-call roster staffed
• Integration contract tests green
• Security scan no criticals
• Compliance logging verified sample

PRR signed by engineering lead + product owner + ops—not solo developer go/no-go.

SLI/SLO examples for AI orchestration

SLI	SLO (monthly)	Error budget
Successful chat completions	99.5%	0.5%
p95 time-to-first-token	<2.5s	5% of requests may exceed
Golden eval pass rate (staging)	≥92%	Block release if below
Tool call success (non-degraded)	99%	1%

Burn-rate alerts at 2× and 5× budget consumption—page on-call before users flood helpdesk.

Static analysis rules for AI-specific code smells

Configure linters and custom rules where possible:

• Flag hardcoded prompts outside prompts/ directory
• Flag print() in production paths—use structured logging
• Flag broad except swallowing adapter errors
• Flag requests to URLs not in allow-list constant file
• Require type hints on public API handlers (FastAPI strict)

Custom grep in CI: sk-, AKIA patterns for accidental secrets. Pre-commit blocks commits matching secret patterns.

Contract between ML and application teams

Document handoff:

Artifact	ML owner	App owner	Update trigger
Golden eval set	ML/QA	Consumes in CI	New failure mode found
Prompt bundle	Product+ML	Deploys	Eval regression
Model deployment name	ML	Config	Model upgrade ADR
Retrieval index version	Data/platform	Config	Corpus publish

Weekly sync during pilot—async Slack only fails when eval threshold changed without app team noticing.

Record decisions in shared engineering decision log linked from ADR index so ML and app leads share one source of truth.

Practice exercises

Primary exercise: Test strategy and CI outline (3–4 hours)

For HR policy assistant or invoice exception copilot, deliver:

1. API design sketch (OpenAPI outline): ≥4 endpoints with auth and error model.
2. Test strategy table covering unit, integration, golden eval, e2e, load, security.
3. CI/CD pipeline diagram with stages, gates, and rollback trigger.
4. Golden set design: 20 example cases with assertion type (schema, citation, forbidden phrase).
5. Observability plan: logs, metrics, traces—what is not logged (PII).

Acceptance criteria: Prompt change explicitly triggers eval gate; write endpoint has idempotency note cross-ref topic 24.

Stretch exercise: Incident simulation and runbook (3 hours)

Assume retrieval index corruption causes empty results but HTTP 200. Write:

1. Detection alerts (metrics/logs)
2. Mitigation steps in 15 min, 1 hr, 4 hr
3. Rollback to previous index_version
4. Post-incident eval backlog
5. CI improvement to prevent recurrence

Acceptance criteria: Runbook includes comms template for compliance; SLO error budget impact estimated.

Reflection exercise: Notebook-to-production gap (45 minutes)

List 10 gaps between a Jupyter RAG demo and production service for a regulated use case. Map each gap to a Learn subsection. File in pattern library.

Questions you should be able to answer

1. What is the API contract and versioning strategy for orchestration and tools?
2. Which authn/z pattern applies to users vs service-to-service calls?
3. What tests run on every PR vs nightly vs pre-release?
4. What golden eval threshold blocks merge—and who can waive?
5. How are prompt, model, and index versions pinned and logged per request?
6. What is the rollback procedure and target time to last good release?
7. How does clean architecture separate domain from LLM provider adapters?
8. What timeouts, retries, and circuit breakers apply per dependency?
9. What degradation modes exist when model or retrieval fails?
10. What load test results justify NFR concurrency targets?
11. How are secrets managed and rotated per environment?
12. What security tests cover prompt injection and tool abuse?
13. What observability signals detect quality drift before users complain?
14. What is code coverage target and on which layers?
15. How does CI enforce contract tests with integration partners?

Negative cases

Prompt prod edit. No version control; Friday surprise. Fix: prompt bundle in git + eval gate.

No golden set. Quality judged by demo questions. Fix: regression suite in CI.

Retry on write. Duplicate tickets/incidents. Fix: idempotency keys.

Log the prompt. PII in Splunk. Fix: redact; ADR on logging.

God service. Untestable monolith. Fix: ports/adapters.

Staging ≠ prod. Private link missing in staging. Fix: prod-like env parity.

Exact match eval. Flaky CI random fail. Fix: property-based assertions.

Load test theatre. Test 10 users; promise 10K. Fix: NFR-driven k6.

Tool registry no authz. Any user invokes dangerous tool. Fix: ABAC + tests.

Dependency unpinned. Reproducible build fails. Fix: lockfiles + scan.

Skip eval on "small change". Retrieval chunk change breaks recall. Fix: corpus hash triggers eval.

Canary skip. Big-bang prod deploy. Fix: progressive rollout + SLO monitor.

Integration with Delivery Guide

Topic 15 capability	Guide chapter
Release trains, environments	Delivery Guide
Architecture boundaries	Architecture Guide
Eval design detail	Topic 21 + Guide QA sections
Integration contracts	Topic 24

Output

Expected outputs for this topic:

• API design sketch (OpenAPI outline with auth and errors)
• Test strategy document (pyramid + AI regression)
• CI/CD pipeline outline with gates and rollback
• Golden set sample (≥20 cases) with assertion types
• Observability and SLO sketch for orchestration service

Practice checklist

• API spec includes versioning, limits, and idempotency on writes
• Test strategy names eval threshold and waiver process
• CI diagram shows prompt/retrieval/model change triggers
• Rollback runbook target time documented
• Security tests include injection and tool authorization
• Load test linked to NFR concurrency from topic 08
• Practice artefacts filed in pattern library

Related playbook content

• Architecture Guide — service boundaries and NFRs
• Delivery Guide — release and environment patterns
• Forward-Deployed Engineer roadmap
• The Complete Forward-Deployed Engineer Roadmap
• AI Evaluation and Quality Assurance — eval design depth
• Integration and Enterprise Systems — idempotency and contracts
• 8D Framework
• Architecture and cloud

Key takeaways

• Prompts, retrieval configs, and model pins are code—review, version, evaluate in CI.
• Test pyramid plus golden eval is the minimum bar for AI production—not optional polish.
• APIs, auth, and observability must be designed before scale—retrofit is expensive.
• Rollback and degradation paths are reliability features, not ops afterthoughts.
• Tool surfaces are security boundaries—test authorization like any admin API.
• Pair with Delivery Guide for release workflow depth.

What to do next

1. Complete primary test strategy before production traffic.
2. Continue with Cloud and Platform Engineering for hosting orchestration services.
3. Align integration contracts with Integration and Enterprise Systems.
4. Deepen eval design in AI Evaluation and Quality Assurance.