Cloud and Platform Engineering

Executive summary

Cloud and platform engineering for AI is how approved infrastructure hosts inference, orchestration, retrieval, and batch embedding at scale—with landing zones, identity, network isolation, GPU capacity, IaC, observability, DR, and explicit multi-cloud tradeoffs—not a developer's personal subscription with public endpoints. An AI Solution Engineer must produce landing-zone fit notes, environment topology, and control checklists that satisfy enterprise security and residency while meeting latency and cost NFRs. Without this capability, pilots leak data through public model APIs, exhaust GPU quotas at go-live, fail DR audits, or require £200K+ redeployments when discovered too late.

When to use this page

• Stage 3 requires where and how AI workloads run—not only what they do.
• Security mandates private connectivity, CMK/BYOK, or sovereign cloud regions.
• Programme needs GPU for embedding batch, fine-tune, or self-hosted inference.
• Enterprise provides landing zone (Cloud Factory, CAF, GCP Organization)—you must map workload into it.
• Multi-cloud or hybrid discussed—need tradeoff ADR, not resume-driven three-cloud design.
• FinOps asks for cost envelope and tagging before scale approval.
• DR and RTO/RPO questions arise for conversation state and vector indexes.
• Dev team deployed to wrong subscription—platform review blocks production.

Executive view

Focus on **residency**, **blast radius**, **run cost**, and rework risk in scenarios. Ask: are we on approved runway, and what happens if region fails?

Technical view

Design subscriptions/VPCs, IAM, peering, endpoints, IaC modules, and capacity plans before enabling production traffic.

Why this matters

AI workloads are not normal web apps. They combine high egress to model APIs, GPU-scarce compute, large vector indexes, sensitive corpus storage, and bursty batch embedding jobs that starve interactive inference if shared naively. A UK insurer deployed a copilot to a developer sandbox: US region, public model endpoint, shared admin password, no CMK—discovered at production gate. Redeploy to enterprise landing zone took 14 weeks and £220K rework; pilot credibility damaged.

Landing zones exist because enterprises learned painful lessons: centralized identity, guardrails (policy-as-code), network hub-spoke, separation of prod/non-prod, and audit logging cannot be reinvented per project. Platform engineering provides runway; product teams land aircraft (topic 08). Blurring the boundary creates security findings and duplicate VPCs.

Cloud choices also lock operational models. Multi-cloud sounds resilient but triples IaC, observability, and skilled headcount—often unjustified for a single use case. Hybrid connects to on-prem ERP and core banking—latency and split-brain DR complexity. This topic teaches when to accept complexity, not default to it.

Pair with Architecture Guide for technology view and topic 15 for what runs on the platform.

Learn

Landing zones and account topology

Definition. A landing zone is a pre-configured cloud foundation: organization hierarchy, subscriptions/accounts/projects, guardrails (policies), networking hub, logging, identity integration, and often workload placement rules (AI in sub-prod-ai, prod in sub-prod-ai-prd).

Engagement use. Request landing zone blueprint day one. Map AI components to approved segments: orchestration in app subnet, vector DB in data subnet, batch jobs in compute subnet, model API via private endpoint only. Document fit gaps—need waiver ADR if non-standard (e.g., temporary public endpoint in dev only).

Pitfalls.

• Personal subscription "for speed."
• Prod and dev in same subscription—blast radius.
• Ignoring quota limits in landing zone region.
• Landing zone without AI service allow-list—OpenAI blocked by policy.

Worked example. Azure CAF: Management group AI-Workload → subscriptions ai-nonprod, ai-prod; policy denies public IP on NICs; Azure OpenAI allowed in westeurope only; deployment via approved pipeline service principal.

Identity and access management (IAM)

Definition. IAM covers human access (PIM/PAM just-in-time admin), workload identity (managed identity, IRSA on EKS, GCP SA), RBAC on resources, and conditional access (MFA, device compliance). AI pipelines need service principals for embed jobs, orchestrators, CI/CD—with least privilege.

Engagement use. No long-lived keys in repos. Humans get Contributor on non-prod via PIM; prod read-only except break-glass. Orchestrator identity: get secret, call model, query vector—not Owner. Separate identities for batch embed vs online serving for audit. Federated CI (OIDC) to cloud—no static CI keys.

Pitfalls.

• One mega service principal owns subscription.
• Admin creds on embedding CronJob pod.
• Missing ABAC tags for env enforcement.
• Identity not propagated to data plane (vector ACL plugin misconfigured).

Worked example. AWS: EKS IRSA role orchestrator-prod with policy allowing bedrock:InvokeModel on specific ARN, secretsmanager:GetSecretValue for two secrets, aoss:APIAccessAll on one collection—nothing else.

Networking: VPC/VNet, subnets, peering, and private endpoints

Definition. Network design isolates tiers: public edge (load balancer only), application subnets (orchestration), data subnets (databases, vector), private endpoints for PaaS (model API, storage, key vault) avoiding public internet paths.

Engagement use. Hub-spoke or landing zone virtual WAN for on-prem/hybrid. Egress via central firewall with allow-list domains for model API if not fully private. NSG/security groups: deny east-west except required ports. Document DNS for private link zones. Latency budget includes peering hops to on-prem ERP.

Pitfalls.

• Public model endpoint because "private link hard."
• Flat network—all pods reach all data.
• Missing SNAT exhaustion on outbound-heavy embed jobs.
• Hybrid without bandwidth plan—RAG over WAN unusable.

Worked example. EU bank: orchestration in 10.1.2.0/24; vector in 10.1.3.0/24; Azure OpenAI private endpoint in 10.1.4.0/24; egress to Salesforce via private link + proxy; no default route to internet from app subnet.

Environment separation and promotion

Definition. Environments (dev, test, staging, prod) are isolated accounts or subscriptions with promotion of immutable artifacts (containers, IaC tags)—not copy-paste config edits.

Engagement use. Parity: staging mirrors prod network path (private endpoints), identity pattern, and similar data (masked). Separate model quotas per env to prevent dev burning prod quota. Config drift detected by IaC plan in CI. Secrets per env in separate vault paths.

Pitfalls.

• Prod keys in dev vault "for testing real responses."
• Staging on cheaper tier without private link—false confidence.
• Manual console hotfix in prod not backported to IaC.
• Shared vector index across envs—test pollutes prod metrics.

Worked example. Three subscriptions; Argo CD promotes same image tag 1.8.4 dev→staging→prod; staging runs full eval; prod CMK keys differ; dev uses synthetic corpus only.

GPU capacity planning and scheduling

Definition. GPUs accelerate embedding, fine-tuning, and self-hosted inference. Capacity planning covers SKU choice (A100, H100, L4), quota, node pools, time-slicing, batch vs realtime queues, and cost per token/job.

Engagement use. Separate interactive and batch pools. Request quota 12 weeks early. Right-size: many RAG systems need CPU embed on small models + GPU only for fine-tune bursts. Monitor GPU utilisation—<30% sustained suggests over-provision. Use spot/preemptible for batch with checkpointing.

Pitfalls.

• Fine-tune and serving share pool—serving starved.
• No quota fallback plan—launch day blocked.
• GPU nodes for orchestration that is CPU-bound—waste.
• Ignore driver/CUDA pin in container—silent perf loss.

Worked example. Insurer: L4 node pool 2–8 nodes autoscale for nightly embed of 2M chunks; interactive inference on managed API (no self-host GPU); fine-tune quarterly on dedicated 1×A100 job queue max 72 hr—booked via platform ticket.

Infrastructure as Code (IaC) and policy-as-code

Definition. IaC (Terraform, Bicep, CDK, Pulumi) defines resources declaratively; policy-as-code (OPA, Azure Policy, SCPs) enforces guardrails automatically.

Engagement use. All resources via modules: network, aks, openai, vector, monitoring. PR plan in CI; apply via pipeline service principal only. Tag mandatory: cost_centre, use_case, env, owner. Policies deny public blob, unencrypted disks, non-approved regions.

Pitfalls.

• ClickOps prod changes—not in Terraform state drift hell.
• Monolithic state file—blast radius on apply.
• Secrets in tfvars committed to git.
• Module fork per project—no standardisation.

Worked example. Terraform module genai-workload params: region, subnet ids, model deployment name, index SKU; CI terraform plan on PR; apply to prod requires two approvers; drift detect nightly.

Hyperscaler AI services vs self-hosted models

Definition. Managed AI APIs (Azure OpenAI, Bedrock, Vertex) offer operational simplicity, enterprise agreements, private link—cost per token. Self-hosted (vLLM, TGI on GPU) offers control, air-gapped options—ops burden.

Engagement use. Default managed unless: air-gap, extreme scale cost crossover, or custom model weights. Document data processing terms (training opt-out, residency). Self-host needs SRE model: patching, scaling, model registry, A/B traffic split.

Pitfalls.

• Self-host to "save money" at <1B tokens/month—ops cost higher.
• Managed API without PTU/reservation plan—latency spikes at scale.
• Wrong region deployment name hardcoded—DR fail.

Worked example. Bank ADR: Azure OpenAI EU for inference; self-host embedding MiniLM on CPU cluster for cost; crossover analysis at 8B tokens/month revisits self-host LLM.

Multi-cloud, hybrid, and edge tradeoffs

Definition. Multi-cloud runs across AWS+Azure+GCP—rarely needed for single AI product. Hybrid connects cloud AI to on-prem ERP/core systems. Edge runs inference near device/plant for latency/offline.

Engagement use. Multi-cloud only with explicit drivers: acquisition, regulator-mandated split, or avoid lock-in at proven scale—document cost of complexity (2–3× platform engineering). Hybrid: place orchestration cloud-side or on-prem based on data gravity and latency; often retrieval index cloud, authoritative data on-prem via API. Edge for factory vision—cloud for training only.

Pitfalls.

• Multi-cloud disaster recovery fantasy without data replication cost.
• Hybrid dual writes between cloud index and on-prem DB.
• Edge model update orphaned—version skew.
• Egress fees kill hybrid sync of full corpus nightly.

Worked example. ADR-009: Single cloud (Azure) for EU programme; hybrid read from on-prem SAP via ExpressRoute; no multi-cloud; edge not in scope year 1—revisit if plant latency NFR <50ms.

Observability, FinOps, and platform SLOs

Definition. Platform provides central logs, metrics, traces, cost dashboards, and SLOs on shared services (index API, model endpoint proxy).

Engagement use. Tag all resources for chargeback. Monitor GPU util, token usage, 429 rates, index lag. Platform SLO example: embedding job completion within 4 hr of corpus publish 95% of time. Alert on budget thresholds—FinOps weekly review in scale phase.

Pitfalls.

• Untagged resources—"AI cost mystery."
• No token per use case attribution.
• Platform team blind to product eval fail—not operational metric.
• Log retention unbounded cost.

Worked example. Dashboard: use_case=servicing-copilot monthly £23K inference, £4K vector, £2K embed batch; alert at 120% of budget; platform SLO breach triggers incident to platform on-call not product intern.

Disaster recovery, backup, and regional residency

Definition. DR plans RTO/RPO for stateful components: vector index, conversation store, config registry. Residency constrains primary and DR regions (EU data stays EU—even DR pair).

Engagement use. Vector index: async replica or rebuild from gold corpus runbook (often acceptable RPO hours if gold durable). Conversation state: replicate or accept session loss with UX message. Test restore quarterly. DR failover may need manual approval in regulated industries—document runbook time 4 hr not 4 min if true.

Pitfalls.

• DR region US when primary EU—regulator block.
• Backup never restored—corrupt index undetected.
• Single-AZ vector for "cost"—AZ outage = outage.
• Model deployment missing in DR region—failover works except brain.

Worked example. RPO 1 hr index replica EU-West→EU-North; RTO 4 hr manual failover approval; conversation Redis 15 min snapshot; runbook RB-DR-03 last tested Q2, pass.

Security controls: encryption, keys, and zero trust

Definition. Encryption at rest (CMK in Key Vault/KMS/HSM), in transit (TLS 1.2+), BYOK/HYOK where required, zero trust (verify every call, no flat trust inside VPC).

Engagement use. CMK for storage, vector, logs if mandated. Rotate keys annually. Private Link mandatory prod. WAF on public edge only. JIT admin. Vulnerability scan on AMIs/containers. Confidential computing niche for extreme cases.

Pitfalls.

• Microsoft-managed keys when contract requires CMK.
• TLS terminate at load balancer then HTTP internal—acceptable only with mTLS inner or strict NSG.
• Key vault access too broad—same SP decrypts all envs.

Worked example. CMK per env; orchestrator decrypt via managed identity; HSM-backed key for prod; annual rotation runbook; penetration test validates no public storage endpoints.

Frameworks and methods

Landing zone fit assessment template

Control area	Required	Current fit	Gap	Waiver?
Subscription placement	AI prod in approved MG	Yes	—	No
Region	EU only	Yes	—	No
Private model access	Private endpoint	Partial dev	Prod PE week 3	Temp dev
CMK	Mandatory prod	No	Implement	No
Tagging	Mandatory tags	Partial	Fix IaC	No
IAM	No shared admin	Yes	—	No

Environment topology diagram (logical)

GPU capacity worksheet

Workload	GPU SKU	Peak nodes	Hours/day	Quota needed	Fallback
Nightly embed	L4	8	3	8×L4	CPU embed slower
Fine-tune quarterly	A100	1	72	1×A100	Delay tune

Multi-cloud decision matrix

Factor	Single cloud	Multi-cloud
Ops headcount	Lower	2–3×
Residency control	Easier	Complex
Vendor outage	DR within cloud	Theoretical cross-cloud
Cost at <£500K/yr AI	Usually lower	Usually higher

Default single cloud unless matrix score forced.

Well-Architected review (platform lens)

Pillars applied to platform product: reliability of shared index API, security of private endpoints, cost of GPU pools, operational excellence of IaC pipelines.

8D alignment

Design: landing zone fit. Deploy: IaC apply, DR test. See 8D Framework.

Real-world scenarios

Scenario A: EU banking private deployment

EU bank mandates no public internet from app tier to model. Initial POC used public Azure OpenAI—blocked at ARB. Redesign: private endpoint, CMK, EU West only, egress deny default, ExpressRoute to on-prem core banking read API.

Numbers: Redesign 11 weeks, £185K; incremental run +£3K/month private link vs public; zero public egress findings at audit; model latency +40ms vs public—within NFR. Lesson: landing zone fit week 1, not week 20.

Scenario B: GPU quota crunch at launch (US SaaS)

SaaS analytics copilot launches Black Friday; embed batch needs 24×A10; quota 12; backlog 6 hours; stale index for peak pricing docs.

Fix: Quota request 8 weeks earlier; CPU fallback embed path; priority queue for tier-1 corpus; reserved capacity PTU for inference. Outcome: Peak day index lag p95 22 min vs prior 6 hr; revenue-impacting stale answers <0.3%.

Scenario C: Hybrid manufacturer (SAP on-prem)

Manufacturer RAG over SAP PM notes + SharePoint manuals. Cloud orchestration; SAP via on-prem API over VPN; 300ms RTT adds to latency budget.

Design: Cache read-heavy SAP extracts in silver layer hourly sync; live SAP write only via approved BAPI async. Numbers: Full cloud move rejected—£2M SAP rewrite; hybrid ops +0.4 FTE platform; p95 user latency 2.8s within 3s NFR.

Scenario D: Multi-cloud retreat (global insurer)

Insurer attempted AWS Bedrock + Azure OpenAI per region "for resilience." 18 months, triple IaC teams, inconsistent eval, £900K over budget. EA review consolidates to Azure EU single platform with DR pair.

ADR-010: retire AWS path; save £420K/year ops; migration 9 months. Lesson: multi-cloud without forced driver is tax.

Scenario E (stretch): Sovereign cloud requirement (public sector)

Government workload requires sovereign cloud region with staff citizenship controls on ops. Standard landing zone not fit—waived to sovereign provider; model hosting air-gapped approval; GPU pool small—batch windows restricted.

Numbers: +25% infra cost; +6 week procurement; compliance acceptance achieved—generic hyperscaler rejected by authority.

Kubernetes patterns for AI workloads

Definition. Kubernetes (EKS, AKS, GKE) orchestrates containerised orchestration APIs, retrieval services, and workers. Patterns: HPA on CPU/RPS/custom metrics (tokens_per_second), KEDA for queue-driven embed jobs, taints/tolerations for GPU nodes, PodDisruptionBudgets for availability during node drain.

Engagement use. Separate namespaces per env (ai-prod, ai-staging). Network policies deny cross-namespace except Istio/Linkerd mTLS mesh where mandated. Resource requests/limits on all pods—GPU jobs request nvidia.com/gpu: 1. Init containers warm caches before readiness passes.

Pitfalls.

• GPU node pool without autoscaler—queue forever.
• Single replica orchestration—AZ outage = outage.
• ConfigMap stores secrets—use vault CSI.
• Massive emptyDir for model weights—node disk full.

Worked example. AKS: orchestration 3–30 pods HPA on p95 latency; embed jobs KEDA on Service Bus queue depth; GPU pool 0–6 nodes; PDB minAvailable: 2 for orchestration.

Serverless vs containers for AI components

Definition. Serverless (Lambda, Cloud Functions, Azure Functions) suits spiky, short tasks: webhooks, format conversion, lightweight routing. Containers suit long-lived connections (SSE streams), consistent latency, GPU, and complex dependencies.

Engagement use. Hybrid: API container on AKS; serverless for CMS webhook → queue message; avoid 15 min Lambda limit for large embed batches—use Batch/AKS job instead. Cold start breaks <2s first-token NFR—size provisioned concurrency if serverless edge.

Pitfalls.

• Streaming chat on Lambda without provisioned concurrency—cold start UX fail.
• Container everything—including once-a-day 10s job—ops overhead.
• Serverless VPC attach misconfig—cannot reach private vector DB.

Worked example. Functions handle SharePoint webhook (200ms avg); AKS handles chat SSE; nightly embed AKS Job 2 hr wall clock 800K chunks.

Cost optimisation and FinOps levers

Definition. FinOps for AI: reserved capacity (Azure PTU, Bedrock provisioned), spot batch GPU, right-size indexes, content-hash skip re-embed, tiered model routing (small model FAQ, large model complex), log retention caps.

Engagement use. Monthly review: cost per successful session, cost per 1K tokens, GPU utilisation %. Autoscale down off-peak. Budget alerts at 80/100/120%. Chargeback tags mandatory.

Pitfalls.

• PTU purchased before traffic proof—£ wasted.
• Re-embed entire corpus nightly—hash skip saves 40–70% typically.
• Verbose debug logging to expensive log analytics.
• Cross-AZ egress for chatty vector queries—architecture tax.

Worked example. Programme starts pay-as-you-go £31K/month; after hash-skip + FAQ routing + PTU partial reservation → £19K/month at same volume −39%; FinOps report to steering monthly.

Private DNS, certificates, and certificate lifecycle

Definition. Private DNS zones resolve private endpoints (openai.privatelink.azure.com). Certificate management (ACME internal, Key Vault certs) for mTLS to bank APIs—auto-renew with 30-day expiry alert.

Engagement use. Document every cert: owner, expiry, renewal runbook. Service mesh mTLS for east-west. Fail CI if cert <14 days on staging mirror of prod.

Pitfalls.

• Manual cert paste—expires holiday weekend.
• Wildcard cert overuse—blast radius.
• DNS split-brain hybrid—intermittent ERP failures.

Worked example. Key Vault auto-renew mTLS cert for core banking; Datadog synthetic fails 15 days before expiry; last incident 2022 before automation.

Platform onboarding checklist for product teams

Product team requests GenAI platform access—platform engineering verifies:

1. Landing-zone subscription allocated and tagged
2. Identity templates applied (orchestrator SP, CI OIDC)
3. Network peering / private endpoints validated
4. Quota (GPU, TPM) confirmed with dates
5. Observability dashboard cloned and use_case label set
6. Runbooks linked in service catalog
7. Cost budget owner signed

Gate: no production DNS cutover until 7/7 complete—record in ARB minutes.

Data plane vs control plane separation

Definition. Control plane manages configuration, IAM policies, IaC, GitOps. Data plane handles user traffic: orchestration requests, retrieval queries, model inference.

Engagement use. Changes to control plane (policy, network) require change window and rollback; data plane scales independently. Blast radius: compromised control plane credential must not imply data plane exfiltration—separate roles.

Pitfalls.

• Single admin role for both—audit finding.
• GitOps applies network policy mistake—prod outage.
• Data plane pods with cluster-admin—escape risk.

Worked example. Azure: platform team Contributor on management subscription; product team Deployer on workload subscription only; policy assignments at MG level—product cannot create public IP.

Secrets rotation and vault architecture

Definition. Secrets (API keys, client secrets, mTLS keys) live in Key Vault / Secrets Manager / Parameter Store; injected at runtime via CSI driver or init; rotated on schedule with dual-active period.

Engagement use. Rotation runbook: create new secret version → deploy consumers with dual read → disable old version after 24 hr traffic verify. Never rotate during Black Friday without freeze window.

Pitfalls.

• Single secret version referenced everywhere—rotation outage.
• Secrets in env vars logged on crash dump.
• Manual rotation spreadsheet—missed expiry.

Worked example. Key Vault secret crm-client-secret versions 3→4; Argo roll restart orchestrator pods; synthetic transaction verifies; version 3 disabled; calendar reminder 80 days before next rotation.

Compliance scanning and audit evidence collection

Definition. Cloud compliance dashboards (Azure Policy, AWS Config, Security Hub) prove controls continuously—not only annual audit screenshots.

Engagement use. Export evidence pack monthly: policy compliance %, private endpoint coverage, CMK usage, admin PIM logs, vulnerability scan summary. Map to control framework (ISO 27001, SOC2) rows.

Pitfalls.

• Compliance tool enabled but 0 policies assigned—theatre.
• Evidence manual screenshot—stale.
• Exceptions not linked to ADR waiver IDs.

Worked example. Monthly pack auto-generated to SharePoint Audit/AI-Platform/2024-03; 4 policy non-compliance items with remediation tickets JIRA-4412 etc.; auditor accepts continuous export.

Edge cases: air-gapped and classified environments

Definition. Air-gapped clouds have no internet; model weights and corpora sneakernet or internal mirror; updates quarterly. Classified environments add personnel clearance and physical controls.

Engagement use. Architecture radically different: self-host models, internal artifact registry, manual CVE triage. Plan 6–12 month lead times for hardware and accreditation—do not promise agile cadence identical to public cloud.

Pitfalls.

• Assuming Azure OpenAI available—it's not.
• USB transfer without malware scan—incident.
• Same IaC modules without air-gap parameterization—deploy fail.

Worked example. Defence research assistant: vLLM on 2×A100 air-gap cluster; corpus updates via scanned media; eval suite shipped with corpus on same media version tag corpus-2024.02.

Capacity planning for model rate limits (TPM/RPM)

Definition. Cloud model APIs enforce tokens per minute (TPM) and requests per minute (RPM) per deployment/region. Provisioned throughput units (PTU) reserve capacity.

Engagement use. Model traffic forecast: avg tokens/request × peak RPS × headroom 1.5×. Request quota increase 8 weeks ahead; multi-deployment routing if single deployment saturates. Monitor 429 as leading indicator.

Pitfalls.

• Single deployment for prod+staging—staging burn prod quota.
• Burst marketing campaign without quota plan—hard outage.
• Ignore embedding TPM separate from chat TPM.

Worked example. Forecast 12M tokens/day peak; request PTU 300K TPM; secondary deployment fallback on 429 with quality disclaimer in degraded mode; 429 rate <0.1% post-change.

Shared services catalogue for AI platform teams

Platform team publishes catalogue consumed by product squads:

Service	SLA	Onboarding time	Owner
Model endpoint (EU)	99.9%	2 days	Platform
Vector index API	99.9%	5 days	Platform
CI eval runner	99.5%	1 day	Platform
Prompt registry	99.5%	1 day	Platform
CRM adapter template	Best effort	10 days	Integration COE

Product teams request via ticket—no bespoke VPC per copilot without exception ADR.

Network traffic inspection and egress control

Definition. Egress filtering via firewall, DNS filtering, or secure web gateway restricts outbound destinations to approved model API FQDNs and SaaS integrations—prevents data exfiltration if orchestration compromised.

Engagement use. Default deny egress from app subnets; allow-list maintenance via ticketed change. Log blocked egress attempts—alert on anomaly. Split horizon DNS for private endpoints.

Pitfalls.

• Allow 0.0.0.0/0 "temporarily" for debug—forgotten.
• Model SDK bypasses corporate proxy—shadow IT path.
• DNS over HTTPS bypasses filtering—policy gap.

Worked example. Palo Alto rule: app subnet → allow *.openai.azure.com via private link only; block general internet; 3 blocked attempts/week investigated; zero public exfil paths in annual pen test.

Platform metrics executive summary template

Monthly one-page for steering committee:

Metric	This month	Target	Trend
Platform availability	99.94%	99.9%	↑
GPU utilisation (avg)	41%	35–70%	OK
AI infra cost / use case	£23K	£28K cap	OK
Policy violations (Azure Policy)	2 open	0	↓ action
MTTR platform incidents	38 min	<60 min	OK

Narrative: 3 sentences on incidents, cost drivers, upcoming quota changes—no raw cloud bill dump.

Hybrid DNS and split-horizon resolution

Definition. Hybrid DNS resolves the same name differently on-prem vs cloud—critical when ERP hostname must point to internal IP from corporate network and private endpoint from cloud VNet.

Engagement use. Document conditional forwarders; test from both sides in staging. AI orchestration in cloud must use cloud-side resolver—on-prem DNS servers not reachable breaks lookups silently if misconfigured.

Pitfalls.

• Hardcoded IP bypasses DR failover.
• Split DNS misconfiguration—intermittent 502 only from cloud.
• Certificate SAN mismatch on internal vs external name.

Worked example. erp.bank.internal → 10.50.1.20 on-prem; 10.60.2.40 private endpoint from Azure VNet via Azure Private DNS zone linked to VNet; integration tests assert both paths quarterly.

Sustainability and carbon awareness (optional NFR)

Some enterprises add carbon or renewable energy preferences to region selection. GPU-heavy workloads have higher carbon intensity—document trade-off ADR when sustainability conflicts with latency (e.g., chosen region farther from users but greener grid).

Worked example. ADR-014: primary EU North (lower carbon grid factor published by provider) vs EU West (lower latency for UK users)—chose EU West with carbon offset programme; documented in sustainability report appendix.

Backup regions and game-day exercises

Schedule annual game day for region failure: simulate primary region unavailable; execute DR runbook; measure RTO achieved vs target; document gaps. Include model deployment availability in DR region—not only infrastructure.

Checklist: DNS/traffic shift, index replica promotion or re-index job start, secrets available in DR vault, integration paths still reach on-prem via same ExpressRoute (often forgotten—DR region cloud cannot reach on-prem if networking not duplicated).

Tagging schema reference (minimum keys)

Tag key	Example	Purpose
use_case	servicing-copilot	Cost allocation
env	prod	Policy scope
cost_centre	CC-8842	Chargeback
owner	team-servicing-ai	Escalation
data_class	confidential	DLP policies
dr_tier	tier-1	DR priority

Deny resource deploy if mandatory tags missing—Azure Policy / SCP enforced at platform level, not voluntary developer memory.

ExpressRoute / VPN hybrid capacity planning

Hybrid AI programmes must size WAN for peak retrieval sync and API calls—not average chat traffic.

Estimate: peak RPS × avg payload × business hours × headroom 1.4×. If corpus sync adds 200 GB/night, ensure circuit not saturated with other batch jobs—coordinate with network team before ARB promises real-time hybrid answers.

Worked example. Manufacturer: 500 Mbps ExpressRoute; nightly 80 GB SAP extract + 40 GB document sync fits 02:00–05:00 window with 35% peak utilisation—headroom for daytime 120 TPS API reads documented in network runbook.

Platform engineering operating model (RACI)

Activity	Platform team	Product team	Security	FinOps
Landing zone policy	A/R	C	C	I
Subscription request	R	C	A	C
GPU quota	R	C	I	C
IaC module maintain	A/R	C	C	I
App deploy manifest	C	A/R	C	I
Incident platform layer	A/R	C	C	I
Incident app layer	C	A/R	C	I
Cost optimise shared	R	C	I	A

A = accountable, R = responsible, C = consulted, I = informed. Prevents product teams editing org-level policies without platform review—and prevents platform team owning product eval thresholds.

Azure / AWS / GCP service selection notes (AI workloads)

Without vendor-first framing, typical pattern mappings:

Need	Azure	AWS	GCP
Managed LLM API	Azure OpenAI	Bedrock	Vertex AI
Vector search	AI Search	OpenSearch Serverless	Vertex Vector Search
Orchestration	AKS	EKS	GKE
Identity	Entra + MI	IAM + IRSA	IAM + WI
Secrets	Key Vault	Secrets Manager	Secret Manager
Private PaaS access	Private Link	PrivateLink	Private Service Connect

Choose one primary column per programme ADR—mixing managed LLM across clouds without driver is an ops tax (topic 08 ADR-010 pattern). Revisit the mapping when residency, quota, or enterprise agreement constraints change—service catalogs evolve quarterly on all three hyperscalers. Document the chosen column in the landing-zone fit note appendix for auditor traceability.

Practice exercises

Primary exercise: Landing-zone fit and env topology (3–4 hours)

For UK insurer copilot or EU bank servicing assistant, deliver:

1. Landing-zone fit table (≥8 control areas) with gaps and remediation dates.
2. Environment topology diagram: subnets, private endpoints, identity flows.
3. IAM matrix: human roles, workload identities, CI federated access.
4. GPU/CPU capacity plan with quota timeline.
5. DR sketch: RTO/RPO per component; residency statement.

Acceptance criteria: Prod path has no mandatory public internet to model; CMK addressed; tags listed.

Stretch exercise: Multi-cloud ADR (2 hours)

Write ADR for and against multi-cloud for a £400K/year AI programme with EU residency. Include 3-year TCO rough order and ops FTE. Recommend single or multi with decision.

Acceptance criteria: Decision cites specific drivers—not generic "avoid lock-in."

Reflection exercise: Sandbox anti-pattern (45 minutes)

Document 10 findings wrong with personal cloud subscription production path. Map to Learn subsections. File in pattern library.

Questions you should be able to answer

1. Which landing zone/subscription hosts prod vs non-prod—and who owns it?
2. What region(s) satisfy residency—and where is DR?
3. How does traffic reach the model API without forbidden public paths?
4. What IAM identities call model, vector, secrets, ERP—and with what scopes?
5. What network segments isolate app, data, integration, management?
6. What GPU/CPU capacity and quota are required by when?
7. How is IaC structured, reviewed, and drift-detected?
8. What tags drive cost allocation and policy enforcement?
9. What are RTO/RPO for index, sessions, and config?
10. What CMK/BYOK requirements apply—and rotation process?
11. What environment promotion path deploys artifacts to prod?
12. What platform SLOs apply to shared services?
13. Why single vs multi-cloud for this programme—ADR reference?
14. What hybrid connectivity latency is in the end-to-end budget?
15. What Well-Architected gaps remain before production gate?

Negative cases

Developer subscription prod. Blocked or rehosted expensive. Fix: landing zone day one.

Public model endpoint. Data path compliance fail. Fix: private link + policy.

Shared admin credential. Key rotation impossible. Fix: managed identity + PIM.

No quota plan. Launch blocked. Fix: GPU/TPM request early.

Flat VPC. Lateral movement risk. Fix: tiered subnets + NSG.

Staging without PE. Prod-only failure mode. Fix: parity networking.

ClickOps drift. Terraform untrusted. Fix: pipeline-only apply.

US DR for EU data. Regulator reject. Fix: residency-aware DR.

GPU for everything. Cost blowout. Fix: right-size workload to CPU/managed API.

Untagged spend. FinOps blind. Fix: mandatory tags policy.

Multi-cloud default. Ops tax. Fix: ADR with drivers.

Backup never tested. DR theatre. Fix: quarterly restore drill.

Integration with Architecture Guide

Topic 16 capability	Guide chapter
Technology view, hosting patterns	Architecture Guide
Platform vs product	Topic 08
CI deploy targets	Topic 15
Hybrid integration paths	Topic 24

Output

Expected outputs for this topic:

• Landing-zone fit note with gap/remediation table
• Environment topology diagram (network + identity)
• IAM and service identity matrix
• GPU/quota capacity plan with dates
• DR sketch with RTO/RPO and residency statement
• Optional: IaC module list and tagging schema

Practice checklist

• Prod placement in approved landing zone documented
• Private connectivity to model and data services on prod path
• CMK/encryption requirements addressed
• IAM follows least privilege; no shared long-lived admin keys
• GPU quota request timeline aligned to go-live
• DR/residency consistent with topic 08 NFRs
• Multi-cloud/hybrid choice has ADR if non-obvious
• Practice artefacts filed in pattern library

Related playbook content

• Architecture Guide — technology view and review gates
• AWS Learning Roadmap · Full guide
• Azure Learning Roadmap · Full guide
• Google Cloud Learning Roadmap · Full guide
• Enterprise Architecture — platform vs product framing
• Software Engineering — what deploys on platform
• Integration and Enterprise Systems — hybrid connectivity
• Architecture and cloud

Key takeaways

• Landing zones are non-negotiable runway for enterprise AI—sandbox prod is rework waiting to happen.
• Identity, network, and encryption must be designed together—fixing one in isolation fails audits.
• GPU capacity is a programme milestone, not a ticket filed at launch.
• IaC and policy-as-code make guardrails repeatable; ClickOps makes incidents repeatable.
• Multi-cloud is usually ops tax—require explicit ADR drivers.
• Pair with Architecture Guide for hosting pattern depth.

What to do next

1. Complete landing-zone fit before requesting production subscription access.
2. Continue with Security Engineering for Stage 4 depth.
3. Align hybrid paths with Integration and Enterprise Systems.
4. Follow cloud roadmaps (AWS, Azure, GCP) for certification depth.