Performance Engineering and AI FinOps
Executive view
Technical view
Why this matters
Users experience end-to-end latency—retrieval, reranking, model TTFT, streaming, tool calls, post-processing—not model marketing benchmarks. A 400ms model TTFT plus 1.2s retrieval plus 800ms tool round-trip fails a 3s interactive SLA before the answer streams. Performance engineering decomposes the path, sets budgets per segment, and optimises the dominant segment first—often retrieval or unnecessary agent steps, not the frontier model.
FinOps exists because AI spend is variable, opaque, and bursty. Token invoices do not explain which product, team, or intent drives cost. Without cost per successful task, leaders optimise the wrong thing—cutting requests while failure rate rises, or caching unsafe answers to save money. FinOps ties telemetry (tokens, latency, outcome flags) to business units and use cases, enabling showback, budgets, and engineering backlogs grounded in £/outcome.
Caching is the highest-leverage FinOps tool when safe: prefix caching reduces prompt cost; semantic response caches eliminate repeat LLM calls; KV cache optimizations improve throughput on self-hosted stacks. Regulated environments require cache eligibility rules—never cache personalized advice without TTL and invalidation on corpus version change. See LLM Caching Ultimate Guide for mechanism depth.
Batching improves throughput for offline jobs but must not starve interactive traffic—separate queues, schedules, and capacity pools. Capacity planning translates expected tokens/sec and concurrent users into API quotas, GPU counts, and autoscaling policies—with headroom for eval jobs and failover.
Pair with Model FinOps and AI Cost Engineering, topic 21 (eval—optimisation must not break quality), and topic 07 (TCO inputs).
Learn
Latency fundamentals: TTFT, TPOT, and end-to-end SLA
Definition. TTFT (time to first token) measures delay until streaming begins—dominates perceived responsiveness. TPOT (time per output token) affects total completion time. End-to-end latency includes retrieval, orchestration, tools, and UI render—not model only.
Engagement use. Define performance budget per journey type: interactive assist vs batch summarisation. Target p50/p95/p99; document excluded segments (client VPN). Instrument each segment with trace IDs. SLA example: interactive p95 ≤3.0s to first token visible; batch ≤4h for 10K documents.
Pitfalls.
- Reporting vendor benchmark TTFT without your prompt size.
- Ignoring retrieval/rerank in SLA.
- Measuring lab only—not production percentile.
- Optimizing TTFT while total task time still 45s due to agent loops.
Worked example. Servicing assistant trace: retrieval 420ms, rerank 180ms, LLM TTFT 890ms, completion 1.1s (380 tokens)—end-to-end first useful content ~1.5s OK; agent variant adds 2 tool calls +2.4s—fails SLA without routing fix.
Concurrency, rate limits, and queuing
Definition. Concurrency = simultaneous in-flight requests; rate limits cap requests/tokens per minute by provider or platform. Queuing smooths bursts but adds wait time—must be visible to users.
Engagement use. Model peak concurrent users × requests/session. Request quota increases from cloud vendor early; implement client-side backoff and priority queues (interactive > batch). Load test at 1.5× expected peak.
Pitfalls.
- Hitting 429 errors in production without graceful degradation.
- Batch jobs consuming shared quota.
- Unbounded client retries multiplying cost.
- No circuit breaker on slow dependencies.
Worked example. 800 concurrent agents; avg 0.12 req/min each → 96 req/min mean, peaks 280 req/min—reserve 350 req/min token quota; batch queue separate worker pool 50 req/min cap.
Streaming and perceived performance
Definition. Streaming displays partial tokens as generated—improves perceived latency even when total time unchanged.
Engagement use. Stream to UI after minimal retrieval validation; show "searching sources…" states. Combine with progressive disclosure (citations first, narrative second) where UX allows.
Pitfalls.
- Streaming before safety checks complete on regulated content.
- UI flicker on partial JSON breaking parsers.
- Users interpret first tokens as final—disclaim draft state.
Worked example. Show citation cards at 1.2s while summary streams—users start verification early; perceived wait drops ~40% in usability tests despite same total time.
Model routing and right-sizing
Definition. Routing sends intents to appropriate models/tools—small/fast for classification and templated replies; large/accurate for complex reasoning. Fallbacks degrade gracefully when primary unavailable.
Engagement use. Build intent classifier or rules router with eval on routing accuracy. Track quality regression when cheap path misfires. Document fallback chain in runbook.
Pitfalls.
- Single frontier model for all traffic—3–10× overspend.
- Router errors silent—wrong model wrong task.
- Fallback to non-compliant region/model.
- No eval on routed cohorts separately.
Worked example. Tier-1 FAQ 8B regional model: 78% traffic, 92% golden pass; complex frontier 22% traffic. Blended cost $0.0042/request vs all-frontier $0.011—62% savings; quality within SLA on stratified eval.
Quantisation, distillation, and self-host economics
Definition. Quantisation lowers weight precision for speed/memory; distillation trains smaller models from larger teachers. Self-host shifts capex/opex—GPU amortization vs API elasticity.
Engagement use. Compare $/1M tokens self-host vs API at your volume breakeven. Include ops FTE, idle GPU, failover capacity. Run eval regression on quantised models before cutover.
Pitfalls.
- GPU sizing on peak only—empty GPU still costs.
- Ignoring engineering cost of serving stack.
- Quantisation hurting tool-call JSON reliability.
- Self-host without autoscaling—latency spikes or waste.
Worked example. At <80M tokens/month, managed API cheaper; above 120M, private 2×L4 cluster breaks even including 0.4 FTE ops—FinOps review quarterly as volume grows.
Caching layers: KV, prefix, and response caches
Definition. KV cache reuses attention states within a session/server (PagedAttention, vLLM). Prefix/prompt caching reuses identical prompt prefixes across requests (system prompt, static docs). Exact response cache returns identical answers for identical queries. Semantic response cache returns similar prior answers when embedding distance < threshold.
Engagement use. Eligibility matrix: public FAQ → semantic cache OK with corpus version key; personalized account data → no cache or short TTL user-scoped. Invalidate on corpus publish event. Monitor cache hit rate and stale answer rate on eval sample.
Pitfalls.
- Semantic cache without version key—stale policy answers.
- Caching pre-moderation unsafe content.
- Hit rate vanity metric while quality drops.
- Prefix cache on dynamic timestamps breaking hit rate.
Worked example. Policy assistant: 34% semantic hit rate on lookup intents; API cost −28%; stale-answer eval 0.3% post version-key fix (was 2.1%).
Batching strategies
Definition. Batching groups multiple inference requests for throughput—dynamic batching in servers, scheduled offline batches for bulk summarisation/embeddings.
Engagement use. Nightly embedding refresh batch 500 docs/job; interactive path no batch wait. Separate job queues with concurrency limits. Price batch SLA separately in business case (topic 07).
Pitfalls.
- Batch jobs during business hours on shared GPU.
- Large batches missing SLA on tail items.
- Re-embed entire corpus nightly when delta small.
Worked example. Claims batch: 8,400 summaries/night, batch size 64, completes 2h 10m by 06:00 SLA; cost $0.003/doc vs interactive $0.009/doc—67% cheaper due to batching + smaller quantised model.
Cost per request vs cost per successful task
Definition. Cost per request = spend / requests. Cost per successful task = spend / tasks meeting success criteria (eval pass, user marked resolved, ticket closed without reopen).
Engagement use. Define success with product and ops—link to topic 06 metrics. Instrument task_success flag on traces. FinOps dashboards show £/success by BU, intent, model route.
Pitfalls.
- Success = HTTP 200—ignores wrong answers.
- Averaging failed multi-step agent runs— hides loop waste.
- No attribution tags on requests—cannot charge back.
Worked example. Agent helpdesk: $12,400/week spend; 186K requests; $0.067/request looks fine—but only 41K successful tasks → $0.30/success—unacceptable vs $0.08 target; optimisation focuses on loop reduction not token discount.
Token budgets, caps, and anomaly detection
Definition. Token budgets cap spend per user/BU/day. Anomaly detection flags spikes (runaway agent, prompt injection loop, misconfigured job).
Engagement use. Soft cap warnings at 80%; hard cap with graceful message at 100%. Alert on 3σ daily spend vs trailing 14-day median. Runbooks for kill switch on feature flag.
Pitfalls.
- Caps without UX—users see opaque errors.
- No owner on alerts—pager fatigue ignored.
- Budget per request not per task—wrong incentives.
Worked example. BU marketing chatbot spikes 340% Sunday—alert traced to cron calling agent without max_steps; fix cap 12 steps saves $4.1K/week.
Showback, chargeback, and unit economics governance
Definition. Showback reports costs to consumers without financial transfer; chargeback bills internal BUs. Unit economics governance = regular forum reviewing £/success, latency, quality jointly.
Engagement use. Monthly FinOps review: top five expensive intents, optimisation backlog status, eval regression from last deploy. Align with AI FinOps and Commercial Design.
Pitfalls.
- Chargeback without quality metrics—race to cheap bad answers.
- Showback reports only totals—no actionable decomposition.
- FinOps separate from engineering prioritisation.
Worked example. Platform showback: Claims BU £48K/month, £0.19/success vs target £0.12—backlog item semantic cache approved; Legal BU low volume high cost per success due to 100% review—accepted quality trade-off documented.
Capacity planning and headroom
Definition. Capacity planning projects infrastructure needed for target volume, concurrency, and growth—with headroom for failover, eval, and peaks.
Engagement use. Spreadsheet: volume growth 3×/12mo; tokens/request; peak RPM; GPU tokens/sec; API TPM limits. Plan N+1 for critical path. Game-day load test quarterly.
Pitfalls.
- Planning average not p95 peak.
- Forgetting embedding/index growth with corpus.
- No failover capacity—regional outage = total outage.
- Eval jobs excluded—steal production quota unexpectedly.
Worked example. Year-end volume 3.2× normal for 72 hours—pre-negotiate TPM bump +120% for window; pre-warm 2 extra GPU replicas; batch deferred 48h—p95 latency stays <3.2s vs 6.1s in unplanned prior year.
Fallback, degradation, and graceful latency modes
Definition. Fallback switches to alternate model, cache, or static response when primary path fails or exceeds latency budget. Degradation reduces features (no tools, shorter context) to meet SLA under stress.
Engagement use. Define degradation tiers in runbook: Tier 0 full; Tier 1 drop rerank; Tier 2 cache-only FAQ; Tier 3 static "try again" message. Automate tier selection on saturation signals—never silent wrong answers.
Pitfalls.
- Fallback to non-compliant region/model.
- Degradation without user-visible notice.
- No eval on fallback paths—quality cliff hidden.
- Infinite retry loops on fallback chain.
Worked example. Primary route timeout 2.5s → fallback small model cached prefix only; user sees banner "Reduced detail mode—sources verified." p95 3.4s vs 7.1s hard fail; eval on fallback cohort −1.2% pass—acceptable vs outage.
Cold start, warm pools, and serverless inference
Definition. Cold start delay when scaling from zero GPU/serverless instances. Warm pools keep minimum instances ready—trade cost for latency stability.
Engagement use. Size warm pool for p95 concurrent not mean; pre-warm before known peaks (Monday AM, month-end batch). Serverless attractive at low volume; breaks at sustained peak—model breakeven in capacity worksheet.
Pitfalls.
- Autoscale min=0 for interactive user-facing—TTFT spikes.
- Warm pool sized for demo traffic only.
- Ignoring model load time on version deploy.
Worked example. Serverless cold start 4.2s TTFT unacceptable; move to min 2 warm replicas—idle cost +$3.8K/month but p95 TTFT 1.1s; FinOps accepts trade on £/success improvement from abandonment reduction.
Embedding, vector index, and retrieval FinOps
Definition. Retrieval costs include embedding API, re-embed on corpus change, vector storage, query compute, and reranker calls—often 20–40% of total at RAG scale.
Engagement use. Track embed cost per doc and query embed cost per request separately. Delta pipelines: embed only changed docs (hash diff). Tier storage: hot/warm index; archive stale collections.
Pitfalls.
- Full corpus re-embed nightly by cron habit.
- Vector DB cluster oversized "just in case."
- Ignoring reranker doubling model calls per request.
Worked example. 1.2M chunks; full re-embed $1,400/run daily → delta $90/run avg; annual save ~$470K; index storage tiering saves $2.1K/month on cold collections rarely queried.
Multi-tenant cost allocation
Definition. Shared platforms must allocate spend fairly across BUs, products, or clients—by tagged usage, successful tasks, or negotiated weights.
Engagement use. Mandatory tags at gateway; untagged traffic to overhead bucket reviewed weekly. Allocation disputes resolved in FinOps forum with product attendance.
Pitfalls.
- Shared platform cost invisible—BUs overuse.
- Allocation by request count not success—rewards failure loops.
- Missing tags on batch jobs—mystery spend.
Worked example. Platform £180K/month shared; 92% tagged to 6 BUs; 8% overhead drives tagging initiative; BU Claims 41% of successful tasks, charged £74K—aligns with benefits realisation story in topic 07.
Error budgets and latency SLOs (SRE for AI)
Borrow error budget thinking: if p95 latency SLO is 3s with 99% monthly attainment, you have ~7.2 hours of "bad latency" per month before breaching—spend it deliberately on risky deploys, not accidentally on agent loops.
Engagement use. Pair latency SLO with quality SLO (eval pass rate). Exhausting latency budget triggers FinOps freeze on new features until optimisation backlog clears—prevents simultaneous quality and cost crises.
Worked example. March deploy raised TTFT +400ms; burned 35% of monthly latency budget in first week—automatic rollback policy triggered; team finishes cache work before retry—users never saw compounding +1.2s degradation.
Frameworks and methods
Performance budget template
| Segment | Budget (p95) | Owner | Measure |
|---|---|---|---|
| Auth / gateway | 50ms | Platform | APM |
| Retrieval | 450ms | Search | Trace |
| Rerank | 200ms | Search | Trace |
| LLM TTFT | 900ms | ML Ops | Inference log |
| Tool call (each) | 600ms | Integration | Trace |
| UI render | 100ms | Frontend | RUM |
FinOps metrics hierarchy
- North Star: cost per successful task
- Drivers: tokens/task, requests/task, model route mix, cache hit rate, review rate
- Guardrails: p95 latency, eval pass rate, error rate
Caching eligibility matrix
| Data type | Exact cache | Semantic cache | TTL |
|---|---|---|---|
| Public FAQ | Yes | Yes | Corpus version |
| Policy general | Yes | Yes with version key | Until publish |
| Account-specific | No | No | — |
| Draft generation | No | No | — |
Optimisation backlog prioritisation (ICE)
Impact on £/success or latency p95; Confidence from profiling data; Effort engineering days. Rank weekly in FinOps forum.
Agent loop cost control
- max_steps hard cap
- Tool allowlist per intent
- Classifier early exit to FAQ path
- Reuse prior tool results in session
- Summarize history instead of full re-prompt
Link to commercial model
Cost per successful task feeds Commercial and Financial Modelling TCO and Business case sensitivity. Document breakeven volume for self-host vs API.
Real-world scenarios
Scenario A: Multi-tool customer service agent (telecom)
Context. 1.8M customer chats/month post-launch; spend $410K/month; p95 latency 8.4s; CSAT down −0.4.
Diagnosis. Avg 5.2 model calls per session; 38% sessions hit tool timeout retry; semantic cache 0% (not deployed); all traffic frontier model.
Interventions. Intent router → 71% to small model + template path; max_steps 8 → 5; semantic cache on top 120 FAQs with version key; tool timeout 3s → 2s with fallback message.
Results (12 weeks). Spend $410K → $178K (−57%); cost per successful resolution $0.24 → $0.09; p95 latency 8.4s → 3.1s; CSAT recovery −0.1 vs baseline; eval pass 91.2% → 90.8% (within tolerance).
Scenario B: Insurer batch + interactive mixed workload
Context. 340 adjusters interactive; nightly 6,200 claim summaries; shared API quota.
Problem. Batch starts 02:00 but runs long into 09:00—interactive p95 5.8s morning peak.
Interventions. Separate TPM allocation contract: interactive pool 70%, batch 30%; batch window moved 00:30–05:30 with dynamic batching; embed refresh delta-only (−82% embed tokens).
Results. Interactive p95 5.8s → 2.9s; batch cost $0.008/doc → $0.003/doc; total monthly $52K → $31K; no eval regression on golden set.
Scenario C: Internal legal research assistant (professional services — stretch)
Context. 900 lawyers; long prompts (~12K tokens) with matter context; strict confidentiality—no external semantic cache vendor.
FinOps approach. Private deploy; prefix caching on firm-wide system prompt + practice guides; KV tuning on vLLM; matter-specific chunks never cached cross-user.
Numbers. TTFT 2.1s → 1.3s; cost per research session $1.85 → $0.94; prefix hit rate 52%; incorrect answer rate unchanged on 150 matter goldens.
Shows safe caching under confidentiality—not all cache types allowed, still major savings.
Practice exercises
Primary exercise: Performance budget + FinOps dashboard spec (3 hours)
For a RAG HR assistant (25K employees, 8 queries/user/month, 25% multi-turn agent path):
-
Performance budget table — segments with p95 targets totaling ≤3.5s interactive.
-
Cost model — tokens/request assumptions; compute monthly API + review cost; cost per successful task with 88% success rate.
-
Three optimisations — caching, routing, batching—with estimated £/% impact and eval risk note.
-
Capacity plan — peak RPM and TPM request calculation with 1.5× headroom.
-
Alert rules — two anomaly alerts with thresholds.
Acceptance criteria: Multi-turn path identified as dominant cost driver; at least one optimisation includes cache eligibility rules; cost per success calculated not just per request.
Stretch exercise: Agent spend post-mortem (2 hours)
Given: overnight spend +$18K in 6 hours; 940K requests; stack traces show looping tool. Write post-mortem: root cause, immediate controls, structural fixes, updated FinOps policy (max_steps, budgets, alerts).
Acceptance criteria: Quantifies cost per successful vs failed loop; proposes eval gate before re-enable.
Reflection exercise: Self-host vs API breakeven (45 minutes)
Pick volume curve 20M–200M tokens/month; sketch breakeven including 0.5 FTE ops; state what metric triggers quarterly revisit.
Questions you should be able to answer
- What is p95 end-to-end latency for the primary user journey—and per segment?
- What is TTFT vs total completion time—and which dominates UX?
- What is cost per successful task—and how is success defined?
- What percentage of spend is API vs review vs infrastructure vs embed/index?
- Which intents routes use small vs large models—and what is eval pass by route?
- What is cache hit rate—and how do you detect stale cached answers?
- Where are batch and interactive workloads separated?
- What happens at rate limit 429—retries, queue, or degrade?
- What token budgets exist per BU/user—and who gets alerted at 80%?
- What is the optimisation backlog top item—and expected £/success impact?
- What capacity headroom exists for 3× peak day?
- What agent loop controls (max_steps, tool allowlist) are enforced?
- How does FinOps data feed the TCO model in topic 07?
- What eval regression ran after last cost optimisation deploy?
- What would break SLA or £/success at 3× volume without changes?
Negative cases
Model-only optimisation. Swap GPT variants; ignore retrieval slowness. Fix: segment budgets; profile traces.
Unsafe caching. Semantic cache without corpus version—compliance incident. Fix: eligibility matrix + invalidation.
Cost per request vanity. Cheap requests that fail tasks. Fix: success-defined unit economics.
Shared quota starvation. Batch kills interactive morning SLA. Fix: separate pools/schedules.
Runaway agent. No max_steps; overnight $18K spike. Fix: caps, alerts, kill switch.
Router neglect. Misfires to cheap model on hard intents. Fix: stratified eval by route.
FinOps without engineering backlog. Reports ignored. Fix: ICE-ranked backlog in monthly forum.
Chargeback races to bottom. BU cuts quality to cut bill. Fix: quality guardrails in showback.
Ignoring embed/index cost. Vector DB and refresh dominate at scale. Fix: delta pipelines; tiered storage.
Load test fantasy. Lab TTFT only. Fix: production percentile monitoring.
Optimisation without eval. Cache saves money; pass rate drops 4%. Fix: regression gate on deploy.
GPU over-provision idle. Fix: autoscaling; breakeven review vs API.
Observability and profiling for performance + FinOps
You cannot optimise what you cannot decompose. Instrument from pilot week one.
Trace structure. Propagate trace_id from API gateway through retrieval, model, tools, post-process. Span attributes: bu_id, use_case, intent, model_id, route_tier, cache_hit, tokens_in, tokens_out, success, failure_class.
Golden signals (SRE adapted for AI).
| Signal | Why it matters |
|---|---|
| Latency p95 per segment | Finds dominant wait |
| Error rate | Distinguish infra vs quality failures |
| Tokens per successful task | FinOps driver |
| Cost per successful task | Executive metric |
| Eval smoke pass rate | Quality guardrail on optimisations |
| Saturation (TPM/RPM/GPU util) | Capacity planning |
Profiling cadence. Weekly: top 10 expensive traces by cost; top 10 slowest p99 traces. Monthly: load test at 1.5× peak; FinOps review with product on success definition drift.
Tools (pattern-agnostic). OpenTelemetry-compatible tracing; cost allocation tags on cloud billing; log aggregation with trace correlation; eval pipeline posting regression IDs to same dashboard.
TTFT optimisation playbook (ordered)
- Reduce prompt size — compress system prompt, trim few-shots, summarise history.
- Prefix caching — stabilize system + corpus prefix blocks.
- Retrieval budget — fewer chunks with better rerank vs many chunks.
- Model route — smaller model for draft/classify; frontier only on escalate.
- Parallelize — retrieval + user context fetch concurrent.
- Regional placement — inference close to user and corpus.
- Quantisation / faster SKU — after eval pass.
- Streaming UX — perceived win while continuing 1–6.
Do not skip straight to 7 while retrieval adds 1.2s—profile first.
Capacity planning worksheet (template)
Inputs (fill for your engagement):
- Monthly requests: ____ ; growth 3× scenario: ____
- Tokens/request p50: ____ ; p95: ____
- Concurrent users peak: ____
- Interactive share vs batch: ____ / ____ %
- Target p95 latency: ____ s
- Model mix: ____ % small / ____ % large
Derived:
- Peak TPM = concurrent × (tokens/request) × (60 / avg session duration seconds) × safety factor 1.3
- GPU tokens/sec needed = peak TPM / 60 × (large model share)
- API quota request = peak RPM × 1.5 headroom
Example (filled). 280 concurrent; 3,900 tokens p50; 12 min avg session; 75% small model:
- Peak TPM ≈ 280 × 3900 × (60/720) × 1.3 ≈ 118K TPM interactive
- Batch add 40K TPM off-peak pool
- Request 200K TPM committed use with burst to 260K
Document assumptions in FinOps appendix—finance and ops debate numbers, not vibes.
FinOps operating rhythm (monthly)
| Week | Activity | Output |
|---|---|---|
| 1 | Billing close + tag cleanup | Allocated spend by BU/intent |
| 2 | Optimisation backlog grooming (ICE) | Ranked engineering tickets |
| 3 | Eval regression review post-deploys | Quality sign-off |
| 4 | Steering slice: £/success, p95, adoption | Continue/stop optimisations |
Escalation triggers (examples):
- Cost per success >20% over plan 2 consecutive months
- p95 latency >15% over SLA 48 hours
- Cache stale-answer eval >1%
- Anomaly spend >3σ daily
Scenario D: Global bank — multilingual translation + summarisation
Context. APAC + EU trading support; 6 languages; 42K messages/day; mix short classify (80%) and long summarize (20%).
Challenge. TTFT spiked to 6.2s when EU users hit US endpoint; translation step added 900ms serial.
Interventions. Regional inference endpoints; parallel language detect + retrieval; classify path on 3B model (TTFT 220ms); summarize path frontier with prefix cache on desk procedures; semantic cache only for EN internal FAQs.
Results. Blended p95 6.2s → 2.7s; cost per successful task $0.14 → $0.06; translation errors −18% (better locale routing); FinOps dashboard by region prevents repeat US-routing misconfig.
Demonstrates latency + FinOps are regional architecture problems—not single global model choice.
Scenario E: Insurance fraud investigation agent (high loop risk)
Context. SIU team; 45 investigators; agent prototype chains search → graph query → summarizer → email drafter; pilot spend $38K/week for 120 active users—unsustainable.
FinOps diagnosis. Average 7.8 LLM calls per investigation session; 43% sessions exceed max_steps without resolution; graph tool returns large JSON inflating tokens (12K avg prompt).
Controls. Compress tool JSON schema; max_steps 6; summarizer only on pinned evidence subset; session-level token budget 40K with hard stop; route classify to 8B model (92% accuracy on intent eval).
Outcome. Spend $38K → $11K/week (−71%); cost per successful investigation $18 → $5.40; median session time 28 min → 19 min; investigator satisfaction +0.6 on 5-pt scale—quality up as noise reduced.
Lesson. Agent FinOps is workflow design first—step caps and schema diet beat haggling over API list price.
Agent and tool FinOps patterns (deep dive)
Multi-step agents dominate runaway spend. Treat each pattern explicitly in architecture and FinOps policy.
Pattern: ReAct loop. Plan → act → observe cycles burn tokens. Controls: max_steps, duplicate action detection, tool result truncation, progressive summarization of observations.
Pattern: Router + specialists. Cheap router; specialists only when needed. FinOps metric: router accuracy × specialist cost avoided.
Pattern: Human approval gate. Expensive actions pause for human—counts as successful task only after approval, preventing false success metrics.
Pattern: Tool fan-out. Parallel tools reduce latency but multiply cost if uncapped—limit parallelism to 3 with priority queue.
Pattern: Self-reflection / critique loops. Second pass "check your work" doubles cost—require eval proof that +X% quality worth 2× tokens.
Token budget policy example (per session):
| Tier | Max tokens | Max steps | Models allowed |
|---|---|---|---|
| Standard user | 24K | 5 | Small + medium |
| Power user (approved) | 48K | 8 | + frontier |
| Batch job | 120K | 12 | Medium + batch SKU |
Log budget exhaustion rate—high rate indicates product UX forcing retries or unclear tasks.
Tool output hygiene checklist:
- JSON fields limited to schema—no raw dump of entire CRM record
- Pagination on list tools—max 50 rows default
- Summarize tool output before re-injection into prompt
- Cache idempotent read tools within session (60s TTL)
- Idempotency keys on write tools—prevent duplicate side effects and duplicate LLM recovery loops
Reserved capacity, committed use, and cloud FinOps discounts
Cloud providers offer committed use discounts (CUD), reserved throughput, and provisioned capacity—trade flexibility for 15–45% savings at predictable volume.
Engagement use. When FinOps forecast shows ≥12 months above breakeven volume, model commit vs on-demand in TCO (topic 07). Include utilization risk: unused commit is sunk cost. Review quarterly; 80% utilization minimum before new commit.
Pitfalls.
- Commit based on pilot volume—massive overcommit.
- Commit on wrong region/model family after routing change.
- Ignoring early termination fees in downside scenario.
- Separate teams buy duplicate commits on same account.
Worked example. 200K TPM commit saves 28% vs on-demand ($47K/month → $33.8K/month); breakeven vs on-demand at ≥142K TPM utilization—FinOps dashboard alerts at 130K sustained to trigger scale-down review before renewal.
API enterprise agreements. Negotiate tiered $/1M tokens with annual true-up; include model deprecation migration support clause—commercial and FinOps jointly own renewal 90 days before expiry.
Eval and red-team compute as FinOps line items
Evaluation is recurring cost, not one-time project expense. Budget: golden set regression (weekly), red-team campaigns (quarterly), embed eval on corpus publish (per release), human rater panels (sampled).
Worked example. 2000 golden cases × 4K tokens × $2.50/1M × 52 weekly runs ≈ $1,040/year API— trivial alone—but add human review of failures 15% × 200 cases × 8 min × £30/hour ≈ £12K/year plus 2 days/quarter red-team vendor £40K/year → ~£55K/year eval opex must appear in TCO or "free quality" illusion breaks at scale.
Include eval compute in capacity plan—regression Sunday window 30K TPM reserved so production interactive traffic unaffected.
FinOps maturity model (self-assessment)
| Level | Characteristics | Typical gap |
|---|---|---|
| 0 Ad hoc | Monthly invoice surprise | No tags, no cost per success |
| 1 Visible | Showback by BU | No optimisation backlog |
| 2 Managed | Budgets, alerts, ICE backlog | Quality not in FinOps forum |
| 3 Optimised | Cost per success + eval gates | Chargeback + unit pricing |
| 4 Strategic | Portfolio trade-offs with finance | Benefits realisation closed loop |
Most enterprises reach Level 1 after first production incident; target Level 2 before scale funding, Level 3 before internal chargeback. Honest self-assessment prevents selling consumption pricing when client is still Level 0—recipe for political fallout when bill arrives.
Pair maturity assessment with Model FinOps roadmap skills planning: Level 2 requires tracing and tagging engineering; Level 3 requires product-defined success metrics and eval regression discipline—not finance tooling alone.
Publish a weekly FinOps digest to product and engineering leads: top three expensive intents, one optimisation shipped, one eval regression result—keeps cost engineering visible without waiting for monthly steering.
Optimisation vs quality decision log (template)
| Change | £/success Δ | p95 Δ | Eval Δ | Ship? | Date |
|---|---|---|---|---|---|
| Semantic cache v1 | −22% | −8% | −0.2% | Yes | 2026-03 |
| Aggressive quant | −15% | −12% | −2.1% | No | 2026-04 |
| max_steps 8→5 | −31% | −18% | −0.4% | Yes | 2026-05 |
Require row for every production FinOps change—prevents "optimisation heroics" that erode quality silently.
Integration with commercial model (topic 07)
FinOps metrics feed and validate the business case:
- Plan: TCO run-rate uses cost per success × expected successful tasks/month.
- Track: Monthly variance decomposition updates benefits and run-rate forecast.
- Gate: Phase 2 funding requires ≤£X/success and ≥Y% eval pass for 90 days.
- Price: Internal chargeback rates reset quarterly from actual marginal cost, not initial guess.
When FinOps and commercial teams use different success definitions, executives see conflicting stories—align in the operating forum described above.
Related playbook content
- Model FinOps roadmap — structured learning path for cost engineering
- Model FinOps and AI Cost Engineering — deep article companion
- AI FinOps and Commercial Design — chargeback, pricing, commercial operating model
- LLM Caching Ultimate Guide — KV, prefix, semantic caching mechanisms
- Business case and prioritisation — feed FinOps numbers into funding cases
- Commercial and Financial Modelling — TCO and sensitivity inputs
- Product Management — success metrics define cost per successful task
- AI Evaluation and Quality Assurance — quality gates for optimisations
- Generative AI and LLM Fundamentals — tokens, routing, quantisation basics
- Agentic AI and Workflow Automation — agent loop cost patterns
- Operations and Production Support — runbooks, incidents, SLOs
- 8D Framework — Operate gate evidence
- How to use this Learning Map — study loop and artefact standards
Practice checklist
- I can state p95 end-to-end latency and top segment contributor
- I calculate cost per successful task—not requests alone
- Caching rules include corpus/version keys for regulated content
- Batch and interactive workloads have separate capacity
- Agent loops have max_steps and tool governance
- Optimisation backlog links to eval regression requirement
- FinOps metrics connect to business case TCO (topic 07)
- I completed the primary exercise and filed artefacts in my pattern library
Discussion
Comments
Share feedback or questions about this page. No account required.
Loading comments…