LLM Cost Tracking and FinOps Across Azure, AWS, and Google Cloud

Generative AI cost management is more complicated than ordinary cloud cost management. A conventional application might be charged according to CPU hours, memory, storage, and network usage. An LLM application can accumulate costs through input tokens, output tokens, cached tokens, reasoning tokens, embeddings, retrieval, reranking, guardrails, model evaluation, agent tool calls, retries, observability, and the infrastructure supporting the application.

The main FinOps challenge is therefore not simply:

How can we use the cheapest model?

The correct question is:

What is the lowest sustainable cost at which the application can meet its quality, latency, reliability, security, and business-outcome requirements?

A cheap model that produces inaccurate answers, requires repeated calls, or causes users to abandon a workflow may be more expensive than a premium model that completes the task correctly on the first attempt.

As of July 2026, Azure, AWS, and Google Cloud all provide several consumption models for generative AI, including pay-per-token inference, discounted batch processing, prompt or context caching, premium-priority processing, and reserved or provisioned throughput. The terminology differs, but the underlying FinOps decision is similar: use shared capacity for variable demand, reserved capacity for predictable baseline demand, and discounted asynchronous capacity for work that does not require immediate responses.

1. What LLM FinOps Means​

LLM FinOps is the practice of creating financial accountability for generative AI systems while allowing engineering, product, finance, security, and business teams to make informed trade-offs between:

• Cost
• Response quality
• Latency
• Throughput
• Reliability
• Security
• Compliance
• Business value

Traditional FinOps metrics such as monthly cloud expenditure remain useful, but they are insufficient for AI. An organisation must connect infrastructure spending with measurable outcomes.

The FinOps Foundation describes unit economics as the connection between technology spending and the value created by that spending. For AI applications, useful units can include cost per request, cost per conversation, cost per agent action, cost per case resolved, or cost per successful business outcome.

For example, a customer-service chatbot should not be judged only by its monthly token bill. It should also be measured through:

• Cost per customer conversation
• Cost per successfully resolved case
• Percentage of conversations resolved without a human agent
• Average handling-time reduction
• Customer satisfaction
• Hallucination or error rate
• Escalation rate
• Revenue protected or generated
• Compliance incidents avoided

A system costing $40,000 per month that resolves 500,000 customer issues may be commercially superior to a system costing $10,000 that resolves only 50,000 issues and sends the remainder to human agents.

2. The Full Cost Anatomy of an LLM Application​

An LLM FinOps programme must track more than the model invoice.

2.1 Model inference costs​

For managed foundation-model APIs, the primary costs usually include:

Input tokens​

These include:

• System instructions
• Developer prompts
• User messages
• Conversation history
• Retrieved documents
• Few-shot examples
• Tool definitions
• Structured-output schemas
• Images, audio, or video converted into model input units

Long system prompts and unnecessarily large retrieved contexts can become major cost drivers.

Output tokens​

Output tokens are frequently more expensive than input tokens. Long answers, verbose reasoning, unnecessary explanations, and repeated structured data can therefore materially increase spending.

Cached input tokens​

Azure OpenAI, Amazon Bedrock, and Vertex AI support mechanisms that reduce the cost and latency of repeatedly processing identical or reusable context.

The savings depend on the model, deployment type, cache policy, cache-write cost, and cache-read price. Prompt design therefore becomes part of financial engineering.

Reasoning tokens​

Some reasoning models use hidden or visible intermediate reasoning tokens. These can produce better results for complex tasks but may increase:

• Total token consumption
• Response time
• Cost per request
• Variability between requests

Reasoning models should not be used indiscriminately for classification, extraction, formatting, or simple question answering.

Multimodal consumption​

Image, document, audio, and video models can be priced using tokens, images, seconds, pages, or modality-specific units.

A document-processing solution may therefore have separate costs for:

• OCR or document extraction
• Image tokens
• LLM reasoning
• Embeddings
• Storage
• Search
• Output generation

2.2 Agentic workflow costs​

Agentic systems can be significantly more expensive than single-call applications because one user request may trigger:

1. A planner call
2. A retrieval call
3. Several tool executions
4. A critic or verifier call
5. A response-generation call
6. A retry following a tool failure
7. A safety-validation call

The cost of an agent run can be represented as:

C_agent_run = Σ_i (C_model,i + C_retrieval,i + C_tool,i + C_validation,i)
            + C_shared_infrastructure

Agent FinOps must therefore track the complete trace, not only the final model response.

Important controls include:

• Maximum number of agent steps
• Maximum tokens per run
• Maximum monetary cost per run
• Maximum retry count
• Tool-call timeout
• Duplicate tool-call detection
• Circular-agent-loop detection
• Human approval before expensive actions
• A fallback response when the cost budget is exhausted

2.3 Retrieval and RAG costs​

A production RAG system may incur costs from:

• Document ingestion
• Document parsing
• OCR
• Chunking
• Embedding generation
• Vector storage
• Vector search
• Hybrid keyword search
• Graph databases
• Reranking models
• LLM generation
• Re-embedding after document changes
• Backup and replication
• Data transfer

Retrieving more documents does not always improve quality. Sending 20 chunks to a model when only four are relevant increases cost, latency, and the risk of confusing the model.

A better approach is:

1. Apply tenant and metadata filters.
2. Run hybrid retrieval.
3. Retrieve a moderately sized candidate set.
4. Rerank the candidates.
5. Send only the most relevant evidence.
6. Dynamically increase context only when confidence is low.

2.4 Safety, governance, and observability costs​

A financially complete model must also include:

• Content moderation
• Prompt-injection detection
• PII detection and redaction
• Guardrails
• Groundedness checks
• Hallucination evaluation
• Automated reasoning checks
• Human review
• Logging
• Distributed tracing
• Metrics storage
• Evaluation datasets
• Red-team testing
• Security scanning
• Audit retention

For example, Amazon Bedrock Guardrails charges separately for several safeguards, including content filters, sensitive-information filters, contextual-grounding checks, and automated-reasoning checks. These charges should be included in the cost of the use case rather than treated as an unexpected platform overhead.

2.5 Self-hosted model costs​

When an organisation hosts an open-weight model on Azure Machine Learning, Amazon SageMaker, Vertex AI, Kubernetes, or virtual machines, costs usually include:

• GPU or accelerator hours
• CPU and memory
• Model storage
• Container registry
• Load balancers
• Persistent disks
• Network traffic
• Kubernetes control-plane costs
• Monitoring
• Redundant replicas
• Disaster recovery
• Model-serving software
• Engineering and operational support
• Security patching
• Idle capacity
• Model optimisation

The cost formula is closer to:

C_self_hosted = C_GPU + C_CPU/RAM + C_storage + C_network
              + C_platform + C_operations

The most important metric is not merely GPU price per hour. It is:

Cost per generated token = Total serving cost / Useful tokens generated

A powerful GPU operating at 15% utilisation can be more expensive per token than a premium managed API.

3. Establishing a Request-Level LLM Cost Ledger​

Cloud billing reports are necessary, but they normally do not contain enough application context to explain why costs occurred.

Every model request should generate an internal usage record containing fields such as:

Do not place unrestricted personal information into cost tags. Use pseudonymous or hashed identifiers where user-level attribution is necessary.

3.1 Basic request-cost formula​

For a text model:

C_request = (T_i / 1,000,000) × P_i
          + (T_c / 1,000,000) × P_c
          + (T_o / 1,000,000) × P_o
          + C_tools + C_retrieval + C_guardrails

Where:

• T_i = uncached input tokens
• T_c = cached input tokens
• T_o = output tokens
• P_i = input-token price per million
• P_c = cached-token price per million
• P_o = output-token price per million

For provisioned capacity:

C_request = Allocated provisioned cost / Number of completed requests
          + C_variable_services

3.2 Fully loaded cost per successful outcome​

The strongest FinOps measure is:

Cost per successful outcome =
  (Model cost + retrieval cost + platform cost + operations cost)
  / Number of successful outcomes

A request should count as successful only when it achieves the intended business result.

For example:

• Correct answer delivered
• Case resolved
• Document successfully processed
• Lead qualified
• Code accepted
• Transaction completed
• Employee task completed
• Escalation correctly initiated

This prevents teams from reporting an artificially low “cost per API call” while ignoring failed or low-quality results.

4. Cost, Latency, and Performance Must Be Optimised Together​

An LLM platform should maintain at least five service objectives:

Optimisation should be rejected when it violates one of the mandatory quality or reliability thresholds.

For example:

• Reducing retrieved context may lower cost but reduce groundedness.
• Moving to a smaller model may lower latency but increase classification errors.
• Increasing batch size may improve throughput but delay individual responses.
• Scaling to zero may eliminate idle costs but introduce cold-start latency.
• Provisioning excessive capacity may improve p99 latency but create poor utilisation.

The objective is to find the Pareto-efficient frontier: architectures where cost cannot be reduced further without materially damaging another required outcome.

5. Cross-Cloud Consumption Models​

Azure documents that Global Standard is a common starting point for general workloads, while provisioned deployments are intended for predictable throughput and lower latency variance. AWS exposes Reserved, Priority, Standard, and Flex service tiers. Google Cloud similarly distinguishes Provisioned Throughput, Standard, Priority, Flex, and Batch consumption.

6. Microsoft Azure LLM Cost Tracking and FinOps​

6.1 Azure OpenAI and Microsoft Foundry consumption options​

Azure offers several deployment patterns.

Global Standard​

Global Standard uses pay-per-token billing and routes requests across Azure infrastructure.

It is appropriate for:

• Variable traffic
• Prototypes
• General production applications
• Workloads that can tolerate some latency variation
• Applications requiring broad model availability

It generally offers high quota and availability, but sustained high-volume workloads may experience more latency variation than provisioned deployments.

Regional or Data Zone Standard​

These options are useful when processing-location requirements limit where inference may occur.

Data residency requirements can influence both cost and performance. A globally routed deployment may have broader capacity, while a regional deployment may offer stronger location control but lower available quota.

Provisioned Throughput Units​

Provisioned Throughput Units, or PTUs, reserve model-processing capacity.

Azure charges for the deployed PTU quantity, regardless of whether the capacity is fully used. Azure supports hourly provisioned billing and reservations for sustained workloads. Reservations can be purchased for fixed PTU quantities and scoped across qualifying deployments.

Provisioned throughput is suitable when:

• Traffic is predictable
• Throughput is consistently high
• Latency variance must be controlled
• Capacity availability is business-critical
• Token volume justifies a fixed commitment

The principal financial risk is low utilisation.

Track:

PTU utilisation = Consumed throughput / Purchased throughput

Low utilisation means the organisation is paying for idle capacity. Sustained saturation can create queues or require overflow capacity.

A practical architecture is:

• Provisioned capacity for predictable baseline traffic
• Standard or priority capacity for bursts
• Batch capacity for offline work

Do not reserve capacity based only on the highest observed peak. Measure a representative traffic percentile and permit bursts to use pay-as-you-go capacity.

Azure Batch​

Azure Batch is designed for asynchronous, high-volume inference. Microsoft describes a 24-hour target turnaround and a price approximately 50% lower than Global Standard, with separate batch quota that does not disrupt online inference quota.

Good batch workloads include:

• Overnight document classification
• Evaluation-dataset execution
• Synthetic-data generation
• Embedding preparation
• Report generation
• Product-description generation
• Historical conversation summarisation
• Compliance review backlogs

Do not use expensive online endpoints for work that users do not need immediately.

6.2 Azure prompt caching​

Azure prompt caching can reduce both cost and latency when requests contain a sufficiently long identical prefix. Azure currently documents a minimum 1,024-token prompt and requires the first 1,024 tokens to be identical for cache eligibility. Cache usage is exposed through cached-token fields in the response.

Place the following near the beginning of the prompt:

• Stable system instructions
• Tool definitions
• Output schemas
• Reusable few-shot examples
• Common policy documents
• Stable reference context

Place variable elements near the end:

• User question
• Current date
• Session-specific data
• Retrieved evidence
• Transaction details

Avoid inserting variable identifiers, timestamps, random values, or dynamically reordered tools at the beginning. A small prefix change can reduce the cache-hit rate.

Track:

Cache hit rate = Cached input tokens / Cache-eligible input tokens

Also track cache-write costs for models where writes are charged. Azure recommends stable prefixes and a steady request pattern so that cache writes are followed by sufficient cache reads.

6.3 Azure cost allocation and monitoring​

Use Azure Cost Management to establish:

• Subscription and resource-group budgets
• Cost alerts
• Forecast alerts
• Anomaly alerts
• Cost exports
• Amortised reservation reporting

Azure Cost Management supports budget alerts and cost-anomaly alerts. Scheduled cost exports can write detailed actual or amortised usage data to Azure Storage for downstream analysis.

Recommended Azure tagging dimensions include:

• business_unit
• cost_center
• product
• application
• environment
• owner
• data_classification
• model_purpose

Cloud billing data should then be joined with application-level token telemetry using deployment, resource, model, date, region, and environment.

6.4 Self-hosted LLMs on Azure​

For open-weight models hosted through Azure Machine Learning:

• Use autoscaling for online endpoints.
• Scale using request latency, queue depth, GPU utilisation, or schedules.
• Maintain enough warm replicas to satisfy latency objectives.
• Use scale-to-zero for non-interactive environments where cold starts are acceptable.
• Place data, models, and endpoints in the same region where possible.
• Remove failed deployments that continue consuming compute.
• Use separate production and experimentation capacity.

Azure Machine Learning integrates online-endpoint autoscaling with Azure Monitor and supports rules based on deployment or endpoint metrics, including request latency. Azure also recommends reducing unused compute and setting minimum cluster nodes to zero for workloads that do not require continuously available capacity.

7. AWS LLM Cost Tracking and FinOps​

7.1 Amazon Bedrock service tiers​

Amazon Bedrock provides several inference tiers.

Flex​

Flex is intended for cost-sensitive workloads that can tolerate slower processing or higher throttling risk.

Examples include:

• Development
• Testing
• Model evaluations
• Background summarisation
• Low-priority agent tasks
• Non-urgent content generation

Standard​

Standard is the general-purpose option for ordinary production traffic.

Priority​

Priority gives requests preferential processing and is intended for high-value, customer-facing, or mission-critical interactions where faster and more reliable responses justify a price premium.

Reserved​

Reserved capacity is intended for predictable, mission-critical workloads. AWS permits organisations to reserve token-per-minute capacity, and excess traffic can overflow to Standard capacity rather than simply failing when the reserved allocation is exceeded.

A useful request-level policy is:

This prevents every request from using the highest-cost tier.

7.2 Amazon Bedrock batch processing​

AWS states that supported Bedrock models can use batch inference at approximately 50% lower pricing than ordinary on-demand inference. Batch workloads are asynchronous and write their results to Amazon S3.

Batch is appropriate for:

• Large document collections
• Evaluation runs
• Content enrichment
• Bulk classification
• Synthetic examples
• Data migration
• Historical conversation processing

Some interactive features, such as tool-calling exchanges, may not be available in batch workflows, so the workflow must be redesigned as independent records.

7.3 Bedrock prompt caching​

Amazon Bedrock supports model-dependent prompt-caching behaviour, including explicit cache checkpoints for supported models.

Cache policies can differ by model:

• Minimum prefix length
• Number of cache checkpoints
• Cache TTL
• Cache-write price
• Cache-read discount
• Supported prompt fields

AWS documentation notes that cache writes may cost more than ordinary input for some models, while cache reads receive a discount. A cache is financially beneficial only when a written prefix is reused enough times during its TTL.

A simplified break-even formula is:

N_break-even = (C_cache_write − C_normal_input)
             / (C_normal_input − C_cache_read)

Where N_break-even is the number of cache reads needed after a write.

Use explicit cache checkpoints around:

• Large tool definitions
• Common knowledge bases
• Policy documents
• Code repositories
• Agent instructions
• Repeated conversation context

Do not cache highly personalised or rapidly changing information unless the cache isolation and expiry behaviour meets security requirements.

7.4 AWS cost allocation​

Amazon Bedrock Application Inference Profiles allow costs to be attributed by application, team, or workload. Their tags can flow to AWS Cost Explorer and Cost and Usage Reports. AWS notes that this gives aggregated billing attribution, while per-prompt details require request metadata and model invocation logs.

A recommended AWS structure is:

Use:

• Cost allocation tags
• Cost Categories
• AWS Cost Explorer
• Cost and Usage Report or Data Exports
• AWS Budgets
• Cost Anomaly Detection
• CloudWatch metrics
• Model invocation logging

AWS recommends workload-level budgets, cost reporting, and anomaly-detection mechanisms so that unexpected usage is identified quickly.

7.5 Self-hosted models on SageMaker​

Amazon SageMaker offers several serving patterns:

AWS states that asynchronous inference can scale to zero and that serverless inference prevents payment for continuously idle instances. SageMaker Savings Plans can reduce eligible committed usage costs for steady workloads.

For open-source LLMs:

• Benchmark GPU and accelerator types.
• Compare GPU memory requirements after quantisation.
• Use tensor parallelism only when required.
• Use continuous batching.
• Use paged key-value caches.
• Monitor tokens generated per GPU-second.
• Use asynchronous queues to smooth bursts.
• Consider Inferentia where the model and runtime are supported.
• Consolidate compatible small models where isolation requirements permit.
• Scale asynchronous endpoints to zero.

8. Google Cloud LLM Cost Tracking and FinOps​

8.1 Vertex AI consumption options​

Google Cloud provides:

• Standard PayGo
• Priority PayGo
• Flex PayGo
• Batch inference
• Provisioned Throughput

Google recommends matching each option to workload criticality. Flex is suitable for latency-tolerant work, while Priority provides increased reliability for important workloads. Provisioned Throughput is intended for predictable requirements where guaranteed throughput is necessary.

Provisioned Throughput and GSUs​

Google Cloud measures provisioned generative-AI capacity using Generative AI Scale Units, or GSUs.

Required throughput depends on:

• Model
• Model version
• Region
• Input size
• Output size
• Modality
• Requests per second
• Model-specific token burndown rates

Unused provisioned throughput does not accumulate or carry forward. Provisioning is tied to a project, region, model, and model version, so FinOps teams must avoid assuming that unused capacity can automatically cover unrelated deployments.

Google recommends a hybrid model:

• Cover an appropriate baseline percentile with Provisioned Throughput.
• Use Standard or Priority PayGo for bursts.
• Avoid provisioning exclusively for the highest peak.
• Regularly monitor GSU utilisation.

Google also documents lower prices for longer commitments and advises moving latency-tolerant activity to Flex or Batch, which can offer substantial discounts compared with ordinary online processing.

8.2 Context caching​

Vertex AI context caching reduces repeated processing of common input and can improve both latency and cost.

Useful cached content includes:

• Large source documents
• Video or audio context
• Common system instructions
• Product catalogues
• Policy manuals
• Shared code or schemas
• Repeated examples

Google Cloud supports implicit and explicit caching for supported models. Cached tokens can receive discounted pricing, while explicit caches may also incur storage charges based on cached tokens and retention duration.

Calculate whether the cache is worthwhile:

Net cache value = Avoided token processing cost
                − Cache storage and write cost

A long-lived cache is not automatically economical. If the content is accessed only once or twice, cache storage can outweigh the benefit.

8.3 Google Cloud cost allocation and billing export​

Google Cloud Billing can export standard, detailed, pricing, commitment, and FOCUS-format data to BigQuery.

The export can contain:

• Billing account
• Project
• Service
• SKU
• Region
• Cost
• Credits
• Currency
• Labels
• Tags
• Resource-level information

Google recommends the detailed export when resource-level analysis is required and supports a FOCUS-formatted export for standardised FinOps analysis.

A typical architecture is:

Join the billing data with application telemetry using:

• Project ID
• Region
• Model
• Endpoint
• SKU
• Date and hour
• Labels
• Application ID

Billing exports can be delayed and are not designed as the only real-time cost-control mechanism. Use request-level estimates for immediate controls and reconcile them later against authoritative billing data.

8.4 Vertex AI monitoring and self-hosted models​

Vertex AI and Cloud Monitoring expose metrics that can help teams analyse request volume, token throughput, latency, errors, and capacity constraints.

For custom models:

• Autoscale according to CPU or GPU utilisation.
• Track request queue depth and latency.
• Use scale-to-zero for endpoints with long idle periods when cold-start latency is acceptable.
• Use Spot VMs for fault-tolerant or asynchronous processing.
• Apply committed-use discounts to predictable compute.
• Separate batch and interactive capacity.
• Ensure quotas can support maximum replica counts.

Vertex AI supports endpoint autoscaling and, for qualifying endpoint types, scale-to-zero. Google Cloud also supports Spot VMs for fault-tolerant inference workloads, with the trade-off that instances can be preempted.

9. The Most Effective Cross-Cloud Optimisation Strategies​

9.1 Route requests by complexity​

Do not send every request to the largest model.

Create complexity classes such as:

Tier 1: Deterministic processing​

Use rules or conventional code for:

• Date formatting
• Exact calculations
• Database lookups
• Validation
• Template filling
• Simple routing

Tier 2: Small or efficient model​

Use a smaller model for:

• Classification
• Intent detection
• Entity extraction
• Summarisation
• Query rewriting
• Simple question answering
• Moderation pre-checks

Tier 3: General premium model​

Use for:

• Customer-facing responses
• Complex RAG
• Multi-document synthesis
• High-value content generation
• Tool selection

Tier 4: Advanced reasoning model​

Use only for:

• Complex planning
• Difficult coding
• Multi-step analysis
• High-risk decisions requiring stronger reasoning
• Cases where the smaller model reports low confidence

The router itself should be inexpensive. A rules-first router followed by a small classifier is often better than asking a premium model to choose which premium model should answer.

9.2 Use a model cascade​

A model cascade attempts the cheapest qualified method first.

Track:

• Percentage resolved at each layer
• Accuracy at each layer
• Escalation rate
• False-confidence rate
• Cost avoided
• Added routing latency

The cascade should be evaluated on end-to-end quality, not simply the number of requests handled by the cheapest model.

9.3 Reduce output before input​

Output tokens are commonly more expensive and also increase end-to-end latency because they must be generated sequentially.

Controls include:

• Explicit response-length requirements
• Structured outputs
• Short default answers with optional expansion
• Stop sequences
• Maximum output tokens
• Avoiding repeated explanations
• Returning identifiers instead of repeated records
• Performing formatting in code rather than the LLM

Do not ask the LLM to generate large JSON structures when the same structure can be assembled programmatically from extracted fields.

9.4 Optimise the prompt prefix​

For prompt caching:

This increases the probability that Azure, AWS, or Google Cloud can reuse the stable prefix.

Version stable prompts deliberately. Unnecessary wording changes can invalidate caches and make cost comparisons difficult.

9.5 Control conversation history​

Sending an entire conversation on every turn creates steadily increasing cost and latency.

Use:

• Recent-message windows
• Structured user state
• Periodic conversation summaries
• Separate long-term memory
• Retrieval over older messages
• Field-level state rather than raw transcripts

For example, retain:

{
  "customer_intent": "mortgage_payment_support",
  "verified": true,
  "preferred_language": "English",
  "open_case": "CASE-10492",
  "summary": "Customer requested a temporary payment arrangement."
}

This is cheaper than repeatedly transmitting 30 previous chat turns.

9.6 Optimise RAG context dynamically​

Use a context budget rather than a fixed number of chunks.

A dynamic policy might be:

Remove:

• Duplicate chunks
• Navigation text
• Headers repeated across documents
• Boilerplate legal notices
• Irrelevant tables
• Low-confidence retrievals

Use metadata filtering before vector search whenever possible.

9.7 Cache at several levels​

A mature application can use four different caches.

Exact response cache​

Keyed by a deterministic representation of the request.

Best for:

• FAQ responses
• Repeated classifications
• Static policy questions

Semantic response cache​

Uses embedding similarity to find a sufficiently similar previously answered question.

It must have:

• Similarity threshold
• Tenant isolation
• Data-freshness checks
• Policy-version checks
• Model-version metadata
• Security filtering

Prompt or context cache​

Provided by the model platform to reuse the computation associated with repeated prefixes.

Retrieval cache​

Stores retrieval results for repeated search queries or document sets.

Do not cache responses for rapidly changing account balances, medical decisions, market prices, or permission-dependent data without strong validation.

9.8 Separate interactive and asynchronous workloads​

A common mistake is routing everything through the same online endpoint.

Use interactive capacity for:

• User conversations
• Real-time copilots
• Transaction support
• Interactive search

Use batch, flex, spot, or asynchronous capacity for:

• Evaluations
• Nightly summarisation
• Document ingestion
• Data enrichment
• Synthetic data
• Reporting
• Index maintenance
• Historical reprocessing

Azure, AWS, and Google Cloud all document discounted batch options, commonly around 50% below standard online pricing for supported models.

9.9 Reserve baseline demand, not peak demand​

A hybrid capacity model is often financially superior:

Track provisioned-capacity:

• Coverage
• Utilisation
• Waste
• Overage
• Spillover
• Queue time
• Throttling
• Cost per token

Reservation coverage = Eligible usage covered by reservation
                     / Total eligible usage

Reservation utilisation = Used reserved capacity
                        / Purchased reserved capacity

High coverage with low utilisation indicates over-purchasing. High utilisation with frequent premium overflow may indicate under-provisioning.

9.10 Use streaming correctly​

Streaming improves perceived latency because the interface can display tokens before the complete answer is generated.

Streaming normally does not reduce the number of generated tokens by itself. However, it can improve user experience and may allow users to cancel an unhelpful response before completion.

Track separately:

• Time to first token
• Inter-token latency
• Total generation time
• Cancellation rate
• Tokens avoided through cancellation

9.11 Prevent retry storms​

Retries can silently multiply cost.

Use:

• Exponential backoff
• Jitter
• Idempotency keys
• Maximum retry limits
• Circuit breakers
• Queue-based load smoothing
• Regional failover policies
• Separate retry budgets
• No retries for deterministic client errors

Track original and retry requests separately.

A request that succeeds after four model calls should not be reported as a single inexpensive success.

10. Optimising Open-Source LLM Hosting​

10.1 Quantisation​

Quantisation reduces model memory and compute requirements by representing weights at lower precision, such as:

• FP16 or BF16
• FP8
• INT8
• INT4

Potential benefits include:

• Smaller GPU requirement
• Higher model density
• Lower memory bandwidth
• Higher throughput
• Lower cost per token

Potential disadvantages include:

• Reduced accuracy
• Lower stability for some tasks
• Runtime compatibility constraints
• Quantisation overhead
• Different performance across hardware

Quantisation must be validated against use-case-specific evaluations rather than only public benchmarks.

10.2 Continuous and dynamic batching​

Batching multiple requests increases GPU utilisation.

NVIDIA Triton supports dynamic batching and concurrent model execution, allowing several requests or model instances to use accelerator resources more efficiently. The batch-delay setting must be tuned because waiting too long to form a batch can increase user latency.

Measure:

• Batch size
• Batch formation delay
• Requests per second
• Tokens per second
• Time to first token
• p95 latency
• GPU utilisation
• Memory utilisation

A suitable target may be 70–85% sustained accelerator utilisation rather than 100%, because operating continuously at saturation can produce severe queueing and p99 latency.

10.3 Prefix and key-value cache reuse​

Transformer inference stores key-value representations of previously processed tokens.

Systems can reduce repeated prefill computation through:

• Prefix-aware routing
• Shared prompt prefixes
• Session affinity
• Prefix caching
• Reused system instructions
• Cached document context

Requests with the same prefix should be routed consistently where the serving platform supports prefix reuse.

10.4 Speculative decoding​

Speculative decoding uses a smaller draft model to propose tokens that a larger model validates.

When well matched, it can:

• Increase generated tokens per second
• Reduce latency
• Improve accelerator utilisation

However, the benefit depends on draft acceptance rates. A poorly matched draft model adds computation without sufficient acceleration.

10.5 Right-size accelerators​

Benchmark model configurations using real production prompts.

Compare:

• Model load time
• Maximum context
• Concurrent requests
• Time to first token
• Output tokens per second
• GPU memory
• Power and hourly price
• Cost per 1,000 completed requests
• Cost per successful outcome

The fastest GPU is not automatically the most economical. A less expensive accelerator with adequate memory may produce a lower cost per token for medium-volume workloads.

10.6 Avoid idle high-availability replicas​

High availability may require multiple replicas, but replicas should be justified by:

• Failure-domain requirements
• Upgrade strategy
• Traffic volume
• Recovery-time objectives
• Cold-start duration
• Regional resilience

Development and test environments generally should not mirror the complete production replica count continuously.

Use schedules to reduce or stop non-production endpoints outside working hours.

11. A Practical LLM FinOps Dashboard​

Executive dashboard​

Display:

• Total monthly AI cost
• Forecast versus budget
• Cost by business unit
• Cost by use case
• Cost per successful outcome
• Business value delivered
• Quality and reliability status
• Top anomalies
• Commitment utilisation

Product dashboard​

Display:

• Cost per conversation
• Cost per active user
• Cost per tenant
• Resolution or completion rate
• Human-escalation rate
• Model-routing distribution
• Cache-hit rate
• Revenue or productivity impact

Engineering dashboard​

Display:

• Input, output and cached tokens
• Tokens per request
• Time to first token
• p50, p95 and p99 latency
• Tokens generated per second
• Requests per second
• Throttling and 429 rates
• Retry rate
• Error rate
• Agent steps per run
• Tool calls per run
• GPU utilisation
• Provisioned-capacity utilisation

Finance dashboard​

Display:

• Actual and amortised cost
• Budget variance
• Forecast
• Cost allocation coverage
• Untagged spending
• Reservation coverage
• Reservation utilisation
• Overage spending
• Discount effectiveness
• Unit-cost trend

12. Cost Guardrails​

A mature platform should enforce guardrails before cost is incurred.

Request-level guardrails​

Examples:

• Maximum input tokens
• Maximum output tokens
• Maximum model class
• Maximum number of agent steps
• Maximum tools per run
• Maximum retries
• Maximum request cost
• Maximum retrieved context
• Maximum conversation history
• Maximum reasoning effort

Tenant guardrails​

Examples:

• Requests per minute
• Tokens per minute
• Daily token allowance
• Monthly spending limit
• Concurrent-agent limit
• Premium-model allowance
• Batch-job quota

Environment guardrails​

Examples:

• Development cannot use priority capacity.
• Test environments scale to zero.
• Experiments have expiration dates.
• Premium models require an approved use-case tag.
• Untagged resources are blocked.
• Evaluation jobs use batch capacity.
• Provisioned capacity requires a utilisation owner.

Guardrails should degrade gracefully. When a spending threshold is reached, the system could:

1. Reduce maximum output length.
2. Route low-risk work to a smaller model.
3. Disable optional critic calls.
4. Move background work to a queue.
5. Preserve premium capacity for critical users.
6. Require approval for further spending.

Do not abruptly route high-risk regulated decisions to an unvalidated model solely to save money.

13. Worked Cost-Optimisation Example​

Consider an AI customer-service platform processing one million requests each month.

Baseline architecture​

Assumptions:

• Two premium-model calls per user request
• 3,000 input tokens per call
• 400 output tokens per call
• Illustrative input price: $2 per million tokens
• Illustrative output price: $8 per million tokens

These rates are illustrative rather than quotations from a particular provider.

Input cost​

1,000,000 × 2 × 3,000 = 6,000,000,000 input tokens
6,000 × $2 = $12,000

Output cost​

1,000,000 × 2 × 400 = 800,000,000 output tokens
800 × $8 = $6,400

Baseline model cost​

$12,000 + $6,400 = $18,400

Optimised architecture​

Simple requests: 55%​

• Small model
• One call
• 1,800 input tokens
• 250 output tokens
• $0.30 per million input tokens
• $1.20 per million output tokens

Cost:

$462

Medium requests: 35%​

• Premium model
• One call
• 2,300 input tokens
• 350 output tokens
• 40% of input served from cache
• Cached input assumed to receive a 90% discount

Cost:

$2,010.40

Complex requests: 10%​

• Premium model
• Two calls
• 3,500 input tokens
• 700 output tokens
• 30% of input served from cache

Cost:

$2,142

Optimised model subtotal​

$462 + $2,010.40 + $2,142 = $4,614.40

Assume retrieval, guardrails, observability, and other platform services add 15%:

$4,614.40 × 1.15 = $5,306.56

Estimated saving​

1 − ($5,306.56 / $18,400) = 71.2%

The saving is achieved through:

• Model routing
• Fewer model calls
• Smaller context
• Shorter outputs
• Prompt caching
• Eliminating unnecessary premium-model usage

The optimisation should be released only after evaluations confirm that task success, safety, and customer satisfaction remain above required thresholds.

Latency may also improve because 55% of requests now use a smaller model with fewer tokens. Complex requests retain the stronger model rather than sacrificing quality.

14. A 90-Day LLM FinOps Implementation Roadmap​

Days 1–30: Inform​

1. Inventory all models, endpoints, agents and RAG applications.
2. Define a mandatory cost-tagging standard.
3. Enable Azure exports, AWS CUR/Data Exports, or Google Cloud Billing exports.
4. Capture request-level token and latency telemetry.
5. Calculate cost per request and conversation.
6. Establish quality and latency baselines.
7. Create budgets and anomaly alerts.
8. Identify untagged and idle resources.

Days 31–60: Optimise​

1. Introduce model routing.
2. Reduce system-prompt and context size.
3. Implement prompt caching.
4. Add response and retrieval caching.
5. Move offline workflows to batch.
6. Move low-priority synchronous work to flex capacity.
7. Tune maximum output lengths.
8. Add conversation summarisation.
9. Optimise RAG retrieval and reranking.
10. Remove unnecessary agent steps.
11. Autoscale self-hosted endpoints.
12. Schedule non-production shutdowns.

Days 61–90: Operate​

1. Calculate provisioned-throughput break-even points.
2. Reserve only measured baseline demand.
3. Implement chargeback or showback.
4. Define cost per successful business outcome.
5. Add quality-adjusted cost reporting.
6. Introduce request-level spending guardrails.
7. Review commitment utilisation monthly.
8. Add cost regression testing to CI/CD.
9. Establish a model and pricing review board.
10. Reassess model routing whenever a new model or price is introduced.

15. Cost Regression Testing​

Every prompt, model, retrieval, or agent change should be tested for financial impact.

A regression suite should compare:

• Average input tokens
• Average output tokens
• Cache-hit rate
• Average agent steps
• Tool calls
• Accuracy
• Groundedness
• p95 latency
• Estimated cost per successful task

A deployment should fail a release gate when:

• Cost increases without a justified quality improvement.
• Quality falls below the accepted threshold.
• Latency exceeds the service objective.
• Agent loops or retries materially increase.
• Cache-hit rate unexpectedly decreases.
• A premium model begins receiving traffic intended for a cheaper route.

Example policy:

Release the change when:

Task success does not fall by more than 1 percentage point,
p95 latency does not increase by more than 10%,
and cost per successful outcome does not increase by more than 5%,

unless the product owner approves a documented value or risk justification.

16. Final Recommendations​

The most effective LLM cost strategy is not a single discount or provider feature. It is a layered architecture.

The recommended design is:

1. Measure every request.
2. Allocate every cost to an owner and use case.
3. Measure cost per successful outcome, not merely per token.
4. Use deterministic code before an LLM where possible.
5. Route simple work to smaller models.
6. Use premium reasoning models only when complexity requires them.
7. Place stable prompt content first to maximise caching.
8. Reduce conversation history and RAG context.
9. Move asynchronous workloads to batch or flex capacity.
10. Reserve predictable baseline demand rather than peak demand.
11. Use standard or priority capacity for bursts according to criticality.
12. Autoscale self-hosted models and eliminate idle non-production GPUs.
13. Control agent loops, retries and tool fan-out.
14. Evaluate cost, quality and latency together.
15. Continuously reconsider model choices as prices and capabilities change.

Azure, AWS, and Google Cloud now provide broadly comparable financial levers: discounted batch inference, context or prompt caching, multiple service tiers, provisioned capacity, billing exports, budgets, and anomaly detection. The differentiator is not merely which provider has the lowest published token rate. It is how effectively the organisation combines these capabilities into a measurable, governed, and workload-aware operating model.

A well-designed LLM FinOps practice does not constrain innovation. It allows teams to scale generative AI confidently by ensuring that every additional dollar of AI expenditure is connected to measurable performance, reliability, and business value.