Architecting a Production-Grade Multi-Tenant AI Platform on AWS
A platform with one million registered users does not need one-million-user capacity. It needs enough capacity for peak concurrent work, enough isolation to keep one tenant from harming another, and enough evidence to prove that each AI response is safe, grounded, affordable and attributable.
This article is an AWS-native reference architecture with explicit trade-offs — not a one-size-fits-all blueprint. It is written as a field guide for platforms serving 1M+ registered users, where the hard problems are concurrency maths, tenant isolation, inference routing, RAG authorization, unit economics and recoverability.
This architecture is intentionally opinionated. It chooses managed services where they reduce undifferentiated operations, but it preserves explicit control over tenant context, authorization, model routing, data boundaries, deployment gates and unit economics. AWS's SaaS guidance emphasises that isolation must be enforced across every layer, not inferred from the fact that tenants share a platform.1
1. Start with workload mathematics, not the user count
The phrase "one million users" is useful for business storytelling and almost useless for infrastructure sizing. Two platforms with the same registered-user count can differ by orders of magnitude in peak traffic and inference cost. The sizing model must begin with observed or forecast arrival rate, concurrency, prompt length, output length, streaming duration, tenant skew and cacheability.
Suppose 50,000 users are active during the busiest hour, 10 percent initiate an AI task within a five-minute window, and each task streams for eight seconds. That is roughly 16.7 requests per second and 134 concurrent generations before retries, background work or safety checks. If the average request contains 3,000 input tokens and produces 600 output tokens, token throughput becomes the binding constraint long before the web tier does.
| Design rule | Why it matters |
|---|---|
| Load-test the shape of demand | Model arrivals, context sizes and streaming lifetimes; do not replay uniform HTTP requests that ignore token throughput. |
| Separate online and offline work | Interactive chat needs a bounded latency path. Corpus ingestion, long summaries, evaluations and durable agents should enter queues with explicit age and retry limits. |
| Design for skew | The largest tenant, hottest partition or most expensive prompt usually defines failure behaviour. Per-tenant metrics and quotas are architectural requirements. |
2. The end-to-end AWS reference architecture
The design below uses a multi-account landing zone and a three-Availability-Zone primary Region. Stateless APIs and the AI orchestrator run on EKS in private subnets. Managed data services hold state, while Amazon Bedrock provides managed model inference. A warm secondary Region contains minimum viable application capacity and replicated data services.
Figure 1. End-to-end multi-tenant AI platform on AWS.
At the edge, Route 53, CloudFront, AWS WAF and Shield protect and accelerate traffic. Amazon Cognito or a federated enterprise identity provider authenticates the user. The API tier never trusts a tenant identifier supplied in a request body; it derives tenant context from validated identity claims and confirms it against the tenant registry.
The SaaS control plane owns onboarding, tenant configuration, plans, entitlements, residency rules, metering, billing and offboarding. The application plane serves business requests. This separation is crucial: tenant lifecycle operations should not be embedded as ad hoc branches inside every microservice.
3. Multi-tenancy: pool by default, isolate by policy
AWS describes pooled, siloed and mixed isolation models. A production platform should treat these as deployable tiers rather than force every customer into one extreme.2
| Isolation tier | Recommended use | Implementation and trade-off |
|---|---|---|
| Pool | Standard tenants and high-volume self-service | Shared EKS services and data stores with mandatory tenant keys, policy checks and fair-use controls. Lowest unit cost; requires excellent isolation testing. |
| Bridge | Enterprise or data-sensitive customers | Shared application services with dedicated database/schema, queue, vector index, encryption key or model budget. Stronger isolation without duplicating the whole platform. |
| Silo | Regulated or contractually isolated workloads | Dedicated AWS account, VPC, data services, quotas and audit boundary. Maximum separation and operational cost. |
In the pooled tier, tenants share compute and storage resources while every key, row, object and vector record carries a stable tenant identifier. In the bridge tier, compute may remain shared while selected resources — such as a database, KMS key, S3 bucket, vector collection or inference reservation — are dedicated. In the silo tier, an entire account and VPC stack can be deployed for the tenant.
Authorization belongs in several places:
- The edge validates identity.
- A central policy service such as Amazon Verified Permissions evaluates user-to-resource permissions.
- Application code binds tenant context to every downstream call.
- The data layer applies partition, row or index-level controls.
RAG retrieval performs the same authorization twice: inside the search query and again before evidence enters the prompt.
4. The synchronous AI request path
Figure 2. The online path keeps policy and data isolation outside the model.
| Step | What happens |
|---|---|
| Admit and authenticate | CloudFront and WAF absorb edge threats; API Gateway or an ALB terminates the request; the service validates the JWT, schema, payload size and idempotency key. |
| Resolve tenant policy | The registry returns the tenant tier, residency boundary, allowed models, retention policy and feature entitlements. Redis-backed token buckets enforce user, tenant, model and concurrency ceilings. |
| Choose the workflow | The model gateway classifies task complexity, modality and risk. It assigns a latency budget, context budget, tool allowlist and eligible model class. |
| Retrieve authorised evidence | Hybrid keyword and vector search applies tenant and document ACL filters, reranks results, removes duplicates and preserves source identifiers for citations. |
| Protect the request | Prompt-injection checks, PII policy and tool-schema validation run before inference. Guardrails are applied according to the use case, but application authorization remains independent. |
| Invoke with bounded fallback | An application inference profile records usage and cost. Geographic routing can increase available throughput within an approved geography. Timeouts, a small retry budget and a same-class alternate avoid retry storms. |
| Validate and stream | Structured output is parsed, citations are checked, sensitive output is redacted, and tokens stream using SSE or WebSocket. Every response records model, prompt, guardrail and knowledge-index versions. |
5. Inference choice is a routing problem
The production decision is not "Which is the best model?" It is "Which model and capacity tier is the cheapest option that meets this task's quality, safety and latency requirement?" The answer can change by tenant, feature, geography and time of day.
Amazon Bedrock application inference profiles help attribute usage and cost, while cross-Region inference profiles can distribute requests across approved Regions to increase throughput. Bedrock also exposes Reserved, Priority, Standard and Flex service tiers, making capacity a per-workload choice rather than a single platform-wide commitment.345
| Workload | Default model decision | Capacity and cost decision |
|---|---|---|
| Classification, extraction, moderation | Rules or small/fast model first; structured output | Standard tier; batch compatible items; short context |
| Interactive grounded assistant | Balanced model proven by domain evaluations | Standard or Priority; geographic profile; stream responses; cache stable context |
| High-value complex reasoning | Premium model only after router confidence or tenant entitlement | Priority for strict SLO; Reserved only after sustained utilisation is measured |
| Embeddings | Versioned model selected by retrieval recall and vector footprint | Online queries; batch corpus generation; deduplicate and incrementally update |
| Offline summaries and evaluations | Cheapest model meeting an offline quality threshold | Flex/batch with SQS backpressure and completion deadlines |
| Self-hosted / custom | SageMaker only for defensible IP, tuning, compliance or economics | Include idle GPU, batching, warm capacity and on-call burden in the comparison |
Prompt caching is useful when long, stable prefixes — such as policies, tool schemas or uploaded documents — repeat across requests. AWS documents it as a way to reduce both input-token cost and latency for supported models, but cache writes, minimum checkpoint sizes, model support and cross-Region behaviour must be measured.6
| Cache mode | When to use |
|---|---|
| Exact response cache | Safest for deterministic requests with an explicit freshness policy. |
| Semantic response cache | Powerful for repetitive questions, but requires tenant isolation, similarity thresholds and invalidation when source data changes. |
| Prompt cache | Best for repeated long prefixes; track cache reads, writes and effective token savings separately. |
| No cache | Required for highly personalised, rapidly changing or legally sensitive output unless a safe cache key can be proven. |
6. Production RAG: authorization before relevance
Retrieval-augmented generation is a data product, not a vector-database feature. The canonical copy of every document belongs in a versioned S3 data lake. EventBridge and SQS initiate ingestion; Step Functions tracks durable document state; Lambda handles lightweight transformations; ECS tasks handle large files or long-running parsers; and Textract is used where OCR or document structure is required.
The pipeline validates file type, scans for malware, classifies data, removes or tags sensitive content, creates structure-aware chunks, assigns deterministic chunk identifiers, attaches document and tenant ACLs, generates versioned embeddings and publishes into a hybrid index. DynamoDB stores ingestion state and deduplication keys so every operation is idempotent.
OpenSearch Serverless is attractive for variable or intermittent demand because ingestion and search compute scale as OpenSearch Compute Units. A provisioned OpenSearch domain can be more economical for sustained utilisation or when deeper cluster and shard tuning is required. The decision should compare workload-specific OCU or node utilisation, index size, query concurrency, vector dimensionality and operational burden.78
| Gate | Requirement |
|---|---|
| Quality | Track recall@k, precision, reranker lift, groundedness, citation correctness and answer relevance using domain-specific golden sets. |
| Freshness | Upserts, tombstones, deletions and ACL changes must reach every index within a defined service level. |
| Recovery | The index is disposable. S3 remains the source of truth, embedding versions are recorded, and a full rebuild is rehearsed. |
| Privacy | Adversarial tests must prove that users cannot retrieve another tenant's chunks through query rewriting, metadata omissions or indirect prompt injection. |
7. Data services and state placement
A single database rarely matches every access pattern.
| Service | Fit |
|---|---|
| Aurora PostgreSQL | Tenants, plans, billing relationships, workflow records and other relational transactions. RDS Proxy controls connection storms from autoscaled services. |
| DynamoDB | Conversation state, idempotency records, high-volume key-value access and event-processing checkpoints. |
| Redis | Short-lived sessions, quotas and caches. |
| S3 | Durable objects and analytical history. |
For regional recovery, Aurora Global Database provides a primary Region with secondary clusters, while DynamoDB Global Tables replicate key-value state. Application semantics still matter: an eventually consistent multi-Region table can create write conflicts, so a write-to-one-Region strategy or a deliberately selected consistency model is often safer for transactional state.910
8. Scalability and latency engineering
The web tier is usually not the bottleneck. Model quotas, token throughput, vector-search latency, database connections and queue age are more likely to limit scale. The architecture must therefore autoscale each layer on the signal that represents pending work.
For EKS, use HPA for pod-level CPU, memory or application latency; KEDA for SQS queue depth and oldest-message age; and Karpenter for node provisioning. AWS recommends managed node groups and Karpenter for large-scale EKS data planes, and Karpenter is particularly suitable for spiky or diverse compute requirements.1112
| Principle | Practice |
|---|---|
| Protect a stable baseline | Run critical controllers on managed baseline nodes or Fargate. Do not let the autoscaler depend entirely on capacity that it manages itself. |
| Spread across zones | Use topology constraints, pod disruption budgets and zonal capacity tests. A three-AZ diagram is meaningless if most replicas land in one AZ. |
| Scale on demand signals | Queue age and pending work are better than CPU for asynchronous pipelines; concurrent streams and first-token latency are better than average request count for inference gateways. |
| Bound every dependency | Use connection pools, timeouts, circuit breakers, jittered retries, dead-letter queues and maximum queue age. Unbounded retries amplify an outage. |
9. Cost architecture and unit economics
Cloud cost becomes manageable when it is expressed per successful business outcome instead of per service. A low cost per token can still produce an expensive product if prompts are bloated, retries multiply, retrieval is weak or users abandon slow responses. The north-star metric should therefore be cost per successful task, segmented by tenant, feature and model route.
| Lever | Actions |
|---|---|
| Inference | Small-model-first routing, output caps, prompt and semantic caching, narrow retrieval, asynchronous execution for tolerant work, and Reserved capacity only after utilisation proves the commitment. |
| Compute | Graviton where compatible, Savings Plans for baseline nodes and Spot only for interruptible workers. Karpenter should have explicit NodePool resource ceilings and cost alarms. |
| Data | S3 lifecycle policies, TTL ephemeral DynamoDB state, tune vector dimensions and quantisation, and compare OpenSearch Serverless OCUs with provisioned utilisation. |
| Tenancy | Price bridge and silo tiers to cover their dedicated databases, indexes, keys, quotas, deployment pipelines, observability and incident-management overhead. |
| Measurement | Track input/output tokens, cache read/write tokens, retrieved chunks, model escalations, retries, queue delay and failure-adjusted cost for every request. |
10. Security and responsible AI controls
Security begins with the AWS account structure. Use Organizations and Control Tower to separate production, non-production, security tooling and immutable log archives. IAM Identity Center governs human access; workload roles and EKS Pod Identity avoid long-lived credentials; VPC endpoints reduce public data paths; KMS and Secrets Manager protect data and credentials.
Bedrock Guardrails can provide content filters, denied topics, sensitive-information controls, prompt-attack detection and contextual grounding checks. Guardrails should be applied both before and after inference according to the use case, but they do not replace authorization, deterministic validation or human escalation for high-impact decisions.13
| Control area | Practice |
|---|---|
| Prompt and tool security | Treat retrieved documents and tool output as untrusted input. Allowlist tools, validate JSON schemas, constrain arguments and require explicit approval for high-impact actions. |
| Data minimisation | Do not log raw prompts or model responses by default. Store redacted samples only under an approved retention and access policy. |
| Supply chain | Generate SBOMs, scan dependencies and containers, sign images in ECR, pin production AMIs and block deployment on critical findings. |
| Auditability | Record policy decisions, model and prompt versions, guardrail versions, tool calls, source citations and operator overrides without storing unnecessary sensitive content. |
11. Multi-Region resilience without accidental complexity
Multi-Region architecture is justified by a business recovery objective, not by diagram aesthetics. A sensible starting point for a critical platform might be an RTO of 30 minutes and an RPO of five minutes, but these values must be negotiated with product, legal and finance teams.
The primary Region runs the full three-AZ stack. The secondary Region keeps minimum viable application capacity, replicated images and configuration, promotable Aurora clusters, DynamoDB replicas and S3 replicas. Route 53 or Route 53 Application Recovery Controller supports traffic recovery, but the failover sequence must fence writes before promoting state to avoid split-brain behaviour.
| Phase | Action |
|---|---|
| Detect | Use external synthetics, regional health signals and multi-window SLO burn alerts. |
| Decide | A documented automatic or human-approved policy determines whether the failure is regional, dependent-service-specific or application-induced. |
| Fence and promote | Stop or redirect writes, promote the database, scale the standby application and verify secrets, quotas and Bedrock model availability. |
| Route and validate | Shift traffic gradually and execute synthetic tenant journeys, authorization checks, RAG queries and billing events. |
| Recover forward | Reconcile asynchronous events, repair replication, establish the new recovery posture and capture evidence from the exercise. |
AWS's Reliability Pillar stresses explicit recovery objectives, dependency management and tested recovery procedures. Quarterly game days should include a regional loss, a model-throttling event, an OpenSearch impairment and a cross-tenant authorization probe.14
12. Observability, evaluation and deployment gates
Traditional observability answers whether the system is available. AI observability must also answer whether the output was useful, grounded, safe and economically sensible. Every trace should correlate edge, application, retrieval, guardrail, model and tool spans while carrying tenant-safe dimensions.
| Signal class | Examples |
|---|---|
| Reliability | Request rate, errors, p95/p99 latency, saturation, queue age, throttles, fallback rate and dependency health. |
| AI | First-token latency, tokens per second, input/output tokens, cache hits, model route, groundedness, citation accuracy, safety detections and task success. |
| Tenant | Consumption, concurrency, throttles, hot partitions, storage, cost and SLO attainment by tenant and plan. |
| Release | Golden-set regression, red-team failures, retrieval recall, schema failures, cost per task and canary user outcomes. |
Application code, prompts, model configuration, guardrails, chunking logic, embedding model and index aliases must all be versioned independently. A canary release should roll back when either system SLOs or AI evaluation thresholds regress. This is the operational difference between experimenting with a model and running an AI product.
13. A practical delivery roadmap
| Phase | Build | Exit criteria |
|---|---|---|
| 1. Foundation | Landing zone, identity, tenant registry, CI/CD, telemetry and threat model | A tenant can be onboarded and offboarded automatically; access and audit tests pass |
| 2. Minimum production AI | Single-Region multi-AZ app, model gateway, Guardrails, quotas and online evaluations | Load, safety and rollback tests pass at expected peak plus headroom |
| 3. Governed RAG | S3 source of truth, ingestion workflow, hybrid retrieval, ACLs and citation checks | Freshness, deletion, rebuild, recall and cross-tenant tests meet targets |
| 4. Economic scale | Model tiers, exact/semantic/prompt caches, Karpenter and per-tenant unit economics | Cost per successful task and SLO attainment are stable by tenant tier |
| 5. Regional resilience | Warm standby, global data replication, failover automation and runbooks | A production-like game day meets approved RTO/RPO without isolation failure |
Conclusion
A production-grade multi-tenant AI platform is not a model endpoint surrounded by a few APIs. It is a SaaS operating model with AI inside it. The architecture must know which tenant is acting, which data they may access, which model and capacity tier they are allowed to consume, how the result will be evaluated, what the request cost, and how the platform behaves when a Region, dependency or model is unavailable.
On AWS, the strongest default is a bridge model built on a multi-account foundation: pooled EKS services for efficiency, targeted data and capacity isolation for enterprise tenants, Amazon Bedrock for managed inference, governed S3 and OpenSearch pipelines for RAG, and a warm secondary Region for workloads whose recovery objectives justify it. The final production design should be driven by measured traffic and evaluation data, not by fashionable services or a registered-user headline.
Sources and further reading
AWS documentation consulted for this architecture. Service availability, model support, quotas and pricing vary by Region and can change; verify them during implementation.
Related playbook reading:
- The Complete AWS Learning Roadmap
- LLM Cost Tracking and FinOps Across Azure, AWS, and Google Cloud
- LLM Caching Ultimate Guide
- Production-Grade Evaluation Strategies for LLMs, RAG and Agents
- Building Production-Grade AI Agents
- RAG Architecture
Discussion
Comments
Share feedback or questions about this page. No account required.
Loading comments…