Provision identical, production-grade EKS clusters across multiple regions with a single command. Eliminate per-region manual configuration and reduce deployment time from weeks to minutes.
Overview
This Guidance helps organizations deploy and manage machine learning inference workloads across multiple AWS Regions by automating capacity orchestration and providing a unified API for global Kubernetes cluster management. Platform engineers deploy a single AWS CDK application that provisions identical Amazon Elastic Kubernetes Service Auto Mode clusters across configured regions, connected through AWS Global Accelerator for intelligent traffic routing. Users submit Kubernetes manifests through a REST API or CLI with AWS IAM authentication—no manual cluster configuration required—while the platform automatically provisions GPU, AWS Trainium, AWS Inferentia, or CPU nodes on demand based on workload requirements. You gain simplified multi-region operations with automatic failover, reduced latency through global traffic optimization, and cost efficiency through on-demand node provisioning without pre-scaling infrastructure.
Benefits
Deploy global AI infrastructure instantly
Scale accelerators on demand automatically
Provision GPU, Trainium, and Inferentia nodes only when workloads require them, then scale to zero when idle. Reduce compute costs by avoiding pre-provisioned accelerator capacity.
Ensure continuous AI workload availability
Route training and inference requests to the nearest healthy region automatically using anycast routing with built-in failover. Maintain workload continuity during regional outages without manual intervention.
How it works
This architecture diagram shows how to deploy and manage machine learning inference workloads across multiple AWS Regions using EKS Auto Mode clusters with automated global capacity orchestration.
Download the architecture diagram.Deployment Architecture
Step 1
DevOps / Platform Engineers author and deploy one AWS Cloud Development Kit (AWS CDK) app; regions are configured in the cdk.json.
The command gco stacks deploy-all is used to provision identical
AWS CloudFormation stacks in every configured AWS Region
(auto-bootstrap included).
Users submit Kubernetes manifests through one REST API or CLI signed
with AWS SigV4 (no per-cluster kubeconfig configuration is necessary).
Amazon API Gateway (edge-optimized) validates IAM credentials at Amazon CloudFront edges and routes the request to the AWS Lambda proxy.
The AWS Lambda proxy injects a rotating secret from AWS Secrets Manager into user request before forwarding it to AWS Global Accelerator.
AWS Global Accelerator routes each request over the AWS backbone to the nearest healthy AWS Region, with automatic failover.
Per-region deployed Application Load Balancers accept traffic only from Global Accelerator IP addresses and route into respective Amazon Elastic Kubernetes Service clusters.
Each AWS Region runs an Amazon Elastic Kubernetes Service Auto Mode cluster that provisions GPU, AWS Trainium, AWS Inferentia, and CPU based nodes on demand via node pools configurations.
EKS Auto Mode in a multi-AZ VPC
Step 1
An internet-facing Application Load Balancer deployed in Amazon VPC public subnets accepts only from Global Accelerator IPs and routes via Kubernetes Ingress.
Amazon Elastic Kubernetes Service Auto Mode cluster runs a managed control plane and auto-provisions nodes with no pre-scaling.
Karpenter nodepools include system, general CPU (AMD64+ARM64), NVIDIA GPU, AWS Trainium/AWS Inferentia, inference, and Amazon Elastic Fabric Adapter (EFA) components.
Platform pods in gco-system namespace operate the workloads: Manifest
Processor applies manifests, Queue Processor consumes the job queue,
Inference Monitor reconciles endpoints, and Health Monitor tracks
cluster health.
Workloads persist data to Amazon EFS (shared RWX access) and optionally FSx for Lustre; inference workloads use Valkey cache, Amazon Aurora PostgreSQL pgvector vector storage, and Amazon S3 model weights.
A Regional API Gateway with VPC Link fronts an internal Network Load Balancer for private, in-VPC access to AWS services.
The internal Network Load Balancer forwards private traffic to in-cluster services without any public exposure.
Regional AWS services: Queue Processor drains Amazon Simple Queue Service, Inference Monitor reconciles Amazon DynamoDB, and Health Monitor publishes to Amazon CloudWatch.
End-to-end security
Step 1
IAM Authentication: Amazon API Gateway validates users' AWS credentials with SigV4 on every request.
Secret Header: an AWS Lambda proxy injects a rotating token from AWS Secrets Manager, rotated daily.
IP Restriction: ALB security groups accept traffic only from AWS Global Accelerator IPs.
Header Validation: backend services reject any request missing the valid secret token.
IAM Roles for Service Accounts (IRSA): pods assume IAM roles for AWS access, no static credentials are stored in the cluster.
Compliance is validated with CDK-nag validation utility against AWS Solutions, HIPAA, NIST 800-53, and PCI DSS standards.
Default-deny Kubernetes network policies plus encryption at rest (AWS KMS) and in transit (TLS 1.2+) protect all service communication.
Deploy with confidence
Everything you need to launch this Guidance in your account is right here.
Let's make it happen
Ready to deploy? Review the sample code on GitHub for detailed deployment instructions to deploy as-is or customize to fit your needs.