Set up Amazon EKS cluster for AI/ML workloads using Terraform

This section walks you through the steps to create the infrastructure required to run training or inference workloads on Amazon EKS by using Terraform. The steps include creating an EKS cluster, GPU-enabled nodes with EKS Auto Mode or Karpenter, a monitoring stack with Prometheus and Grafana, and Amazon S3 storage for model weights.

See the documentation for EKS Auto Mode and Karpenter for more information on how those features provision and auto-scale EC2 instances in EKS clusters.

High-level architecture and workflow

The diagram shows the AWS high-level architecture for this section’s setup.

Prerequisites

Verify your tool versions:

terraform --version
aws --version
kubectl version --client
jq --version

Step 1: Download and deploy the Terraform code

This walkthrough uses the Terraform code in the sample-eks-docs AWS Samples GitHub repository. Clone the repository into a working directory:

git clone git@github.com:aws-samples/sample-eks-docs.git
cd sample-eks-docs/ai-ml/set-up-cluster

The repository has the following structure under the ai-ml/set-up-cluster/ directory you just changed into:

set-up-cluster/
├── scripts/
│   └── cleanup.sh
└── terraform/
    ├── auto-mode/
    └── karpenter/

The repository provides two deployment paths. Choose only one and use it throughout the guide.

EKS cluster options: EKS Auto Mode and self-managed Karpenter

export MY_CIDR="$(curl -s https://checkip.amazonaws.com)/32"
terraform apply -var "my_cidr=${MY_CIDR}"

Deploy the cluster

Change into the directory for your chosen path, initialize Terraform, and apply:

Both variants default to the us-east-2 region. To deploy in a different region, add -var "region=region-code" to the terraform apply command in the following step, where region-code is the AWS Region you want to deploy in.

The Terraform code uses all available Availability Zones in the target region, excluding use1-az3, usw1-az2, and cac1-az3 because Amazon EKS does not support control plane placement in those zones.

Review the Terraform outputs

When the apply completes, Terraform prints the following outputs (values vary based on your configuration):

Apply complete! Resources: 74 added, 0 changed, 0 destroyed.

Outputs:

cluster_name         = "ai-eks-docs"
configure_kubectl    = "aws eks update-kubeconfig --region us-east-2 --name ai-eks-docs --alias ai-eks-docs"
configure_model_bucket = "export MODEL_BUCKET=ai-eks-docs-models-20250612abc1"
model_bucket         = "ai-eks-docs-models-20250612abc1"
node_iam_role_name   = "ai-eks-docs-eks-auto-20250612..."
region               = "us-east-2"

The configure_kubectl output is a ready-to-run command that points kubectl at the cluster. The model_bucket output contains the S3 bucket name for model weights. The node_iam_role_name output shows the IAM role that nodes use.

Configure kubectl

Point kubectl at the new cluster. The configure_kubectl output is a ready-to-run command:

eval "$(terraform output -raw configure_kubectl)"

Verify the cluster

Step 2: Create dynamic GPU NodePool

GPU NodePools are opt-in. By default, terraform apply creates the cluster and monitoring stack with no GPU capacity and no GPU billing. To provision GPU nodes, pass the nodepools variable with a strategy name.

Enable the spot-ondemand strategy, which provisions G-family GPU instances with a generation greater than 4, using Spot capacity with On-Demand as a fallback:

terraform apply -var 'nodepools={"spot-ondemand"={}}'

This command applies the NodePool and NodeClass templates from the nodepools/spot-ondemand/ directory. Both paths use the same NodePool API, but they differ in the NodeClass the NodePool references.

Both paths show 0 nodes for gpu-inf until a GPU workload is scheduled. EKS Auto Mode and Karpenter only launch nodes when pending Pods require them.

Step 3: Test with a sample Pod

Test your GPU NodePool setup with an nvidia-smi Pod:

cat << EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: nvidia-smi
  labels:
    guide: ai-eks-docs
spec:
  restartPolicy: OnFailure
  tolerations:
    - key: "nvidia.com/gpu"
      operator: "Exists"
      effect: "NoSchedule"
  containers:
    - name: nvidia-smi
      image: public.ecr.aws/amazonlinux/amazonlinux:2023-minimal
      command: ["nvidia-smi"]
      resources:
        limits:
          nvidia.com/gpu: 1
EOF

Verify the Pod is scheduled and completed successfully:

kubectl get pods nvidia-smi

Expected output:

NAME         READY   STATUS      RESTARTS   AGE
nvidia-smi   0/1     Completed   0          67s

The STATUS: Completed means the nvidia-smi command ran and exited. Check the Pod logs to see the GPU detected by the node:

kubectl logs nvidia-smi

Expected output:

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.159.03             Driver Version: 580.159.03     CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA L4                      On  |   00000000:31:00.0 Off |                    0 |
| N/A   41C    P8             13W /   72W |       0MiB /  23034MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

The output shows the GPU model, driver version, CUDA version, and available memory. In this example, Karpenter provisioned a G6 instance which has an NVIDIA L4 GPU with 24 GB of memory. The GPU model and memory vary depending on the instance type Karpenter selects. G5 instances have NVIDIA A10G GPUs (24 GB), G6 instances have NVIDIA L4 GPUs (24 GB), and G6e instances have NVIDIA L40S GPUs (48 GB).

To understand how Karpenter and the Kubernetes scheduler coordinated to provision a node and place the Pod, check the Pod’s lifecycle events:

kubectl describe pod nvidia-smi

Expected output:

Events:
  Type     Reason            Age   From                   Message
  ----     ------            ----  ----                   -------
  Warning  FailedScheduling  75s   default-scheduler      0/2 nodes are available: 2 node(s) had untolerated taint(s).
  Normal   Nominated         74s   eks-auto-mode/compute  Pod should schedule on: nodeclaim/gpu-inf-z6q75
  Normal   Scheduled         35s   default-scheduler      Successfully assigned default/nvidia-smi to i-0eb897a8302551589
  Normal   Pulling           27s   kubelet                spec.containers{nvidia-smi}: Pulling image "public.ecr.aws/amazonlinux/amazonlinux:2023-minimal"
  Normal   Pulled            22s   kubelet                spec.containers{nvidia-smi}: Successfully pulled image "public.ecr.aws/amazonlinux/amazonlinux:2023-minimal" in 5.625s (5.626s including waiting). Image size: 37440620 bytes.
  Normal   Created           22s   kubelet                spec.containers{nvidia-smi}: Container created
  Normal   Started           21s   kubelet                spec.containers{nvidia-smi}: Container started

These events show the Pod scheduling sequence: the Pod initially fails to schedule because no GPU nodes exist (FailedScheduling), Karpenter nominates a new NodeClaim (Nominated), the scheduler assigns the Pod once the node is ready (Scheduled), and then the container image is pulled and started. EKS Auto Mode has SOCI (Seekable OCI) parallel pull installed and configured by default on G, P, and Trn instances, and the self-managed Karpenter path configures it explicitly through the FastImagePull feature gate.

A NodeClaim is a request Karpenter creates to provision a specific node. It shows the instance type, capacity type, AZ, and whether the node is ready:

kubectl get nodeclaims

Expected output:

NAME            TYPE        CAPACITY   ZONE         NODE                  READY   AGE
gpu-inf-z6q75   g6.xlarge   spot       us-east-2a   i-0eb897a8302551589   True    5m

The instance type and AZ vary. Any G-family instance with a generation greater than 4 is eligible.

kubectl get events | grep InsufficientCapacityError

Step 4: Add reserved capacity to the NodePool (optional)

While the GPU NodePool from Step 2 provisions Spot or On-Demand instances dynamically, some use cases require guaranteed capacity. You can create an On-Demand Capacity Reservation (ODCR) to ensure GPU capacity is available when needed.

With Terraform, a single command creates the ODCR, a custom NodeClass that references the reservation by tag, and updates the NodePool to include reserved as a capacity type. Terraform tags the ODCR with nodepool=reserved-spot-ondemand and the NodeClass selects it by that tag.

Use defaults (g6e.4xlarge, 1 instance, first cluster AZ):

terraform apply -var 'nodepools={"reserved-spot-ondemand"={reservation={}}}'

Pick the instance type, count, and AZ:

terraform apply -var 'nodepools={"reserved-spot-ondemand"={reservation={instance_type="g6e.2xlarge",instance_count=1,az="us-east-2a"}}}'

The reservation object supports the following fields:

If you get an InsufficientInstanceCapacity error, the reservation cannot be fulfilled in the specified AZ. Cancel the Terraform operation (Ctrl+C), then re-run with a different az value:

terraform apply -var 'nodepools={"reserved-spot-ondemand"={reservation={instance_type="g6e.4xlarge",az="us-east-2b"}}}'

After applying, Terraform updates the NodePool to include reserved, spot, and on-demand in the capacity-type requirements. Karpenter treats reserved as the most cost-efficient option and launches it first. Once the reservation is full, it falls back to Spot or On-Demand.

On the EKS Auto Mode path, Terraform creates a custom gpu-inf NodeClass (because the bundled default NodeClass is read-only) that references the ODCR by tag through capacityReservationSelectorTerms. On the self-managed Karpenter path, Terraform re-applies the gpu-inf EC2NodeClass with capacityReservationSelectorTerms added and updates the NodePool to include reserved.

Verify the ODCR was created:

aws ec2 describe-capacity-reservations \
  --filters "Name=state,Values=active" "Name=tag:nodepool,Values=reserved-spot-ondemand" \
  --query 'CapacityReservations[0].{Id:CapacityReservationId,State:State,InstanceType:InstanceType,AvailableCount:AvailableInstanceCount}' \
  --output table \
  --region $(terraform output -raw region)

Verify the NodeClass references the ODCR:

Verify the NodePool is ready:

kubectl get nodepools gpu-inf

Expected output:

NAME      NODECLASS   NODES   READY   AGE
gpu-inf   gpu-inf     0       True    30s

cat << 'EOF' | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: gpu-overflow-test
  labels:
    guide: ai-eks-docs
spec:
  replicas: 2
  selector:
    matchLabels:
      app: gpu-overflow-test
  template:
    metadata:
      labels:
        app: gpu-overflow-test
        guide: ai-eks-docs
    spec:
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
      containers:
        - name: nvidia-smi
          image: public.ecr.aws/amazonlinux/amazonlinux:2023-minimal
          command: ["sh", "-c", "nvidia-smi && sleep infinity"]
          resources:
            limits:
              nvidia.com/gpu: 1
EOF

kubectl get pods -l app=gpu-overflow-test -o wide

NAME                                 READY   STATUS    RESTARTS   AGE     IP            NODE                  NOMINATED NODE   READINESS GATES
gpu-overflow-test-55d55ff5b9-dvg52   1/1     Running   0          4m42s   10.0.75.210   i-08741a36089ff2088   <none>           <none>
gpu-overflow-test-55d55ff5b9-hw4m9   1/1     Running   0          4m43s   10.0.82.49    i-0f50cdbacb2017202   <none>           <none>

kubectl get nodeclaims

NAME            TYPE          CAPACITY   ZONE         NODE                  READY   AGE
gpu-inf-vw99m   g6e.4xlarge   reserved   us-east-2c   i-0f50cdbacb2017202   True    6m
gpu-inf-s65s6   g6.xlarge     spot       us-east-2b   i-08741a36089ff2088   True    5m59s

kubectl delete deployment gpu-overflow-test

Monitoring

Terraform already provisioned the full monitoring stack during terraform apply in Step 1. The stack includes an Amazon Managed Service for Prometheus (AMP) workspace, IAM policies and EKS Pod Identity Associations for Prometheus remote-write and Grafana query access, the kube-prometheus-stack Helm chart (Prometheus, Grafana, kube-state-metrics, node-exporter), and the NVIDIA DCGM Exporter for GPU metrics.

This section covers verification of the deployed monitoring components.

Verify monitoring pods

Wait for all monitoring pods to be ready:

kubectl wait --for=condition=Ready pod --all -n monitoring --timeout=300s
kubectl get pods -n monitoring

Expected output:

NAME                                                       READY   STATUS    RESTARTS   AGE
kube-prometheus-stack-grafana-7c58f54f77-rftrj             3/3     Running   0          5m
kube-prometheus-stack-kube-state-metrics-d68dcbc84-5smxq   1/1     Running   0          5m
kube-prometheus-stack-operator-5895df479f-ttm47            1/1     Running   0          5m
kube-prometheus-stack-prometheus-node-exporter-t9q7s       1/1     Running   0          5m
kube-prometheus-stack-prometheus-node-exporter-x6vfb       1/1     Running   0          5m
prometheus-kube-prometheus-stack-prometheus-0              2/2     Running   0          5m

Access Grafana

Grafana is exposed through an internet-facing AWS Application Load Balancer (ALB), restricted to the CIDR you set in var.my_cidr. Print the load balancer URL (allow a minute or two for the ALB to provision):

echo "http://$(kubectl get ingress kube-prometheus-stack-grafana -n monitoring -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')"

Open the URL in your browser. Log in with username admin and the password from the following command:

kubectl --namespace monitoring get secrets kube-prometheus-stack-grafana -o jsonpath="{.data.admin-password}" | base64 -d ; echo

Verify the metrics pipeline

To verify the metrics pipeline is working end to end:

If up shows results, the pipeline (cluster → Prometheus → AMP → Grafana) is working.

Validate DCGM GPU metrics

The DCGM Exporter DaemonSet runs on GPU nodes and reports GPU utilization, memory, temperature, power draw, NVLink bandwidth, and tensor activity metrics.

Verify the DCGM exporter DaemonSet:

kubectl get daemonset dcgm-exporter -n monitoring

Once a GPU node is running (from Step 2 or Step 4), you should see one or more ready Pods. To validate DCGM metrics, navigate to Drilldown > Metrics in Grafana and search for DCGM_.

Validate DCGM metrics in Grafana

To view the dashboard, navigate to Dashboards > GPU Monitoring > NVIDIA DCGM Exporter Dashboard.

NVIDIA DCGM Exporter Dashboard in Grafana

Model weights S3 bucket

Terraform already created an Amazon S3 bucket for storing model weights, a model-storage-sa ServiceAccount in the default namespace, an IAM policy scoped to the bucket, and an EKS Pod Identity Association that links them. Workload Pods that set serviceAccountName: model-storage-sa can read from and write to the bucket.

Verify the bucket

Retrieve the bucket name from Terraform outputs:

MODEL_BUCKET=$(terraform output -raw model_bucket)
echo ${MODEL_BUCKET}

Verify the bucket exists:

aws s3api head-bucket --bucket ${MODEL_BUCKET}

Expected output:

{
    "BucketArn": "arn:aws:s3:::ai-eks-docs-models-20250612abc1",
    "BucketRegion": "us-east-2",
    "AccessPointAlias": false
}

cat << EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: s3-test
  labels:
    guide: ai-eks-docs
spec:
  serviceAccountName: model-storage-sa
  containers:
    - name: aws-cli
      image: public.ecr.aws/aws-cli/aws-cli:2.27.0
      command:
        - sh
        - -c
        - |
          echo "=== Caller Identity ==="
          aws sts get-caller-identity
          echo ""
          echo "=== S3 Write Test ==="
          echo "pod identity works" | aws s3 cp - s3://${MODEL_BUCKET}/test.txt
          echo ""
          echo "=== S3 List Test ==="
          aws s3 ls s3://${MODEL_BUCKET}/
          echo ""
          echo "=== S3 Delete Test ==="
          aws s3 rm s3://${MODEL_BUCKET}/test.txt
  restartPolicy: Never
EOF

kubectl wait --for=jsonpath='{.status.phase}'=Succeeded pod/s3-test --timeout=300s
kubectl logs s3-test

=== Caller Identity ===
{
    "UserId": "AROA...:eks-ai-eks-docs-model-s-...",
    "Account": "123456789012",
    "Arn": "arn:aws:sts::123456789012:assumed-role/ai-eks-docs-models-.../eks-ai-eks-docs-..."
}

=== S3 Write Test ===
upload: - to s3://ai-eks-docs-models-20250612abc1/test.txt

=== S3 List Test ===
2026-07-15 12:00:00         19 test.txt

=== S3 Delete Test ===
delete: s3://ai-eks-docs-models-20250612abc1/test.txt

kubectl delete pod s3-test

Next steps

With your cluster ready, you can proceed to Load & Serve Model to deploy a large language model and interact with the inference endpoint.

Cleanup

Delete the test workloads so no Pods are holding GPU nodes:

kubectl delete pod nvidia-smi --ignore-not-found
kubectl delete deployment gpu-overflow-test --ignore-not-found

terraform apply -var 'nodepools={"spot-ondemand"={}}'

kubectl delete pdb -A --all --ignore-not-found
kubectl delete nodeclaim --all --wait=true --timeout=900s

terraform destroy

aws ec2 describe-capacity-reservations \
  --filters "Name=state,Values=active" "Name=tag:nodepool,Values=reserved-spot-ondemand" \
  --query 'CapacityReservations[].CapacityReservationId' \
  --output text