Deploy workloads from Azure DevOps pipelines to private Amazon EKS clusters

Find the Azure DevOps organization GUID.

Sign in to your Azure DevOps account, and then use the following URL to find the organization GUID: https://dev.azure.com/{DevOps_Org_ID}/_apis/projectCollections?api-version=6.0 In the URL, replace {DevOps_org_ID} with your Azure DevOps organization ID.

Configure an IdP in the AWS account.

To configure an Identity provider (IdP) in the AWS account for an Azure service connection, use the following steps:

For more details, see How to federate into AWS from Azure DevOps using OpenID Connect.

Create an IAM policy in the AWS account.

To create an IAM policy to provide the required permissions to the IAM role used by the Azure DevOps pipeline, use the following steps:

Create an IAM role in the AWS account.

To configure an IAM role in the AWS account for the Azure service connection, use the following steps:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::{account_id}:oidc-provider/vstoken.dev.azure.com/{OrganizationGUID}"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "vstoken.dev.azure.com/{OrganizationGUID}:aud": "api://AzureADTokenExchange",
          "vstoken.dev.azure.com/{OrganizationGUID}:sub": "sc://{OrganizationName}/{ProjectName}/{ServiceConnectionName}"
        }
      }
    }
  ]
}

In the policy, provide your information for the following placeholders:

Create a service connection in the Azure DevOps account.

To configure an Azure service connection, use the following steps:

For more details, see Create a service connection in the Microsoft documentation.

Add IAM role to Amazon EKS configuration file.

The IAM role must have the necessary permissions to perform the required operations on the Amazon EKS cluster. Because it’s a pipeline role, the IAM role must be able to manage almost all types of resources on the cluster. Therefore, the system:masters group permission is appropriate for this role.

To add the required configuration to the aws-auth ConfigMap within Kubernetes, use the following code:

- groups:
  - system:masters
  rolearn: arn:aws:iam::{account_id}:role/ADO-role
  username: ADO-role

Replace {account_id} with your AWS account ID.

For more information, see How Amazon EKS works with IAM in the Amazon EKS documentation.

Create a self-hosted agent pool.

To configure a self-hosted agent pool in the Azure DevOps account, use the following steps:

For more details, see Create and manage agent pools in the Microsoft documentation.

Create an Amazon ECR repository.

The Docker images that are used to deploy the Azure DevOps agent and sample application (webapp) on the private Amazon EKS cluster must be stored in an Amazon ECR repository. To create an Amazon ECR repository, use the following steps:

For more details, see Creating an Amazon ECR private repository to store images in the Amazon ECR documentation.

Create a Dockerfile to build the Azure DevOps agent.

Create a Dockerfile to build the Docker image that has the Azure DevOps agent installed. Store the following content in a file named Dockerfile:

FROM ubuntu:22.04 
ENV TARGETARCH="linux-x64"
RUN apt update && apt upgrade -y && apt install -y curl git jq libicu70 unzip wget

RUN curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
RUN unzip awscliv2.zip
RUN ./aws/install
RUN rm -rf aws awscliv2.zip

RUN curl -sSL https://get.docker.com/ | sh

RUN curl -sL https://aka.ms/InstallAzureCLIDeb | bash
RUN mkdir -p azp 
WORKDIR /azp/

COPY ./start.sh ./ 
RUN chmod +x ./start.sh

RUN useradd -m -d /home/agent agent 
RUN chown -R agent:agent /azp /home/agent
RUN groupadd -f docker 
RUN usermod -aG docker agent
USER agent

ENTRYPOINT [ "./start.sh" ]

Create script for the Azure DevOps agent.

To create the start.sh script, use the following steps:

Build a Docker image with the Azure DevOps agent.

To create a Docker image to install the Azure DevOps agent, use the Dockerfile that you created earlier to build the image. In the same directory where the Dockerfile is stored, run the following commands:

aws ecr get-login-password --region region | docker login --username AWS --password-stdin aws_account_id.dkr.ecr.region.amazonaws.com

docker build --platform linux/amd64 -t ado-agent:latest .

docker tag ado-agent:latest aws_account_id.dkr.ecr.region.amazonaws.com/webapp:latest

docker push aws_account_id.dkr.ecr.region.amazonaws.com/webapp:latest

Replace aws_account_id and region with your AWS account ID and AWS Region.

Generate an Azure personal access token.

The agent running on the private Amazon EKS cluster requires a personal access token (PAT) so that it can authenticate with the Azure DevOps account. To generate a PAT, use the following steps:

apiVersion: v1
kind: Secret
metadata:
  name: azdevops-pat
  namespace: default
type: Opaque
stringData:
  AZP_TOKEN: <PAT Token>

kubectl create -f ado-secret.yaml

For more details, see Register an agent using a personal access token (PAT) in the Microsoft documentation.

Use the Kubernetes manifest file for agent deployment.

To deploy the Azure DevOps agent on the private Amazon EKS cluster, copy the following manifest file and store the file as agent-deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: azure-pipelines-agent-eks
  labels:
    app: azure-pipelines-agent
spec:
  replicas: 1
  selector:
    matchLabels:
      app: azure-pipelines-agent
  template:
    metadata:
      labels:
        app: azure-pipelines-agent
    spec:
      containers:
      - name: docker
        image: docker:dind
        securityContext: 
          privileged: true
        volumeMounts:
        - name: shared-workspace
          mountPath: /workspace
        - name: dind-storage
          mountPath: /var/lib/docker
        env:
        - name: DOCKER_TLS_CERTDIR
          value: ""
      - name: azure-pipelines-agent
        image: aws_account_id.dkr.ecr.region.amazonaws.com/webapp:latest
        env:
        - name: AZP_URL
          value: "<Azure account URL>"
        - name: AZP_POOL
          value: "eks-agent"
        - name: AZP_TOKEN
          valueFrom:
            secretKeyRef:
              name: azdevops-pat
              key: AZP_TOKEN
        - name: AZP_AGENT_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: DOCKER_HOST
          value: tcp://localhost:2375
        volumeMounts:
        - mountPath: /workspace
          name: shared-workspace
      volumes:
      - name: dind-storage
        emptyDir: {}
      - name: shared-workspace
        emptyDir: {}

Replace aws_account_id and <Azure account URL> with your AWS account ID and Azure DevOps account URL.

Deploy the agent on the private Amazon EKS cluster.

To deploy the Azure Devops agent on the private Amazon EKS cluster, use the following command:

kubectl create -f agent-deployment.tf

Verify the agent is running.

To verify that the Azure DevOps agent is running, use the following command:

kubectl get deploy azure-pipelines-agent-eks

The expected output should be similar to the following:

NAME                        READY   UP-TO-DATE   AVAILABLE   AGE
azure-pipelines-agent-eks   1/1     1            1           58s

Make sure that the READY column shows 1/1.

Verify the agent is registered with the Azure DevOps agent pool.

To verify that the agent is deployed on the private Amazon EKS cluster and is registered with the agent pool eks-agent, use the following steps:

You should see one agent listed with a Status of Online, and the name of the agent should start with azure-pipelines-agent-eks-*.

Fork the sample application repository to your GitHub account.

Fork the following AWS Samples repository to your GitHub account:

https://github.com/aws-samples/deploy-kubernetes-resources-to-amazon-eks-using-azure-devops

Create a pipeline.

To create a pipeline in your Azure DevOps account, use the following steps:

pool:
  name: eks-agent
#pool: self-hosted # If you are running self-hosted Azure DevOps Agents

stages:
# Refering the pipeline template, input parameter that are not specified will be added with defaults
- template: ./pipeline_templates/main_template.yaml
  parameters:
    serviceConnectionName: aws-sc
    awsRegion: <your region>
    awsEKSClusterName: <name of your EKS cluster>
    projectName: webapp

Verify that the sample application deployed.

After the pipeline completes, verify the successful deployment of the sample application by checking both the Amazon ECR repository and the Amazon EKS cluster.

To verify artifacts in the Amazon ECR repository, use the following steps:

For example, 20250501.1-image and 20250501.1-helm.

To verify deployment on the private Amazon EKS cluster in the namespace webapp, use the following command:

kubectl get deploy -n webapp 

The expected output is as follows:

NAME     READY   UP-TO-DATE   AVAILABLE
webapp   1/1     1            1           

Note: If this is your first pipeline run, you might need to authorize the service connection and agent pool. Look for permission requests in the Azure DevOps pipeline interface, and approve them to proceed.