# Deploy workloads from Azure DevOps pipelines to private Amazon EKS clusters *Mahendra Revanasiddappa, Amazon Web Services* ## Summary This pattern demonstrates how to implement continuous integration and continuous delivery (CI/CD) from Azure DevOps pipelines to private Amazon Elastic Kubernetes Service (Amazon EKS) clusters. It addresses a critical challenge faced by organizations that are enhancing their security posture by transitioning to private API server endpoints for their Amazon EKS clusters. A public endpoint exposes the Kubernetes API server directly to the internet, creating a larger attack surface that malicious actors could potentially target. By switching to a private endpoint, access to the cluster's control plane is restricted to within the customer's virtual private cloud (VPC). Although transitioning an Amazon EKS cluster to a private API endpoint significantly enhances security, it introduces connectivity challenges for external CI/CD platforms like Azure DevOps. The private endpoint is only accessible from within the cluster's VPC or peered networks. Therefore, standard Microsoft-hosted Azure DevOps agents, operating outside the AWS private network, can’t reach the Kubernetes API server directly. This breaks typical deployment workflows that rely on tools like kubectl or Helm running on these agents because they fail to establish a connection to the cluster. To overcome this problem, this pattern showcases an efficient approach by using self-hosted Azure DevOps agents within private Amazon EKS clusters. This solution offers superior cost optimization, operational efficiency, and scalability while preserving security requirements. This approach particularly benefits enterprises seeking to streamline their multi-cloud DevOps processes without compromising on performance or security. ## Prerequisites and limitations **Prerequisites** + An active AWS account. + AWS Command Line Interface (AWS CLI) version 2.13.17 or later, [installed.](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) + kubectl version 1.25.1 or later, [installed.](https://kubernetes.io/docs/tasks/tools/) + A private Amazon EKS cluster version 1.24 or later [created](https://docs.aws.amazon.com/eks/latest/userguide/create-cluster.html), with permissions to create namespaces, secrets, and deployments[.](https://docs.aws.amazon.com/eks/latest/userguide/create-cluster.html) + Worker nodes in an Amazon EKS cluster with outbound connectivity to the internet so that the Azure DevOps agent running on them can connect to Azure DevOps agent pool. + GitHub account [created](https://github.com/signup). + An Azure DevOps project with access to configure service connections, which are authenticated connections between Azure Pipelines and external or remote services, [created](https://learn.microsoft.com/en-us/azure/devops/user-guide/sign-up-invite-teammates?view=azure-devops&tabs=microsoft-account). + The AWS Toolkit for Azure DevOps version 1.15 or later installed for the Azure DevOps project described in the previous point. For installation instructions, see [AWS Toolkit for Azure DevOps](https://marketplace.visualstudio.com/items?itemName=AmazonWebServices.aws-vsts-tools) in Visual Studio Marketplace. **Limitations** + Some AWS services aren’t available in all AWS Regions. For Region availability, see [AWS Services by Region](https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/). For specific endpoints, see [Service endpoints and quotas](https://docs.aws.amazon.com/general/latest/gr/aws-service-information.html), and choose the link for the service. ## Architecture This pattern creates the following: + **Amazon ECR repository** - The Amazon Elastic Container Registry (Amazon ECR) repository stores the Docker image with the Azure DevOps agent and the sample app that is deployed. + **Azure DevOps agent pool** - An Azure DevOps self-hosted agent pool registers the agent running on the private Amazon EKS cluster. + **IAM role** - An AWS Identity and Access Management (IAM) role for the Azure service connection to provide required access to the agent that’s running on a private Amazon EKS cluster. + **Azure DevOps service connection** - A service connection in an Azure DevOps account to use the IAM role that provides the required access for the pipeline jobs to access AWS services. The following diagram shows the architecture of deploying a self-hosted Azure DevOps agent on a private Amazon EKS cluster and deploying a sample application on the same cluster. ![\[Deployment of self-hosted Azure DevOps agent and sample application on private Amazon EKS cluster.\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/images/pattern-img/a965834f-a1e2-4679-bd8c-15eed4f57b55/images/ee22bd3e-311c-46e0-8024-9b7e7752080a.png) The diagram shows the following workflow: 1. Deploy a self-hosted Azure DevOps agent as a deployment inside an Amazon EKS cluster. 1. An Azure DevOps agent connects to the agent pool on an Azure DevOps account using a personal access token (PAT) for authentication. 1. Azure Pipelines configures a pipeline to deploy by using code from a GitHub repository. 1. The pipeline runs on the agent from the agent pool that was configured in the pipeline configuration. The Azure DevOps agent gets the job information of the pipeline by constantly polling to the Azure DevOps account. 1. The Azure DevOps agent builds a Docker image as part of the pipeline job and pushes the image to the Amazon ECR repository. 1. The Azure DevOps agent deploys the sample application on a private Amazon EKS cluster in a namespace called `webapp`. ## Tools **Tools** + [Amazon Elastic Container Registry (Amazon ECR)](https://docs.aws.amazon.com/AmazonECR/latest/userguide/what-is-ecr.html) is a managed container image registry service that’s secure, scalable, and reliable. + [Amazon Elastic Kubernetes Service (Amazon EKS)](https://docs.aws.amazon.com/eks/latest/userguide/what-is-eks.html) helps you run Kubernetes on AWS without needing to install or maintain your own Kubernetes control plane or nodes. + [AWS Identity and Access Management (IAM)](https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html) helps you securely manage access to your AWS resources by controlling who is authenticated and authorized to use them. **Other tools** + [Docker](https://www.docker.com/) is a set of platform as a service (PaaS) products that use virtualization at the operating-system level to deliver software in containers. + [kubectl](https://kubernetes.io/docs/tasks/tools/) is a command-line interface that helps you run commands against Kubernetes clusters. **Code repository** + The code for this pattern is available in the GitHub [deploy-kubernetes-resources-to-amazon-eks-using-azure-devops](https://github.com/aws-samples/deploy-kubernetes-resources-to-amazon-eks-using-azure-devops) repository. ## Best practices + For Amazon EKS, see the [Amazon EKS Best Practices Guide](https://docs.aws.amazon.com/eks/latest/best-practices/introduction.html). + Follow the principle of least privilege and grant the minimum permissions required to perform a task. For more information, see [Grant least privilege](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html#grant-least-priv) and [Security best practices](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html) in the IAM documentation. ## Epics ### Create a service connection | Task | Description | Skills required | | --- | --- | --- | | Find the Azure DevOps organization GUID. | Sign in to your Azure DevOps account, and then use the following URL to find the organization GUID: `https://dev.azure.com/{DevOps_Org_ID}/_apis/projectCollections?api-version=6.0` In the URL, replace `{DevOps_org_ID}` with your Azure DevOps organization ID. | AWS DevOps | | Configure an IdP in the AWS account. | To configure an Identity provider (IdP) in the AWS account for an Azure service connection, use the following steps:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/deploy-workloads-from-azure-devops-pipelines-to-private-amazon-eks-clusters.html)For more details, see [How to federate into AWS from Azure DevOps using OpenID Connect](https://aws.amazon.com/blogs/modernizing-with-aws/how-to-federate-into-aws-from-azure-devops-using-openid-connect/). | AWS DevOps | | Create an IAM policy in the AWS account. | To create an IAM policy to provide the required permissions to the IAM role used by the Azure DevOps pipeline, use the following steps:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/deploy-workloads-from-azure-devops-pipelines-to-private-amazon-eks-clusters.html) | AWS DevOps | | Create an IAM role in the AWS account. | To configure an IAM role in the AWS account for the Azure service connection, use the following steps:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/deploy-workloads-from-azure-devops-pipelines-to-private-amazon-eks-clusters.html)
{
  "Version": "2012-10-17",		 	 	 
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::{account_id}:oidc-provider/vstoken.dev.azure.com/{OrganizationGUID}"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "vstoken.dev.azure.com/{OrganizationGUID}:aud": "api://AzureADTokenExchange",
          "vstoken.dev.azure.com/{OrganizationGUID}:sub": "sc://{OrganizationName}/{ProjectName}/{ServiceConnectionName}"
        }
      }
    }
  ]
}
In the policy, provide your information for the following placeholders:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/deploy-workloads-from-azure-devops-pipelines-to-private-amazon-eks-clusters.html) | AWS DevOps | | Create a service connection in the Azure DevOps account. | To configure an Azure service connection, use the following steps:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/deploy-workloads-from-azure-devops-pipelines-to-private-amazon-eks-clusters.html)For more details, see [Create a service connection](https://learn.microsoft.com/en-us/azure/devops/pipelines/library/service-endpoints?view=azure-devops#create-a-service-connection) in the Microsoft documentation. | AWS DevOps | | Add IAM role to Amazon EKS configuration file. | The IAM role must have the necessary permissions to perform the required operations on the Amazon EKS cluster. Because it’s a pipeline role, the IAM role must be able to manage almost all types of resources on the cluster. Therefore, the `system:masters` group permission is appropriate for this role.To add the required configuration to the `aws-auth ConfigMap` within Kubernetes, use the following code:
- groups:
  - system:masters
  rolearn: arn:aws:iam::{account_id}:role/ADO-role
  username: ADO-role
Replace `{account_id}` with your AWS account ID.For more information, see [How Amazon EKS works with IAM](https://docs.aws.amazon.com/eks/latest/userguide/security-iam-service-with-iam.html#security-iam-service-with-iam-roles) in the Amazon EKS documentation. | AWS DevOps | ### Create an agent pool | Task | Description | Skills required | | --- | --- | --- | | Create a self-hosted agent pool. | To configure a self-hosted agent pool in the Azure DevOps account, use the following steps:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/deploy-workloads-from-azure-devops-pipelines-to-private-amazon-eks-clusters.html)For more details, see [Create and manage agent pools](https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/pools-queues?view=azure-devops&tabs=yaml%2Cbrowser) in the Microsoft documentation. | | ### Build Azure DevOps agent image and push to Amazon ECR | Task | Description | Skills required | | --- | --- | --- | | Create an Amazon ECR repository. | The Docker images that are used to deploy the Azure DevOps agent and sample application (`webapp`) on the private Amazon EKS cluster must be stored in an Amazon ECR repository. To create an Amazon ECR repository, use the following steps:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/deploy-workloads-from-azure-devops-pipelines-to-private-amazon-eks-clusters.html)For more details, see [Creating an Amazon ECR private repository to store images](https://docs.aws.amazon.com/AmazonECR/latest/userguide/repository-create.html) in the Amazon ECR documentation. | AWS DevOps | | Create a Dockerfile to build the Azure DevOps agent. | Create a Dockerfile to build the Docker image that has the Azure DevOps agent installed. Store the following content in a file named `Dockerfile`:

FROM ubuntu:22.04 
ENV TARGETARCH="linux-x64"
RUN apt update && apt upgrade -y && apt install -y curl git jq libicu70 unzip wget

RUN curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
RUN unzip awscliv2.zip
RUN ./aws/install
RUN rm -rf aws awscliv2.zip

RUN curl -sSL https://get.docker.com/ | sh

RUN curl -sL https://aka.ms/InstallAzureCLIDeb | bash
RUN mkdir -p azp 
WORKDIR /azp/

COPY ./start.sh ./ 
RUN chmod +x ./start.sh

RUN useradd -m -d /home/agent agent 
RUN chown -R agent:agent /azp /home/agent
RUN groupadd -f docker 
RUN usermod -aG docker agent
USER agent

ENTRYPOINT [ "./start.sh" ]
| AWS DevOps | | Create script for the Azure DevOps agent. | To create the `start.sh` script, use the following steps:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/deploy-workloads-from-azure-devops-pipelines-to-private-amazon-eks-clusters.html) | AWS DevOps | | Build a Docker image with the Azure DevOps agent. | To create a Docker image to install the Azure DevOps agent, use the Dockerfile that you created earlier to build the image. In the same directory where the Dockerfile is stored, run the following commands:
aws ecr get-login-password --region region | docker login --username AWS --password-stdin aws_account_id.dkr.ecr.region.amazonaws.com

docker build --platform linux/amd64 -t ado-agent:latest .

docker tag ado-agent:latest aws_account_id.dkr.ecr.region.amazonaws.com/webapp:latest

docker push aws_account_id.dkr.ecr.region.amazonaws.com/webapp:latest
Replace `aws_account_id` and `region` with your AWS account ID and AWS Region. | AWS DevOps | ### Deploy the Azure DevOps agent to a private Amazon EKS cluster | Task | Description | Skills required | | --- | --- | --- | | Generate an Azure personal access token. | The agent running on the private Amazon EKS cluster requires a personal access token (PAT) so that it can authenticate with the Azure DevOps account. To generate a PAT, use the following steps:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/deploy-workloads-from-azure-devops-pipelines-to-private-amazon-eks-clusters.html)[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/deploy-workloads-from-azure-devops-pipelines-to-private-amazon-eks-clusters.html)
apiVersion: v1
kind: Secret
metadata:
  name: azdevops-pat
  namespace: default
type: Opaque
stringData:
  AZP_TOKEN:
[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/deploy-workloads-from-azure-devops-pipelines-to-private-amazon-eks-clusters.html)
kubectl create -f ado-secret.yaml
For more details, see [Register an agent using a personal access token (PAT)](https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/personal-access-token-agent-registration?view=azure-devops) in the Microsoft documentation. | AWS DevOps | | Use the Kubernetes manifest file for agent deployment. | To deploy the Azure DevOps agent on the private Amazon EKS cluster, copy the following manifest file and store the file as `agent-deployment.yaml`: 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: azure-pipelines-agent-eks
  labels:
    app: azure-pipelines-agent
spec:
  replicas: 1
  selector:
    matchLabels:
      app: azure-pipelines-agent
  template:
    metadata:
      labels:
        app: azure-pipelines-agent
    spec:
      containers:
      - name: docker
        image: docker:dind
        securityContext: 
          privileged: true
        volumeMounts:
        - name: shared-workspace
          mountPath: /workspace
        - name: dind-storage
          mountPath: /var/lib/docker
        env:
        - name: DOCKER_TLS_CERTDIR
          value: ""
      - name: azure-pipelines-agent
        image: aws_account_id.dkr.ecr.region.amazonaws.com/webapp:latest
        env:
        - name: AZP_URL
          value: ""
        - name: AZP_POOL
          value: "eks-agent"
        - name: AZP_TOKEN
          valueFrom:
            secretKeyRef:
              name: azdevops-pat
              key: AZP_TOKEN
        - name: AZP_AGENT_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: DOCKER_HOST
          value: tcp://localhost:2375
        volumeMounts:
        - mountPath: /workspace
          name: shared-workspace
      volumes:
      - name: dind-storage
        emptyDir: {}
      - name: shared-workspace
        emptyDir: {}
Replace `aws_account_id` and `` with your AWS account ID and Azure DevOps account URL. | AWS DevOps | | Deploy the agent on the private Amazon EKS cluster. | To deploy the Azure Devops agent on the private Amazon EKS cluster, use the following command:
kubectl create -f agent-deployment.tf
| AWS DevOps | | Verify the agent is running. | To verify that the Azure DevOps agent is running, use the following command:
kubectl get deploy azure-pipelines-agent-eks
The expected output should be similar to the following:

NAME                        READY   UP-TO-DATE   AVAILABLE   AGE
azure-pipelines-agent-eks   1/1     1            1           58s
Make sure that the `READY `column shows `1/1`. | AWS DevOps | | Verify the agent is registered with the Azure DevOps agent pool. | To verify that the agent is deployed on the private Amazon EKS cluster and is registered with the agent pool `eks-agent`, use the following steps:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/deploy-workloads-from-azure-devops-pipelines-to-private-amazon-eks-clusters.html)You should see one agent listed with a **Status** of **Online**, and the name of the agent should start with **azure-pipelines-agent-eks-\$1**. | AWS DevOps | ### Deploy sample application | Task | Description | Skills required | | --- | --- | --- | | Fork the sample application repository to your GitHub account. | Fork the following AWS Samples repository to your GitHub account:[https://github.com/aws-samples/deploy-kubernetes-resources-to-amazon-eks-using-azure-devops](https://github.com/aws-samples/deploy-kubernetes-resources-to-amazon-eks-using-azure-devops) | AWS DevOps | | Create a pipeline. | To create a pipeline in your Azure DevOps account, use the following steps:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/deploy-workloads-from-azure-devops-pipelines-to-private-amazon-eks-clusters.html)
pool:
  name: eks-agent
#pool: self-hosted # If you are running self-hosted Azure DevOps Agents

stages:
# Refering the pipeline template, input parameter that are not specified will be added with defaults
- template: ./pipeline_templates/main_template.yaml
  parameters:
    serviceConnectionName: aws-sc
    awsRegion: 
    awsEKSClusterName: 
    projectName: webapp
[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/deploy-workloads-from-azure-devops-pipelines-to-private-amazon-eks-clusters.html) | AWS DevOps | | Verify that the sample application deployed. | After the pipeline completes, verify the successful deployment of the sample application by checking both the Amazon ECR repository and the Amazon EKS cluster.To verify artifacts in the Amazon ECR repository, use the following steps:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/deploy-workloads-from-azure-devops-pipelines-to-private-amazon-eks-clusters.html)[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/deploy-workloads-from-azure-devops-pipelines-to-private-amazon-eks-clusters.html)For example, `20250501.1-image` and `20250501.1-helm`.To verify deployment on the private Amazon EKS cluster in the namespace `webapp`, use the following command:
kubectl get deploy -n webapp
The expected output is as follows:

NAME     READY   UP-TO-DATE   AVAILABLE
webapp   1/1     1            1
Note: If this is your first pipeline run, you might need to authorize the service connection and agent pool. Look for permission requests in the Azure DevOps pipeline interface, and approve them to proceed. | AWS DevOps | ## Troubleshooting | Issue | Solution | | --- | --- | | Pipeline fails when Amazon ECR repository name doesn’t match `webapp` | The sample application expects the Amazon ECR repository name to match the `projectName: webapp` parameter in `azure_pipeline.yml`.To resolve this issue, rename your Amazon ECR repository to `webapp`, or update the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/deploy-workloads-from-azure-devops-pipelines-to-private-amazon-eks-clusters.html) | | Error: Kubernetes cluster unreachable: the server has asked for the client to provide credentials | If you encounter this error in the "Pull and Deploy Helm Chart" step in your Azure pipeline, the root cause typically stems from an incorrect IAM role configuration in your Amazon EKS cluster's `aws-auth ConfigMap`.To resolve this issue, check the following:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/deploy-workloads-from-azure-devops-pipelines-to-private-amazon-eks-clusters.html) | ## Related resources **AWS Blogs** + [How to federate into AWS from Azure DevOps using OpenID Connect](https://aws.amazon.com/blogs/modernizing-with-aws/how-to-federate-into-aws-from-azure-devops-using-openid-connect/) **AWS services documentation** + [Amazon ECR](https://docs.aws.amazon.com/AmazonECR/latest/userguide/what-is-ecr.html) + [Amazon EKS](https://docs.aws.amazon.com/eks/latest/userguide/what-is-eks.html) **Microsoft documentation** + [What is Azure DevOps?](https://learn.microsoft.com/en-us/azure/devops/user-guide/what-is-azure-devops?view=azure-devops) + [What is Azure Pipelines?](https://learn.microsoft.com/en-us/azure/devops/pipelines/get-started/what-is-azure-pipelines?view=azure-devops)