Amazon CloudWatch solution: Prometheus metric collection on Amazon ECS

This solution helps you collect Prometheus-compatible metrics from your Amazon ECS tasks without running or scaling your own collection infrastructure. Your tasks expose metrics through Prometheus exporters, and a Amazon CloudWatch managed Prometheus collector discovers the tasks in your VPC through DNS-based service discovery, scrapes their /metrics endpoints, and delivers the metrics to CloudWatch. Because Amazon ECS tasks are ephemeral and their IP addresses change, this solution uses AWS Cloud Map service discovery so that the collector always scrapes the current set of tasks. For general information about all CloudWatch observability solutions, see CloudWatch observability solutions. For more information about managed collectors, see Amazon CloudWatch managed Prometheus collectors.

Requirements

This solution is relevant for the following conditions:

Enable Prometheus exporters

A Prometheus exporter is a process that exposes metrics in Prometheus exposition format on an HTTP /metrics endpoint. On Amazon ECS, you typically run an exporter as a sidecar container in the same task definition as your application, or instrument the application to expose metrics directly. The following exporters are commonly used:

Add the exporter to your task definition and expose its metrics port by following the exporter's official documentation. For a complete catalog of available exporters, see Exporters and integrations in the Prometheus documentation.

Step 1: Set up security groups

Confirm that the security group for your Amazon ECS tasks allows inbound traffic from the collector on the metrics port. When your tasks and the collector share a security group, add a self-referencing ingress rule so that resources in the same security group can communicate. The following command allows inbound traffic on port 80 from the same security group:

aws ec2 authorize-security-group-ingress \
  --group-id sg-security-group-id \
  --ip-permissions IpProtocol=tcp,FromPort=80,ToPort=80,UserIdGroupPairs='[{GroupId=sg-security-group-id}]'

Step 2: Set up the scrape configuration

For Amazon ECS tasks that you register with AWS Cloud Map, use DNS-based service discovery (dns_sd_configs) to automatically find and scrape your tasks as they start and stop. The dns_sd_configs section directs the collector to query the AWS Cloud Map DNS name and scrape all returned IP addresses. When Amazon ECS replaces a task, the collector picks up the change on its next DNS query. The relabel_configs section adds consistent labels to your metrics so that you can query and filter across services. For the full set of supported configuration options, see Scraper configuration.

global:
  scrape_interval: 30s
  scrape_timeout: 10s

scrape_configs:
  - job_name: 'ecs-payforadoption'
    dns_sd_configs:
      - names: ['payforadoption-go.Workshop-space']
        type: A
        port: 80
    metrics_path: '/metrics'
    relabel_configs:
      - target_label: service_name
        replacement: 'payforadoption-go'
      - target_label: cloudmap_namespace
        replacement: 'Workshop-space'
      - target_label: environment
        replacement: 'production'
      - target_label: compute_platform
        replacement: 'ecs-fargate'

Step 3: Set up the scraper

Create a VPC-connected managed collector that scrapes your Amazon ECS tasks and delivers metrics to your CloudWatch dataset:

You can use GetDefaultScraperConfiguration to retrieve a general-purpose scraper configuration, or provide your own.

AWS API

Use the CreateScraper API operation to create a scraper with a CloudWatch destination. Replace the subnet, security group, and dataset information with your own values.

POST /scrapers HTTP/1.1

{
  "alias": "ecs-payforadoption-scraper",
  "source": {
    "vpcConfiguration": {
      "subnetIds": ["subnet-subnet-id-1", "subnet-subnet-id-2"],
      "securityGroupIds": ["sg-security-group-id"]
    }
  },
  "destination": {
    "cloudWatchConfiguration": {
      "datasetArn": "arn:aws:cloudwatch:us-west-2:123456789012:dataset/default"
    }
  },
  "scrapeConfiguration": {
    "configurationBlob": "base64-encoded-blob"
  }
}

AWS CLI

Use the create-scraper command to create a scraper with a CloudWatch destination. Replace the subnet, security group, and dataset information with your own values.

aws amp create-scraper \
  --alias "ecs-payforadoption-scraper" \
  --source '{
    "vpcConfiguration": {
      "subnetIds": ["subnet-subnet-id-1", "subnet-subnet-id-2"],
      "securityGroupIds": ["sg-security-group-id"]
    }
  }' \
  --scrape-configuration configurationBlob=$(cat ecs-scraper.yml | base64 -w 0) \
  --destination '{
    "cloudWatchConfiguration": {
      "datasetArn": "arn:aws:cloudwatch:us-west-2:123456789012:dataset/default"
    }
  }'

Validate metrics collection

After the collector begins delivering metrics, run an ad-hoc query in CloudWatch using Query Studio to confirm that your metrics arrive. For example, the following query returns the average requests per second to the Amazon ECS task over the last five minutes:

rate(payforadoption_requests_total[5m])

Build a custom dashboard

After the collector begins delivering your Amazon ECS metrics to CloudWatch, you can build a custom CloudWatch dashboard to visualize them. With custom dashboards, you can combine metrics from your Prometheus exporters into widgets, add PromQL-based queries, and organize the widgets to fit your monitoring needs. For more information, see Using CloudWatch dashboards.

Costs

We charge for Prometheus collectors by the hour, and CloudWatch OpenTelemetry metric ingestion pricing applies. For more information about CloudWatch pricing, see Amazon CloudWatch Pricing.