# Customer managed collectors
This section contains information about ingesting data by setting up your own collectors that send metrics to Amazon Managed Service for Prometheus using Prometheus remote write.
When you use your own collectors to send metrics to Amazon Managed Service for Prometheus, you are responsible for securing your metrics and making sure that the ingestion process meets your availability needs.
Most customer managed collectors use one of the following tools:
+ **AWS Distro for OpenTelemetry (ADOT)** – ADOT is a fully supported, secure, production-ready open source distribution of OpenTelemetry that provides agents to collect metrics. You can use ADOT to collect metrics and send them to your Amazon Managed Service for Prometheus workspace. For more information about the ADOT Collector, see [AWS Distro for OpenTelemetry](https://aws.amazon.com/otel/).
+ **Prometheus agent** – You can set up your own instance of the open source Prometheus server, running as an agent, to collect metrics and forward them to your Amazon Managed Service for Prometheus workspace.
The following topics describe using both of these tools and include general information about setting up your own collectors.
**Topics**
+ [Secure the ingestion of your metrics](AMP-secure-metric-ingestion.md)
+ [Using AWS Distro for OpenTelemetry as a collector](AMP-ingest-with-adot.md)
+ [Using a Prometheus instance as a collector](AMP-ingest-with-prometheus.md)
+ [Set up Amazon Managed Service for Prometheus for high availability data](AMP-ingest-high-availability.md)
# Secure the ingestion of your metrics
Amazon Managed Service for Prometheus provides ways of helping you secure the ingestion of your metrics.
## Using AWS PrivateLink with Amazon Managed Service for Prometheus
The network traffic of ingesting the metrics into Amazon Managed Service for Prometheus can be done over a public internet endpoint, or by a VPC endpoint through AWS PrivateLink. Using AWS PrivateLink ensures that the network traffic from your VPCs is secured within the AWS network without going over the public internet. To create an AWS PrivateLink VPC endpoint for Amazon Managed Service for Prometheus, see [Using Amazon Managed Service for Prometheus with interface VPC endpoints](AMP-and-interface-VPC.md).
## Authentication and authorization
AWS Identity and Access Management (IAM) is a web service that helps you securely control access to AWS resources. You use IAM to control who is authenticated (signed in) and authorized (has permissions) to use resources. Amazon Managed Service for Prometheus integrates with IAM to help you keep your data secure. When you set up Amazon Managed Service for Prometheus, you need to create some IAM roles that enable it to ingest metrics from Prometheus servers, and that enable Grafana servers to query the metrics that are stored in your Amazon Managed Service for Prometheus workspaces. For more information about IAM, see [What is IAM?](https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html).
Another AWS security feature that can help you set up Amazon Managed Service for Prometheus is the AWS Signature Version 4 signing process (AWS SigV4). Signature Version 4 is the process to add authentication information to AWS requests sent by HTTP. For security, most requests to AWS must be signed with an access key, which consists of an access key ID and secret access key. These two keys are commonly referred to as your security credentials. For more information about SigV4, see [Signature Version 4 signing process](https://docs.aws.amazon.com/general/latest/gr/signature-version-4.html).
# Using AWS Distro for OpenTelemetry as a collector
This section describes how to configure the AWS Distro for OpenTelemetry (ADOT) Collector to scrape from a Prometheus-instrumented application, and send the metrics to Amazon Managed Service for Prometheus. For more information about the ADOT Collector, see [AWS Distro for OpenTelemetry](https://aws.amazon.com/otel/).
The following topics describe three different ways to set up ADOT as a collector for your metrics, based on whether your metrics are coming from Amazon EKS, Amazon ECS, or an Amazon EC2 instance.
**Topics**
+ [Set up metrics ingestion using AWS Distro for OpenTelemetry on an Amazon Elastic Kubernetes Service cluster](AMP-onboard-ingest-metrics-OpenTelemetry.md)
+ [Set up metrics ingestion from Amazon ECS using AWS Distro for Open Telemetry](AMP-onboard-ingest-metrics-OpenTelemetry-ECS.md)
+ [Set up metrics ingestion from an Amazon EC2 instance using remote write](AMP-onboard-ingest-metrics-remote-write-EC2.md)
# Set up metrics ingestion using AWS Distro for OpenTelemetry on an Amazon Elastic Kubernetes Service cluster
You can use the AWS Distro for OpenTelemetry (ADOT) collector to scrape metrics from a Prometheus-instrumented application, and send the metrics to Amazon Managed Service for Prometheus.
**Note**
For more information about the ADOT collector, see [AWS Distro for OpenTelemetry](https://aws.amazon.com/otel/).
For more information about Prometheus-instrumented applications, see [What are Prometheus-compatible metrics?](prom-compatible-metrics.md).
Collecting Prometheus metrics with ADOT involves three OpenTelemetry components: the Prometheus Receiver, the Prometheus Remote Write Exporter, and the Sigv4 Authentication Extension.
You can configure the Prometheus Receiver using your existing Prometheus configuration to perform service discovery and metric scraping. The Prometheus Receiver scrapes metrics in the Prometheus exposition format. Any applications or endpoints that you want to scrape should be configured with the Prometheus client library. The Prometheus Receiver supports the full set of Prometheus scraping and re-labeling configurations described in [Configuration ](https://prometheus.io/docs/prometheus/latest/configuration/configuration/) in the Prometheus documentation. You can paste these configurations directly into your ADOT Collector configurations.
The Prometheus Remote Write Exporter uses the `remote_write` endpoint to send the scraped metrics to your management portal workspace. The HTTP requests to export data will be signed with AWS SigV4, the AWS protocol for secure authentication, with the Sigv4 Authentication Extension. For more information, see [Signature Version 4 signing process](https://docs.aws.amazon.com/general/latest/gr/signature-version-4.html).
The collector automatically discovers Prometheus metrics endpoints on Amazon EKS and uses the configuration found in [.](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config)
The following demo is an example of this configuration on a cluster running Amazon Elastic Kubernetes Service or self-managed Kubernetes. To perform these steps, you must have AWS credentials from any of the potential options in the default AWS credentials chain. For more information, see [Configuring the AWS SDK for Go](https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html). This demo uses a sample app that is used for integration tests of the process. The sample app exposes metrics at the `/metrics` endpoint, like the Prometheus client library.
## Prerequisites
Before you begin the following ingestion setup steps, you must set up your IAM role for the service account and trust policy.
**To set up the IAM role for service account and trust policy**
1. Create the IAM role for the service account by following the steps in [Set up service roles for the ingestion of metrics from Amazon EKS clusters](set-up-irsa.md#set-up-irsa-ingest).
The ADOT Collector will use this role when it scrapes and exports metrics.
1. Next, edit the trust policy. Open the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/home).
1. In the left navigation pane, choose **Roles** and find the **amp-iamproxy-ingest-role** that you created in step 1.
1. Choose the **Trust relationships** tab and choose **Edit trust relationship**.
1. In the trust relationship policy JSON, replace `aws-amp` with `adot-col` and then choose **Update Trust Policy**. Your resulting trust policy should look like the following:
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::111122223333:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:sub": "system:serviceaccount:adot-col:amp-iamproxy-ingest-service-account",
"oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:aud": "sts.amazonaws.com"
}
}
}
]
}
```
------
1. Choose the **Permissions** tab and make sure that the following permissions policy is attached to the role.
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"aps:RemoteWrite",
"aps:GetSeries",
"aps:GetLabels",
"aps:GetMetricMetadata"
],
"Resource": "*"
}
]
}
```
------
## Enabling Prometheus metric collection
**Note**
When you create a namespace in Amazon EKS, `alertmanager` and node exporter are disabled by default.
**To enable Prometheus collection on an Amazon EKS or Kubernetes cluster**
1. Fork and clone the sample app from the repository at [aws-otel-community](https://github.com/aws-observability/aws-otel-community).
Then run the following commands.
```
cd ./sample-apps/prometheus-sample-app
docker build . -t prometheus-sample-app:latest
```
1. Push this image to a registry such as Amazon ECR or DockerHub.
1. Deploy the sample app in the cluster by copying this Kubernetes configuration and applying it. Change the image to the image that you just pushed by replacing `{{PUBLIC_SAMPLE_APP_IMAGE}}` in the `prometheus-sample-app.yaml` file.
```
curl https://raw.githubusercontent.com/aws-observability/aws-otel-collector/main/examples/eks/aws-prometheus/prometheus-sample-app.yaml -o prometheus-sample-app.yaml
kubectl apply -f prometheus-sample-app.yaml
```
1. Enter the following command to verify that the sample app has started. In the output of the command, you will see `prometheus-sample-app` in the `NAME` column.
```
kubectl get all -n aoc-prometheus-pipeline-demo
```
1. Start a default instance of the ADOT Collector. To do so, first enter the following command to pull the Kubernetes configuration for ADOT Collector.
```
curl https://raw.githubusercontent.com/aws-observability/aws-otel-collector/main/examples/eks/aws-prometheus/prometheus-daemonset.yaml -o prometheus-daemonset.yaml
```
Then edit the template file, substituting the **remote\$1write** endpoint for your Amazon Managed Service for Prometheus workspace for `YOUR_ENDPOINT` and your Region for `YOUR_REGION`. Use the **remote\$1write** endpoint that is displayed in the Amazon Managed Service for Prometheus console when you look at your workspace details.
You'll also need to change `YOUR_ACCOUNT_ID` in the service account section of the Kubernetes configuration to your AWS account ID.
In this example, the ADOT Collector configuration uses an annotation (`scrape=true`) to tell which target endpoints to scrape. This allows the ADOT Collector to distinguish the sample app endpoint from kube-system endpoints in your cluster. You can remove this from the re-label configurations if you want to scrape a different sample app.
1. Enter the following command to deploy the ADOT collector.
```
kubectl apply -f prometheus-daemonset.yaml
```
1. Enter the following command to verify that the ADOT collector has started. Look for `adot-col` in the `NAMESPACE` column.
```
kubectl get pods -n adot-col
```
1. Verify that the pipeline works by using the logging exporter. Our example template is already integrated with the logging exporter. Enter the following commands.
```
kubectl get pods -A
kubectl logs -n adot-col name_of_your_adot_collector_pod
```
Some of the scraped metrics from the sample app will look like the following example.
```
Resource labels:
-> service.name: STRING(kubernetes-service-endpoints)
-> host.name: STRING(192.168.16.238)
-> port: STRING(8080)
-> scheme: STRING(http)
InstrumentationLibraryMetrics #0
Metric #0
Descriptor:
-> Name: test_gauge0
-> Description: This is my gauge
-> Unit:
-> DataType: DoubleGauge
DoubleDataPoints #0
StartTime: 0
Timestamp: 1606511460471000000
Value: 0.000000
```
1. To test whether Amazon Managed Service for Prometheus received the metrics, use `awscurl`. This tool enables you to send HTTP requests through the command line with AWS Sigv4 authentication, so you must have AWS credentials set up locally with the correct permissions to query from Amazon Managed Service for Prometheus For instructions on installing `awscurl`, see [awscurl](https://github.com/okigan/awscurl).
In the following command, replace `AMP_REGION`, and `AMP_ENDPOINT` with the information for your Amazon Managed Service for Prometheus workspace.
```
awscurl --service="aps" --region="AMP_REGION" "https://AMP_ENDPOINT/api/v1/query?query=adot_test_gauge0"
{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"adot_test_gauge0"},"value":[1606512592.493,"16.87214000011479"]}]}}
```
If you receive a metric as the response, that means your pipeline setup has been successful and the metric has successfully propagated from the sample app into Amazon Managed Service for Prometheus.
**Cleaning up**
To clean up this demo, enter the following commands.
```
kubectl delete namespace aoc-prometheus-pipeline-demo
kubectl delete namespace adot-col
```
## Advanced configuration
The Prometheus Receiver supports the full set of Prometheus scraping and re-labeling configurations described in [Configuration ](https://prometheus.io/docs/prometheus/latest/configuration/configuration/) in the Prometheus documentation. You can paste these configurations directly into your ADOT Collector configurations.
The configuration for the Prometheus Receiver includes your service discovery, scraping configurations, and re-labeling configurations. The receiver configuration looks like the following.
```
receivers:
prometheus:
config:
[[Your Prometheus configuration]]
```
The following is an example configuration.
```
receivers:
prometheus:
config:
global:
scrape_interval: 1m
scrape_timeout: 10s
scrape_configs:
- job_name: kubernetes-service-endpoints
sample_limit: 10000
kubernetes_sd_configs:
- role: endpoints
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: true
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
```
If you have an existing Prometheus configuration, you must replace the `$` characters with `$$` to avoid having the values replaced with environment variables. \$1This is especially important for the replacement value of the relabel\$1configurations. For example, if you start with the following relabel\$1configuration:
```
relabel_configs:
- source_labels: [__meta_kubernetes_ingress_scheme,__address__,__meta_kubernetes_ingress_path]
regex: (.+);(.+);(.+)
replacement: ${1}://${2}${3}
target_label: __param_target
```
It would become the following:
```
relabel_configs:
- source_labels: [__meta_kubernetes_ingress_scheme,__address__,__meta_kubernetes_ingress_path]
regex: (.+);(.+);(.+)
replacement: $${1}://${2}${3}
target_label: __param_target
```
**Prometheus remote write exporter and Sigv4 authentication extension**
The configuration for the Prometheus Remote Write Exporter and Sigv4 Authentication Extension are simpler than the Prometheus receiver. At this stage in the pipeline, metrics have already been ingested, and we’re ready to export this data to Amazon Managed Service for Prometheus. The minimum requirement for a successful configuration to communicate with Amazon Managed Service for Prometheus is shown in the following example.
```
extensions:
sigv4auth:
service: "aps"
region: "user-region"
exporters:
prometheusremotewrite:
endpoint: "https://aws-managed-prometheus-endpoint/api/v1/remote_write"
auth:
authenticator: "sigv4auth"
```
This configuration sends an HTTPS request that is signed by AWS SigV4 using AWS credentials from the default AWS credentials chain. For more information, see [Configuring the AWS SDK for Go](https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html). You must specify the service to be `aps`.
Regardless of the method of deployment, the ADOT collector must have access to one of the listed options in the default AWS credentials chain. The Sigv4 Authentication Extension depends on the AWS SDK for Go and uses it to fetch credentials and authenticate. You must ensure that these credentials have remote write permissions for Amazon Managed Service for Prometheus.
# Set up metrics ingestion from Amazon ECS using AWS Distro for Open Telemetry
This section explains how to collect metrics from Amazon Elastic Container Service (Amazon ECS) and ingest them into Amazon Managed Service for Prometheus using AWS Distro for Open Telemetry (ADOT). It also describes how to visualize your metrics in Amazon Managed Grafana.
## Prerequisites
**Important**
Before you begin, you must have an Amazon ECS environment on an AWS Fargate cluster with default settings, an Amazon Managed Service for Prometheus workspace, and an Amazon Managed Grafana workspace. We assume that you are familiar with container workloads, Amazon Managed Service for Prometheus, and Amazon Managed Grafana.
For more information, see the following links:
+ For information about how to create an Amazon ECS environment on a Fargate cluster with default settings, see [Creating a cluster](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/create_cluster.html) in the *Amazon ECS Developer Guide*.
+ For information about how to create an Amazon Managed Service for Prometheus workspace, see [Create a workspace](https://docs.aws.amazon.com/prometheus/latest/userguide/AMP-onboard-create-workspace.html) in the *Amazon Managed Service for Prometheus User Guide*.
+ For information about how to create an Amazon Managed Grafana workspace, see [Creating a workspace](https://docs.aws.amazon.com/grafana/latest/userguide/AMG-create-workspace.html) in the *Amazon Managed Grafana User Guide*.
## Step 1: Define a custom ADOT collector container image
Use the following config file as a template to define your own ADOT collector container image. Replace *my-remote-URL* and *my-region* with your `endpoint` and `region` values. Save the config in a file called *adot-config.yaml*.
**Note**
This configuration uses the `sigv4auth` extension to authenticate calls to Amazon Managed Service for Prometheus. For more information about configuring `sigv4auth`, see [Authenticator - Sigv4](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/sigv4authextension) on GitHub.
```
receivers:
prometheus:
config:
global:
scrape_interval: 15s
scrape_timeout: 10s
scrape_configs:
- job_name: "prometheus"
static_configs:
- targets: [ 0.0.0.0:9090 ]
awsecscontainermetrics:
collection_interval: 10s
processors:
filter:
metrics:
include:
match_type: strict
metric_names:
- ecs.task.memory.utilized
- ecs.task.memory.reserved
- ecs.task.cpu.utilized
- ecs.task.cpu.reserved
- ecs.task.network.rate.rx
- ecs.task.network.rate.tx
- ecs.task.storage.read_bytes
- ecs.task.storage.write_bytes
exporters:
prometheusremotewrite:
endpoint: my-remote-URL
auth:
authenticator: sigv4auth
logging:
loglevel: info
extensions:
health_check:
pprof:
endpoint: :1888
zpages:
endpoint: :55679
sigv4auth:
region: my-region
service: aps
service:
extensions: [pprof, zpages, health_check, sigv4auth]
pipelines:
metrics:
receivers: [prometheus]
exporters: [logging, prometheusremotewrite]
metrics/ecs:
receivers: [awsecscontainermetrics]
processors: [filter]
exporters: [logging, prometheusremotewrite]
```
## Step 2: Push your ADOT collector container image to an Amazon ECR repository
Use a Dockerfile to create and push your container image to an Amazon Elastic Container Registry (ECR) repository.
1. Build the Dockerfile to copy and add your container image to the OTEL Docker image.
```
FROM public.ecr.aws/aws-observability/aws-otel-collector:latest
COPY adot-config.yaml /etc/ecs/otel-config.yaml
CMD ["--config=/etc/ecs/otel-config.yaml"]
```
1. Create an Amazon ECR repository.
```
# create repo:
COLLECTOR_REPOSITORY=$(aws ecr create-repository --repository aws-otel-collector \
--query repository.repositoryUri --output text)
```
1. Create your container image.
```
# build ADOT collector image:
docker build -t $COLLECTOR_REPOSITORY:ecs .
```
**Note**
This assumes you are building your container in the same environment that it will run in. If not, you may need to use the `--platform` parameter when building the image.
1. Sign in to the Amazon ECR repository. Replace *my-region* with your `region` value.
```
# sign in to repo:
aws ecr get-login-password --region my-region | \
docker login --username AWS --password-stdin $COLLECTOR_REPOSITORY
```
1. Push your container image.
```
# push ADOT collector image:
docker push $COLLECTOR_REPOSITORY:ecs
```
## Step 3: Create an Amazon ECS task definition to scrape Amazon Managed Service for Prometheus
Create an Amazon ECS task definition to scrape Amazon Managed Service for Prometheus. Your task definition should include a container named `adot-collector` and a container named `prometheus`. `prometheus` generates metrics, and `adot-collector` scrapes `prometheus`.
**Note**
Amazon Managed Service for Prometheus runs as a service, collecting metrics from containers. The containers in this case run Prometheus locally, in Agent mode, which send the local metrics to Amazon Managed Service for Prometheus.
**Example: Task definition**
The following is an example of how your task definition might look. You can use this example as a template to create your own task definition. Replace the `image` value of `adot-collector` with your repository URL and image tag (`$COLLECTOR_REPOSITORY:ecs`). Replace the `region` values of `adot-collector` and `prometheus` with your `region` values.
```
{
"family": "adot-prom",
"networkMode": "awsvpc",
"containerDefinitions": [
{
"name": "adot-collector",
"image": "account_id.dkr.ecr.region.amazonaws.com/image-tag",
"essential": true,
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "/ecs/ecs-adot-collector",
"awslogs-region": "my-region",
"awslogs-stream-prefix": "ecs",
"awslogs-create-group": "True"
}
}
},
{
"name": "prometheus",
"image": "prom/prometheus:main",
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "/ecs/ecs-prom",
"awslogs-region": "my-region",
"awslogs-stream-prefix": "ecs",
"awslogs-create-group": "True"
}
}
}
],
"requiresCompatibilities": [
"FARGATE"
],
"cpu": "1024"
}
```
## Step 4: Give your task permissions to access Amazon Managed Service for Prometheus
To send the scraped metrics to Amazon Managed Service for Prometheus, your Amazon ECS task must have the correct permissions to call the AWS API operations for you. You must create an IAM role for your tasks and attach the `AmazonPrometheusRemoteWriteAccess` policy to it. For more information about creating this role and attaching the policy, see [Creating an IAM role and policy for your tasks](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html#create_task_iam_policy_and_role).
After you attach `AmazonPrometheusRemoteWriteAccess` to your IAM role, and use that role for your tasks, Amazon ECS can send your scraped metrics to Amazon Managed Service for Prometheus.
## Step 5: Visualize your metrics in Amazon Managed Grafana
**Important**
Before you begin, you must run a Fargate task on your Amazon ECS task definition. Otherwise, Amazon Managed Service for Prometheus can't consume your metrics.
1. From the navigation pane in your Amazon Managed Grafana workspace, choose **Data sources** under the AWS icon.
1. On the **Data sources** tab, for **Service**, select **Amazon Managed Service for Prometheus** and choose your **Default Region**.
1. Choose **Add data source**.
1. Use the `ecs` and `prometheus` prefixes to query and view your metrics.
# Set up metrics ingestion from an Amazon EC2 instance using remote write
This section explains how to run a Prometheus server with remote write in an Amazon Elastic Compute Cloud (Amazon EC2) instance. It explains how to collect metrics from a demo application written in Go and send them to an Amazon Managed Service for Prometheus workspace.
## Prerequisites
**Important**
Before you start, you must have installed Prometheus v2.26 or later. We assume that you're familiar with Prometheus, Amazon EC2, and Amazon Managed Service for Prometheus. For information about how to install Prometheus, see [Getting started](https://prometheus.io/docs/prometheus/latest/getting_started/) on the Prometheus website.
If you're unfamiliar with Amazon EC2 or Amazon Managed Service for Prometheus, we recommend that you start by reading the following sections:
+ [What is Amazon Elastic Compute Cloud?](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/concepts.html)
+ [What is Amazon Managed Service for Prometheus?](https://docs.aws.amazon.com/prometheus/latest/userguide/what-is-Amazon-Managed-Service-Prometheus.html)
## Create an IAM role for Amazon EC2
To stream metrics, you must first create an IAM role with the AWS managed policy **AmazonPrometheusRemoteWriteAccess**. Then, you can launch an instance with the role and stream metrics into your Amazon Managed Service for Prometheus workspace.
1. Open the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/).
1. From the navigation pane, choose **Roles**, and then choose **Create role**.
1. For the type of trusted entity, choose **AWS service**. For the use case, choose **EC2**. Choose **Next: Permissions**.
1. In the search bar, enter **AmazonPrometheusRemoteWriteAccess**. For **Policy name**, select **AmazonPrometheusRemoteWriteAccess**, and then choose **Attach policy**. Choose **Next:Tags**.
1. (Optional) Create IAM tags for your IAM role. Choose **Next: Review**.
1. Enter a name for your role. Choose **Create policy**.
## Launch an Amazon EC2 instance
To launch an Amazon EC2 instance, follow the instructions at [Launch an instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html#launch-instance-with-role) in the *Amazon Elastic Compute Cloud User Guide for Linux Instances*.
## Run the demo application
After creating your IAM role, and launching an EC2 instance with the role, you can run a demo application to see it work.
**To run a demo application and test metrics**
1. Use the following template to create a Go file named `main.go`.
```
package main
import (
"github.com/prometheus/client_golang/prometheus/promhttp"
"net/http"
)
func main() {
http.Handle("/metrics", promhttp.Handler())
http.ListenAndServe(":8000", nil)
}
```
1. Run the following commands to install the correct dependencies.
```
sudo yum update -y
sudo yum install -y golang
go get github.com/prometheus/client_golang/prometheus/promhttp
```
1. Run the demo application.
```
go run main.go
```
The demo application should run on port 8000 and show all of the exposed Prometheus metrics. The following is an example of these metrics.
```
curl -s http://localhost:8000/metrics
...
process_max_fds 4096# HELP process_open_fds Number of open file descriptors.# TYPE process_open_fds gauge
process_open_fds 10# HELP process_resident_memory_bytes Resident memory size in bytes.# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 1.0657792e+07# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.61131955899e+09# HELP process_virtual_memory_bytes Virtual memory size in bytes.# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 7.77281536e+08# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.# TYPE process_virtual_memory_max_bytes gauge
process_virtual_memory_max_bytes -1# HELP promhttp_metric_handler_requests_in_flight Current number of scrapes being served.# TYPE promhttp_metric_handler_requests_in_flight gauge
promhttp_metric_handler_requests_in_flight 1# HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.# TYPE promhttp_metric_handler_requests_total counter
promhttp_metric_handler_requests_total{code="200"} 1
promhttp_metric_handler_requests_total{code="500"} 0
promhttp_metric_handler_requests_total{code="503"} 0
```
## Create an Amazon Managed Service for Prometheus workspace
To create an Amazon Managed Service for Prometheus workspace, follow the instructions at [Create a workspace](AMP-create-workspace.md).
## Run a Prometheus server
1. Use the following example YAML file as a template to create a new file named `prometheus.yaml`. For `url`, replace *my-region* with your Region value and *my-workspace-id* with the workspace ID that Amazon Managed Service for Prometheus generated for you. For `region`, replace *my-region* with your Region value.
**Example: YAML file**
```
global:
scrape_interval: 15s
external_labels:
monitor: 'prometheus'
scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:8000']
remote_write:
-
url: https://aps-workspaces.my-region.amazonaws.com/workspaces/my-workspace-id/api/v1/remote_write
queue_config:
max_samples_per_send: 1000
max_shards: 200
capacity: 2500
sigv4:
region: my-region
```
1. Run the Prometheus server to send the demo application’s metrics to your Amazon Managed Service for Prometheus workspace.
```
prometheus --config.file=prometheus.yaml
```
The Prometheus server should now send the demo application’s metrics to your Amazon Managed Service for Prometheus workspace.
# Using a Prometheus instance as a collector
You can use a Prometheus instance, running in *agent* mode (known as a *Prometheus agent*), to scrape metrics and send them to your Amazon Managed Service for Prometheus workspace.
The following topics describe different ways to set up a Prometheus instance running in agent mode as a collector for your metrics.
**Warning**
When you create a Prometheus agent, you are responsible for its configuration and maintenance. Avoid exposing Prometheus scrape endpoints to the public internet by [enabling security features](https://prometheus.io/docs/prometheus/latest/configuration/https/).
If you set up multiple Prometheus instances that monitor the same set of metrics and sent them to a single Amazon Managed Service for Prometheus workspace for high availability, you need to set up deduplication. If you don't follow the steps to set up deduplication, you will be charged for all data samples sent to Amazon Managed Service for Prometheus, including duplicate samples. For instructions about setting up deduplication, see [Deduplicating high availability metrics sent to Amazon Managed Service for Prometheus](AMP-ingest-dedupe.md).
**Topics**
+ [Set up ingestion from a new Prometheus server using Helm](AMP-onboard-ingest-metrics-new-Prometheus.md)
+ [Set up ingestion from an existing Prometheus server in Kubernetes on EC2](AMP-onboard-ingest-metrics-existing-Prometheus.md)
+ [Set up ingestion from an existing Prometheus server in Kubernetes on Fargate](AMP-onboard-ingest-metrics-existing-Prometheus-fargate.md)
# Set up ingestion from a new Prometheus server using Helm
The instructions in this section get you up and running with Amazon Managed Service for Prometheus quickly. You set up a new Prometheus server in an Amazon EKS cluster, and the new server uses a default configuration to send metrics to Amazon Managed Service for Prometheus. This method has the following prerequisites:
+ You must have an Amazon EKS cluster from which the new Prometheus server will collect metrics.
+ Your Amazon EKS cluster must have an [Amazon EBS CSI driver](https://docs.aws.amazon.com/eks/latest/userguide/ebs-csi.html) installed (required by Helm).
+ You must use Helm CLI 3.0 or later.
+ You must use a Linux or macOS computer to perform the steps in the following sections.
## Step 1: Add new Helm chart repositories
To add new Helm chart repositories, enter the following commands. For more information about these commands, see [Helm Repo](https://helm.sh/docs/helm/helm_repo/).
```
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add kube-state-metrics https://kubernetes.github.io/kube-state-metrics
helm repo update
```
## Step 2: Create a Prometheus namespace
Enter the following command to create a Prometheus namespace for the Prometheus server and other monitoring components. Replace *prometheus-namespace* with the name that you want for this namespace.
```
kubectl create namespace prometheus-namespace
```
## Step 3: Set up IAM roles for service accounts
For the method of onboarding that we are documenting, you need to use IAM roles for service accounts in the Amazon EKS cluster where the Prometheus server is running.
With IAM roles for service accounts, you can associate an IAM role with a Kubernetes service account. This service account can then provide AWS permissions to the containers in any pod that uses that service account. For more information, see [IAM roles for service accounts](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html).
If you have not already set up these roles, follow the instructions at [Set up service roles for the ingestion of metrics from Amazon EKS clusters](set-up-irsa.md#set-up-irsa-ingest) to set up the roles. The instructions in that section require the use of `eksctl`. For more information, see [Getting started with Amazon Elastic Kubernetes Service – `eksctl`](https://docs.aws.amazon.com/eks/latest/userguide/getting-started-eksctl.html).
**Note**
When you are not on EKS or AWS and using just access key and secret key to access Amazon Managed Service for Prometheus, you cannot use the `EKS-IAM-ROLE` based SigV4.
## Step 4: Set up the new server and start ingesting metrics
To install the new Prometheus server that sends metrics to your Amazon Managed Service for Prometheus workspace, follow these steps.
**To install a new Prometheus server to send metrics to your Amazon Managed Service for Prometheus workspace**
1. Use a text editor to create a file named `my_prometheus_values_yaml` with the following content.
+ Replace *IAM\$1PROXY\$1PROMETHEUS\$1ROLE\$1ARN* with the ARN of the **amp-iamproxy-ingest-role** that you created in [Set up service roles for the ingestion of metrics from Amazon EKS clusters](set-up-irsa.md#set-up-irsa-ingest).
+ Replace *WORKSPACE\$1ID* with the ID of your Amazon Managed Service for Prometheus workspace.
+ Replace *REGION* with the Region of your Amazon Managed Service for Prometheus workspace.
```
## The following is a set of default values for prometheus server helm chart which enable remoteWrite to AMP
## For the rest of prometheus helm chart values see: https://github.com/prometheus-community/helm-charts/blob/main/charts/prometheus/values.yaml
##
serviceAccounts:
server:
name: amp-iamproxy-ingest-service-account
annotations:
eks.amazonaws.com/role-arn: ${IAM_PROXY_PROMETHEUS_ROLE_ARN}
server:
remoteWrite:
- url: https://aps-workspaces.${REGION}.amazonaws.com/workspaces/${WORKSPACE_ID}/api/v1/remote_write
sigv4:
region: ${REGION}
queue_config:
max_samples_per_send: 1000
max_shards: 200
capacity: 2500
```
1. Enter the following command to create the Prometheus server.
+ Replace *prometheus-chart-name* with your Prometheus release name.
+ Replace *prometheus-namespace* with the name of your Prometheus namespace.
```
helm install prometheus-chart-name prometheus-community/prometheus -n prometheus-namespace \
-f my_prometheus_values_yaml
```
**Note**
You can customize the `helm install` command in many ways. For more information, see [Helm install](https://helm.sh/docs/helm/helm_install/) in the *Helm documentation*.
# Set up ingestion from an existing Prometheus server in Kubernetes on EC2
Amazon Managed Service for Prometheus supports ingesting metrics from Prometheus servers in clusters running Amazon EKS and in self-managed Kubernetes clusters running on Amazon EC2. The detailed instructions in this section are for a Prometheus server in an Amazon EKS cluster. The steps for a self-managed Kubernetes cluster on Amazon EC2 are the same, except that you will need to set up the OIDC provider and IAM roles for service accounts yourself in the Kubernetes cluster.
The instructions in this section use Helm as the Kubernetes package manager.
**Topics**
+ [Step 1: Set up IAM roles for service accounts](#AMP-onboard-existing-Prometheus-IRSA)
+ [Step 2: Upgrade your existing Prometheus server using Helm](#AMP-onboard-ingest-metrics-existing-remotewrite)
## Step 1: Set up IAM roles for service accounts
For the method of onboarding that we are documenting, you need to use IAM roles for service accounts in the Amazon EKS cluster where the Prometheus server is running. These roles are also called *service roles*.
With service roles, you can associate an IAM role with a Kubernetes service account. This service account can then provide AWS permissions to the containers in any pod that uses that service account. For more information, see [IAM roles for service accounts](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html).
If you have not already set up these roles, follow the instructions at [Set up service roles for the ingestion of metrics from Amazon EKS clusters](set-up-irsa.md#set-up-irsa-ingest) to set up the roles.
## Step 2: Upgrade your existing Prometheus server using Helm
The instructions in this section include setting up remote write and sigv4 to authenticate and authorize the Prometheus server to remote write to your Amazon Managed Service for Prometheus workspace.
### Using Prometheus version 2.26.0 or later
Follow these steps if you are using a Helm chart with Prometheus Server image of version 2.26.0 or later.
**To set up remote write from a Prometheus server using Helm chart**
1. Create a new remote write section in your Helm configuration file:
+ Replace `${IAM_PROXY_PROMETHEUS_ROLE_ARN}` with the ARN of the **amp-iamproxy-ingest-role** that you created in [Step 1: Set up IAM roles for service accounts](#AMP-onboard-existing-Prometheus-IRSA). The role ARN should have the format of `arn:aws:iam::your account ID:role/amp-iamproxy-ingest-role`.
+ Replace `${WORKSPACE_ID}` with your Amazon Managed Service for Prometheus workspace ID.
+ Replace `${REGION}` with the Region of the Amazon Managed Service for Prometheus workspace (such as `us-west-2`).
```
## The following is a set of default values for prometheus server helm chart which enable remoteWrite to AMP
## For the rest of prometheus helm chart values see: https://github.com/prometheus-community/helm-charts/blob/main/charts/prometheus/values.yaml
##
serviceAccounts:
server:
name: amp-iamproxy-ingest-service-account
annotations:
eks.amazonaws.com/role-arn: ${IAM_PROXY_PROMETHEUS_ROLE_ARN}
server:
remoteWrite:
- url: https://aps-workspaces.${REGION}.amazonaws.com/workspaces/${WORKSPACE_ID}/api/v1/remote_write
sigv4:
region: ${REGION}
queue_config:
max_samples_per_send: 1000
max_shards: 200
capacity: 2500
```
1. Update your existing Prometheus Server configuration using Helm:
+ Replace `prometheus-chart-name` with your Prometheus release name.
+ Replace `prometheus-namespace` with the Kubernetes namespace where your Prometheus Server is installed.
+ Replace `my_prometheus_values_yaml` with the path to your Helm configuration file.
+ Replace `current_helm_chart_version` with the current version of your Prometheus Server Helm chart. You can find the current chart version by using the [helm list](https://helm.sh/docs/helm/helm_list/) command.
```
helm upgrade prometheus-chart-name prometheus-community/prometheus \
-n prometheus-namespace \
-f my_prometheus_values_yaml \
--version current_helm_chart_version
```
### Using earlier versions of Prometheus
Follow these steps if you are using a version of Prometheus earlier than 2.26.0. These steps use a sidecar approach, because earlier versions of Prometheus don't natively support AWS Signature Version 4 signing process (AWS SigV4).
These instructions assume that you are using Helm to deploy Prometheus.
**To set up remote write from a Prometheus server**
1. On your Prometheus server, create a new remote write configuration. First, create a new update file. We will call the file `amp_ingest_override_values.yaml`.
Add the following values to the YAML file.
```
serviceAccounts:
server:
name: "amp-iamproxy-ingest-service-account"
annotations:
eks.amazonaws.com/role-arn: "${SERVICE_ACCOUNT_IAM_INGEST_ROLE_ARN}"
server:
sidecarContainers:
- name: aws-sigv4-proxy-sidecar
image: public.ecr.aws/aws-observability/aws-sigv4-proxy:1.0
args:
- --name
- aps
- --region
- ${REGION}
- --host
- aps-workspaces.${REGION}.amazonaws.com
- --port
- :8005
ports:
- name: aws-sigv4-proxy
containerPort: 8005
statefulSet:
enabled: "true"
remoteWrite:
- url: http://localhost:8005/workspaces/${WORKSPACE_ID}/api/v1/remote_write
```
Replace `${REGION}` with the Region of the Amazon Managed Service for Prometheus workspace.
Replace `${SERVICE_ACCOUNT_IAM_INGEST_ROLE_ARN}` with the ARN of the **amp-iamproxy-ingest-role** that you created in [Step 1: Set up IAM roles for service accounts](#AMP-onboard-existing-Prometheus-IRSA). The role ARN should have the format of `arn:aws:iam::your account ID:role/amp-iamproxy-ingest-role`.
Replace `${WORKSPACE_ID}` with your workspace ID.
1. Upgrade your Prometheus Helm chart. First, find your Helm chart name by entering the following command. In the output from this command, look for a chart with a name that includes `prometheus`.
```
helm ls --all-namespaces
```
Then enter the following command.
```
helm upgrade --install prometheus-helm-chart-name prometheus-community/prometheus -n prometheus-namespace -f ./amp_ingest_override_values.yaml
```
Replace *prometheus-helm-chart-name* with the name of the Prometheus helm chart returned in the previous command. Replace *prometheus-namespace* with the name of your namespace.
#### Downloading Helm charts
If you don't already have Helm charts downloaded locally, you can use the following command to download them.
```
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm pull prometheus-community/prometheus --untar
```
# Set up ingestion from an existing Prometheus server in Kubernetes on Fargate
Amazon Managed Service for Prometheus supports ingesting metrics from Prometheus servers in self-managed Kubernetes clusters running on Fargate. To ingest metrics from Prometheus servers in Amazon EKS clusters running on Fargate, override the default configs in a config file named amp\$1ingest\$1override\$1values.yaml as follows:
```
prometheus-node-exporter:
enabled: false
alertmanager:
enabled: false
serviceAccounts:
server:
name: amp-iamproxy-ingest-service-account
annotations:
eks.amazonaws.com/role-arn: ${IAM_PROXY_PROMETHEUS_ROLE_ARN}
server:
persistentVolume:
enabled: false
remoteWrite:
- url: https://aps-workspaces.${REGION}.amazonaws.com/workspaces/${WORKSPACE_ID}/api/v1/remote_write
sigv4:
region: ${REGION}
queue_config:
max_samples_per_send: 1000
max_shards: 200
capacity: 2500
```
Install Prometheus using the overrides with the following command:
```
helm install prometheus-for-amp prometheus-community/prometheus \
-n prometheus \
-f amp_ingest_override_values.yaml
```
Note that in the Helm chart configuration we disabled the node exporter and the alert manager as well as running the Prometheus server deployment.
You can verify the install with the following example test query.
```
$ awscurl --region region --service aps "https://aps-workspaces.region_id.amazonaws.com/workspaces/workspace_id/api/v1/query?query=prometheus_api_remote_read_queries"
{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"prometheus_api_remote_read_queries","instance":"localhost:9090","job":"prometheus"},"value":[1648461236.419,"0"]}]}}21
```
# Set up Amazon Managed Service for Prometheus for high availability data
When you send data to Amazon Managed Service for Prometheus, it is automatically replicated across AWS Availability Zones in the Region, and is served to you from a cluster of hosts that provide scalability, availability, and security. You might want to add additional high availability fail-safes, depending on your particular setup. There are two common ways that you might additional high availability safeties to your setup:
+ If you have multiple containers or instances that have the same data, you can send that data to Amazon Managed Service for Prometheus and have the data automatically de-duplicated. This helps to ensure that your data will be sent to your Amazon Managed Service for Prometheus workspace.
For more information about de-duplicating high-availability data, see [Deduplicating high availability metrics sent to Amazon Managed Service for Prometheus](AMP-ingest-dedupe.md).
+ If you want to ensure that you have access to your data, even when the AWS Region is not available, you can send your metrics to a second workspace, in another Region.
For more information about sending metrics data to multiple workspaces, see [Use cross Region workspaces to add high availability in Amazon Managed Service for Prometheus](AMP-send-to-multiple-workspaces.md).
**Topics**
+ [Deduplicating high availability metrics sent to Amazon Managed Service for Prometheus](AMP-ingest-dedupe.md)
+ [Send high availability data to Amazon Managed Service for Prometheus with Prometheus](Send-high-availability-data.md)
+ [Set up high availability data to Amazon Managed Service for Prometheus using the Prometheus Operator Helm chart](Send-high-availability-data-operator.md)
+ [Send high-availability data to Amazon Managed Service for Prometheus with AWS Distro for OpenTelemetry](Send-high-availability-data-ADOT.md)
+ [Send high availability data to Amazon Managed Service for Prometheus with the Prometheus community Helm chart](Send-high-availability-prom-community.md)
+ [Answers to common questions about high availability configuration in Amazon Managed Service for Prometheus](HA_FAQ.md)
+ [Use cross Region workspaces to add high availability in Amazon Managed Service for Prometheus](AMP-send-to-multiple-workspaces.md)
# Deduplicating high availability metrics sent to Amazon Managed Service for Prometheus
You can send data from multiple Prometheus *agents* (Prometheus instances running in Agent mode) to your Amazon Managed Service for Prometheus workspace. If some of these instances are recording and sending the same metrics, your data will have a higher availability (even if one of the agents stops sending data, the Amazon Managed Service for Prometheus workspace will still receive the data from another instance). However, you want your Amazon Managed Service for Prometheus workspace to automatically de-duplicate the metrics so that you don't see the metrics multiple times, and aren't charged for the data ingestion and storage multiple times.
For Amazon Managed Service for Prometheus to automatically de-duplicate data from multiple Prometheus agents, you give the set of agents that are sending the duplicate data a single *cluster name*, and each of the instances a *replica name*. The cluster name identifies the instances as having shared data, and the replica name allows Amazon Managed Service for Prometheus to identify the source of each metric. The final stored metrics include the cluster label, but not the replica, so the metrics appear to be coming from a single source.
**Note**
Certain versions of Kubernetes (1.28 and 1.29) may emit their own metric with a `cluster` label. This can cause issues with Amazon Managed Service for Prometheus deduplication. See the [High availability FAQ](HA_FAQ.md#HA_FAQ_cluster-label) for more information.
The following topics show how to send data and include the `cluster` and `__replica__` labels, so that Amazon Managed Service for Prometheus de-duplicates the data automatically.
**Important**
If you do not set up deduplication, you will be charged for all data samples that are sent to Amazon Managed Service for Prometheus. These data samples include duplicate samples.
# Send high availability data to Amazon Managed Service for Prometheus with Prometheus
To set up a high availability configuration with Prometheus, you must apply external labels on all instances of a high availability group, so Amazon Managed Service for Prometheus can identify them. Use the `cluster` label to identify a Prometheus instance agent as part of a high availability group. Use the `__replica__` label to identify each replica in the group separately. You need to apply both `__replica__` and `cluster` labels for de-duplication to work.
**Note**
The `__replica__` label is formatted with two underscore symbols before and after the word `replica`.
**Example: code snippets**
In the following code snippets, the `cluster` label identifies the Prometheus instance agent `prom-team1`, and the `_replica_` label identifies the replicas `replica1` and `replica2`.
```
cluster: prom-team1
__replica__: replica1
```
```
cluster: prom-team1
__replica__: replica2
```
As Amazon Managed Service for Prometheus stores data samples from high availability replicas with these labels, it strips the `replica` label when the samples are accepted. This means that you will only have a 1:1 series mapping for your current series instead of a series per replica. The `cluster` label is kept.
**Note**
Certain versions of Kubernetes (1.28 and 1.29) may emit their own metric with a `cluster` label. This can cause issues with Amazon Managed Service for Prometheus deduplication. See the [High availability FAQ](HA_FAQ.md#HA_FAQ_cluster-label) for more information.
# Set up high availability data to Amazon Managed Service for Prometheus using the Prometheus Operator Helm chart
To set up a high availability configuration with the Prometheus Operator in Helm, you must apply external labels on all instances of a high availability group, so Amazon Managed Service for Prometheus can identify them. You also must set the attributes `replicaExternalLabelName` and `externalLabels` on the Prometheus Operator Helm chart.
**Example: YAML header**
In the following YAML header, `cluster` is added to `externalLabel` to identify a Prometheus instance agent as part of a high-availability group, and `replicaExternalLabels` identifies each replica in the group.
```
replicaExternalLabelName: __replica__
externalLabels:
cluster: prom-dev
```
**Note**
Certain versions of Kubernetes (1.28 and 1.29) may emit their own metric with a `cluster` label. This can cause issues with Amazon Managed Service for Prometheus deduplication. See the [High availability FAQ](HA_FAQ.md#HA_FAQ_cluster-label) for more information.
# Send high-availability data to Amazon Managed Service for Prometheus with AWS Distro for OpenTelemetry
AWS Distro for OpenTelemetry (ADOT) is a secure and production-ready distribution of the OpenTelemetry project. ADOT provides you with source APIs, libraries, and agents, so you can collect distributed traces and metrics for application monitoring. For information about ADOT, see [ About AWS Distro for Open Telemetry](https://aws-otel.github.io/about).
To set up ADOT with a high availability configuration, you must configure an ADOT collector container image and apply the external labels `cluster` and `__replica__` to the AWS Prometheus remote write exporter. This exporter sends your scraped metrics to your Amazon Managed Service for Prometheus workspace via the `remote_write` endpoint. When you set these labels on the remote write exporter, you prevent duplicate metrics from being kept while redundant replicas run. For more information about the AWS Prometheus remote write exporter, see [Getting started with Prometheus remote write exporter for Amazon Managed Service for Prometheus](https://aws-otel.github.io/docs/getting-started/prometheus-remote-write-exporter).
**Note**
Certain versions of Kubernetes (1.28 and 1.29) may emit their own metric with a `cluster` label. This can cause issues with Amazon Managed Service for Prometheus deduplication. See the [High availability FAQ](HA_FAQ.md#HA_FAQ_cluster-label) for more information.
# Send high availability data to Amazon Managed Service for Prometheus with the Prometheus community Helm chart
To set up a high availability configuration with the Prometheus community Helm chart, you must apply external labels on all instances of a high availability group, so Amazon Managed Service for Prometheus can identify them. Here is an example of how you could add the `external_labels` to a single instance of Prometheus from the Prometheus community Helm chart.
```
server:
global:
external_labels:
cluster: monitoring-cluster
__replica__: replica-1
```
**Note**
If you want multiple replicas, you have to deploy the chart multiple times with different replica values, because the Prometheus community Helm chart does not let you dynamically set the replica value when increasing the number of replicas directly from the controller group. If you prefer to have the `replica` label auto-set, use the prometheus-operator Helm chart.
**Note**
Certain versions of Kubernetes (1.28 and 1.29) may emit their own metric with a `cluster` label. This can cause issues with Amazon Managed Service for Prometheus deduplication. See the [High availability FAQ](HA_FAQ.md#HA_FAQ_cluster-label) for more information.
# Answers to common questions about high availability configuration in Amazon Managed Service for Prometheus
## Should I include the value *\$1\$1replica\$1\$1* into another label to track the sample points?
In a high availability setting, Amazon Managed Service for Prometheus ensures data samples are not duplicated by electing a leader in the cluster of Prometheus instances. If the leader replica stops sending data samples for 30 seconds, Amazon Managed Service for Prometheus automatically makes another Prometheus instance a leader replica and ingests data from the new leader, including any missed data. Therefore, the answer is no, it is not recommended. Doing so may cause issues like:
+ Querying a `count` in **PromQL** may return higher than expected value during the period of electing a new leader.
+ The number of `active series` gets increased during a period of electing a new leader and it reaches the `active series limits`. See [ AMP Quotas ](https://docs.aws.amazon.com/prometheus/latest/userguide/AMP_quotas.html) for more info.
## Kubernetes seems to have it's own *cluster* label, and is not deduplicating my metrics. How can I fix this?
A new metric, `apiserver_storage_size_bytes` was introduced in Kubernetes 1.28, with a `cluster` label. This can cause issues with deduplication in Amazon Managed Service for Prometheus, which depends on the `cluster` label. In Kubernetes 1.3, the label is renamed to `storage-cluster_id` (it is also renamed in later patches of 1.28 and 1.29). If your cluster is emitting this metric with the `cluster` label, Amazon Managed Service for Prometheus can't dedupe the associated time series. We recommend you upgrade your Kubernetes cluster to the latest patched version to avoid this problem. Alternately, you can relabel the `cluster` label on your `apiserver_storage_size_bytes` metric before ingesting it into Amazon Managed Service for Prometheus.
**Note**
For more details about the change to Kubernetes, see [Rename Label cluster to storage\$1cluster\$1id for apiserver\$1storage\$1size\$1bytes metric](https://github.com/kubernetes/kubernetes/pull/124283) in the *Kubernetes GitHub project*.
# Use cross Region workspaces to add high availability in Amazon Managed Service for Prometheus
To add cross-Region availability to your data, you can send metrics to multiple workspaces across AWS Regions. Prometheus supports both multiple writers and cross-Region writing.
The following example shows how to set up a Prometheus server running in Agent mode to send metrics to two workspaces in different Regions with Helm.
```
extensions:
sigv4auth:
service: "aps"
receivers:
prometheus:
config:
scrape_configs:
- job_name: 'kubernetes-kubelet'
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: true
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
- role: node
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- target_label: __address__
replacement: kubernetes.default.svc.cluster.local:443
- source_labels: [__meta_kubernetes_node_name]
regex: (.+)
target_label: __metrics_path__
replacement: /api/v1/nodes/$${1}/proxy/metrics
exporters:
prometheusremotewrite/one:
endpoint: "https://aps-workspaces.workspace_1_region.amazonaws.com/workspaces/ws-workspace_1_id/api/v1/remote_write"
auth:
authenticator: sigv4auth
prometheusremotewrite/two:
endpoint: "https://aps-workspaces.workspace_2_region.amazonaws.com/workspaces/ws-workspace_2_id/api/v1/remote_write"
auth:
authenticator: sigv4auth
service:
extensions: [sigv4auth]
pipelines:
metrics/one:
receivers: [prometheus]
exporters: [prometheusremotewrite/one]
metrics/two:
receivers: [prometheus]
exporters: [prometheusremotewrite/two]
```