Using an OpenSearch Ingestion pipeline with Amazon Managed Service for Prometheus

You can use Amazon Managed Service for Prometheus as a destination for your OpenSearch Ingestion pipeline to store metrics in time series format. The Prometheus sink allows you to send OpenTelemetry metrics or other time series data from your pipeline to an Amazon Managed Service for Prometheus workspace for monitoring, alerting, and analysis.

The prometheus sink plugin enables OpenSearch Ingestion pipelines to write metrics data to Amazon Managed Service for Prometheus workspaces using the Prometheus remote write protocol. This integration allows you to:

Prerequisites

Before you configure the Prometheus sink, ensure you have the following:

Step 1: Configure the pipeline role

The Prometheus sink automatically inherits the pipeline role's IAM permissions for authentication, so no additional role configuration (like sts_role_arn) is required in the sink settings.

The following sample policy shows the required permissions for using Amazon Managed Service for Prometheus as a sink:

{
  "Version": "2012-10-17",		 	 	 
  "Statement": [
    {
      "Sid": "AMPRemoteWrite",
      "Effect": "Allow",
      "Action": [
        "aps:RemoteWrite"
      ],
      "Resource": "arn:aws:aps:region:account-id:workspace/workspace-id"
    }
  ]
}

Replace the following placeholders:

You must attach these permissions to your pipeline role.

Ensure your pipeline role has a trust relationship that allows OpenSearch Ingestion to assume it:

{
  "Version": "2012-10-17",		 	 	 
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "osis-pipelines.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Step 2: Create the pipeline

After you've set up your permissions, you can configure an OpenSearch Ingestion pipeline to use Amazon Managed Service for Prometheus as a sink.

Basic configuration

The following example shows a minimal Prometheus sink configuration:

version: "2"
sink:
  - prometheus:
      url: "https://aps-workspaces.region.amazonaws.com/workspaces/workspace-id/api/v1/remote_write"
      aws:
        region: "region"

You must specify the url option within the prometheus sink configuration, which is the Amazon Managed Service for Prometheus remote write endpoint. To format the URL, locate your workspace ID in the Amazon Managed Service for Prometheus console and construct the URL as follows: https://aps-workspaces.region.amazonaws.com/workspaces/workspace-id/api/v1/remote_write.

Configuration options

Use the following options to configure batching and flushing behavior for the Prometheus sink:

Example pipelines

Example 1: OpenTelemetry metrics to Amazon Managed Service for Prometheus

This pipeline receives OpenTelemetry metrics from an external agent and writes them to Amazon Managed Service for Prometheus:

version: "2"
source:
  otel_metrics_source:
    path: "/v1/metrics"
    output_format: otel

sink:
  - prometheus:
      url: "https://aps-workspaces.us-east-1.amazonaws.com/workspaces/ws-a1b2c3d4-5678-90ab-cdef-EXAMPLE11111/api/v1/remote_write"
      aws:
        region: "us-east-1"

Example 2: Dual sink - OpenSearch and Amazon Managed Service for Prometheus

This pipeline routes metrics to both OpenSearch and Amazon Managed Service for Prometheus:

version: "2"
source:
  otel_metrics_source:
    path: "/v1/metrics"
    output_format: otel

sink:
  - opensearch:
      hosts:
        - "https://search-domain-endpoint.us-east-1.es.amazonaws.com"
      index: "metrics-%{yyyy.MM.dd}"
      aws:
        region: "us-east-1"
        sts_role_arn: "arn:aws:iam::123456789012:role/OSI-Pipeline-Role"

  - prometheus:
      url: "https://aps-workspaces.us-east-1.amazonaws.com/workspaces/ws-a1b2c3d4-5678-90ab-cdef-EXAMPLE11111/api/v1/remote_write"
      aws:
        region: "us-east-1"

Example 3: Metrics with filtering

This pipeline filters metrics before sending to Amazon Managed Service for Prometheus:

version: "2"
source:
  otel_metrics_source:
    path: "/v1/metrics"
    output_format: otel

processor:
  - drop_events:
      drop_when: '/name != "http.server.duration" and /name != "http.client.duration"'

sink:
  - prometheus:
      url: "https://aps-workspaces.us-east-1.amazonaws.com/workspaces/ws-a1b2c3d4-5678-90ab-cdef-EXAMPLE11111/api/v1/remote_write"
      aws:
        region: "us-east-1"

You can use a preconfigured Amazon Managed Service for Prometheus blueprint to create these pipelines. For more information, see Working with blueprints.

Creating a pipeline with Amazon Managed Service for Prometheus sink

Using the AWS Console

Using the AWS CLI

Create a pipeline configuration file (for example, amp-pipeline.yaml) with your desired configuration, then run:

aws osis create-pipeline \
  --pipeline-name my-amp-pipeline \
  --min-units 2 \
  --max-units 4 \
  --pipeline-configuration-body file://amp-pipeline.yaml

Using AWS CloudFormation

Resources:
  MyAMPPipeline:
    Type: AWS::OSIS::Pipeline
    Properties:
      PipelineName: my-amp-pipeline
      MinUnits: 2
      MaxUnits: 4
      PipelineConfigurationBody: |
        version: "2"
        source:
          otel_metrics_source:
            path: "/v1/metrics"
            output_format: otel
        sink:
          - prometheus:
              url: "https://aps-workspaces.us-east-1.amazonaws.com/workspaces/ws-a1b2c3d4-5678-90ab-cdef-EXAMPLE11111/api/v1/remote_write"
              aws:
                region: "us-east-1"

Monitoring and troubleshooting

CloudWatch metrics

Monitor your pipeline's performance using CloudWatch metrics:

Common issues

Issue: Pipeline fails to write to Amazon Managed Service for Prometheus

Solutions:

Issue: Authentication errors

Solutions:

Issue: High latency or throttling

Solutions:

Limitations

Consider the following limitations when you set up an OpenSearch Ingestion pipeline for Amazon Managed Service for Prometheus:

Best practices