# Managing and forwarding alerts in Amazon Managed Service for Prometheus with alert manager When the [alerting rules](AMP-ruler-rulesfile.md) that Amazon Managed Service for Prometheus runs are firing, alert manager handles the alerts that are sent. It de-duplicates, groups, and routes the alerts to downstream receivers. Amazon Managed Service for Prometheus supports only Amazon Simple Notification Service as a receiver, and can route messages to Amazon SNS topics in the same account. You can also use alert manager to silence and inhibit alerts. Alert manager provides similar functionality to Alertmanager in Prometheus. You can use alert manager's configuration file for the following: + **Grouping** – Grouping collects similar alerts into a single notification. This is especially useful during larger outages when many systems fail at once and hundreds of alerts might fire simultaneously. For example, suppose that a network failure causes many of your nodes to fail at the same time. If these types of alerts are grouped, alert manager sends you a single notification. Alert grouping and the timing for the grouped notifications are configured by a routing tree in the alert manager configuration file. For more information, see [](https://prometheus.io/docs/alerting/latest/configuration/#route). + **Inhibition** – Inhibition suppresses notifications for certain alerts if certain other alerts are already firing. For example, if an alert is firing about a cluster being unreachable, you can configure alert manager to mute all other alerts concerning this cluster. This prevents notifications for hundreds or thousands of firing alerts that are unrelated to the actual issue. For more information about how to write inhibition rules, see [](https://prometheus.io/docs/alerting/latest/configuration/#inhibit_rule). + **Silences** – Silences mute alerts for a specified time, such as during a maintenance window. Incoming alerts are checked for whether they match all the equality or regular expression matchers of an active silence. If they do, no notifications are sent for that alert. To create a silence, you use the `PutAlertManagerSilences` API. For more information, see [PutAlertManagerSilences](AMP-APIReference-PutAlertManagerSilences.md). **Prometheus templating** Standalone Prometheus supports templating, using separate template files. Templates can use conditionals and format data, among other things. In Amazon Managed Service for Prometheus, you put your templating in the same alert manager configuration file as your [alert manager configuration](AMP-alertmanager-config.md). **Topics** + [Understanding IAM permissions needed for working with alert manager](AMP-alertmanager-IAM-permissions.md) + [Create an alert manager configuration in Amazon Managed Service for Prometheus to manage and route alerts](AMP-alertmanager-config.md) + [Forward alerts to an alert receiver with alert manager in Amazon Managed Service for Prometheus](AMP-alertmanager-receiver.md) + [Upload your alert manager configuration file to Amazon Managed Service for Prometheus](AMP-alertmanager-upload.md) + [Integrate alerts with Amazon Managed Grafana or open source Grafana](integrating-grafana.md) + [Troubleshoot alert manager with CloudWatch Logs](Troubleshooting-alerting.md) # Understanding IAM permissions needed for working with alert manager You must give users permissions to use alert manager in Amazon Managed Service for Prometheus. Create an AWS Identity and Access Management (IAM) policy with the following permissions, and assign the policy to your users, groups, or roles. # Create an alert manager configuration in Amazon Managed Service for Prometheus to manage and route alerts To use alert manager and templating in Amazon Managed Service for Prometheus, you create an alert manager configuration YAML file. An Amazon Managed Service for Prometheus alert manager file has two main sections: + `template_files:` contains the templates used for messages sent by receivers. For more information, see [Template Reference](https://prometheus.io/docs/prometheus/latest/configuration/template_reference/) and [Template Examples](https://prometheus.io/docs/prometheus/latest/configuration/template_examples/) in the Prometheus documentation. + `alertmanager_config:` contains the alert manager configuration. This uses the same structure as an alert manager config file in standalone Prometheus. For more information, see [Configuration](https://prometheus.io/docs/alerting/latest/configuration/) in the Alertmanager documentation. **Note** The `repeat_interval` configuration described in the Prometheus documentation above has an additional limitation in Amazon Managed Service for Prometheus. The maximum allowed value is five days. If you set it higher than five days, it will be treated as five days and notifications will be sent again after the five day period has passed. **Note** You can also edit the configuration file directly in the Amazon Managed Service for Prometheus console, but it must still follow the format specified here. For more information on uploading or editing a configuration file, see [Upload your alert manager configuration file to Amazon Managed Service for Prometheus](AMP-alertmanager-upload.md). In Amazon Managed Service for Prometheus, your alert manager configuration file must have all your alert manager configuration content inside of an `alertmanager_config` key at the root of the YAML file. The following is a basic example alert manager configuration file: ``` alertmanager_config: | route: receiver: 'default' receivers: - name: 'default' sns_configs: - topic_arn: arn:aws:sns:us-east-2:123456789012:My-Topic sigv4: region: us-east-2 attributes: key: key1 value: value1 ``` The only receiver currently supported is Amazon Simple Notification Service (Amazon SNS). If you have other types of receivers listed in the configuration, it will be rejected. Here is another sample alert manager configuration file that uses both the `template_files` block and the `alertmanager_config` block. ``` template_files: default_template: | {{ define "sns.default.subject" }}[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}]{{ end }} {{ define "__alertmanager" }}AlertManager{{ end }} {{ define "__alertmanagerURL" }}{{ .ExternalURL }}/#/alerts?receiver={{ .Receiver | urlquery }}{{ end }} alertmanager_config: | global: templates: - 'default_template' route: receiver: default receivers: - name: 'default' sns_configs: - topic_arn: arn:aws:sns:us-east-2:accountid:My-Topic sigv4: region: us-east-2 attributes: key: severity value: SEV2 ``` **Default Amazon SNS template block** The default Amazon SNS configuration uses the following template unless you explicitly override it. ``` {{ define "sns.default.message" }}{{ .CommonAnnotations.SortedPairs.Values | join " " }} {{ if gt (len .Alerts.Firing) 0 -}} Alerts Firing: {{ template "__text_alert_list" .Alerts.Firing }} {{- end }} {{ if gt (len .Alerts.Resolved) 0 -}} Alerts Resolved: {{ template "__text_alert_list" .Alerts.Resolved }} {{- end }} {{- end }} ``` # Forward alerts to an alert receiver with alert manager in Amazon Managed Service for Prometheus When an alert is raised by an alert rule, it is sent to alert manager. Alert manager performs functions such as de-duplicating alerts, inhibiting alerts during maintenance, or grouping them as needed. It then forwards the alert as a message to an *alert receiver*. You can set up an alert receiver that can notify operators, have automated responses, or respond to the alerts in other ways. You can configure Amazon Simple Notification Service (Amazon SNS) and PagerDuty as alert receivers in Amazon Managed Service for Prometheus. The following topics describe how to create and configure your alert receiver. **Topics** + [Use Amazon SNS as an alert receiver](AMP-alertmanager-receiver-createtopic.md) + [Use PagerDuty as an alert receiver](AMP-alertmanager-pagerduty.md) # Use Amazon SNS as an alert receiver You can use an existing Amazon SNS topic as an alert receiver for Amazon Managed Service for Prometheus, or you can create a new one. We recommend that you use a topic of the **Standard** type, so that you can forward alerts from the topic to email, SMS, or HTTP. To create a new Amazon SNS topic to use as your alert manager receiver, follow the steps in [ Step 1: Create a topic](https://docs.aws.amazon.com/sns/latest/dg/sns-getting-started.html#step-create-queue). Be sure to choose **Standard** for the topic type. If you want to receive emails every time a message is sent to that Amazon SNS topic, follow the steps in [ Step 2: Create a subscription to the topic](https://docs.aws.amazon.com/sns/latest/dg/sns-getting-started.html#step-send-message). Whether you use a new or existing Amazon SNS topic, you will need the Amazon Resource Name (ARN) of your Amazon SNS topic to complete the following tasks. **Topics** + [Giving Amazon Managed Service for Prometheus permission to send alert messages to your Amazon SNS topic](AMP-alertmanager-receiver-AMPpermission.md) + [Configure alert manager to send messages to your Amazon SNS topic](AMP-alertmanager-receiver-config.md) + [Configure alert manager to send messages to Amazon SNS as JSON](AMP-alertmanager-receiver-JSON.md) + [Configure Amazon SNS to send messages for alerts to other destinations](AMP-alertmanager-SNS-otherdestinations.md) + [Understanding Amazon SNS message validation rules](AMP-alertmanager-receiver-validation-truncation.md) # Giving Amazon Managed Service for Prometheus permission to send alert messages to your Amazon SNS topic You must give Amazon Managed Service for Prometheus permission to send messages to your Amazon SNS topic. The following policy statement will give that permission. It includes a `Condition` statement to help prevent a security problem known as the *confused deputy* problem. The `Condition` statement restricts access to the Amazon SNS topic to allow only operations coming from this specific account and Amazon Managed Service for Prometheus workspace. For more information about the confused deputy problem, see [Cross-service confused deputy prevention](#cross-service-confused-deputy-prevention). **To give Amazon Managed Service for Prometheus permission to send messages to your Amazon SNS topic** 1. Open the Amazon SNS console at [https://console.aws.amazon.com/sns/v3/home](https://console.aws.amazon.com/sns/v3/home). 1. In the navigation pane, choose **Topics**. 1. Choose the name of the topic that you are using with Amazon Managed Service for Prometheus. 1. Choose **Edit**. 1. Choose **Access policy** and add the following policy statement to the existing policy. ``` { "Sid": "Allow_Publish_Alarms", "Effect": "Allow", "Principal": { "Service": "aps.amazonaws.com" }, "Action": [ "sns:Publish", "sns:GetTopicAttributes" ], "Condition": { "ArnEquals": { "aws:SourceArn": "workspace_ARN" }, "StringEquals": { "AWS:SourceAccount": "account_id" } }, "Resource": "arn:aws:sns:region:account_id:topic_name" } ``` [Optional] If your Amazon SNS topic is service side encryption (SSE) enabled, you need to allow Amazon Managed Service for Prometheus to send messages to this encrypted topic by adding the `kms:GenerateDataKey*` and `kms:Decrypt` permissions to the AWS KMS key policy of the key used to encrypt the topic. For example, you could add the following to the policy: ``` { "Statement": [{ "Effect": "Allow", "Principal": { "Service": "aps.amazonaws.com" }, "Action": [ "kms:GenerateDataKey*", "kms:Decrypt" ], "Resource": "*" }] } ``` For more information, see [AWS KMS Permissions for SNS Topic](https://docs.aws.amazon.com/sns/latest/dg/sns-key-management.html#sns-what-permissions-for-sse). 1. Choose **Save changes**. **Note** By default, Amazon SNS creates the access policy with condition on `AWS:SourceOwner`. For more information, see [SNS Access Policy](https://docs.aws.amazon.com/sns/latest/dg/sns-access-policy-use-cases.html#source-account-versus-source-owner). **Note** IAM follows the [ Most-restrictive policy first ](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_evaluation-logic.html) rule. In your SNS topic, if there is a policy block that is more restrictive than the documented Amazon SNS policy block, the permission for the topic policy is not granted. To evaluate your policy and find out what's been granted, see [ Policy evaluation logic](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_evaluation-logic.html). ## SNS topic configuration for opt-in regions You can use `aps.amazonaws.com` to configure an Amazon SNS topic in the same AWS Region as your Amazon Managed Service for Prometheus workspace. To use an SNS topic from a non-opt-in Region (such as us-east-1) with an opt-in Region (such as af-south-1), you need to use the Regional service principal format. In the Regional service principle, replace *us-east-1* with the non-opt-in Region you want to use: **aps.*us-east-1*.amazonaws.com**. The following table lists the opt-in Regions and their corresponding Regional service principals: **Opt-in Regions and their Regional service principals** | Region name | Region | Regional service principal | | --- | --- | --- | | Africa (Cape Town) | af-south-1 | af-south-1.aps.amazonaws.com | | Asia Pacific (Hong Kong) | ap-east-1 | ap-east-1.aps.amazonaws.com | | Asia Pacific (Thailand) | ap-southeast-7 | ap-southeast-7.aps.amazonaws.com | | Europe (Milan) | eu-south-1 | eu-south-1.aps.amazonaws.com | | Europe (Zurich) | eu-central-2 | eu-central-2.aps.amazonaws.com | | Middle East (UAE) | me-central-1 | me-central-1.aps.amazonaws.com | | Asia Pacific (Malaysia) | ap-southeast-5 | ap-southeast-5.aps.amazonaws.com | For information on enabling an opt-in Region, see [Managing AWS Regions](https://docs.aws.amazon.com/accounts/latest/reference/manage-acct-regions.html) in the *IAM User Guide* in the Amazon Web Services General Reference. When configuring your Amazon SNS topic for these opt-in Regions, ensure you use the correct Regional service principal to enable cross-region delivery of alerts. ## Cross-service confused deputy prevention The confused deputy problem is a security issue where an entity that doesn't have permission to perform an action can coerce a more-privileged entity to perform the action. In AWS, cross-service impersonation can result in the confused deputy problem. Cross-service impersonation can occur when one service (the *calling service*) calls another service (the *called service*). The calling service can be manipulated to use its permissions to act on another customer's resources in a way it should not otherwise have permission to access. To prevent this, AWS provides tools that help you protect your data for all services with service principals that have been given access to resources in your account. We recommend using the [https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_condition-keys.html#condition-keys-sourcearn](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_condition-keys.html#condition-keys-sourcearn) and [https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_condition-keys.html#condition-keys-sourceaccount](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_condition-keys.html#condition-keys-sourceaccount) global condition context keys in resource policies to limit the permissions that Amazon Managed Service for Prometheus gives to Amazon SNS to the resource. If you use both global condition context keys, the `aws:SourceAccount` value and the account in the `aws:SourceArn` value must use the same account ID when used in the same policy statement. The value of `aws:SourceArn` must be the ARN of the Amazon Managed Service for Prometheus workspace. The most effective way to protect against the confused deputy problem is to use the `aws:SourceArn` global condition context key with the full ARN of the resource. If you don't know the full ARN of the resource or if you are specifying multiple resources, use the `aws:SourceArn` global context condition key with wildcards (`*`) for the unknown portions of the ARN. For example, `arn:aws:servicename::123456789012:*`. The policy shown in [Giving Amazon Managed Service for Prometheus permission to send alert messages to your Amazon SNS topic](#AMP-alertmanager-receiver-AMPpermission) shows how you can use the `aws:SourceArn` and `aws:SourceAccount` global condition context keys in Amazon Managed Service for Prometheus to prevent the confused deputy problem. # Configure alert manager to send messages to your Amazon SNS topic After you have a (new or existing) **Standard** type Amazon SNS topic, you can add it to your alert manager configuration as an alert receiver. Alert manager can forward your alerts to a configured alert receiver. To complete this, you must know the Amazon Resource Name (ARN) of your Amazon SNS topic. For more information about Amazon SNS receiver configuration, see [](https://prometheus.io/docs/alerting/latest/configuration/#sns_configs) in the Prometheus configuration documentation. **Unsupported properties** Amazon Managed Service for Prometheus supports Amazon SNS as the alert receiver. However, because of service constraints, not all of the properties of the Amazon SNS receiver are supported. The following properties are not allowed in an Amazon Managed Service for Prometheus alert manager configuration file: + `api_url:` – Amazon Managed Service for Prometheus sets the `api_url` for you, so this property is not allowed. + `Http_config` – This property allows you to set external proxies. Amazon Managed Service for Prometheus does not currently support this feature. Additionally, SigV4 settings are required to have a Region property. Without the Region property, Amazon Managed Service for Prometheus doesn't have enough information to make the authorization request. **To configure alert manager with your Amazon SNS topic as the receiver** 1. If you are using an existing alert manager configuration file, open it in a text editor. 1. If there are current receivers other than Amazon SNS in the `receivers` block, remove them. You can configure multiple Amazon SNS topics to be receivers by putting them in separate `sns_config` blocks within the `receivers` block. 1. Add the following YAML block within the `receivers` section. ``` - name: name_of_receiver sns_configs: - sigv4: region: AWS Region topic_arn: ARN_of_SNS_topic subject: yoursubject attributes: key: yourkey value: yourvalue ``` If a `subject`is not specified, by default, a subject would be generated with the default template with the label name and values, which may result in a value that is too long for SNS. To change the template that is applied to the subject, refer to [Configure alert manager to send messages to Amazon SNS as JSON](AMP-alertmanager-receiver-JSON.md) in this guide. Now you must upload your alert manager configuration file to Amazon Managed Service for Prometheus. For more information, see [Upload your alert manager configuration file to Amazon Managed Service for Prometheus](AMP-alertmanager-upload.md). # Configure alert manager to send messages to Amazon SNS as JSON By default, Amazon Managed Service for Prometheus alert manager outputs messages in a plain text list format. This can be more difficult for other services to parse. You can configure alert manager to send alerts in JSON format instead. JSON can make it simpler to process the messages downstream from Amazon SNS in AWS Lambda or in webhook-receiving endpoints. Instead of using the default template, you can define a custom template to output the message contents in JSON, making it easier to parse in downstream functions. To output messages from alert manager to Amazon SNS in JSON format, update your alert manager configuration to contain the following code inside your `template_files` root section: ``` default_template: | {{ define "sns.default.message" }}{{ "{" }}"receiver": "{{ .Receiver }}","status": "{{ .Status }}","alerts": [{{ range $alertIndex, $alerts := .Alerts }}{{ if $alertIndex }}, {{ end }}{{ "{" }}"status": "{{ $alerts.Status }}"{{ if gt (len $alerts.Labels.SortedPairs) 0 -}},"labels": {{ "{" }}{{ range $index, $label := $alerts.Labels.SortedPairs }}{{ if $index }}, {{ end }}"{{ $label.Name }}": "{{ $label.Value }}"{{ end }}{{ "}" }}{{- end }}{{ if gt (len $alerts.Annotations.SortedPairs ) 0 -}},"annotations": {{ "{" }}{{ range $index, $annotations := $alerts.Annotations.SortedPairs }}{{ if $index }}, {{ end }}"{{ $annotations.Name }}": "{{ $annotations.Value }}"{{ end }}{{ "}" }}{{- end }},"startsAt": "{{ $alerts.StartsAt }}","endsAt": "{{ $alerts.EndsAt }}","generatorURL": "{{ $alerts.GeneratorURL }}","fingerprint": "{{ $alerts.Fingerprint }}"{{ "}" }}{{ end }}]{{ if gt (len .GroupLabels) 0 -}},"groupLabels": {{ "{" }}{{ range $index, $groupLabels := .GroupLabels.SortedPairs }}{{ if $index }}, {{ end }}"{{ $groupLabels.Name }}": "{{ $groupLabels.Value }}"{{ end }}{{ "}" }}{{- end }}{{ if gt (len .CommonLabels) 0 -}},"commonLabels": {{ "{" }}{{ range $index, $commonLabels := .CommonLabels.SortedPairs }}{{ if $index }}, {{ end }}"{{ $commonLabels.Name }}": "{{ $commonLabels.Value }}"{{ end }}{{ "}" }}{{- end }}{{ if gt (len .CommonAnnotations) 0 -}},"commonAnnotations": {{ "{" }}{{ range $index, $commonAnnotations := .CommonAnnotations.SortedPairs }}{{ if $index }}, {{ end }}"{{ $commonAnnotations.Name }}": "{{ $commonAnnotations.Value }}"{{ end }}{{ "}" }}{{- end }}{{ "}" }}{{ end }} {{ define "sns.default.subject" }}[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}]{{ end }} ``` **Note** This template creates JSON from alphanumeric data. If your data has special characters, encode them before using this template. To make sure that this template is used in outgoing notifications, reference it in your `alertmanager_config` block as follows: ``` alertmanager_config: | global: templates: - 'default_template' ``` **Note** This template is for the entire message body as JSON. This template overwrites the entire message body. You cannot override the message body if you wish to use this specific template. Any overrides that are manually done will take precedence over the template. For more information about: + The alert manager configuration file, see [Create an alert manager configuration in Amazon Managed Service for Prometheus to manage and route alerts](AMP-alertmanager-config.md). + Uploading your configuration file, see [Upload your alert manager configuration file to Amazon Managed Service for Prometheus](AMP-alertmanager-upload.md). # Configure Amazon SNS to send messages for alerts to other destinations Amazon Managed Service for Prometheus can only send alert messages to Amazon Simple Notification Service (Amazon SNS). To send those messages to other destinations, such as email, webhook, Slack, or OpsGenie, you must configure Amazon SNS to forward the messages on to those endpoints. The following sections describing configuring Amazon SNS to forward alerts to other destinations. **Topics** + [Email](#AMP-alertmanager-SNS-otherdestinations-email) + [Webhook](#AMP-alertmanager-SNS-otherdestinations-webhook) + [Slack](#AMP-alertmanager-SNS-otherdestinations-Slack) + [OpsGenie](#AMP-alertmanager-SNS-otherdestinations-OpsGenie) ## Email To configure an Amazon SNS topic to output messages to email, create a subscription. In the Amazon SNS console, choose the **Subscriptions** tab to open the **Subscriptions** list page. Choose **Create Subscription** and select **Email**. Amazon SNS sends a confirmation email to the listed email address. After you accept the confirmation, you are able to receive Amazon SNS notifications as emails from the topic you subscribed to. For more information, see [Subscribing to an Amazon SNS topic](https://docs.aws.amazon.com/sns/latest/dg/sns-create-subscribe-endpoint-to-topic.html). ## Webhook To configure an Amazon SNS topic to output messages to a webhook endpoint, create a subscription. In the Amazon SNS console, choose the **Subscriptions** tab to open the **Subscriptions** list page. Choose **Create Subscription** and select **HTTP/HTTPS**. After you create the subscription, you must follow the confirmation steps to activate it. When it is active, your HTTP endpoint should receive the Amazon SNS notifications. For more information, see [Subscribing to an Amazon SNS topic](https://docs.aws.amazon.com/sns/latest/dg/sns-create-subscribe-endpoint-to-topic.html). For more information about using Slack webhooks to publish messages to various destinations, see [ How do I use webhooks to publish Amazon SNS messages to Amazon Chime, Slack, or Microsoft Teams?](https://aws.amazon.com/premiumsupport/knowledge-center/sns-lambda-webhooks-chime-slack-teams/) ## Slack To configure an Amazon SNS topic to output messages to Slack, you have two options. You can either integrate with Slack’s email-to-channel integration, which allows Slack to accept email messages and forward them to a Slack channel, or you can use a Lambda function to rewrite the Amazon SNS notification to Slack. For more information about forwarding emails to slack channels, see [Confirming AWS SNS Topic Subscription for Slack Webhook](https://stackoverflow.com/questions/49341187/confirming-aws-sns-topic-subscription-for-slack-webhook). For more information about constructing a Lambda function to convert Amazon SNS messages to Slack, see [ How to integrate Amazon Managed Service for Prometheus with Slack](https://aws.amazon.com/blogs/mt/how-to-integrate-amazon-managed-service-for-prometheus-with-slack/). ## OpsGenie For information about how to configure an Amazon SNS topic to output messages to OpsGenie, see [Integrate Opsgenie with Incoming Amazon SNS](https://support.atlassian.com/opsgenie/docs/integrate-opsgenie-with-incoming-amazon-sns/). # Understanding Amazon SNS message validation rules Amazon Simple Notification Service (Amazon SNS) requires messages to meet certain standards. Messages that don't meed these standards will be modified when they are received. The alert messages will be validated, truncated, or modified, if necessary, by the Amazon SNS receiver based on the following rules: + Message contains non-utf characters. + Message will be replaced by **Error - not a valid UTF-8 encoded string**. + One message attribute will be added with the key of **truncated** and the value of **true**. + One message attribute will be added with the key of **modified** and the value of **Message: Error - not a valid UTF-8 encoded string**. + Message is empty. + Message will be replaced by **Error - Message should not be empty**. + One message attribute will be added with the key of **modified** and the value of **Message: Error - Message should not be empty**. + Message has been truncated. + Message will have the truncated content. + One message attribute will be added with the key of **truncated** and the value of **true**. + One message attribute will be added with the key of "modified" and the value of **Message: Error - Message has been truncated from *X* KB, because it exceeds the 256 KB size limit**. + Subject contains control or non-ASCII characters. + If the subject contains control characters or non-ASCII characters, SNS replaces the subject with **Error - contains control- or non-ASCII characters**. + For SNS email subjects, remove control characters, such as newlines: `\n`. + Subject is not ASCII. + Subject will be replaced by **Error - contains non printable ASCII characters**. + One message attribute will be added with the key of **modified** and the value of **Subject: Error - contains non-printable ASCII characters**. + Subject has been truncated. + Subject will have the truncated content. + One message attribute will be added with the key of **modified** and the value of **Subject: Error - Subject has been truncated from *X* characters, because it exceeds the 100 character size limit**. + Message attribute has invalid key/value. + Invalid message attribute will be removed. + One message attribute will be added with the key of **modified** and the value of **MessageAttribute: Error - *X* of the message attributes have been removed because of invalid MessageAttributeKey or MessageAttributeValue**. + Message attribute has been truncated. + Extra message attributes will be removed. + One message attribute will be added with the key of **modified** and the value of **MessageAttribute: Error - *X* of the message attributes have been removed, because it exceeds the 256KB size limit**. # Use PagerDuty as an alert receiver You can configure Amazon Managed Service for Prometheus to send alerts directly to PagerDuty. This integration requires you to store your PagerDuty integration key in AWS Secrets Manager and grant Amazon Managed Service for Prometheus permission to read the secret. PagerDuty integration enables automated incident response workflows and ensures critical alerts reach the right team members at the right time. When you use PagerDuty as an alert receiver, you can take advantage of PagerDuty's escalation policies, on-call scheduling, and incident management features to ensure that alerts are acknowledged and resolved quickly. This integration is particularly valuable for production environments where rapid response to system issues is essential for maintaining service availability and meeting SLA requirements. For more information, see [PagerDuty Knowledge Base](https://support.pagerduty.com/) on the *PagerDuty website*. ## PagerDuty configuration options | Option | Description | Required | | --- | --- | --- | | `routing_key` | The PagerDuty routing key for an integration on a service. You must specify this as an Secrets Manager ARN | Yes | | `service_key` | The PagerDuty service key for an integration on a service. You must specify this as an Secrets Manager ARN | Yes (for Events API v1) | | `client` | The client identification of the notifier | No | | `client_url` | A backlink to the sender of the notification | No | | `description` | Description of the incident | No | | `details` | A set of arbitrary key/value pairs that provide further detail about the incident | No | | `severity` | Severity of the incident | No | | `class` | The class, or type, of the event | No | | `component` | Component of the source machine that is responsible for the event | No | | `group` | Logical grouping of components | No | | `source` | The unique location of the affected system | No | **Note** The `url`, `service_key_file`, `routing_key_file`, and `http_config` options are not supported. The following topics describe how to configure PagerDuty as an alert receiver in Amazon Managed Service for Prometheus. **Topics** + [PagerDuty configuration options](#AMP-alertmanager-pagerduty-configuration-options) + [Configure AWS Secrets Manager and permissions](AMP-alertmanager-pagerduty-permissions.md) + [Configure alert manager to send alerts to PagerDuty](AMP-alertmanager-pagerduty-configure-alertmanager.md) # Configure AWS Secrets Manager and permissions Before you can send alerts to PagerDuty, you must securely store your PagerDuty integration key and configure the necessary permissions. This process involves creating a secret in AWS Secrets Manager, encrypting it with a customer-managed AWS Key Management Service (AWS KMS) key, and granting Amazon Managed Service for Prometheus the required permissions to access both the secret and its encryption key. The following procedures guide you through each step of this configuration process. **To create a secret in Secrets Manager for PagerDuty** To use PagerDuty as an alert receiver, you must store your PagerDuty integration key in Secrets Manager. Follow these steps: 1. Open the [Secrets Manager console](https://console.aws.amazon.com/secretsmanager/). 1. Choose **Store a new secret**. 1. For **Secret type**, choose **Other type of secret**. 1. For **Key/value pairs**, enter your PagerDuty integration key as the secret value. This is either the routing key or service key from your PagerDuty integration. 1. Choose **Next**. 1. Enter a name and description for your secret, then choose **Next**. 1. Configure rotation settings if desired, then choose **Next**. 1. Review your settings and choose **Store**. 1. After creating the secret, note its ARN. You'll need this when configuring the alert manager. **To encrypt your secret with a customer-managed AWS KMS key** You must grant Amazon Managed Service for Prometheus permission to access your secret and its encryption key: 1. **Secret resource policy**: Open your secret in the [Secrets Manager console](https://console.aws.amazon.com/secretsmanager/). 1. Choose **Resource permissions**. 1. Choose **Edit permissions**. 1. Add the following policy statement. In the statement, replace the *highlighted values* with your specific values. ``` { "Effect": "Allow", "Principal": { "Service": "aps.amazonaws.com" }, "Action": "secretsmanager:GetSecretValue", "Resource": "*", "Condition": { "ArnEquals": { "aws:SourceArn": "arn:aws:aps:aws-region:123456789012:workspace/WORKSPACE_ID" }, "StringEquals": { "aws:SourceAccount": "123456789012" } } } ``` 1. Choose **Save**. 1. **KMS key policy**: Open your AWS KMS key in the [AWS KMS console](https://console.aws.amazon.com/kms). 1. Choose **Key policy**. 1. Choose **Edit**. 1. Add the following policy statement. In the statement, replace the *highlighted values* with your specific values. ``` { "Effect": "Allow", "Principal": { "Service": "aps.amazonaws.com" }, "Action": "kms:Decrypt", "Resource": "*", "Condition": { "ArnEquals": { "aws:SourceArn": "arn:aws:aps:aws-region:123456789012:workspace/WORKSPACE_ID" }, "StringEquals": { "aws:SourceAccount": "123456789012" } } } ``` 1. Choose **Save**. **Next steps** – Continue to the next topic, [Configure alert manager to send alerts to PagerDuty](AMP-alertmanager-pagerduty-configure-alertmanager.md). # Configure alert manager to send alerts to PagerDuty To configure alert manager to send alerts to PagerDuty, you need to update your alert manager definition. You can do this using the AWS Management Console, AWS CLI, or AWS SDKs. **Example alert manager configuration** Following, is an example alert manager configuration that sends alerts to PagerDuty. In the example, replace the *highlighted values* with your specific values. ``` alertmanager_config: | route: receiver: 'pagerduty-receiver' group_by: ['alertname'] group_wait: 30s group_interval: 5m repeat_interval: 1h receivers: - name: 'pagerduty-receiver' pagerduty_configs: - routing_key: aws_secrets_manager: secret_arn: 'arn:aws:secretsmanager:aws-region:123456789012:secret:YOUR_SECRET_NAME' secret_key: 'YOUR_SECRET_KEY' refresh_interval: 5m description: '{{ .CommonLabels.alertname }}' severity: 'critical' details: firing: '{{ .Alerts.Firing | len }}' status: '{{ .Status }}' instance: '{{ .CommonLabels.instance }}' ``` **Example AWS CLI** Following, is an AWS CLI command used to update your alert manager definition. In the example, replace the *highlighted values* with your specific values. ``` aws amp put-alert-manager-definition \ --workspace-id WORKSPACE_ID \ --data file://alertmanager-config.yaml ``` ## Troubleshooting PagerDuty integration If alerts are not being sent to PagerDuty, check the following items: + Verify that your secret exists and contains the correct PagerDuty integration key. + Confirm that your secret is encrypted with a customer-managed KMS key. + Ensure that the resource policies for both the secret and the KMS key grant the necessary permissions to Amazon Managed Service for Prometheus. + Check that the ARN in your alert manager configuration correctly references your secret. + Verify that your PagerDuty integration key is valid and active in your PagerDuty account. Amazon Managed Service for Prometheus supports Amazon CloudWatch Logs, and the following CloudWatch metrics, to help with troubleshooting. For more information, see [Monitor Amazon Managed Service for Prometheus events with CloudWatch Logs](CW-logs.md) and [Use CloudWatch metrics to monitor Amazon Managed Service for Prometheus resources](AMP-CW-usage-metrics.md).CloudWatch metrics + `SecretFetchFailure` + `AlertManagerNotificationsThrottledByIntegration` + `AlertManagerNotificationsFailedByIntegration` # Upload your alert manager configuration file to Amazon Managed Service for Prometheus Once you know what you want in your Alert manager configuration file, you can create and edit it within the console, or you can upload an existing file with the Amazon Managed Service for Prometheus console or AWS CLI. **Note** If you are running an Amazon EKS cluster, you can also upload an Alert manager configuration file using [AWS Controllers for Kubernetes](integrating-ack.md). **To use the Amazon Managed Service for Prometheus console to edit or replace your alert manager configuration** 1. Open the Amazon Managed Service for Prometheus console at [https://console.aws.amazon.com/prometheus/](https://console.aws.amazon.com/prometheus/home). 1. In the upper left corner of the page, choose the menu icon, and then choose **All workspaces**. 1. Choose the workspace ID of the workspace, and then choose the **Alert manager** tab. 1. If the workspace doesn't already have an alert manager definition, choose **Add definition**. **Note** If the workspace has an alert manager definition that you want to replace, choose **Modify** instead. 1. Choose **Choose file**, select the alert manager definition file, and choose **Continue**. **Note** Alternately, you can create a new file and edit it directly in the console, by choosing the **Create definition** option. This will create a sample default configuration that you edit before uploading. **To use the AWS CLI to upload an alert manager configuration to a workspace for the first time** 1. Base64 encode the contents of your alert manager file. On Linux, you can use the following command: ``` base64 input-file output-file ``` On macOS, you can use the following command: ``` openssl base64 input-file output-file ``` 1. To upload the file, enter one of the following commands. On AWS CLI version 2, enter: ``` aws amp create-alert-manager-definition --data file://path_to_base_64_output_file --workspace-id my-workspace-id --region region ``` On AWS CLI version 1, enter: ``` aws amp create-alert-manager-definition --data fileb://path_to_base_64_output_file --workspace-id my-workspace-id --region region ``` 1. It takes a few seconds for your alert manager configuration to become active. To check the status, enter the following command: ``` aws amp describe-alert-manager-definition --workspace-id workspace_id --region region ``` If the `status` is `ACTIVE`, your new alert manager definition has taken effect. **To use the AWS CLI to replace a workspace's alert manager configuration with a new one** 1. Base64 encode the contents of your alert manager file. On Linux, you can use the following command: ``` base64 input-file output-file ``` On macOS, you can use the following command: ``` openssl base64 input-file output-file ``` 1. To upload the file, enter one of the following commands. On AWS CLI version 2, enter: ``` aws amp put-alert-manager-definition --data file://path_to_base_64_output_file --workspace-id my-workspace-id --region region ``` On AWS CLI version 1, enter: ``` aws amp put-alert-manager-definition --data file://path_to_base_64_output_file --workspace-id my-workspace-id --region region ``` 1. It takes a few seconds for your new alert manager configuration to become active. To check the status, enter the following command: ``` aws amp describe-alert-manager-definition --workspace-id workspace_id --region region ``` If the `status` is `ACTIVE`, your new alert manager definition has taken effect. Until that time, your previous alert manager configuration is still active. # Integrate alerts with Amazon Managed Grafana or open source Grafana Alert rules that you have created in Alertmanager within Amazon Managed Service for Prometheus can be forwarded and viewed in [Amazon Managed Grafana](https://aws.amazon.com/grafana/) and [Grafana](https://grafana.com), unifying your alert rules and alerts in a single environment. Within Amazon Managed Grafana, you can view your alert rules and the alerts that are generated. ## Prerequisites Before starting to integrate Amazon Managed Service for Prometheus into Amazon Managed Grafana, you must have completed the following prerequisites: + You must have an existing AWS account and IAM credentials to create Amazon Managed Service for Prometheus and IAM roles programmatically. For more information about creating an AWS account and IAM credentials, see [Set up AWS](AMP-setting-up.md). + You must have an Amazon Managed Service for Prometheus workspace, and be ingesting data into it. To set up a new workspace, see [Create an Amazon Managed Service for Prometheus workspace](AMP-onboard-create-workspace.md). You should also be familiar with the Prometheus concepts such as Alertmanager and Ruler. For more information about these topics, see the [Prometheus documentation](https://prometheus.io/docs/introduction/overview/). + You have an Alertmanager configuration and a rules file already configured in Amazon Managed Service for Prometheus. For more information about Alertmanager in Amazon Managed Service for Prometheus, see [Managing and forwarding alerts in Amazon Managed Service for Prometheus with alert manager](AMP-alert-manager.md). For more information about rules, see [Using rules to modify or monitor metrics as they are received](AMP-Ruler.md). + You must either have Amazon Managed Grafana set up, or the open source version of Grafana running. + If you are using Amazon Managed Grafana, you must be using Grafana alerting. For more information see [Migrating legacy dashboard alerts to Grafana alerting](https://docs.aws.amazon.com/grafana/latest/userguide/alert-opt-in.html). + If you are using the open source version of Grafana, you must be running version 9.1 or higher. **Note** You can use earlier versions of Grafana, but you must [enable the unified alerting](https://grafana.com/docs/grafana/v8.4/alerting/unified-alerting/opt-in/) (Grafana alerting) feature, and you might have to set up a [sigv4 proxy](https://github.com/awslabs/aws-sigv4-proxy) to make calls from Grafana to Amazon Managed Service for Prometheus. For more information, see [Set up Grafana open source or Grafana Enterprise for use with Amazon Managed Service for Prometheus](AMP-onboard-query-standalone-grafana.md). + Amazon Managed Grafana must have the following permissions for your Prometheus resources. You must add them to either the service-managed or customer-managed policies described in [https://docs.aws.amazon.com/grafana/latest/userguide/AMG-manage-permissions.html](https://docs.aws.amazon.com/grafana/latest/userguide/AMG-manage-permissions.html). + `aps:ListRules` + `aps:ListAlertManagerSilences` + `aps:ListAlertManagerAlerts` + `aps:GetAlertManagerStatus` + `aps:ListAlertManagerAlertGroups` + `aps:PutAlertManagerSilences` + `aps:DeleteAlertManagerSilence` ## Setting up Amazon Managed Grafana If you have already set up rules and alerts in your Amazon Managed Service for Prometheus instance, the configuration to use Amazon Managed Grafana as a dashboard for those alerts is done entirely within Amazon Managed Grafana. **To configure Amazon Managed Grafana as your alerts dashboard** 1. Open the Grafana console for your workspace. 1. Under **Configurations**, choose **Data sources**. 1. Either create or open your Prometheus data source. If you have not previously set up a Prometheus data source, see [Step 2: Add the Prometheus data source in Grafana](AMP-onboard-query-standalone-grafana.md#AMP-onboard-query-standalone-grafana-datasource) for more information. 1. In the Prometheus data source, select **Manage alerts via Alertmanager UI**. 1. Go back to the **Data sources** interface. 1. Create a new Alertmanager data source. 1. In the Alertmanager data source configuration page, add the following settings: + Set **Implementation** to `Prometheus`. + For the **URL** setting, use the URL for your Prometheus workspace, remove everything after the workspace ID, and append `/alertmanager` to the end. In the following example, replace the *variables* with you own (account specific) information: ``` https://aps-workspaces.US East (N. Virginia).amazonaws.com/workspaces/ws-example-1234-5678-abcd-xyz00000001/alertmanager. ``` + Under **Auth**, turn on **SigV4Auth**. This tells Grafana to use the [AWS authentication](https://docs.aws.amazon.com/general/latest/gr/signature-version-4.html) for the requests. + Under **SigV4Auth Details**, for **Default Region**, provide the region of your Prometheus instance, for example `us-east-1`. + Set the **Default** option to `true`. 1. Choose **Save and test**. 1. Your Amazon Managed Service for Prometheus alerts should now be configured to work with your Grafana instance. Verify that you can see any **Alert rules**, **Alert groups** (including active alerts), and **Silences** from your Amazon Managed Service for Prometheus instance in the Grafana **Alerting** page. # Troubleshoot alert manager with CloudWatch Logs Using [Monitor Amazon Managed Service for Prometheus events with CloudWatch Logs](CW-logs.md), you can troubleshoot Alert Manager and Ruler related issues. This section contains Alert Manager related troubleshooting topics. **Topics** + [Active alerts warning](#Troubleshooting-alerting-active-alerts) + [Alert aggregation group size warning](#Troubleshooting-alerting-aggregation-group-size) + [Alerts size too big warning](#Troubleshooting-alerting-size-too-big) + [Empty content warning](#Troubleshooting-alerting-empty) + [Invalid `key/value` warning](#Troubleshooting-alerting-invalid-keyvalue) + [Message limit warning](#Troubleshooting-alerting-msg-limit) + [No resource based policy error](#Troubleshooting-alerting-no-policy) + [Non ASCII warning](#Troubleshooting-alerting-non-ASCII) + [Not authorized to call KMS](#Troubleshooting-alerting-no-access-kms) + [Template error](#Troubleshooting-template-error) ## Active alerts warning **When the log contains the following warning** ``` { "workspaceId": "ws-efdc5b42-b051-11ec-b123-4567ac120002", "message": { "log": "too many alerts, limit: 1000", "level": "WARN" }, "component": "alertmanager" } ``` This means that the Alert manager **Active alerts** quota is exceeded. **Action to take** Request a quota increase. Sign in to the AWS Management Console and open the Service Quotas console at [https://console.aws.amazon.com/servicequotas/](https://console.aws.amazon.com/servicequotas/). ## Alert aggregation group size warning **When the log contains the following warning** ``` { "workspaceId": "ws-efdc5b42-b051-11ec-b123-4567ac120002", "message": { "log": "Too many aggregation groups, cannot create new group for alert, groups=1000, limit=1000, alert=sample-alert", "level": "WARN" }, "component": "alertmanager" } ``` This means that the Alert manager Alert aggregation group size quota has been exceeded. **Action to take** Reduce the Alert aggregation group size by using the `group_by` parameter. For more information, see [Route-related settings in the](https://prometheus.io/docs/alerting/latest/configuration/)*Prometheus documentation*. You can also request a quota increase. Sign in to the AWS Management Console and open the Service Quotas console at [https://console.aws.amazon.com/servicequotas/](https://console.aws.amazon.com/servicequotas/). ## Alerts size too big warning **When the log contains the following warning** ``` { "workspaceId": "ws-efdc5b42-b051-11ec-b123-4567ac120002", "message": { "log": "alerts too big, total size limit: 20000000 bytes", "level": "WARN" }, "component": "alertmanager" } ``` This means that Alert manager Alerts per workspace, in size quota has been exceeded. **Action to take** Remove unnecessary annotations and labels to reduce alert size. ## Empty content warning **When the log contains the following warning** ``` { "workspaceId": "ws-abcd1234-ef56-78ab-cd90-1234abcd0000", "message": { "log": "Message has been modified because the content was empty." "level": "WARN" }, "component": "alertmanager" } ``` This means that the Alert manager template resolved the outbound alert to an empty message. **Action to take** Validate your Alert manager template and ensure that you have a valid template for all receiver pathways. ## Invalid `key/value` warning **When the log contains the following warning** ``` { "workspaceId": "ws-abcd1234-ef56-78ab-cd90-1234abcd0000", "message": { "log": "MessageAttributes has been removed because of invalid key/value, numberOfRemovedAttributes=1" "level": "WARN" }, "component": "alertmanager" } ``` This means that some of the message attributes have been removed due to keys/values being invalid. **Action to take** Re-evaluate the templates you are using to populate the message attributes, and ensure it is resolving to a valid SNS message attribute. For more information about validating a message to an Amazon SNS topic, see [Validating SNS topic](https://docs.aws.amazon.com/sns/latest/api/API_Publish.html#API_Publish_RequestParameters) ## Message limit warning **When the log contains the following warning** ``` { "workspaceId": "ws-abcd1234-ef56-78ab-cd90-1234abcd0000", "message": { "log": "Message has been truncated because it exceeds size limit, originSize=266K, truncatedSize=12K" "level": "WARN" }, "component": "alertmanager" } ``` This means that some of the message size is too big. **Action to take** Look at the Alert receiver message template and re-work it to fit within the size limit. ## No resource based policy error **When the log contains the following error** ``` { "workspaceId": "ws-abcd1234-ef56-78ab-cd90-1234abcd0000", "message": { "log": "Notify for alerts failed, AMP is not authorized to perform: SNS:Publish on resource: arn:aws:sns:us-west-2:12345:testSnsReceiver because no resource-based policy allows the SNS:Publish action" "level": "ERROR" }, "component": "alertmanager" } ``` This means that Amazon Managed Service for Prometheus does not have the permissions to submit the alert to the SNS topic specified. **Action to take** Validate that the access policy on your Amazon SNS topic grants Amazon Managed Service for Prometheus the ability to send SNS messages to the topic. Create an SNS Access Policy giving the service `aps.amazonaws.com` (Amazon Managed Service for Prometheus) access to your Amazon SNS topic. For more information about SNS Access Policies, see [Using the Access Policy Language](https://docs.aws.amazon.com/sns/latest/dg/sns-access-policy-language-using.html) and [Example cases for Amazon SNS access control](https://docs.aws.amazon.com/sns/latest/dg/sns-access-policy-use-cases.html) in the *Amazon Simple Notification Service Developer Guide*. ## Non ASCII warning **When the log contains the following warning** ``` { "workspaceId": "ws-abcd1234-ef56-78ab-cd90-1234abcd0000", "message": { "log": "Subject has been modified because it contains control or non-ASCII characters." "level": "WARN" }, "component": "alertmanager" } ``` This means that the subject has non-ASCII characters. **Action to take** Remove references in subject field of your template to the labels that might contain non-ASCII characters. ## Not authorized to call KMS **When the log contains the following AWS KMS error** ``` { "workspaceId": "ws-abcd1234-ef56-78ab-cd90-1234abcd0000", "message": { "log": "Notify for alerts failed, AMP is not authorized to call KMS", "level": "ERROR" }, "component": "alertmanager" } ``` **Action to take** Validate that the key policy of the key used to encrypt the Amazon SNS topic allows the Amazon Managed Service for Prometheus service principal `aps.amazonaws.com` to perform the following actions: `kms:GenerateDataKey*`, and `kms:Decrypt`. For more information, see [AWS KMS Permissions for SNS Topic](https://docs.aws.amazon.com/sns/latest/dg/sns-key-management.html#sns-what-permissions-for-sse). ## Template error **When the log contains the following error** ``` { "workspaceId": "ws-efdc5b42-b051-11ec-b123-4567ac120002", "message": { "log": "Notify for alerts failed. There is an error in a receiver that is using templates in the AlertManager definition. Make sure that the syntax is correct and only template functions and variables that exist are used in the receiver 'default', sns_configs position #2, section 'attributes'" "level": "ERROR" }, "component": "alertmanager" } ``` This means that there is an error in a template being used in the AlertManager definition. The error entry contains directions about what receiver, the position in the sns\$1configs and the property that contains errors. **Action to take** Validate your Alert Manager definition. Make sure that the syntax is correct and that you reference template variables and functions that exist. For more information, see the [Notification Template Reference](https://prometheus.io/docs/alerting/latest/notifications/) in the *Prometheus* open-source documentation.