# Metrics in Amazon CloudWatch Metrics Metrics are data about the performance of your systems. Amazon CloudWatch collects metrics through two paths: AWS vended metrics from services such as Amazon EC2, Amazon EBS, and Amazon RDS, and custom metrics that you publish using the OpenTelemetry Protocol (OTLP) or the CloudWatch API. Many AWS services provide free metrics for resources by default. You can also enable detailed monitoring for some resources, such as Amazon EC2 instances. CloudWatch supports two query languages for metrics. The Prometheus Query Language (PromQL) provides flexible analytics for OpenTelemetry metrics and AWS vended metrics published in OpenTelemetry format. CloudWatch Metrics Insights provides a SQL-based query engine for traditional CloudWatch metrics. Both support graphing, dashboards, and alarms. Key Topics: + [Metrics concepts](cloudwatch_concepts.md) + [Basic monitoring and detailed monitoring in CloudWatch](cloudwatch-metrics-basic-detailed.md) + [Publish custom metrics](publishingMetrics.md) + [Send metrics using OpenTelemetry (new)](CloudWatch-OpenTelemetry-Sections.md) + [Publish AWS vended metrics as OpenTelemetry metrics (new)](publishingMetrics.md) + [Query metrics with PromQL (new)](CloudWatch-PromQL.md) + [Query your CloudWatch metrics with CloudWatch Metrics Insights](query_with_cloudwatch-metrics-insights.md) + [Use metrics explorer to monitor resources by their tags and properties](CloudWatch-Metrics-Explorer.md) + [Use metric streams](CloudWatch-Metric-Streams.md) + [View available metrics](viewing_metrics_with_cloudwatch.md) + [Graphing metrics](graph_metrics.md) + [Using CloudWatch anomaly detection](CloudWatch_Anomaly_Detection.md) + [Using math expressions with CloudWatch metrics](using-metric-math.md) + [Use search expressions in graphs](using-search-expressions.md) + [Get statistics for a metric](getting-metric-statistics.md) # Metrics concepts Concepts The following terminology and concepts are central to your understanding and use of Amazon CloudWatch: + [OpenTelemetry metrics](#OpenTelemetry_metrics) + [Namespaces](#Namespace) + [Metrics](#Metric) + [Dimensions](#Dimension) + [Resolution](#Resolution_definition) + [Statistics](#Statistic) + [Percentiles](#Percentiles) + [Alarms](#CloudWatchAlarms) For information about the service quotas for CloudWatch metrics, alarms, API requests, and alarm email notifications, see [CloudWatch service quotas](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/cloudwatch_limits.html). ## OpenTelemetry metrics CloudWatch also supports metrics sent using the OpenTelemetry Protocol (OTLP). OpenTelemetry metrics use a different data model from traditional CloudWatch metrics. Instead of namespaces and dimensions, OpenTelemetry metrics use metric names with descriptive labels (key-value pairs) that follow OpenTelemetry semantic conventions. OpenTelemetry metrics support up to 150 labels per metric and support metric types including gauge, sum, histogram, and exponential histogram. OpenTelemetry metrics are queried using the Prometheus Query Language (PromQL) in CloudWatch Query Studio or through the Prometheus-compatible query API. You can set PromQL-based CloudWatch Alarms on OpenTelemetry metrics. The following table summarizes the key differences between OpenTelemetry metrics and traditional CloudWatch metrics. | Concept | Traditional CloudWatch metrics | OpenTelemetry metrics | | --- | --- | --- | | Identity | Namespace, metric name, up to 30 dimensions | Metric name, up to 150 labels | | Metric types | Single values, statistic sets | Gauge, sum, histogram, exponential histogram | | Ingestion | PutMetricData API or AWS CLI | OpenTelemetry Protocol (OTLP) | | Query language | GetMetricStatistics, Metrics Insights | Prometheus Query Language (PromQL) | | Alarms | Standard CloudWatch Alarms | PromQL-based CloudWatch Alarms | | Console experience | CloudWatch Metrics console | CloudWatch Query Studio | | Retention | Up to 15 months with automatic rollup | 30 days (public preview) | For more information, see [Send metrics using OpenTelemetry](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-OpenTelemetry-Sections.html). ## Namespaces A *namespace* is a container for CloudWatch metrics. Metrics in different namespaces are isolated from each other, so that metrics from different applications are not mistakenly aggregated into the same statistics. There is no default namespace. You must specify a namespace for each data point you publish to CloudWatch. You can specify a namespace name when you create a metric. These names must contain valid ASCII characters, and be 255 or fewer characters. Possible characters are: alphanumeric characters (0-9A-Za-z), period (.), hyphen (-), underscore (\$1), forward slash (/), hash (\$1), colon (:), and the space character. A namespace must contain at least one non-whitespace character. The AWS namespaces typically use the following naming convention: `AWS/service`. For example, Amazon EC2 uses the `AWS/EC2` namespace. For the list of AWS namespaces, see [AWS services that publish CloudWatch metrics](aws-services-cloudwatch-metrics.md). ## Metrics *Metrics* are the fundamental concept in CloudWatch. A metric represents a time-ordered set of data points that are published to CloudWatch. Think of a metric as a variable to monitor, and the data points as representing the values of that variable over time. For example, the CPU usage of a particular EC2 instance is one metric provided by Amazon EC2. The data points themselves can come from any application or business activity from which you collect data. By default, many AWS services provide metrics at no charge for resources (such as Amazon EC2 instances, Amazon EBS volumes, and Amazon RDS DB instances). For a charge, you can also enable detailed monitoring for some resources, such as your Amazon EC2 instances, or publish your own application metrics. For custom metrics, you can add the data points in any order, and at any rate you choose. You can retrieve statistics about those data points as an ordered set of time-series data. Metrics exist only in the Region in which they are created. Metrics cannot be deleted, but they automatically expire after 15 months if no new data is published to them. Data points older than 15 months expire on a rolling basis; as new data points come in, data older than 15 months is dropped. Metrics are uniquely defined by a name, a namespace, and zero or more dimensions. Each data point in a metric has a time stamp, and (optionally) a unit of measure. You can retrieve statistics from CloudWatch for any metric. For more information, see [View available metrics](viewing_metrics_with_cloudwatch.md) and [Publish custom metrics](publishingMetrics.md). ### Time stamps Each metric data point must be associated with a time stamp. The time stamp can be up to two weeks in the past and up to two hours into the future. If you do not provide a time stamp, CloudWatch creates a time stamp for you based on the time the data point was received. Time stamps are `dateTime` objects, with the complete date plus hours, minutes, and seconds (for example, 2016-10-31T23:59:59Z). For more information, see [dateTime](http://www.w3.org/TR/xmlschema-2/#dateTime). Although it is not required, we recommend that you use Coordinated Universal Time (UTC). When you retrieve statistics from CloudWatch, all times are in UTC. CloudWatch alarms check metrics based on the current time in UTC. Custom metrics sent to CloudWatch with time stamps other than the current UTC time can cause alarms to display the **Insufficient Data** state or result in delayed alarms. ### Metrics retention CloudWatch retains metric data as follows: + Data points with a period of less than 60 seconds are available for 3 hours. These data points are high-resolution custom metrics. + Data points with a period of 60 seconds (1 minute) are available for 15 days + Data points with a period of 300 seconds (5 minutes) are available for 63 days + Data points with a period of 3600 seconds (1 hour) are available for 455 days (15 months) Data points that are initially published with a shorter period are aggregated together for long-term storage. For example, if you collect data using a period of 1 minute, the data remains available for 15 days with 1-minute resolution. After 15 days this data is still available, but is aggregated and is retrievable only with a resolution of 5 minutes. After 63 days, the data is further aggregated and is available with a resolution of 1 hour. **Note** Metrics that have not had any new data points in the past two weeks do not appear in the console. They also do not appear when you type their metric name or dimension names in the search box in the **All metrics** tab in the console, and they are not returned in the results of a [list-metrics](https://docs.aws.amazon.com/cli/latest/reference/cloudwatch/list-metrics.html) command. The best way to retrieve these metrics is with the [get-metric-data](https://docs.aws.amazon.com/cli/latest/reference/cloudwatch/get-metric-data.html) or [get-metric-statistics](https://docs.aws.amazon.com/cli/latest/reference/cloudwatch/get-metric-statistics.html) commands in the AWS CLI. ## Dimensions A *dimension* is a name/value pair that is part of the identity of a metric. You can assign up to 30 dimensions to a metric. Every metric has specific characteristics that describe it, and you can think of dimensions as categories for those characteristics. Dimensions help you design a structure for your statistics plan. Because dimensions are part of the unique identifier for a metric, whenever you add a unique name/value pair to one of your metrics, you are creating a new variation of that metric. AWS services that send data to CloudWatch attach dimensions to each metric. You can use dimensions to filter the results that CloudWatch returns. For example, you can get statistics for a specific EC2 instance by specifying the `InstanceId` dimension when you search for metrics. For metrics produced by certain AWS services, such as Amazon EC2, CloudWatch can aggregate data across dimensions. For example, if you search for metrics in the `AWS/EC2` namespace but do not specify any dimensions, CloudWatch aggregates all data for the specified metric to create the statistic that you requested. CloudWatch does not aggregate across dimensions for your custom metrics. ### Dimension combinations CloudWatch treats each unique combination of dimensions as a separate metric, even if the metrics have the same metric name. You can only retrieve statistics using combinations of dimensions that you specifically published. When you retrieve statistics, specify the same values for the namespace, metric name, and dimension parameters that were used when the metrics were created. You can also specify the start and end times for CloudWatch to use for aggregation. For example, suppose that you publish four distinct metrics named ServerStats in the DataCenterMetric namespace with the following properties: ``` Dimensions: Server=Prod, Domain=Frankfurt, Unit: Count, Timestamp: 2016-10-31T12:30:00Z, Value: 105 Dimensions: Server=Beta, Domain=Frankfurt, Unit: Count, Timestamp: 2016-10-31T12:31:00Z, Value: 115 Dimensions: Server=Prod, Domain=Rio, Unit: Count, Timestamp: 2016-10-31T12:32:00Z, Value: 95 Dimensions: Server=Beta, Domain=Rio, Unit: Count, Timestamp: 2016-10-31T12:33:00Z, Value: 97 ``` If you publish only those four metrics, you can retrieve statistics for these combinations of dimensions: + `Server=Prod,Domain=Frankfurt` + `Server=Prod,Domain=Rio` + `Server=Beta,Domain=Frankfurt` + `Server=Beta,Domain=Rio` You can't retrieve statistics for the following dimensions or if you specify no dimensions. (The exception is by using the metric math **SEARCH** function, which can retrieve statistics for multiple metrics. For more information, see [Use search expressions in graphs](using-search-expressions.md).) + `Server=Prod` + `Server=Beta` + `Domain=Frankfurt` + `Domain=Rio` **Note** OpenTelemetry metrics use labels instead of dimensions. Labels serve a similar purpose but follow OpenTelemetry semantic conventions and support up to 150 labels per metric. For more information, see [OpenTelemetry metrics](#OpenTelemetry_metrics). ## Resolution Each metric is one of the following: + Standard resolution, with data having a one-minute granularity + High resolution, with data at a granularity of one second Metrics produced by AWS services are standard resolution by default. When you publish a custom metric, you can define it as either standard resolution or high resolution. When you publish a high-resolution metric, CloudWatch stores it with a resolution of 1 second, and you can read and retrieve it with a period of 1 second, 5 seconds, 10 seconds, 30 seconds, or any multiple of 60 seconds. High-resolution metrics can give you more immediate insight into your application's sub-minute activity. Keep in mind that every `PutMetricData` call for a custom metric is charged, so calling `PutMetricData` more often on a high-resolution metric can lead to higher charges. For more information about CloudWatch pricing, see [Amazon CloudWatch Pricing](https://aws.amazon.com/cloudwatch/pricing/). If you set an alarm on a high-resolution metric, you can specify a high-resolution alarm with a period of 10 seconds or 30 seconds, or you can set a regular alarm with a period of any multiple of 60 seconds. There is a higher charge for high-resolution alarms with a period of 10 or 30 seconds. **Note** OpenTelemetry metrics are ingested at the resolution they are sent, with no minimum granularity restriction. Resolution and rollup behavior for OpenTelemetry metrics differs from traditional CloudWatch metrics. ## Statistics *Statistics* are metric data aggregations over specified periods of time. CloudWatch provides statistics based on the metric data points provided by your custom data or provided by other AWS services to CloudWatch. Aggregations are made using the namespace, metric name, dimensions, and the data point unit of measure, within the time period you specify. For detailed definitions of the statistics supported by CloudWatch, see [CloudWatch statistics definitions](Statistics-definitions.md). ## Units Each statistic has a unit of measure. Example units include `Bytes`, `Seconds`, `Count`, and `Percent`. For the complete list of the units that CloudWatch supports, see the [MetricDatum](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_MetricDatum.html) data type in the *Amazon CloudWatch API Reference*. You can specify a unit when you create a custom metric. If you do not specify a unit, CloudWatch uses `None` as the unit. Units help provide conceptual meaning to your data. Though CloudWatch attaches no significance to a unit internally, other applications can derive semantic information based on the unit. Metric data points that specify a unit of measure are aggregated separately. When you get statistics without specifying a unit, CloudWatch aggregates all data points of the same unit together. If you have two otherwise identical metrics with different units, two separate data streams are returned, one for each unit. ## Periods A *period* is the length of time associated with a specific Amazon CloudWatch statistic. Each statistic represents an aggregation of the metrics data collected for a specified period of time. Periods are defined in numbers of seconds, and valid values for period are 1, 5, 10, 30, or any multiple of 60. For example, to specify a period of six minutes, use 360 as the period value. You can adjust how the data is aggregated by varying the length of the period. The default value of a period is 60 seconds. A period can be as short as one second, and must be a multiple of 60 if it is greater than the default value of 60 seconds. Only custom metrics that you define with a storage resolution of 1 second support sub-minute periods. Even though the option to set a period below 60 is always available in the console, you should select a period that aligns to how the metric is stored. For more information about metrics that support sub-minute periods, see [High-resolution metrics](publishingMetrics.md#high-resolution-metrics). When you retrieve statistics, you can specify a period, start time, and end time. These parameters determine the overall length of time associated with the statistics. The default values for the start time and end time get you the last hour's worth of statistics. The values that you specify for the start time and end time determine how many periods CloudWatch returns. For example, retrieving statistics using the default values for the period, start time, and end time returns an aggregated set of statistics for each minute of the previous hour. If you prefer statistics aggregated in ten-minute blocks, specify a period of 600. For statistics aggregated over the entire hour, specify a period of 3600. When statistics are aggregated over a period of time, they are stamped with the time corresponding to the beginning of the period. For example, data aggregated from 7:00pm to 8:00pm is stamped as 7:00pm. Additionally, data aggregated between 7:00pm and 8:00pm begins to be visible at 7:00pm, then the values of that aggregated data may change as CloudWatch collects more samples during the period. Periods are also important for CloudWatch alarms. When you create an alarm to monitor a specific metric, you are asking CloudWatch to compare that metric to the threshold value that you specified. You have extensive control over how CloudWatch makes that comparison. Not only can you specify the period over which the comparison is made, but you can also specify how many evaluation periods are used to arrive at a conclusion. For example, if you specify three evaluation periods, CloudWatch compares a window of three data points. CloudWatch only notifies you if the oldest data point is breaching and the others are breaching or missing. ## Aggregation Amazon CloudWatch aggregates statistics according to the period length that you specify when retrieving statistics. You can publish as many data points as you want with the same or similar time stamps. CloudWatch aggregates them according to the specified period length. CloudWatch does not automatically aggregate data across Regions, but you can use metric math to aggregate metrics from different Regions. You can publish data points for a metric that share not only the same time stamp, but also the same namespace and dimensions. CloudWatch returns aggregated statistics for those data points. You can also publish multiple data points for the same or different metrics, with any time stamp. For large datasets, you can insert a pre-aggregated dataset called a *statistic set*. With statistic sets, you give CloudWatch the Min, Max, Sum, and SampleCount for a number of data points. This is commonly used when you need to collect data many times in a minute. For example, suppose you have a metric for the request latency of a web page. It doesn't make sense to publish data with every web page hit. We suggest that you collect the latency of all hits to that web page, aggregate them once a minute, and send that statistic set to CloudWatch. Amazon CloudWatch doesn't differentiate the source of a metric. If you publish a metric with the same namespace and dimensions from different sources, CloudWatch treats this as a single metric. This can be useful for service metrics in a distributed, scaled system. For example, all the hosts in a web server application could publish identical metrics representing the latency of requests they are processing. CloudWatch treats these as a single metric, allowing you to get the statistics for minimum, maximum, average, and sum of all requests across your application. ## Percentiles A *percentile* indicates the relative standing of a value in a dataset. For example, the 95th percentile means that 95 percent of the data is lower than this value and 5 percent of the data is higher than this value. Percentiles help you get a better understanding of the distribution of your metric data. Percentiles are often used to isolate anomalies. In a normal distribution, 95 percent of the data is within two standard deviations from the mean and 99.7 percent of the data is within three standard deviations from the mean. Any data that falls outside three standard deviations is often considered to be an anomaly because it differs so greatly from the average value. For example, suppose that you are monitoring the CPU utilization of your EC2 instances to ensure that your customers have a good experience. If you monitor the average, this can hide anomalies. If you monitor the maximum, a single anomaly can skew the results. Using percentiles, you can monitor the 95th percentile of CPU utilization to check for instances with an unusually heavy load. Some CloudWatch metrics support percentiles as a statistic. For these metrics, you can monitor your system and applications using percentiles as you would when using the other CloudWatch statistics (Average, Minimum, Maximum, and Sum). For example, when you create an alarm, you can use percentiles as the statistical function. You can specify the percentile with up to ten decimal places (for example, p95.0123456789). Percentile statistics are available for custom metrics as long as you publish the raw, unsummarized data points for your custom metric. Percentile statistics are not available for metrics when any of the metric values are negative numbers. CloudWatch needs raw data points to calculate percentiles. If you publish data using a statistic set instead, you can only retrieve percentile statistics for this data if one of the following conditions is true: + The SampleCount value of the statistic set is 1 and Min, Max, and Sum are all equal. + The Min and Max are equal, and Sum is equal to Min multiplied by SampleCount. The following AWS services include metrics that support percentile statistics. + API Gateway + Application Load Balancer + Amazon EC2 + Elastic Load Balancing + Kinesis + Lambda + Amazon RDS CloudWatch also supports trimmed mean and other performance statistics, which can have a similar use as percentiles. For more information, see [CloudWatch statistics definitions](Statistics-definitions.md). ## Alarms You can use an *alarm* to automatically initiate actions on your behalf. An alarm watches a single metric over a specified time period, and performs one or more specified actions, based on the value of the metric relative to a threshold over time. The action is a notification sent to an Amazon SNS topic or an Auto Scaling policy. You can also add alarms to dashboards. Alarms invoke actions for sustained state changes only. CloudWatch alarms do not invoke actions simply because they are in a particular state. The state must have changed and been maintained for a specified number of periods. When creating an alarm, select an alarm monitoring period that is greater than or equal to the metrics resolution. For example, basic monitoring for Amazon EC2 provides metrics for your instances every 5 minutes. When setting an alarm on a basic monitoring metric, select a period of at least 300 seconds (5 minutes). Detailed monitoring for Amazon EC2 provides metrics for your instances with a resolution of 1 minute. When setting an alarm on a detailed monitoring metric, select a period of at least 60 seconds (1 minute). If you set an alarm on a high-resolution metric, you can specify a high-resolution alarm with a period of 10 seconds or 30 seconds, or you can set a regular alarm with a period of any multiple of 60 seconds. There is a higher charge for high-resolution alarms. For more information about high-resolution metrics, see [Publish custom metrics](publishingMetrics.md). For more information, see [Using Amazon CloudWatch alarms](CloudWatch_Alarms.md) and [Create an alarm from a metric on a graph](create_alarm_metric_graph.md). For OpenTelemetry metrics, you can create PromQL-based CloudWatch Alarms. These alarms use PromQL queries to define alarm conditions, using the same query language available in CloudWatch Query Studio. # Basic monitoring and detailed monitoring in CloudWatch Basic and detailed monitoring CloudWatch provides two categories of monitoring: *basic monitoring* and *detailed monitoring*. Many AWS services offer basic monitoring by publishing a default set of metrics to CloudWatch with no charge to customers. By default, when you start using one of these AWS services, basic monitoring is automatically enabled. For a list of services that offer basic monitoring, see [AWS services that publish CloudWatch metrics](aws-services-cloudwatch-metrics.md). Detailed monitoring is offered by only some services. It also incurs charges. To use it for an AWS service, you must choose to activate it. For more information about pricing, see [Amazon CloudWatch pricing](http://aws.amazon.com/cloudwatch/pricing). Detailed monitoring options differ based on the services that offer it. For example, Amazon EC2 detailed monitoring provides more frequent metrics, published at one-minute intervals, instead of the five-minute intervals used in Amazon EC2 basic monitoring. Detailed monitoring for Amazon S3 and Amazon Managed Streaming for Apache Kafka means more fine-grained metrics. In different AWS services, detailed monitoring also has different names. For example, in Amazon EC2 it is called detailed monitoring, in AWS Elastic Beanstalk it is called enhanced monitoring, and in Amazon S3 it is called request metrics. Using detailed monitoring for Amazon EC2 helps you better manage your Amazon EC2 resources, so that you can find trends and take action faster. For Amazon S3 request metrics are available at one-minute intervals to help you quickly identify and act on operational issues. On Amazon MSK, when you enable the `PER_BROKER`, `PER_TOPIC_PER_BROKER`, or `PER_TOPIC_PER_PARTITION` level monitoring, you get additional metrics that provide more visibility. The following table lists the services that offer detailed monitoring. It also includes links to the documentation for those services that explain more about the detailed monitoring and provide instructions for how to activate it. | Service | Documentation | | --- | --- | | Amazon API Gateway | [ Dimensions for API Gateway metrics](https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-metrics-and-dimensions.html#api-gateway-metricdimensions) | | AWS AppSync | [ CloudWatch metrics](https://docs.aws.amazon.com/appsync/latest/devguide/monitoring.html#cw-metrics) | | Amazon CloudFront | [ Viewing additional CloudFront distribution metrics](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/viewing-cloudfront-metrics.html#monitoring-console.distributions-additional) | | Amazon EC2 | [ Manage detailed monitoring for your EC2 instances](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/manage-detailed-monitoring.html) | | Elastic Beanstalk | [ Enhanced health reporting and monitoring](https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/health-enhanced.html) | | Amazon Kinesis Data Streams | [ Enhanced Shard-level Metrics](https://docs.aws.amazon.com/streams/latest/dev/monitoring-with-cloudwatch.html#kinesis-metrics-shard) | | AWS Lambda | [ Event source mapping metrics](https://docs.aws.amazon.com/lambda/latest/dg/monitoring-metrics-types.html#event-source-mapping-metrics) | | Amazon MSK | [Amazon MSK Metrics for Monitoring with CloudWatch](https://docs.aws.amazon.com/msk/latest/developerguide/metrics-details.html) | | Amazon S3 | [Amazon S3 request metrics in CloudWatch](https://docs.aws.amazon.com/AmazonS3/latest/userguide/metrics-dimensions.html#s3-request-cloudwatch-metrics) | | Amazon SES | [Collect CloudWatch detailed monitoring metrics using Amazon SES event publishing](https://docs.aws.amazon.com/ses/latest/dg/event-publishing-add-event-destination-cloudwatch.html#event-publishing-add-event-destination-cloudwatch-dimensions). | Additionally, CloudWatch offers out-of-the-box monitoring solutions with more detailed metrics and pre-created dashboards for some AWS services, as shown in the following table. | Service | Feature documentation | | --- | --- | | Lambda | [ Lambda Insights](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Lambda-Insights.html) | | Amazon ECS | [ Container Insights for Amazon ECS](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/deploy-container-insights-ECS.html) | | Amazon EKS | [ Container Insights for Amazon EKS and Kubernetes](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/deploy-container-insights-EKS.html) | | Amazon RDS | [CloudWatch Database Insights](Database-Insights.md) | | Amazon Aurora | [CloudWatch Database Insights](Database-Insights.md) | # Query your CloudWatch metrics with CloudWatch Metrics Insights Query metrics with CloudWatch Metrics Insights CloudWatch Metrics Insights is a powerful high-performance SQL query engine that you can use to query your metrics at scale. You can identify trends and patterns within all of your CloudWatch metrics in real time, with access to up to two weeks of historical data for comprehensive trend analysis. You can also set alarms on any Metrics Insights queries that return a single time series. This can be especially useful to create alarms that watch aggregated metrics across a fleet of your infrastructure or applications. Create the alarm once, and it dynamically adjusts as resources are added to or removed from the fleet. You can perform a CloudWatch Metrics Insights query in the console with the CloudWatch Metrics Insights query editor. You can also perform a CloudWatch Metrics Insights query with the AWS CLI or an AWS SDK by running `GetMetricData` or `PutDashboard`. There's no charge for queries that you run with the CloudWatch Metrics Insights query editor. For more information about CloudWatch pricing, see [Amazon CloudWatch Pricing](https://aws.amazon.com/cloudwatch/pricing/). With the CloudWatch Metrics Insights query editor, you can choose from a variety of prebuilt sample queries and also create your own queries. As you create your queries, you can use a builder view to browse your existing metrics and dimensions. Alternatively, use an editor view to manually write queries. With Metrics Insights, you can run queries at scale. With the **GROUP BY** clause, you can group your metrics in real time into separate time series per specific dimension value. Because Metrics Insights queries include an **ORDER BY** ability, you can use Metrics Insights to make "Top N" type queries. For example, "Top N" type queries can scan millions of metrics in your account and return the 10 instances that consume the most CPU. This can help you pinpoint and remedy latency issues in your applications. To use tags with alarms, opt in through CloudWatch Settings. After tags is enabled, you can also filter and group metrics using AWS resource tags, enabling you to query metrics aligned with your organizational structure, such as by application, environment, or team. **Topics** + [ # Building your queries in CloudWatch Metrics Insights ](cloudwatch-metrics-insights-buildquery.md) + [ # Query components and syntax in CloudWatch Metrics Insights ](cloudwatch-metrics-insights-querylanguage.md) + [ # Alarms on CloudWatch Metrics Insights queries in CloudWatch ](cloudwatch-metrics-insights-alarms.md) + [ # Use Metrics Insights queries with metric math ](cloudwatch-metrics-insights-math.md) + [ # Use natural language to generate and update CloudWatch Metrics Insights queries ](cloudwatch-metrics-insights-query-assist.md) + [ # SQL inference ](cloudwatch-metrics-insights-inference.md) + [ # Metrics Insights quotas ](cloudwatch-metrics-insights-limits.md) + [ # Metrics Insights sample queries ](cloudwatch-metrics-insights-queryexamples.md) + [ # Metrics Insights glossary ](cloudwatch-metrics-insights-glossary.md) + [ # Troubleshooting Metrics Insights ](cloudwatch-metrics-insights-troubleshooting.md) # Building your queries in CloudWatch Metrics Insights Building queries You can run a CloudWatch Metrics Insights query using the CloudWatch console, the AWS CLI, or the AWS SDKs. Queries run in the console are free of charge. For more information about CloudWatch pricing, see [Amazon CloudWatch Pricing](https://aws.amazon.com/cloudwatch/pricing/). With CloudWatch Metrics Insights, you can analyze metric data across extended time periods of up to two weeks, allowing for more comprehensive historical analysis and trend identification compared to shorter retention periods. For optimal performance when querying longer time ranges, consider using larger periods (such as 5 minutes or 1 hour) to reduce the number of data points returned. When analyzing trends over the full two-week period, use aggregate functions like AVG() or MAX() in your ORDER BY clauses to efficiently identify patterns. For more information about using the AWS SDKs to perform a Metrics Insights query, see [GetMetricData](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_GetMetricData.html). To run a query using the CloudWatch console, follow these steps: **To query your metrics using Metrics Insights** 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the navigation pane, choose **Metrics**, **All metrics**. 1. (Optional) To run a pre-built sample query, choose **Add query** and select the query to run. If you are satisfied with this query, you can skip the rest of this procedure. Or, you can choose **Editor** to edit the sample query and then choose **Run** to run the modified query. 1. To create your own query, choose **Multi source query**. You can then use the **Builder** view (default) to get a guided experience, or the **Editor** view if you prefer to see the query syntax. You can switch between the two views anytime and see your work in progress in both views. In the **Builder** view, click on the namespace, metric name, filter, group, order, and limit fields to browse and select possible values. You can start typing any part of the value you are looking for to filter the list presented by the builder. You can reference resource tags in the filter and group inputs. In the **Editor** view, you can write the query using the subset of SQL supported by Metrics Insights. The editor offers auto-complete options based on the characters you have typed so far, including name of resource tags for metrics that support them. CloudWatch Metrics Insights supports querying metrics by AWS resource tags. You can use tags to filter and group your metric data for more targeted monitoring and analysis. The following examples shows how you can use queries with tags. To see the CPU utilization for Amazon EC2 instances within your production environment: ``` SELECT MAX(CPUUtilization) FROM SCHEMA("AWS/EC2") WHERE tag.env='prod' ``` To group the metrics by environment using the GROUP BY clause: ``` SELECT MAX(CPUUtilization) FROM SCHEMA("AWS/EC2") GROUP BY tag.env ``` To use the GROUP BY clause where you specify the tag name: ``` SELECT AVG(CPUUtilization) FROM "AWS/EC2" GROUP BY tag."aws:cloudformation:stack-name" ``` To combine tag queries with existing metric dimensions: ``` SELECT MAX(CPUUtilization) FROM SCHEMA("AWS/EC2") WHERE tag.env='prod' AND InstanceId='i-1234567890abcdef0' ``` 1. When you are satisfied with your query, choose **Run**. 1. (Optional) Another way to edit a query that you have graphed is to choose the **Graphed metrics** tab and choose the edit icon next to the query formula in the **Details** column. 1. (Optional) To remove a query from the graph, choose **Graphed metrics** and choose the **X** icon at the right side of the row that displays your query. # Query components and syntax in CloudWatch Metrics Insights Query components and syntax CloudWatch Metrics Insights syntax is as follows. ``` SELECT FUNCTION(metricName) FROM namespace | SCHEMA(...) [ WHERE labelKey OPERATOR labelValue [AND ... ] ] [ GROUP BY labelKey [ , ... ] ] [ ORDER BY FUNCTION() [ DESC | ASC ] ] [ LIMIT number ] ``` The possible clauses in a Metrics Insights query are as follows. None of the keywords are case sensitive, but the identifiers such as the names of metrics, namespaces, and dimensions are case sensitive. **SELECT** Required. Specifies the function to use to aggregate observations in each time bucket (determined by the provided period). Also specifies the name of the metric to query. The valid values for **FUNCTION** are `AVG`, `COUNT`, `MAX`, `MIN`, and `SUM`. + `AVG` calculates the average of the observations matched by the query. + `COUNT` returns the count of the observations matched by the query. + `MAX` returns the maximum value of the observations matched by the query. + `MIN` returns the minimum value of the observations matched by the query. + `SUM` calculates the sum of the observations matched by the query. **FROM** Required. Specifies the source of the metric. You can specify either the metric namespace that contains the metric that is to be queried, or a **SCHEMA** table function. Examples of metric namespaces include `"AWS/EC2"`, `"AWS/Lambda"`, and metric namespaces that you have created for your custom metrics. Metric namespaces that include **/** or any other character that is not a letter, number, or underscore must be surrounded by double quotation marks. For more information, see [What needs quotation marks or escape characters?](#cloudwatch-metrics-insights-syntaxdetails). **SCHEMA** An optional table function that can be used within a **FROM** clause. Use **SCHEMA** to scope down the query results to only the metrics that exactly match a list of dimensions, or to metrics that have no dimensions. If you use a **SCHEMA** clause, it must contain at least one argument, and this first argument must be the metric namespace being queried. If you specify **SCHEMA** with only this namespace argument, the results are scoped down to only metrics that do not have any dimensions. If you specify **SCHEMA** with additional arguments, the additional arguments after the namespace argument must be *label* keys. Label keys must be dimension names. If you specify one or more of these label keys, the results are scoped down to only those metrics that have that exact set of dimensions. The order of these label keys does not matter. For example: + **SELECT AVG(CPUUtilization) FROM "AWS/EC2"** matches all `CPUUtilization` metrics in the `AWS/EC2` namespace, no matter their dimensions, and returns a single aggregated time series. + **SELECT AVG(CPUUtilization) FROM SCHEMA("AWS/EC2")** matches only the `CPUUtilization` metrics in the `AWS/EC2` namespace that do not have any dimensions defined. + **SELECT AVG(CPUUtilization) FROM SCHEMA("AWS/EC2", InstanceId)** matches only the `CPUUtilization` metrics that were reported to CloudWatch with exactly one dimension, `InstanceId`. + **SELECT SUM(RequestCount) FROM SCHEMA("AWS/ApplicationELB", LoadBalancer, AvailabilityZone)** matches only the `RequestCount` metrics that were reported to CloudWatch from `AWS/ApplicationELB` with exactly two dimensions, `LoadBalancer` and `AvailabilityZone`. **WHERE** Optional. Filters the results to only those metrics that match your specified expression using specific label values for one or more label keys. For example, **WHERE InstanceType = 'c3.4xlarge'** filters the results to only `c3.4xlarge` instance types, and **WHERE InstanceType \$1= 'c3.4xlarge'** filters the results to all instance types except `c3.4xlarge`. When you run a query in a monitoring account, you can use `WHERE AWS.AccountId` to limit results to only the account that you specify. For example, `WHERE AWS.AccountId=444455556666` queries metrics from only account `444455556666`. To limit your query to only metrics in the monitoring account itself, use `WHERE AWS.AccountId=CURRENT_ACCOUNT_ID()`. Label values must always be enclosed with single quotation marks. **Using tags in WHERE clauses** You can filter results by AWS resource tags using the syntax `tag.keyName`. Tag filters follow the same operator rules as dimension filters. For example: + WHERE `tag.env = 'prod'` filters to metrics from resources tagged with *env=prod* + WHERE `tag.department != 'test'` excludes metrics from resources tagged with *department=test* Tag filters can be combined with dimension filters: `WHERE tag.env = 'prod' AND InstanceType = 'm5.large'` **Supported operators** The **WHERE** clause supports the following operators: + **=** Label value must match the specified string. + **\$1=** Label value must not match the specified string. + **AND** Both conditions that are specified must be true to match. You can use multiple **AND** keywords to specify two or more conditions. **GROUP BY** Optional. Groups the query results into multiple time series, each one corresponding to a different value for the specified label key or keys. For example, using `GROUP BY InstanceId` returns a different time series for each value of `InstanceId`. Using `GROUP BY ServiceName, Operation` creates a different time series for each possible combination of the values of `ServiceName` and `Operation`. With a **GROUP BY** clause, by default the results are ordered in alphabetical ascending order, using the sequence of labels specified in the **GROUP BY** clause. To change the order of the results, add an **ORDER BY** clause to your query. When you run a query in a monitoring account, you can use `GROUP BY AWS.AccountId` to group the results based on the accounts they are from. **Using tags in GROUP BY clauses** You can group results by AWS resource tag values using the syntax `tag.keyName`. For example: + *GROUP BY tag.environment* creates separate time series for each environment tag value + *GROUP BY tag.team, InstanceType* groups by both tag and dimension values + *GROUP BY tag.team, AWS.AccountId* groups by both tag and linked source AccountIDs If some of the matching metrics don't include a specific label key specified in the **GROUP BY** clause, a null group named `Other` is returned. For example, if you specify `GROUP BY ServiceName, Operation` and some of the returned metrics don't include `ServiceName` as a dimension, then those metrics are displayed as having `Other` as the value for `ServiceName`. **ORDER BY** Optional. Specifies the order to use for the returned time series, if the query returns more than one time series. The order is based on the values found by the **FUNCTION** that you specify in the **ORDER BY** clause. The **FUNCTION** is used to calculate a single scalar value from each returned time series, and that value is used to determine the order. You also specify whether to use ascending **ASC** or descending **DESC** order. If you omit this, the default is ascending **ASC**. For example, adding an `ORDER BY MAX() DESC` clause orders the results by the maximum data point observed within the time range, in descending order: meaning that the time series that has the highest maximum data point is returned first. The valid functions to use within an **ORDER BY** clause are `AVG()`, `COUNT()`, `MAX()`, `MIN()`, and `SUM()`. If you use an **ORDER BY** clause with a **LIMIT** clause, the resulting query is a "Top N" query. **ORDER BY** is also useful for queries that might return a large number of metrics, because each query can return no more than 500 time series. If a query matches more than 500 time series, and you use an **ORDER BY** clause, the time series are sorted and then the 500 time series that come first in the sort order are the ones that are returned. **LIMIT** Optional. Limits the number of time series returned by the query to the value that you specify. The maximum value that you can specify is 500, and a query that does not specify a **LIMIT** can also return no more than 500 time series. Using a **LIMIT** clause with an **ORDER BY** clause gives you a "Top N" query. ## What needs quotation marks or escape characters? In a query, label values must always be surrounded with single quotation marks. For example, **SELECT MAX(CPUUtilization) FROM "AWS/EC2" WHERE AutoScalingGroupName = 'my-production-fleet'**. Metric namespaces, metric names, and label keys that contain characters other than letters, numbers, and underscore (\$1) must be surrounded by double quote marks. For example, **SELECT MAX("My.Metric")**. If one of these contains a double quotation mark or single quotation mark itself (such as `Bytes"Input"`), you must escape each quotation mark with a backslash, as in **SELECT AVG("Bytes\$1"Input\$1"")**. If a metric namespace, metric name, or label key, contains a word that is a reserved keyword in Metrics Insights, these must also be enclosed in double quotation marks. For example, if you have a metric named `LIMIT`, you would use `SELECT AVG("LIMIT")`. It is also valid to enclose any namespace, metric name, or label in double quotation marks even if it does not include a reserved keyword. For a complete list of reserved keywords, see [Reserved keywords](cloudwatch-metrics-insights-reserved-keywords.md). ## Build a rich query step by step This section illustrates building a full example that uses all possible clauses, step by step. You can start with the following query, which aggregates all of the Application Load Balancer `RequestCount` metrics that are collected with both the dimensions `LoadBalancer` and `AvailabilityZone`. ``` SELECT SUM(RequestCount) FROM SCHEMA("AWS/ApplicationELB", LoadBalancer, AvailabilityZone) ``` To see metrics only from a specific load balancer, you can add a **WHERE** clause to limit the metrics returned to only those metrics where the value of the `LoadBalancer` dimension is `app/load-balancer-1`. ``` SELECT SUM(RequestCount) FROM SCHEMA("AWS/ApplicationELB", LoadBalancer, AvailabilityZone) WHERE LoadBalancer = 'app/load-balancer-1' ``` The preceding query aggregates the `RequestCount` metrics from all Availability Zones for this load balancer into one time series. If you want to see different time series for each Availability Zone, we can add a **GROUP BY** clause. ``` SELECT SUM(RequestCount) FROM SCHEMA("AWS/ApplicationELB", LoadBalancer, AvailabilityZone) WHERE LoadBalancer = 'app/load-balancer-1' GROUP BY AvailabilityZone ``` Next, you can order the results to see the highest values first. The following **ORDER BY** clause orders the time series in descending order, by the maximum value reported by each time series during the query time range: ``` SELECT SUM(RequestCount) FROM SCHEMA("AWS/ApplicationELB", LoadBalancer, AvailabilityZone) WHERE LoadBalancer = 'app/load-balancer-1' GROUP BY AvailabilityZone ORDER BY MAX() DESC ``` You can also use tags to further filter the results. For example, if you want to see results only for load balancers tagged with a specific environment, we can add tag filtering to the WHERE clause: ``` SELECT SUM(RequestCount) FROM SCHEMA("AWS/ApplicationELB", LoadBalancer, AvailabilityZone) WHERE LoadBalancer = 'app/load-balancer-1' AND tag.Environment = 'prod' GROUP BY AvailabilityZone ORDER BY MAX() DESC ``` You can also group the results by tag values instead of (or in addition to) dimensions. For example, grouping by the Application tag: ``` SELECT SUM(RequestCount) FROM SCHEMA("AWS/ApplicationELB", LoadBalancer, AvailabilityZone) WHERE tag.Environment = 'prod' GROUP BY tag.Application ORDER BY MAX() DESC ``` Finally, if we are primarily interested in a "Top N" type of query, we can use a **LIMIT** clause. This final example limits the results to only the time series with the five highest `MAX` values. ``` SELECT SUM(RequestCount) FROM SCHEMA("AWS/ApplicationELB", LoadBalancer, AvailabilityZone) WHERE LoadBalancer = 'app/load-balancer-1' GROUP BY AvailabilityZone ORDER BY MAX() DESC LIMIT 5 ``` ## Cross-account query examples These examples are valid when run in an account set up as a monitoring account in CloudWatch cross-account observability. The following example searches all Amazon EC2 instances in the source account 123456789012 and returns the average. ``` SELECT AVG(CpuUtilization) FROM "AWS/EC2" WHERE AWS.AccountId ='123456789012' ``` The following example queries the `CPUUtilization` metric in `AWS/EC2` in all the linked source accounts, and groups the results by account ID and instance type. ``` SELECT AVG(CpuUtilization) FROM "AWS/EC2" GROUP BY AWS.AccountId, InstanceType ``` The following example queries the `CPUUtilization` in the monitoring account itself. ``` SELECT AVG(CpuUtilization) FROM "AWS/EC2" WHERE AWS.AccountId = CURRENT_ACCOUNT_ID() ``` # Reserved keywords The following are reserved keywords in CloudWatch Metrics Insights. If any of these words are in a namespace, metric name, or label key in a query, you must enclose them in double quote marks. Reserved keywords are not case sensitive. ``` "ABORT" "ABORTSESSION" "ABS" "ABSOLUTE" "ACCESS" "ACCESSIBLE" "ACCESS_LOCK" "ACCOUNT" "ACOS" "ACOSH" "ACTION" "ADD" "ADD_MONTHS" "ADMIN" "AFTER" "AGGREGATE" "ALIAS" "ALL" "ALLOCATE" "ALLOW" "ALTER" "ALTERAND" "AMP" "ANALYSE" "ANALYZE" "AND" "ANSIDATE" "ANY" "ARE" "ARRAY", "ARRAY_AGG" "ARRAY_EXISTS" "ARRAY_MAX_CARDINALITY" "AS" "ASC" "ASENSITIVE" "ASIN" "ASINH" "ASSERTION" "ASSOCIATE" "ASUTIME" "ASYMMETRIC" "AT", "ATAN" "ATAN2" "ATANH" "ATOMIC" "AUDIT" "AUTHORIZATION" "AUX" "AUXILIARY" "AVE" "AVERAGE" "AVG" "BACKUP" "BEFORE" "BEGIN" "BEGIN_FRAME" "BEGIN_PARTITION", "BETWEEN" "BIGINT" "BINARY" "BIT" "BLOB" "BOOLEAN" "BOTH" "BREADTH" "BREAK" "BROWSE" "BT" "BUFFERPOOL" "BULK" "BUT" "BY" "BYTE" "BYTEINT" "BYTES" "CALL", "CALLED" "CAPTURE" "CARDINALITY" "CASCADE" "CASCADED" "CASE" "CASESPECIFIC" "CASE_N" "CAST" "CATALOG" "CCSID" "CD" "CEIL" "CEILING" "CHANGE" "CHAR", "CHAR2HEXINT" "CHARACTER" "CHARACTERS" "CHARACTER_LENGTH" "CHARS" "CHAR_LENGTH" "CHECK" "CHECKPOINT" "CLASS" "CLASSIFIER" "CLOB" "CLONE" "CLOSE" "CLUSTER", "CLUSTERED" "CM" "COALESCE" "COLLATE" "COLLATION" "COLLECT" "COLLECTION" "COLLID" "COLUMN" "COLUMN_VALUE" "COMMENT" "COMMIT" "COMPLETION" "COMPRESS" "COMPUTE", "CONCAT" "CONCURRENTLY" "CONDITION" "CONNECT" "CONNECTION" "CONSTRAINT" "CONSTRAINTS" "CONSTRUCTOR" "CONTAINS" "CONTAINSTABLE" "CONTENT" "CONTINUE" "CONVERT", "CONVERT_TABLE_HEADER" "COPY" "CORR" "CORRESPONDING" "COS" "COSH" "COUNT" "COVAR_POP" "COVAR_SAMP" "CREATE" "CROSS" "CS" "CSUM" "CT" "CUBE" "CUME_DIST", "CURRENT" "CURRENT_CATALOG" "CURRENT_DATE" "CURRENT_DEFAULT_TRANSFORM_GROUP" "CURRENT_LC_CTYPE" "CURRENT_PATH" "CURRENT_ROLE" "CURRENT_ROW" "CURRENT_SCHEMA", "CURRENT_SERVER" "CURRENT_TIME" "CURRENT_TIMESTAMP" "CURRENT_TIMEZONE" "CURRENT_TRANSFORM_GROUP_FOR_TYPE" "CURRENT_USER" "CURRVAL" "CURSOR" "CV" "CYCLE" "DATA", "DATABASE" "DATABASES" "DATABLOCKSIZE" "DATE" "DATEFORM" "DAY" "DAYS" "DAY_HOUR" "DAY_MICROSECOND" "DAY_MINUTE" "DAY_SECOND" "DBCC" "DBINFO" "DEALLOCATE" "DEC", "DECFLOAT" "DECIMAL" "DECLARE" "DEFAULT" "DEFERRABLE" "DEFERRED" "DEFINE" "DEGREES" "DEL" "DELAYED" "DELETE" "DENSE_RANK" "DENY" "DEPTH" "DEREF" "DESC" "DESCRIBE", "DESCRIPTOR" "DESTROY" "DESTRUCTOR" "DETERMINISTIC" "DIAGNOSTIC" "DIAGNOSTICS" "DICTIONARY" "DISABLE" "DISABLED" "DISALLOW" "DISCONNECT" "DISK" "DISTINCT", "DISTINCTROW" "DISTRIBUTED" "DIV" "DO" "DOCUMENT" "DOMAIN" "DOUBLE" "DROP" "DSSIZE" "DUAL" "DUMP" "DYNAMIC" "EACH" "ECHO" "EDITPROC" "ELEMENT" "ELSE" "ELSEIF", "EMPTY" "ENABLED" "ENCLOSED" "ENCODING" "ENCRYPTION" "END" "END-EXEC" "ENDING" "END_FRAME" "END_PARTITION" "EQ" "EQUALS" "ERASE" "ERRLVL" "ERROR" "ERRORFILES", "ERRORTABLES" "ESCAPE" "ESCAPED" "ET" "EVERY" "EXCEPT" "EXCEPTION" "EXCLUSIVE" "EXEC" "EXECUTE" "EXISTS" "EXIT" "EXP" "EXPLAIN" "EXTERNAL" "EXTRACT" "FALLBACK "FALSE" "FASTEXPORT" "FENCED" "FETCH" "FIELDPROC" "FILE" "FILLFACTOR" "FILTER" "FINAL" "FIRST" "FIRST_VALUE" "FLOAT" "FLOAT4" "FLOAT8" "FLOOR" "FOR" "FORCE" "FOREIGN" "FORMAT" "FOUND" "FRAME_ROW" "FREE" "FREESPACE" "FREETEXT" "FREETEXTTABLE" "FREEZE" "FROM" "FULL" "FULLTEXT" "FUNCTION" "FUSION" "GE" "GENERAL" "GENERATED" "GET" "GIVE" "GLOBAL" "GO" "GOTO" "GRANT" "GRAPHIC" "GROUP" "GROUPING" "GROUPS" "GT" "HANDLER" "HASH" "HASHAMP" "HASHBAKAMP" "HASHBUCKET" "HASHROW" "HAVING" "HELP" "HIGH_PRIORITY" "HOLD" "HOLDLOCK" "HOUR" "HOURS" "HOUR_MICROSECOND" "HOUR_MINUTE" "HOUR_SECOND" "IDENTIFIED" "IDENTITY" "IDENTITYCOL" "IDENTITY_INSERT" "IF" "IGNORE" "ILIKE" "IMMEDIATE" "IN" "INCLUSIVE" "INCONSISTENT" "INCREMENT" "INDEX" "INDICATOR" "INFILE" "INHERIT" "INITIAL" "INITIALIZE" "INITIALLY" "INITIATE" "INNER" "INOUT" "INPUT" "INS" "INSENSITIVE" "INSERT" "INSTEAD" "INT" "INT1" "INT2" "INT3" "INT4" "INT8" "INTEGER" "INTEGERDATE" "INTERSECT" "INTERSECTION" "INTERVAL" "INTO" "IO_AFTER_GTIDS" "IO_BEFORE_GTIDS" "IS" "ISNULL" "ISOBID" "ISOLATION" "ITERATE" "JAR" "JOIN" "JOURNAL" "JSON_ARRAY" "JSON_ARRAYAGG" "JSON_EXISTS" "JSON_OBJECT" "JSON_OBJECTAGG" "JSON_QUERY" "JSON_TABLE" "JSON_TABLE_PRIMITIVE" "JSON_VALUE" "KEEP" "KEY" "KEYS" "KILL" "KURTOSIS" "LABEL" "LAG" "LANGUAGE" "LARGE" "LAST" "LAST_VALUE" "LATERAL" "LC_CTYPE" "LE" "LEAD" "LEADING" "LEAVE" "LEFT" "LESS" "LEVEL" "LIKE" "LIKE_REGEX" "LIMIT" "LINEAR" "LINENO" "LINES" "LISTAGG" "LN" "LOAD" "LOADING" "LOCAL" "LOCALE" "LOCALTIME" "LOCALTIMESTAMP" "LOCATOR" "LOCATORS" "LOCK" "LOCKING" "LOCKMAX" "LOCKSIZE" "LOG" "LOG10" "LOGGING" "LOGON" "LONG" "LONGBLOB" "LONGTEXT" "LOOP" "LOWER" "LOW_PRIORITY" "LT" "MACRO" "MAINTAINED" "MAP" "MASTER_BIND" "MASTER_SSL_VERIFY_SERVER_CERT" "MATCH" "MATCHES" "MATCH_NUMBER" "MATCH_RECOGNIZE" "MATERIALIZED" "MAVG" "MAX" "MAXEXTENTS" "MAXIMUM" "MAXVALUE" "MCHARACTERS" "MDIFF" "MEDIUMBLOB" "MEDIUMINT" "MEDIUMTEXT" "MEMBER" "MERGE" "METHOD" "MICROSECOND" "MICROSECONDS" "MIDDLEINT" "MIN" "MINDEX" "MINIMUM" "MINUS" "MINUTE" "MINUTES" "MINUTE_MICROSECOND" "MINUTE_SECOND" "MLINREG" "MLOAD" "MLSLABEL" "MOD" "MODE" "MODIFIES" "MODIFY" "MODULE" "MONITOR" "MONRESOURCE" "MONSESSION" "MONTH" "MONTHS" "MSUBSTR" "MSUM" "MULTISET" "NAMED" "NAMES" "NATIONAL" "NATURAL" "NCHAR" "NCLOB" "NE" "NESTED_TABLE_ID" "NEW" "NEW_TABLE" "NEXT" "NEXTVAL" "NO" "NOAUDIT" "NOCHECK" "NOCOMPRESS" "NONCLUSTERED" "NONE" "NORMALIZE" "NOT" "NOTNULL" "NOWAIT" "NO_WRITE_TO_BINLOG" "NTH_VALUE" "NTILE" "NULL" "NULLIF" "NULLIFZERO" "NULLS" "NUMBER" "NUMERIC" "NUMPARTS" "OBID" "OBJECT" "OBJECTS" "OCCURRENCES_REGEX" "OCTET_LENGTH" "OF" "OFF" "OFFLINE" "OFFSET" "OFFSETS" "OLD" "OLD_TABLE" "OMIT" "ON" "ONE" "ONLINE" "ONLY" "OPEN" "OPENDATASOURCE" "OPENQUERY" "OPENROWSET" "OPENXML" "OPERATION" "OPTIMIZATION" "OPTIMIZE" "OPTIMIZER_COSTS" "OPTION" "OPTIONALLY" "OR" "ORDER" "ORDINALITY" "ORGANIZATION" "OUT" "OUTER" "OUTFILE" "OUTPUT" "OVER" "OVERLAPS" "OVERLAY" "OVERRIDE" "PACKAGE" "PAD" "PADDED" "PARAMETER" "PARAMETERS" "PART" "PARTIAL" "PARTITION" "PARTITIONED" "PARTITIONING" "PASSWORD" "PATH" "PATTERN" "PCTFREE" "PER" "PERCENT" "PERCENTILE" "PERCENTILE_CONT" "PERCENTILE_DISC" "PERCENT_RANK" "PERIOD" "PERM" "PERMANENT" "PIECESIZE" "PIVOT" "PLACING" "PLAN" "PORTION" "POSITION" "POSITION_REGEX" "POSTFIX" "POWER" "PRECEDES" "PRECISION" "PREFIX" "PREORDER" "PREPARE" "PRESERVE" "PREVVAL" "PRIMARY" "PRINT" "PRIOR" "PRIQTY" "PRIVATE" "PRIVILEGES" "PROC" "PROCEDURE" "PROFILE" "PROGRAM" "PROPORTIONAL" "PROTECTION" "PSID" "PTF" "PUBLIC" "PURGE" "QUALIFIED" "QUALIFY" "QUANTILE" "QUERY" "QUERYNO" "RADIANS" "RAISERROR" "RANDOM" "RANGE" "RANGE_N" "RANK" "RAW" "READ" "READS" "READTEXT" "READ_WRITE" "REAL" "RECONFIGURE" "RECURSIVE" "REF" "REFERENCES" "REFERENCING" "REFRESH" "REGEXP" "REGR_AVGX" "REGR_AVGY" "REGR_COUNT" "REGR_INTERCEPT" "REGR_R2" "REGR_SLOPE" "REGR_SXX" "REGR_SXY" "REGR_SYY" "RELATIVE" "RELEASE" "RENAME" "REPEAT" "REPLACE" "REPLICATION" "REPOVERRIDE" "REQUEST" "REQUIRE" "RESIGNAL" "RESOURCE" "RESTART" "RESTORE" "RESTRICT" "RESULT" "RESULT_SET_LOCATOR" "RESUME" "RET" "RETRIEVE" "RETURN" "RETURNING" "RETURNS" "REVALIDATE" "REVERT" "REVOKE" "RIGHT" "RIGHTS" "RLIKE" "ROLE" "ROLLBACK" "ROLLFORWARD" "ROLLUP" "ROUND_CEILING" "ROUND_DOWN" "ROUND_FLOOR" "ROUND_HALF_DOWN" "ROUND_HALF_EVEN" "ROUND_HALF_UP" "ROUND_UP" "ROUTINE" "ROW" "ROWCOUNT" "ROWGUIDCOL" "ROWID" "ROWNUM" "ROWS" "ROWSET" "ROW_NUMBER" "RULE" "RUN" "RUNNING" "SAMPLE" "SAMPLEID" "SAVE" "SAVEPOINT" "SCHEMA" "SCHEMAS" "SCOPE" "SCRATCHPAD" "SCROLL" "SEARCH" "SECOND" "SECONDS" "SECOND_MICROSECOND" "SECQTY" "SECTION" "SECURITY" "SECURITYAUDIT" "SEEK" "SEL" "SELECT" "SEMANTICKEYPHRASETABLE" "SEMANTICSIMILARITYDETAILSTABLE" "SEMANTICSIMILARITYTABLE" "SENSITIVE" "SEPARATOR" "SEQUENCE" "SESSION" "SESSION_USER" "SET" "SETRESRATE" "SETS" "SETSESSRATE" "SETUSER" "SHARE" "SHOW" "SHUTDOWN" "SIGNAL" "SIMILAR" "SIMPLE" "SIN" "SINH" "SIZE" "SKEW" "SKIP" "SMALLINT" "SOME" "SOUNDEX" "SOURCE" "SPACE" "SPATIAL" "SPECIFIC" "SPECIFICTYPE" "SPOOL" "SQL" "SQLEXCEPTION" "SQLSTATE" "SQLTEXT" "SQLWARNING" "SQL_BIG_RESULT" "SQL_CALC_FOUND_ROWS" "SQL_SMALL_RESULT" "SQRT" "SS" "SSL" "STANDARD" "START" "STARTING" "STARTUP" "STAT" "STATE" "STATEMENT" "STATIC" "STATISTICS" "STAY" "STDDEV_POP" "STDDEV_SAMP" "STEPINFO" "STOGROUP" "STORED" "STORES" "STRAIGHT_JOIN" "STRING_CS" "STRUCTURE" "STYLE" "SUBMULTISET" "SUBSCRIBER" "SUBSET" "SUBSTR" "SUBSTRING" "SUBSTRING_REGEX" "SUCCEEDS" "SUCCESSFUL" "SUM" "SUMMARY" "SUSPEND" "SYMMETRIC" "SYNONYM" "SYSDATE" "SYSTEM" "SYSTEM_TIME" "SYSTEM_USER" "SYSTIMESTAMP" "TABLE" "TABLESAMPLE" "TABLESPACE" "TAN" "TANH" "TBL_CS" "TEMPORARY" "TERMINATE" "TERMINATED" "TEXTSIZE" "THAN" "THEN" "THRESHOLD" "TIME" "TIMESTAMP" "TIMEZONE_HOUR" "TIMEZONE_MINUTE" "TINYBLOB" "TINYINT" "TINYTEXT" "TITLE" "TO" "TOP" "TRACE" "TRAILING" "TRAN" "TRANSACTION" "TRANSLATE" "TRANSLATE_CHK" "TRANSLATE_REGEX" "TRANSLATION" "TREAT" "TRIGGER" "TRIM" "TRIM_ARRAY" "TRUE" "TRUNCATE" "TRY_CONVERT" "TSEQUAL" "TYPE" "UC" "UESCAPE" "UID" "UNDEFINED" "UNDER" "UNDO" "UNION" "UNIQUE" "UNKNOWN" "UNLOCK" "UNNEST" "UNPIVOT" "UNSIGNED" "UNTIL" "UPD" "UPDATE" "UPDATETEXT" "UPPER" "UPPERCASE" "USAGE" "USE" "USER" "USING" "UTC_DATE" "UTC_TIME" "UTC_TIMESTAMP" "VALIDATE" "VALIDPROC" "VALUE" "VALUES" "VALUE_OF" "VARBINARY" "VARBYTE" "VARCHAR" "VARCHAR2" "VARCHARACTER" "VARGRAPHIC" "VARIABLE" "VARIADIC" "VARIANT" "VARYING" "VAR_POP" "VAR_SAMP" "VCAT" "VERBOSE" "VERSIONING" "VIEW" "VIRTUAL" "VOLATILE" "VOLUMES" "WAIT" "WAITFOR" "WHEN" "WHENEVER" "WHERE" "WHILE" "WIDTH_BUCKET" "WINDOW" "WITH" "WITHIN" "WITHIN_GROUP" "WITHOUT" "WLM" "WORK" "WRITE" "WRITETEXT" "XMLCAST" "XMLEXISTS" "XMLNAMESPACES" "XOR" "YEAR" "YEARS" "YEAR_MONTH" "ZEROFILL" "ZEROIFNULL" "ZONE" ``` # Alarms on CloudWatch Metrics Insights queries in CloudWatch Alarms on queries You can create alarms on Metrics Insights queries. This helps you have alarms that track multiple resources without needing to be updated later. The query catches new resources and resources that change. For example, you can create an alarm that watches the CPU utilization of your fleet, and the alarm automatically evaluates new instances that you launch after creating the alarm. In a monitoring account that is set up for CloudWatch cross-account observability, your Metrics Insights alarms can watch resources in source accounts and in the monitoring account itself. For more information about how to limit your alarm queries to a specific account or to group the results by account ID, see the `WHERE` and `GROUP BY` sections in [Query components and syntax in CloudWatch Metrics Insights](cloudwatch-metrics-insights-querylanguage.md). **Using tags in alarm queries** You can create alarms on Metrics Insights queries that use AWS resource tags to filter and group metrics. To use tags with alarms, on the [https://console.aws.amazon.com/connect/](https://console.aws.amazon.com/connect/), choose **Settings**. On the **CloudWatch Settings** page, under **Enable resource tags on telemetry**, choose **Enable**. Context-aware alarms monitor specific applications, environments, or teams automatically as resources change. For example, you can create an alarm that monitors CPU utilization for all Amazon EC2 instances tagged with a specific application. ``` SELECT MAX(CPUUtilization) FROM "AWS/EC2" WHERE tag.Application = 'Orders' AND tag.Environment = 'Prod' ``` Tag-based alarms automatically adapt as you add or remove resources with matching tags, providing dynamic monitoring aligned with your operational structure. **Contents** + [ # Creating a Metrics Insights CloudWatch alarm ](cloudwatch-metrics-insights-alarm-create.md) # Creating a Metrics Insights CloudWatch alarm Creating a Metrics Insights alarm **To create an alarm on a Metrics Insights query using the console** 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the navigation pane, choose **Metrics**, **All metrics**. 1. (Optional) To run a pre-built sample query, choose **Add query** and select the query to run. Or, you can choose **Editor** to edit the sample query and then choose **Run** to run the modified query. 1. To create your own query, choose **Multi source query**. You can then use the **Builder** view, the **Editor** view, and also use a combination of both. You can switch between the two views anytime and see your work in progress in both views. In the **Builder** view, you can browse and select the metric namespace, metric name, filter, group, and order options. For each of these options, the query builder offers you a list of possible selections from your environment to choose from. In the **Editor** view, you can start writing your query. As you type, the editor offers suggestions based on the characters that you have typed so far. For example, when creating your Metrics Insights query for the alarm, you can use tags to filter and group metrics for more targeted monitoring. + Filter by tags – Use `WHERE tag.keyName = 'value'` to monitor resources with specific tags ``` SELECT MAX(CPUUtilization) FROM "AWS/EC2" WHERE tag.Environment = 'Prod' ``` + Combine tags with dimensions – Mix tag filters with existing metric dimensions ``` SELECT AVG(Duration) FROM "AWS/Lambda" WHERE tag.Application = 'OrderService' AND FunctionName = 'process%' ``` **Note** When using tags, alarms will match the metrics only if the specified tags existed on the associated resources during the evaluated time period. 1. When you are satisfied with your query, choose **Run**. 1. Choose **Create alarm**. 1. Under **Conditions**, specify the following: 1. For **Whenever *metric* is**, specify whether the metric must be greater than, less than, or equal to the threshold. Under **than...**, specify the threshold value. 1. Choose **Additional configuration**. For **Datapoints to alarm**, specify how many evaluation periods (data points) must be in the `ALARM` state to trigger the alarm. If the two values here match, you create an alarm that goes to `ALARM` state if that many consecutive periods are breaching. To create an M out of N alarm, specify a lower number for the first value than you specify for the second value. For more information, see [Alarm evaluation](alarm-evaluation.md). 1. For **Missing data treatment**, choose how to have the alarm behave when some data points are missing. For more information, see [Configuring how CloudWatch alarms treat missing data](alarms-and-missing-data.md). 1. Choose **Next**. 1. Under **Notification**, select an SNS topic to notify when the alarm is in `ALARM` state, `OK` state, or `INSUFFICIENT_DATA` state. To have the alarm send multiple notifications for the same alarm state or for different alarm states, choose **Add notification**. To have the alarm not send notifications, choose **Remove**. 1. To have the alarm perform Auto Scaling, EC2, or Systems Manager actions, choose the appropriate button and choose the alarm state and action to perform. Alarms can perform Systems Manager actions only when they go into ALARM state. For more information about Systems Manager actions, see [ Configuring CloudWatch to create OpsItems from alarms](https://docs.aws.amazon.com/systems-manager/latest/userguide/OpsCenter-create-OpsItems-from-CloudWatch-Alarms.html) and [ Incident creation](https://docs.aws.amazon.com/incident-manager/latest/userguide/incident-creation.html). **Note** To create an alarm that performs an SSM Incident Manager action, you must have certain permissions. For more information, see [ Identity-based policy examples for AWS Systems Manager Incident Manager](https://docs.aws.amazon.com/incident-manager/latest/userguide/security_iam_id-based-policy-examples.html). 1. When finished, choose **Next**. 1. Enter a name and description for the alarm. The name must contain only ASCII characters. Then choose **Next**. 1. Under **Preview and create**, confirm that the information and conditions are what you want, then choose **Create alarm**. **To create an alarm on a Metrics Insights query using the AWS CLI** Use the `put-metric-alarm` command and specify a Metrics Insights query in the `metrics` parameter. For example, the following command sets an alarm that goes into ALARM state if any of your instances go above 50% in CPU utilization. ``` aws cloudwatch put-metric-alarm —alarm-name Prod-App-CPU-Alarm —evaluation-periods 1 —comparison-operator GreaterThanThreshold —metrics '[{"Id":"m1","Expression":"SELECT MAX(CPUUtilization) FROM \"AWS/EC2\" WHERE tag.Environment = '\''Prod'\'' AND tag.Application = '\''OrderService'\''", "Period":60}]' —threshold 80 ``` # Use Metrics Insights queries with metric math You can use a Metrics Insights query as an input to a metric math function. For more information about metric math, see [Using math expressions with CloudWatch metrics](using-metric-math.md). A Metrics Insights query that does not include a **GROUP BY** clause returns a single time series. Therefore, its returned results can be used with any metric math function that takes a single time series as input. A Metrics Insights query that includes a **GROUP BY** clause returns multiple time series. Therefore, its returned results can be used with any metric math function that takes an array of time series as input. For example, the following query returns the total number of bytes downloaded for each bucket in the Region, as an array of time series: ``` SELECT SUM(BytesDownloaded) FROM SCHEMA("AWS/S3", BucketName, FilterId) WHERE FilterId = 'EntireBucket' GROUP BY BucketName ``` On a graph in the console or in a [GetMetricData](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_GetMetricData.html) operation, the results of this query are `q1`. This query returns the result in bytes, so if you want to see the result as MB instead, you can use the following math function: ``` q1/1024/1024 ``` # Use natural language to generate and update CloudWatch Metrics Insights queries CloudWatch supports a natural language query capability to help you generate and update queries for [CloudWatch Metrics Insights](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/query_with_cloudwatch-metrics-insights.html) and [CloudWatch Logs Insights](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AnalyzingLogData.html). With this capability, you can ask questions about or describe the CloudWatch data you're looking for in plain English. The natural language capability generates a query based on a prompt that you enter and provides a line-by-line explanation of how the query works. You can also update your query to further investigate your data. Depending on your environment, you can enter prompts like "Which Amazon Elastic Compute Cloud instance has the highest network out?" and "Show me the top 10 Amazon DynamoDB Tables by consumed reads." **Note** The natural-language query feature is generally available in 10 Regions. For some Regions, the feature makes cross-Region calls to Regions in the United States to process the query prompts. The following table lists the supported Regions, and shows where each Region processes its prompts. | Supported Region | Region where prompt is processed | | --- | --- | | US East (N. Virginia) | US East (N. Virginia) | | US East (Ohio) | US East (N. Virginia) | | US West (Oregon) | US West (Oregon) | | Asia Pacific (Hong Kong) | US West (Oregon) | | Asia Pacific (Singapore) | US West (Oregon) | | Asia Pacific (Sydney) | US West (Oregon) | | Asia Pacific (Tokyo) | Asia Pacific (Tokyo) | | Europe (Frankfurt) | Europe (Frankfurt) | | Europe (Ireland) | US East (N. Virginia) | | Europe (Stockholm) | US East (N. Virginia) | To generate a CloudWatch Metrics Insights query with this capability, open the CloudWatch Metrics Insights query editor in the *builder* or *editor* view and choose **Generate query**. **Important** To use the natural language query capability, you must use the [CloudWatchFullAccess](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/CloudWatchFullAccess.html), [CloudWatchReadOnlyAccess](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/CloudWatchReadOnlyAccess.html), [CloudWatchFullAccessV2](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/CloudWatchFullAccessV2.html), [AdministratorAccess](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AdministratorAccess.html), or [ReadOnlyAccess](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/ReadOnlyAccess.html) policy. You can also include the `cloudwatch:GenerateQuery` action in a new or existing customer managed or inline policy. ## Example queries The examples in this section describe how to generate and update queries using the natural language capability. **Note** For more information on the CloudWatch Metrics Insights query editor and syntax, see [CloudWatch Metrics Insights query components and syntax](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/cloudwatch-metrics-insights-querylanguage.html). ### Example: Generate a natural language query To generate a query using natural language, enter a prompt and choose **Generate new query**. This example shows a query that performs a basic search. **Prompt** The following is an example of a prompt that directs the capability to search for the top 10 DynamoDB Tables that consume the most read capacity. ``` Show top 10 DynamoDB Tables by consumed reads ``` **Query** The following is an example of a query that the natural language capability generates based on the prompt. Notice how the prompt appears in a comment before the query. After the query, you can read an explanation that describes how the query works. ``` # Show top 10 DynamoDB Tables by consumed reads SELECT SUM("ConsumedReadCapacityUnits") FROM "AWS/DynamoDB" GROUP BY TableName ORDER BY SUM() DESC LIMIT 10 # This query selects the sum of consumed read capacity units for each DynamoDB table, groups the results by table name, orders the results from highest to lowest read capacity consumption, and limits the results to the top 10 tables. ``` **Note** To turn off the appearance of your prompt and the explanation of how the query works, use the gear icon in your editor. ### Example: Update a natural language query You can update a query by editing the initial prompt and then choosing **Update query**. **Updated prompt** The following example shows an updated version of the previous prompt. Instead of a prompt that searches for the top 10 DynamoDB Tables that consume the most read capacity, this prompt now directs the capability to sort the results by the number of bytes returned. ``` Sort by bytes returned instead ``` **Updated query** The following is an example of the updated query. Notice how the updated prompt appears in a comment before the updated query. After the query, you can read an explanation that describes how the original query has been updated. ``` # Sort by bytes returned instead SELECT SUM("ReturnedBytes") FROM "AWS/DynamoDB" GROUP BY TableName ORDER BY SUM() DESC LIMIT 10 # This query modifies the original query to select the sum of returned bytes instead of consumed read capacity units, and orders the results from highest to lowest sum of returned bytes, limiting the results to the top 10 tables. ``` ## Opting out of using your data for service improvement The natural language prompt data you provide to train the AI model and generate relevant queries is used solely to provide and maintain your service. This data might be used to improve the quality of CloudWatch Metrics Insights. Your trust and privacy, as well as the security of your content, is our highest priority. For more information, see [AWS Service Terms](https://aws.amazon.com/service-terms/) and [AWS responsible AI policy](https://aws.amazon.com/machine-learning/responsible-ai/policy/). You can opt out of having your content used to develop or improve the quality of natural language queries by creating an AI service opt-out policy. To opt-out of data collection for all CloudWatch AI features, including the query generation capability, you must create an opt-out policy for CloudWatch. For more information, see [AI services opt-out policies](https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_policies_ai-opt-out.html) in the *AWS Organizations User Guide*. # SQL inference CloudWatch Metrics Insights uses several mechanisms to infer the intention of a given SQL query. **Topics** + [ ## Time bucketing ](#cloudwatch-metrics-insights-inference-timebucketing) + [ ## Fields projection ](#cloudwatch-metrics-insights-inference-fieldsprojection) + [ ## ORDER BY global aggregation ](#cloudwatch-metrics-insights-inference-OrderBy) ## Time bucketing Time series data points resulting from a query are rolled up into time buckets based on the requested period. To aggregate values in standard SQL, an explicit GROUP BY clause must be defined to collect all the observations of a given period together. Because this is the standard way to query time series data, CloudWatch Metrics Insights infers time bucketing without the need to express an explicit **GROUP BY** clause. For example, when a query is performed with a period of one minute, all the observations belonging to that minute until the next (excluded) are rolled up to the start time of the time bucket. This makes Metrics Insights SQL statements more concise and less verbose. ``` SELECT AVG(CPUUtilization) FROM SCHEMA("AWS/EC2", InstanceId) ``` The previous query returns a single time series (timestamp-value pairs), representing the average CPU utilization of all Amazon EC2 instances. Assuming the requested period is one minute, each data point returned represents the average of all observations measured within a specific one-minute interval (start time inclusive, end time exclusive). The timestamp related to the specific data point is the start time of the bucket ## Fields projection Metrics Insights queries always return the timestamp projection. You don’t need to specify a timestamp column in the **SELECT** clause to get the timestamp of each corresponding data point value. For details about how timestamp is calculated, see [Time bucketing](#cloudwatch-metrics-insights-inference-timebucketing). When using **GROUP BY**, each group name is also inferred and projected in the result, so that you can group the returned time series. ``` SELECT AVG(CPUUtilization) FROM SCHEMA("AWS/EC2", InstanceId) GROUP BY InstanceId ``` The previous query returns a time series for each Amazon EC2 instance. Each time series is labeled after the value of the instance ID. ## ORDER BY global aggregation When using **ORDER BY**, **FUNCTION()** infers which aggregate function that you want to order by (the data point values of the queried metrics). The aggregate operation is performed across all the matched data points of each individual time series across the queried time window. ``` SELECT AVG(CPUUtilization) FROM SCHEMA("AWS/EC2", InstanceId) GROUP BY InstanceId ORDER BY MAX() LIMIT 10 ``` The previous query returns the CPU utilization for each Amazon EC2 instance, limiting the result set to 10 entries. The results are ordered based on the maximum value of the individual time series within the requested time window. The **ORDER BY** clause is applied before **LIMIT**, so that the ordering is calculated against more than 10 time series. # Metrics Insights quotas CloudWatch Metrics Insights currently has the following quotas: + Query up to two weeks of data for visualization in your metrics, widgets, and alarm graphs. You can query the most recent three hours of data for alarm condition evaluations. + A single query can process no more than 10,000 metrics. This means that if the **SELECT**, **FROM**, and **WHERE** clauses match more than 10,000 metrics, the query only processes the first 10,000 of these metrics that it finds. + A single query can return no more than 500 time series. This means that if the query would return more than 500 metrics, not all metrics will be returned in the query results. If you use an **ORDER BY** clause, then all the metrics being processed are sorted, and the 500 that have the highest or lowest values according to your **ORDER BY** clause are returned. If you do not include an **ORDER BY** clause, you can't control which 500 matching metrics are returned. + You can have as many as 200 Metrics Insights alarms per Region. + Metrics Insights does not support high-resolution data, which is metric data reported with a granularity of less than one minute. If you request high-resolution data, the request does not fail, but the output is aggregated at one-minute granularity. + Each [GetMetricData](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_GetMetricData.html) operation can have only one query, but you can have multiple widgets in a dashboard that each include a query. + If a query using tag(s) with a **GROUP BY** or **WHERE** matches a metric that has more than 10 tag updates, only the most recent 10 tagged versions of the metric will be included in the query results. # Metrics Insights sample queries Sample queries This section contains examples of useful CloudWatch Metrics Insights queries that you can copy and use directly or copy and modify in query editor. Some of these examples are already available in the console, and you can access them by choosing **Add query** in the **Metrics** view. ## Application Load Balancer examples **Total requests across all load balancers** ``` SELECT SUM(RequestCount) FROM SCHEMA("AWS/ApplicationELB", LoadBalancer) ``` **Top 10 most active load balancers ** ``` SELECT MAX(ActiveConnectionCount) FROM SCHEMA("AWS/ApplicationELB", LoadBalancer) GROUP BY LoadBalancer ORDER BY SUM() DESC LIMIT 10 ``` ## AWS API usage examples **Top 20 AWS APIs by the number of calls in your account** ``` SELECT COUNT(CallCount) FROM SCHEMA("AWS/Usage", Class, Resource, Service, Type) WHERE Type = 'API' GROUP BY Service, Resource ORDER BY COUNT() DESC LIMIT 20 ``` **CloudWatch APIs sorted by calls** ``` SELECT COUNT(CallCount) FROM SCHEMA("AWS/Usage", Class, Resource, Service, Type) WHERE Type = 'API' AND Service = 'CloudWatch' GROUP BY Resource ORDER BY COUNT() DESC ``` ## DynamoDB examples **Top 10 tables by consumed reads** ``` SELECT SUM(ProvisionedWriteCapacityUnits) FROM SCHEMA("AWS/DynamoDB", TableName) GROUP BY TableName ORDER BY MAX() DESC LIMIT 10 ``` **Top 10 tables by returned bytes** ``` SELECT SUM(ReturnedBytes) FROM SCHEMA("AWS/DynamoDB", TableName) GROUP BY TableName ORDER BY MAX() DESC LIMIT 10 ``` **Top 10 tables by user errors** ``` SELECT SUM(UserErrors) FROM SCHEMA("AWS/DynamoDB", TableName) GROUP BY TableName ORDER BY MAX() DESC LIMIT 10 ``` ## Amazon Elastic Block Store examples **Top 10 Amazon EBS volumes by bytes written** ``` SELECT SUM(VolumeWriteBytes) FROM SCHEMA("AWS/EBS", VolumeId) GROUP BY VolumeId ORDER BY SUM() DESC LIMIT 10 ``` **Average Amazon EBS volume write time** ``` SELECT AVG(VolumeTotalWriteTime) FROM SCHEMA("AWS/EBS", VolumeId) ``` ## Amazon EC2 examples **CPU utilization of EC2 instances sorted by highest ** ``` SELECT AVG(CPUUtilization) FROM SCHEMA("AWS/EC2", InstanceId) GROUP BY InstanceId ORDER BY AVG() DESC ``` **Average CPU utilization across the entire fleet** ``` SELECT AVG(CPUUtilization) FROM SCHEMA("AWS/EC2", InstanceId) ``` **Top 10 instances by highest CPU utilization** ``` SELECT MAX(CPUUtilization) FROM SCHEMA("AWS/EC2", InstanceId) GROUP BY InstanceId ORDER BY MAX() DESC LIMIT 10 ``` **In this case, the CloudWatch agent is collecting a `CPUUtilization` metric per application. This query filters the average of this metric for a specific application name.** ``` SELECT AVG(CPUUtilization) FROM "AWS/CWAgent" WHERE ApplicationName = 'eCommerce' ``` ## Amazon Elastic Container Service examples **Average CPU utilization across all ECS clusters ** ``` SELECT AVG(CPUUtilization) FROM SCHEMA("AWS/ECS", ClusterName) ``` **Top 10 clusters by memory utilization** ``` SELECT AVG(MemoryUtilization) FROM SCHEMA("AWS/ECS", ClusterName) GROUP BY ClusterName ORDER BY AVG() DESC LIMIT 10 ``` **Top 10 services by CPU utilization** ``` SELECT AVG(CPUUtilization) FROM SCHEMA("AWS/ECS", ClusterName, ServiceName) GROUP BY ClusterName, ServiceName ORDER BY AVG() DESC LIMIT 10 ``` **Top 10 services by running tasks (Container Insights)** ``` SELECT AVG(RunningTaskCount) FROM SCHEMA("ECS/ContainerInsights", ClusterName, ServiceName) GROUP BY ClusterName, ServiceName ORDER BY AVG() DESC LIMIT 10 ``` ## Amazon Elastic Kubernetes Service Container Insights examples **Average CPU utilization across all EKS clusters ** ``` SELECT AVG(pod_cpu_utilization) FROM SCHEMA("ContainerInsights", ClusterName) ``` **Top 10 clusters by node CPU utilization** ``` SELECT AVG(node_cpu_utilization) FROM SCHEMA("ContainerInsights", ClusterName) GROUP BY ClusterName ORDER BY AVG() DESC LIMIT 10 ``` **Top 10 clusters by pod memory utilization** ``` SELECT AVG(pop_memory_utilization) FROM SCHEMA("ContainerInsights", ClusterName) GROUP BY ClusterName ORDER BY AVG() DESC LIMIT 10 ``` **Top 10 nodes by CPU utilization** ``` SELECT AVG(node_cpu_utilization) FROM SCHEMA("ContainerInsights", ClusterName, NodeName) GROUP BY ClusterName, NodeName ORDER BY AVG() DESC LIMIT 10 ``` **Top 10 pods by memory utilization** ``` SELECT AVG(pod_memory_utilization) FROM SCHEMA("ContainerInsights", ClusterName, PodName) GROUP BY ClusterName, PodName ORDER BY AVG() DESC LIMIT 10 ``` ## EventBridge examples **Top 10 rules by invocations** ``` SELECT SUM(Invocations) FROM SCHEMA("AWS/Events", RuleName) GROUP BY RuleName ORDER BY MAX() DESC LIMIT 10 ``` **Top 10 rules by failed invocations** ``` SELECT SUM(FailedInvocations) FROM SCHEMA("AWS/Events", RuleName) GROUP BY RuleName ORDER BY MAX() DESC LIMIT 10 ``` **Top 10 rules by matched rules** ``` SELECT SUM(MatchedEvents) FROM SCHEMA("AWS/Events", RuleName) GROUP BY RuleName ORDER BY MAX() DESC LIMIT 10 ``` ## Kinesis examples **Top 10 streams by bytes written** ``` SELECT SUM("PutRecords.Bytes") FROM SCHEMA("AWS/Kinesis", StreamName) GROUP BY StreamName ORDER BY SUM() DESC LIMIT 10 ``` **Top 10 streams by earliest items in the stream** ``` SELECT MAX("GetRecords.IteratorAgeMilliseconds") FROM SCHEMA("AWS/Kinesis", StreamName) GROUP BY StreamName ORDER BY MAX() DESC LIMIT 10 ``` ## Lambda examples **Lambda functions ordered by number of invocations** ``` SELECT SUM(Invocations) FROM SCHEMA("AWS/Lambda", FunctionName) GROUP BY FunctionName ORDER BY SUM() DESC ``` **Top 10 Lambda functions by longest runtime** ``` SELECT AVG(Duration) FROM SCHEMA("AWS/Lambda", FunctionName) GROUP BY FunctionName ORDER BY MAX() DESC LIMIT 10 ``` **Top 10 Lambda functions by error count** ``` SELECT SUM(Errors) FROM SCHEMA("AWS/Lambda", FunctionName) GROUP BY FunctionName ORDER BY SUM() DESC LIMIT 10 ``` ## CloudWatch Logs examples **Top 10 log groups by incoming events** ``` SELECT SUM(IncomingLogEvents) FROM SCHEMA("AWS/Logs", LogGroupName) GROUP BY LogGroupName ORDER BY SUM() DESC LIMIT 10 ``` **Top 10 log groups by written bytes** ``` SELECT SUM(IncomingBytes) FROM SCHEMA("AWS/Logs", LogGroupName) GROUP BY LogGroupName ORDER BY SUM() DESC LIMIT 10 ``` ## Amazon RDS examples **Top 10 Amazon RDS instances by highest CPU utilization** ``` SELECT MAX(CPUUtilization) FROM SCHEMA("AWS/RDS", DBInstanceIdentifier) GROUP BY DBInstanceIdentifier ORDER BY MAX() DESC LIMIT 10 ``` **Top 10 Amazon RDS clusters by writes** ``` SELECT SUM(WriteIOPS) FROM SCHEMA("AWS/RDS", DBClusterIdentifier) GROUP BY DBClusterIdentifier ORDER BY MAX() DESC LIMIT 10 ``` ## Amazon Simple Storage Service examples **Average latency by bucket** ``` SELECT AVG(TotalRequestLatency) FROM SCHEMA("AWS/S3", BucketName, FilterId) WHERE FilterId = 'EntireBucket' GROUP BY BucketName ORDER BY AVG() DESC ``` **Top 10 buckets by bytes downloaded** ``` SELECT SUM(BytesDownloaded) FROM SCHEMA("AWS/S3", BucketName, FilterId) WHERE FilterId = 'EntireBucket' GROUP BY BucketName ORDER BY SUM() DESC LIMIT 10 ``` ## Amazon Simple Notification Service examples **Total messages published by SNS topics** ``` SELECT SUM(NumberOfMessagesPublished) FROM SCHEMA("AWS/SNS", TopicName) ``` **Top 10 topics by messages published** ``` SELECT SUM(NumberOfMessagesPublished) FROM SCHEMA("AWS/SNS", TopicName) GROUP BY TopicName ORDER BY SUM() DESC LIMIT 10 ``` **Top 10 topics by message delivery failures** ``` SELECT SUM(NumberOfNotificationsFailed) FROM SCHEMA("AWS/SNS", TopicName) GROUP BY TopicName ORDER BY SUM() DESC LIMIT 10 ``` ## Amazon SQS examples **Top 10 queues by number of visible messages** ``` SELECT AVG(ApproximateNumberOfMessagesVisible) FROM SCHEMA("AWS/SQS", QueueName) GROUP BY QueueName ORDER BY AVG() DESC LIMIT 10 ``` **Top 10 most active queues** ``` SELECT SUM(NumberOfMessagesSent) FROM SCHEMA("AWS/SQS", QueueName) GROUP BY QueueName ORDER BY SUM() DESC LIMIT 10 ``` **Top 10 queues by age of earliest message** ``` SELECT AVG(ApproximateAgeOfOldestMessage) FROM SCHEMA("AWS/SQS", QueueName) GROUP BY QueueName ORDER BY AVG() DESC LIMIT 10 ``` # Metrics Insights glossary **label** In Metrics Insights, a label is a key-value pair that is used to scope a query to return a particular set of data, or to define criteria by which query results are to be separated into separate time series. A label key is similar to a column name in SQL. Labels must be CloudWatch metric dimensions. **observation** An observation is a value recorded for a given metric at a given time. # Troubleshooting Metrics Insights ## The results include "Other," but I don't have this as a dimension This means that the query includes a **GROUP BY** clause that specifies a label key that is not used in some of the metrics that are returned by the query. In this case, a null group named `Other` is returned. The metrics that do not include that label key are probably aggregated metrics that return values aggregated across all values of that label key. For example, suppose we have the following query: ``` SELECT AVG(Faults) FROM MyCustomNamespace GROUP BY Operation, ServiceName ``` If some of the returned metrics don't include `ServiceName` as a dimension, then those metrics are displayed as having `Other` as the value for `ServiceName`. To prevent seeing "Other" in your results, use **SCHEMA** in your **FROM** clause, as in the following example: ``` SELECT AVG(Faults) FROM SCHEMA(MyCustomNamespace, Operation) GROUP BY Operation, ServiceName ``` This limits the returned results to only the metrics that have both the `Operation` and `ServiceName` dimensions. ## The oldest timestamp in my graph has a lower metric value than the others CloudWatch Metrics Insights supports up to two weeks of historical data. When you graph with a period larger than one minute, there could be cases where the oldest data point differs from the expected value. This is because the CloudWatch Metrics Insights queries return only data within the two-week retention period. In this case, the oldest data point in the query returns only the observations that have been measured within the two-week boundary, instead of returning all the observations within the period of that data point. ## Inconsistent metric values across different time periods when using tag-based queries When you use `WHERE` or `GROUP BY` clauses with tags in CloudWatch Metrics Insights queries, you might see different metric values depending on the selected time period. For example, a 6-hour period might show a peak value of 20, while a 1-hour period shows only 2 for the same time window. This occurs because tag timestamps are stored with second-level resolution, while metric data points are aligned to period boundaries (for example, the start of each minute or hour). To determine which data points match a tag time range, CloudWatch adjusts the start of the range by subtracting one period. With larger periods, this adjustment creates a wider gap between the tag timestamp and the earliest included data point, which can cause data points near the start of the range to be excluded. The following example shows how this affects query results. A metric has two tag values: `env=beta` (from 00:00 to 01:30) and `env=gamma` (from 01:30 to 03:00). Each tag covers 90 minutes of data with a SUM of 270. ![\[Two CloudWatch metric graphs comparing tag-based query results with 1-minute and 3-hour periods.\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/images/metrics-insights-tag-alignment.png) | **env=beta with 1-minute period** | Statistic | Expected | Returned | Difference | | --- | --- | --- | --- | --- | | SUM | 270 | 271 | \$11 | | AVG | 3.0 | 3.0 | 0 | | MIN | 1 | 1 | 0 | | MAX | 5 | 5 | 0 | | SAMPLE\$1COUNT | 90 | 91 | \$11 | | **env=gamma with 1-minute period** | Statistic | Expected | Returned | Difference | | --- | --- | --- | --- | --- | | SUM | 270 | 275 | \$15 | | AVG | 3.0 | 3.0 | 0 | | MIN | 1 | 1 | 0 | | MAX | 5 | 5 | 0 | | SAMPLE\$1COUNT | 90 | 91 | \$11 | With a 1-minute period, the alignment adjustment is small (1 minute), so only 1 extra data point is included per tag. With a 3-hour period, the adjustment spans the entire query range: | **env=beta with 3-hour period** | Statistic | Expected | Returned | Difference | | --- | --- | --- | --- | --- | | SUM | 270 | 540 | \$1270 | | AVG | 3.0 | 3.0 | 0 | | MIN | 1 | 1 | 0 | | MAX | 5 | 5 | 0 | | SAMPLE\$1COUNT | 90 | 180 | \$190 | | **env=gamma with 3-hour period** | Statistic | Expected | Returned | Difference | | --- | --- | --- | --- | --- | | SUM | 270 | 540 | \$1270 | | AVG | 3.0 | 3.0 | 0 | | MIN | 1 | 1 | 0 | | MAX | 5 | 5 | 0 | | SAMPLE\$1COUNT | 90 | 180 | \$190 | With the 3-hour period, both tags return the entire dataset (SUM=540, SAMPLE\$1COUNT=180) because the single aggregated data point's timestamp falls within both adjusted ranges. The tag boundary is effectively erased. To reduce the impact of this behaviour, try the following approaches: + **Use smaller aggregation periods.** Smaller periods (such as 1 minute or 5 minutes) more closely match the second-level resolution of tag timestamps, which minimizes the alignment gap and makes it more likely that all relevant data points are included. + **Use dimension-based filtering instead of tags.** If your use case allows it, filter by dimensions rather than tags. Dimension-based queries are not affected by this behaviour. For example, use `WHERE InstanceId = 'i-1234567890abcdef0'` instead of `WHERE tag."my-tag" = 'my-value'`. + **Query at a consistent granularity.** When comparing metric data across different time windows, use the same period to avoid unexpected differences caused by the alignment adjustment. # Query metrics with PromQL **Topics** + [ ## What is Prometheus Query Language (PromQL)? ](#CloudWatch-PromQL-WhatIs) + [ ## PromQL limits and restrictions ](#CloudWatch-PromQL-Limits) + [ ## Supported AWS Regions ](#CloudWatch-PromQL-Regions) + [ ## IAM permissions for PromQL ](#CloudWatch-PromQL-IAM) + [ # PromQL querying ](CloudWatch-PromQL-Querying.md) + [ # Running PromQL queries in Query Studio (Preview) ](CloudWatch-PromQL-QueryStudio.md) + [ # Using PromQL in alarms ](CloudWatch-PromQL-Alarms.md) **Note** OTLP metrics ingestion, PromQL querying, OTel enrichment of vended AWS metrics, and Query Studio are in public preview release, are free of charge, and subject to change. ## What is Prometheus Query Language (PromQL)? Prometheus Query Language (PromQL) is a functional query language that lets you select, aggregate, and transform time series data in real time. PromQL was originally designed for Prometheus and has become a popular query language for metrics. Amazon CloudWatch supports PromQL for querying metrics including metrics ingested via OpenTelemetry Line Protocol (OTLP) and [AWS enriched vended metrics](https://docs.aws.amazon.com//AmazonCloudWatch/latest/monitoring/aws-services-cloudwatch-metrics.html). When you ingest OTLP metrics, CloudWatch preserves the full semantic structure of your telemetry, including resource attributes, instrumentation scope, datapoint attributes, and AWS-specific metadata, and exposes them as queryable PromQL labels. With PromQL you can do the following: + Select time series by metric name and label matchers. + Apply mathematical functions and operators across time series. + Aggregate metrics across dimensions such as service, region, or account. + Compute rates, histograms, quantiles, and moving averages. You can use PromQL queries interactively in [Running PromQL queries in Query Studio (Preview)](CloudWatch-PromQL-QueryStudio.md) and also to create CloudWatch Alarms. For more information, see [PromQL querying](CloudWatch-PromQL-Querying.md) and [Using PromQL in alarms](CloudWatch-PromQL-Alarms.md). **Note** CloudWatch uses PromQL based on the Prometheus 3.0 specification. This includes support for UTF-8 metric names and label names. The following concepts are fundamental to working with PromQL in CloudWatch. | Concept | Description | | --- | --- | | **Time series** | A stream of timestamped values identified by a metric name and a set of key-value pairs called *labels*. Each unique combination of metric name and labels forms a distinct time series. | | **Instant vector** | A set of time series containing a single sample for each series, all sharing the same timestamp. Returned by queries like `{"http.server.active_requests", "@resource.service.name"="myservice"}`. | | **Range vector** | A set of time series containing a range of data points over time for each series. Created by appending a time duration selector in brackets, for example, `avg_over_time({"http.server.active_requests", "@resource.service.name"="myservice"}[5m])`. | | **Label** | A key-value pair attached to a time series. In OTLP-ingested metrics, labels are derived from resource attributes, instrumentation scope, datapoint attributes, and AWS-specific metadata. | | **Label matcher** | An expression in curly braces that filters time series by label value. Supports exact match (`=`), not equal (`!=`), regex match (`=~`), and negative regex match (`!~`). | | **Aggregation operator** | A function that combines multiple time series into fewer series. Common operators include `sum`, `avg`, `min`, `max`, `count`, and `topk`. | ## PromQL limits and restrictions The following table lists the limits and restrictions for PromQL: | Limit | Value | Additional information | Error code | | --- | --- | --- | --- | | Max TPS for query requests per account | 300 | Maximum number of query requests (/query, /query\$1range) per second allowed per account. | 422 | | Max TPS for discovery requests per account | 10 | Maximum number of discovery requests (/series, /label, /label\$1values) per second allowed per account. | 422 | | Max concurrent query requests per account | 30 | Maximum number of queries (/query, /query\$1range) an account can have actively executing at the same time. | 429 | | Max concurrent discovery requests per account | 30 | Maximum number of discovery requests (/series, /labels, /label\$1values) an account can have actively executing at the same time. | 429 | | Max series returned per query request | 500 | Maximum number of unique time series a query request (/query, /query\$1range) can return. | 200 - truncated response | | Max labels returned per discovery request | 10,000 | Maximum number of unique labels a discovery request (/series, /labels, /label\$1values) can return. | 200 - truncated response | | Max range per request | 7 days | Maximum time range a query can span, including range parameters and lookback periods. | 422 | | Max series scanned per 24h window | 100,000 | Maximum number of unique time series that can be scanned per 24-hour window of query execution. | 422 | | Max samples scanned per 24h window | 300,000,000 | Maximum number of samples that can be scanned per 24-hour window of query execution. | 422 | | Max samples processed per 24h window | 3,000,000,000 | Maximum number of samples that can be processed per 24-hour window of query execution. | 422 | | Execution timeout | 20 seconds | Maximum time the engine can spend evaluating a query, excluding time spent in queue and fetching data from storage. | 422 | ## Supported AWS Regions The following table lists the AWS Regions where OTLP metrics ingestion, PromQL querying, and Query Studio are available. | Region name | Region code | OTLP metrics ingest | PromQL query | Query Studio | | --- | --- | --- | --- | --- | | US East (N. Virginia) | us-east-1 | ✓ | ✓ | ✓ | | US West (Oregon) | us-west-2 | ✓ | ✓ | ✓ | | Europe (Ireland) | eu-west-1 | ✓ | ✓ | ✓ | | Asia Pacific (Singapore) | ap-southeast-1 | ✓ | ✓ | ✓ | | Asia Pacific (Sydney) | ap-southeast-2 | ✓ | ✓ | ✓ | ## IAM permissions for PromQL To execute PromQL queries, you need both `cloudwatch:GetMetricData` and `cloudwatch:ListMetrics` permissions. The following table lists the new PromQL API operations and their required IAM actions: | API operation | Required actions | | --- | --- | | ExecuteMetricQueryPost | `cloudwatch:GetMetricData`, `cloudwatch:ListMetrics` | | ExecuteMetricQueryGet | `cloudwatch:GetMetricData`, `cloudwatch:ListMetrics` | | ExecuteMetricRangeQuery | `cloudwatch:GetMetricData`, `cloudwatch:ListMetrics` | | ExecuteMetricRangeQueryGet | `cloudwatch:GetMetricData`, `cloudwatch:ListMetrics` | | ExecuteMetricSeriesPost | `cloudwatch:ListMetrics` | | ExecuteMetricSeriesGet | `cloudwatch:ListMetrics` | | ExecuteMetricLabelsPost | `cloudwatch:ListMetrics` | | ExecuteMetricLabelsGet | `cloudwatch:ListMetrics` | | ExecuteMetricLabelValuesGet | `cloudwatch:ListMetrics` | # PromQL querying When you ingest OpenTelemetry metrics into CloudWatch via OpenTelemetry Protocol (OTLP), the hierarchical OTLP data model is flattened into PromQL-compatible labels. This section describes the label structure, the PromQL syntax for querying these labels, and the UTF-8 support in PromQL. **Note** PromQL in Prometheus 3 supports full UTF-8 characters in metric names and label names. This is particularly important for OTLP metrics, because OpenTelemetry semantic conventions use dots in attribute names such as `service.name`. Previously, these dots were replaced with underscores during translation, causing discrepancies between what was defined in OTel conventions and what was queryable in Prometheus. When using PromQL in CloudWatch, the `@` prefix convention distinguishes OTLP-scoped labels from standard Prometheus labels. Fields within each scope use a double-`@` prefix (for example, `@resource.@schema_url`), while attributes use a single-`@` scope prefix, for example, `@resource.service.name`. Datapoint attributes also support bare (un-prefixed) access for backward compatibility with standard PromQL queries, for example, `{"http.server.active_requests"}` and `{"@datapoint.@name"="http.server.active_requests"}` are equivalent. A PromQL expression is enclosed in curly braces, specifying the metric name and an optional set of label matchers. The following example selects all time series for the `http.server.active_requests` metric: ``` {"http.server.active_requests"} ``` The following example selects all time series for the metric `http.server.active_requests` where the OpenTelemetry resource attribute `service.name` equals `myservice`: ``` {"http.server.active_requests", "@resource.service.name"="myservice"} ``` You can combine multiple label matchers in a single query. The following example selects all time series for the `http.server.active_requests` metric where the OpenTelemetry resource attribute `service.name` equals `myservice` across all US regions: ``` {"http.server.active_requests", "@resource.service.name"="myservice", "@aws.region"=~"us-.*"} ``` The following example shows a range query. It calculates the average value of all datapoints within a specified time range for each time series: ``` avg_over_time( {"http.server.active_requests", "@resource.service.name"="myservice"}[5m] ) ``` The following table summarizes the prefix conventions for each OTLP scope: | OTLP scope | Fields prefix | Attributes prefix | Example | | --- | --- | --- | --- | | Resource | `@resource.@` | `@resource.` | `@resource.service.name="myservice"` | | Instrumentation Scope | `@instrumentation.@` | `@instrumentation.` | `@instrumentation.@name="otel-go/metrics"` | | Datapoint | `@datapoint.@` | `@datapoint.` or bare | `cpu="cpu0"` or `@datapoint.cpu="cpu0"` | | AWS-reserved | N/A | `@aws.` | `@aws.account_id="123456789"` | ## Querying vended AWS metrics with PromQL To be able to query vended AWS metrics in PromQL, you first need to enable OTel enrichment of vended metrics. See: [Enabling vended metrics in PromQL](CloudWatch-OTelEnrichment.md). After you enable OTel enrichment, vended AWS metrics become queryable via PromQL with additional labels. The metric name is the same as the original CloudWatch metric name, and the original CloudWatch dimensions are available as datapoint attributes. The following labels are available (the example below is for an EC2 instance): | PromQL Label | Description | Example | | --- | --- | --- | | `InstanceId` | Original CloudWatch dimension, as a datapoint attribute | `i-0123456789abcdef0` | | `"@resource.cloud.resource_id"` | Full ARN of the resource | `arn:aws:ec2:us-east-1:123456789012:instance/i-0123456789abcdef0` | | `"@resource.cloud.provider"` | Cloud provider | `aws` | | `"@resource.cloud.region"` | AWS Region where this metric originated | `us-east-1` | | `"@resource.cloud.account.id"` | AWS account ID where this metric originated | `123456789012` | | `"@instrumentation.@name"` | Instrumentation scope name identifying the source service | `cloudwatch.aws/ec2` | | `"@instrumentation.cloudwatch.source"` | Source service identifier | `aws.ec2` | | `"@instrumentation.cloudwatch.solution"` | Enrichment solution identifier | `CloudWatchOTelEnrichment` | | `"@aws.tag.Environment"` | AWS resource tag | `production` | | `"@aws.account"` | AWS account where this metric was ingested (system label) | `123456789012` | | `"@aws.region"` | AWS Region where this metric was ingested (system label) | `us-east-1` | The following example selects `Invocations` for a specific Lambda function: ``` histogram_sum({Invocations, FunctionName="my-api-handler"}) ``` The following example selects Lambda `Errors` for all functions tagged with a specific team: ``` histogram_sum( {Errors, "@instrumentation.@name"="cloudwatch.aws/lambda", "@aws.tag.Team"="backend"} ) ``` The following example computes the total Lambda `Invocations` grouped by team: ``` sum by ("@aws.tag.Team")( {Invocations, "@instrumentation.@name"="cloudwatch.aws/lambda"} ) ``` The following example selects all time series for the EC2 `CPUUtilization` metric. The usage of `"@instrumentation.@name"="cloudwatch.aws/ec2"` is to exclusively match CPUUtilization from EC2 and not from other AWS services such as Amazon Relational Database Service: ``` histogram_avg({CPUUtilization, "@instrumentation.@name"="cloudwatch.aws/ec2"}) ``` # Running PromQL queries in Query Studio (Preview) **Note** Query Studio is currently available as a Preview feature at no additional charge. For supported regions, see [Supported AWS Regions](CloudWatch-PromQL.md#CloudWatch-PromQL-Regions). Query Studio is an interactive query environment in the CloudWatch console where you can write, run, and visualize PromQL queries against your CloudWatch metrics. You can use Query Studio to explore metrics ingested via OTLP and AWS vended metrics, create visualizations, set up alarms, and add widgets to your CloudWatch dashboards. You can run PromQL queries programmatically using the CloudWatch API, or interactively in Query Studio. ## Running a PromQL query in Query Studio **To run a PromQL query in Query Studio** 1. Open the [CloudWatch console](https://console.aws.amazon.com/cloudwatch/). 1. In the navigation pane, choose **Query Studio (Preview)**. 1. In the query editor tool, select **PromQL** from the drop-down menu. 1. Use the **Builder** mode to browse and select metric names, labels, and aggregation functions. 1. Or enter your PromQL query via the **Editor** mode, for example `{"http.server.active_requests"}`. 1. (Optional) Adjust the time range using the time interval selector at the top of the page. 1. Choose **Run** to execute the query and view the results. Query Studio will display the results as a time series graph. You can switch between graph and table views to analyze your data. ## Creating alarms from Query Studio After running a PromQL query that returns a single time series, you can create a CloudWatch Alarm directly from Query Studio. Choose **Create alarm** from the actions menu to configure alarm thresholds, evaluation periods, and notification actions based on your query results. For more information, see [Using PromQL in alarms](CloudWatch-PromQL-Alarms.md). ## Adding visualizations to dashboards You can add any Query Studio visualization to a CloudWatch dashboard. After running a query and viewing the results, choose **Add to dashboard** to save the visualization as a dashboard widget. The widget continues to run the PromQL query at the dashboard's refresh interval, keeping your dashboard up to date. # Using PromQL in alarms You can create CloudWatch alarms that use PromQL queries to monitor your metrics. PromQL alarms evaluate a PromQL expression and trigger alarm state changes based on the query results. For information about alarm concepts, see [Concepts](alarm-concepts.md). For information about alarm data queries, see [Alarm data queries](alarm-data-queries.md). For information about alarm actions, see [Alarm actions](alarm-actions.md). For information about alarm limits, see [Limits](alarm-limits.md). ## Creating a PromQL alarm You can create a PromQL alarm from the CloudWatch console, the AWS CLI, or the CloudWatch API. **To create a PromQL alarm from the console** 1. Open the [CloudWatch console](https://console.aws.amazon.com/cloudwatch/). 1. In the navigation pane, choose **Alarms**, **All alarms**. 1. Choose **Create alarm**. 1. Choose **Select metric**, then choose the **PromQL** tab. 1. Enter your PromQL query. The query must return a single time series for the alarm to evaluate. 1. Configure the alarm conditions, including the threshold, evaluation period, and datapoints to alarm. 1. Configure the alarm actions, such as Amazon SNS notifications. 1. Enter a name and description for the alarm, then choose **Create alarm**. You can also create a PromQL alarm directly from [Running PromQL queries in Query Studio (Preview)](CloudWatch-PromQL-QueryStudio.md) after running a query that returns a single time series. ## Creating a CloudWatch alarm using PromQL for anomaly detection You can create a PromQL alarm that triggers when a metric breaches an expected range defined by statistical bounds. The alarm query combines upper and lower bounds into a single expression that returns only the anomalous data points. Any time series returned by the query is considered breaching. The following example expression detects when an ad request metric exceeds 3 standard deviations from the median over a 60-minute window: ``` 1 * {"app.ads.ad_requests"} > quantile_over_time(0.5, {"app.ads.ad_requests"}[60m] offset 1m) + 3 * stddev_over_time({"app.ads.ad_requests"}[60m] offset 1m) or 1 * {"app.ads.ad_requests"} < clamp_min( quantile_over_time(0.5, {"app.ads.ad_requests"}[60m] offset 1m) - 3 * stddev_over_time({"app.ads.ad_requests"}[60m] offset 1m), 0) ``` This expression works across multiple label values, so the alarm can track anomalies across your entire fleet. Each breaching time series is tracked as a separate contributor. For more information about how PromQL alarms evaluate contributors, see [PromQL alarms](alarm-promql.md). You can adjust the multiplier and time window to match your metric's behavior. A higher multiplier produces wider bounds with fewer false positives. A longer time window smooths out short-term spikes. The `clamp_min` function prevents the lower bound from going negative for metrics that can't have negative values. For more information about building anomaly detection bands with PromQL, see [Anomaly detection using PromQL](CloudWatch_Anomaly_Detection.md#anomaly_detection_promql). # Use metrics explorer to monitor resources by their tags and properties Metrics explorer is a tag-based tool that enables you to filter, aggregate, and visualize your metrics by tags and resource properties, to enhance observability for your services. This gives you a flexible and dynamic troubleshooting experience, so that you can create multiple graphs at a time and use these graphs to build your application health dashboards. Metrics explorer visualizations are dynamic, so if a matching resource is created after you create a metrics explorer widget and add it to a CloudWatch dashboard, the new resource automatically appears in the explorer widget. For example, if all of your EC2 production instances have the **production** tag, you can use metrics explorer to filter and aggregate metrics from all of these instances to understand their health and performance. If a new instance with a matching tag is later created, it's automatically added to the metrics explorer widget. **Note** Metrics explorer provides a point-in-time experience. Resources that have been terminated, or no longer exist with the property or tag that you specified are not displayed in the visualisation. However, you can still find the metrics for these resources in CloudWatch metrics views. With metrics explorer, you can choose how to aggregate metrics from the resources that match the criteria, and whether to show them all in a single graph or on different graphs within one metrics explorer widget. Metrics explorer includes templates that you can use to see useful visualization graphs with one click, and you can also extend these templates to create completely customized metrics explorer widgets. You can use metrics explorer across accounts if you are using the [Cross-account cross-Region CloudWatch console](Cross-Account-Cross-Region.md). However, metrics explorer is not supported for cross-account use from a monitoring account in [CloudWatch cross-account observability](CloudWatch-Unified-Cross-Account.md). Metrics explorer supports metrics emitted by AWS and EC2 metrics that are published by the CloudWatch agent, including memory, disk, and CPU metrics. To use metrics explorer to see the metrics that are published by the CloudWatch agent, you might have to update your CloudWatch agent configuration file. For more information, see [CloudWatch agent configuration for metrics explorer](#CloudWatch-Metrics-Explorer-agent) To create a visualization with metrics explorer and optionally add it to a dashboard, follow these steps. **To create a visualization with metrics explorer** 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the navigation pane, choose **Explorer**. 1. Do one of the following: + To use a template, select it in the box that currently shows **Empty Explorer**. Depending on the template, the explorer might immediately display graphs of metrics. If it doesn't, choose one or more tags or properties in the **From** box and then data should appear. If it doesn't, use the options at the top of the page to display a longer time range in the graphs. + To create a custom visualization, under **Metrics**, choose a single metric or all the available metrics from a service. After you choose a metric, you can optionally repeat this step to add more metrics. 1. For each metric selected, CloudWatch displays the statistic that it will use immediately after the metric name. To change this, choose the statistic name, and then choose the statistic that you want. 1. Under **From**, choose a tag or a resource property to filter your results. After you do this, you can optionally repeat this step to choose more tags or resource properties. If you choose multiple values of the same property, such as two EC2 instance types, the explorer displays all the resources that match either chosen property. It's treated as an OR operation. If you choose different properties or tags, such as the **Production** tag and the M5 instance type, only the resources that match all of these selections are displayed. It's treated as an AND operation. 1. (Optional) For **Aggregate by**, choose a statistic to use to aggregate the metrics. Then, next to **for**, choose how to aggregate the metric from the list. You can aggregate together all the resources that are currently displayed, or aggregate by a single tag or resource property. Depending on how you choose to aggregate, the result may be a single time series or multiple time series. 1. Under **Split by**, you can choose to split a single graph with multiple time series into multiple graphs. The split can be made by a variety of criteria, which you choose under **Split by**. 1. Under **Graph options**, you can refine the graph by changing the period, the type of graph, the legend placement, and the layout. 1. To add this visualization as a widget to a CloudWatch dashboard, choose **Add to dashboard**. ## CloudWatch agent configuration for metrics explorer To enable metrics explorer to discover EC2 metrics published by the CloudWatch agent, make sure that the CloudWatch agent configuration file contains the following values: + In the `metrics` section, make sure that the `aggregation_dimensions` parameter includes `["InstanceId"]`. It can also contain other dimensions. + In the `metrics` section, make sure that the `append_dimensions` parameter includes a `{"InstanceId":"${aws:InstanceId}"}` line. It can also contain other lines. + In the `metrics` section, inside the `metrics_collected` section, check the sections for each resource type that you want metrics explorer to discover, such as the `cpu`, `disk`, and `memory` sections. Make sure that each of these sections has a `"resources": [ "*"] line.`. + In the `cpu` section of the `metrics_collected` section, make sure there is a `"totalcpu": true` line. + You must use the default `CWAgent` namespace for the metrics collected by the CloudWatch agent, instead of a custom namespace. The settings in the previous list cause the CloudWatch agent to publish aggregate metrics for disks, CPUs, and other resources that can be plotted in metrics explorer for all the instances that use it. These settings will republish the metrics that you had previously set up to be published with multiple dimensions, adding to your metric costs. For more information about editing the CloudWatch agent configuration file, see [Manually create or edit the CloudWatch agent configuration file](CloudWatch-Agent-Configuration-File-Details.md). # Use metric streams You can use *metric streams* to continually stream CloudWatch metrics to a destination of your choice, with near-real-time delivery and low latency. Supported destinations include AWS destinations such as Amazon Simple Storage Service and several third-party service provider destinations. There are three main usage scenarios for CloudWatch metric streams: + **Custom setup with Firehose**— Create a metric stream and direct it to an Amazon Data Firehose delivery stream that delivers your CloudWatch metrics to where you want them to go. You can stream them to a data lake such as Amazon S3, or to any destination or endpoint supported by Firehose including third-party providers. JSON, OpenTelemetry 1.0.0, and OpenTelemetry 0.7.0 formats are supported natively, or you can configure transformations in your Firehose delivery stream to convert the data to a different format such as Parquet. With a metric stream, you can continually update monitoring data, or combine this CloudWatch metric data with billing and performance data to create rich datasets. You can then use tools such as Amazon Athena to get insight into cost optimization, resource performance, and resource utilization. + **Quick S3 setup**— Stream to Amazon Simple Storage Service with a quick setup process. By default, CloudWatch creates the resources needed for the stream. JSON, OpenTelemetry 1.0.0, and OpenTelemetry 0.7.0 formats are supported. + **Quick AWS partner setup**— CloudWatch provides a quick setup experience for some third-party partners. You can use third-party service providers to monitor, troubleshoot, and analyze your applications using the streamed CloudWatch data. When you use the quick partner setup workflow, you need to provide only a destination URL and API key for your destination, and CloudWatch handles the rest of the setup. Quick partner setup is available for the following third-party providers: + Datadog + Dynatrace + Elastic + New Relic + Splunk Observability Cloud + SumoLogic You can stream all of your CloudWatch metrics, or use filters to stream only specified metrics. Each metric stream can include up to 1000 filters that either include or exclude metric namespaces or specific metrics. A single metric stream can have only include or exclude filters, but not both. After a metric stream is created, if new metrics are created that match the filters in place, the new metrics are automatically included in the stream. There is no limit on the number of metric streams per account or per Region, and no limit on the number of metric updates being streamed. Each stream can use either JSON format, OpenTelemetry 1.0.0, or OpenTelemetry 0.7.0 format. You can edit the output format of a metric stream at any time, such as for upgrading from OpenTelemetry 0.7.0 to OpenTelemetry 1.0.0. For more information about output formats, see [CloudWatch metric stream output in JSON format](CloudWatch-metric-streams-formats-json.md), [CloudWatch metric stream output in OpenTelemetry 1.0.0 format](CloudWatch-metric-streams-formats-opentelemetry-100.md), and [CloudWatch metric stream output in OpenTelemetry 0.7.0 format](CloudWatch-metric-streams-formats-opentelemetry.md). For metric streams in monitoring accounts, you can choose whether to include metrics from the source accounts linked to that monitoring account. For more information, see [CloudWatch cross-account observability](CloudWatch-Unified-Cross-Account.md). Metric streams always include the `Minimum`, `Maximum`, `SampleCount`, and `Sum` statistics. You can also choose to include additional statistics at an additional charge. For more information, see [Statistics that can be streamed](CloudWatch-metric-streams-statistics.md). Metric streams pricing is based on the number of metric updates. You also incur charges from Firehose for the delivery stream used for the metric stream. For more information, see [Amazon CloudWatch Pricing](https://aws.amazon.com/cloudwatch/pricing/). **Topics** + [ # Set up a metric stream ](CloudWatch-metric-streams-setup.md) + [ # Statistics that can be streamed ](CloudWatch-metric-streams-statistics.md) + [ # Metric stream operation and maintenance ](CloudWatch-metric-streams-operation.md) + [ # Monitoring your metric streams with CloudWatch metrics ](CloudWatch-metric-streams-monitoring.md) + [ # Trust between CloudWatch and Firehose ](CloudWatch-metric-streams-trustpolicy.md) + [ # CloudWatch metric stream output in JSON format ](CloudWatch-metric-streams-formats-json.md) + [ # CloudWatch metric stream output in OpenTelemetry 1.0.0 format ](CloudWatch-metric-streams-formats-opentelemetry-100.md) + [ # CloudWatch metric stream output in OpenTelemetry 0.7.0 format ](CloudWatch-metric-streams-formats-opentelemetry.md) + [ # Troubleshooting metric streams in CloudWatch ](CloudWatch-metric-streams-troubleshoot.md) # Set up a metric stream Use the steps in the following sections to set up a CloudWatch metric stream. After a metric stream is created, the time it takes for metric data to appear at the destination depends on the configured buffering settings on the Firehose delivery stream. The buffering is expressed in maximum payload size or maximum wait time, whichever is reached first. If these are set to the minimum values (60 seconds, 1MB) the expected latency is within 3 minutes if the selected CloudWatch namespaces have active metric updates. In a CloudWatch metric stream, data is sent every minute. Data might arrive at the final destination out of order. All specified metrics in the specified namespaces are sent in the metric stream, except metrics with a timestamp that is more than two days old. For each combination of metric name and namespace that you stream, all dimension combinations of that metric name and namespace are streamed. For metric streams in monitoring accounts, you can choose whether to include metrics from the source accounts linked to that monitoring account. For more information, see [CloudWatch cross-account observability](CloudWatch-Unified-Cross-Account.md). To create and manage metric streams, you must be logged on to an account that has the **CloudWatchFullAccess** policy and the `iam:PassRole` permission, or an account that has the following list of permissions: + `iam:PassRole` + `cloudwatch:PutMetricStream` + `cloudwatch:DeleteMetricStream` + `cloudwatch:GetMetricStream` + `cloudwatch:ListMetricStreams` + `cloudwatch:StartMetricStreams` + `cloudwatch:StopMetricStreams` If you're going to have CloudWatch set up the IAM role needed for metric streams, you must also have the `iam:CreateRole` and `iam:PutRolePolicy` permissions. **Important** A user with the `cloudwatch:PutMetricStream` has access to the CloudWatch metric data that is being streamed, even if they don't have the `cloudwatch:GetMetricData` permission. **Topics** + [ # Custom setup with Firehose ](CloudWatch-metric-streams-setup-datalake.md) + [ # Use Quick Amazon S3 setup ](CloudWatch-metric-streams-setup-Quick-S3.md) + [ # Quick partner setup ](CloudWatch-metric-streams-QuickPartner.md) # Custom setup with Firehose Use this method to create a metric stream and direct it to an Amazon Data Firehose delivery stream that delivers your CloudWatch metrics to where you want them to go. You can stream them to a data lake such as Amazon S3, or to any destination or endpoint supported by Firehose including third-party providers. JSON, OpenTelemetry 1.0.0, and OpenTelemetry 0.7.0 formats are supported natively, or you can configure transformations in your Firehose delivery stream to convert the data to a different format such as Parquet. With a metric stream, you can continually update monitoring data, or combine this CloudWatch metric data with billing and performance data to create rich datasets. You can then use tools such as Amazon Athena to get insight into cost optimization, resource performance, and resource utilization. You can use the CloudWatch console, the AWS CLI, AWS CloudFormation, or the AWS Cloud Development Kit (AWS CDK) to set up a metric stream. The Firehose delivery stream that you use for your metric stream must be in the same account and the same Region where you set up the metric stream. To achieve cross-Region functionality, you can configure the Firehose delivery stream to stream to a final destination that is in a different account or a different Region. ## CloudWatch console This section describes how to use the CloudWatch console to set up a metric stream using Firehose. **To set up a custom metric stream using Firehose** 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the navigation pane, choose **Metrics**, **Streams**. Then choose **Create metric stream**. 1. (Optional) If you are signed in to an account that is set up as a monitoring account in CloudWatch cross-account observability, you can choose whether to include metrics from linked source accounts in this metric stream. To include metrics from source accounts, choose **Include source account metrics**. 1. Choose **Custom setup with Firehose**. 1. For **Select your Kinesis Data Firehose stream**, select the Firehose delivery stream to use. It must be in the same account. The default format for this option is OpenTelemetry 0.7.0, but you can change the format later in this procedure. Then select the Firehose delivery stream to use under **Select your Firehose delivery stream**. 1. (Optional)You can choose **Select existing service role** to use an existing IAM role instead of having CloudWatch create a new one for you. 1. (Optional) To change the output format from the default format for your scenario, choose **Change output format**. The supported formats are JSON, OpenTelemetry 1.0.0, and OpenTelemetry 0.7.0. 1. For **Metrics to be streamed**, choose either **All metrics** or **Select metrics**. If you choose **All metrics**, all metrics from this account will be included in the stream. Consider carefully whether to stream all metrics, because the more metrics that you stream the higher your metric stream charges will be. If you choose **Select metrics**, do one of the following: + To stream most metric namespaces, choose **Exclude** and select the namespaces or metrics to exclude. When you specify a namespace in **Exclude**, you can optionally select some specific metrics from that namespace to exclude. If you choose to exclude a namespace but don't then select metrics in that namespace, all metrics from that namespace are excluded. + To include only a few metric namespaces or metrics in the metric stream, choose **Include** and then select the namespaces or metrics to include. If you choose to include a namespace but don't then select metrics in that namespace, all metrics from that namespace are included. 1. (Optional) To stream additional statistics for some of these metrics beyond Minimum, Maximum, SampleCount, and Sum, choose **Add additional statistics**. Either choose **Add recommended metrics** to add some commonly used statistics, or manually select the namespace and metric name to stream additional statistics for. Next, select the additional statistics to stream. To then choose another group of metrics to stream a different set of additional statistics for, choose **Add additional statistics**. Each metric can include as many as 20 additional statistics, and as many as 100 metrics within a metric stream can include additional statistics. Streaming additional statistics incurs more charges. For more information, see [Statistics that can be streamed](CloudWatch-metric-streams-statistics.md). For definitions of the additional statistics, see [CloudWatch statistics definitions](Statistics-definitions.md). 1. (Optional) Customize the name of the new metric stream under **Metric stream name**. 1. Choose **Create metric stream**. ## AWS CLI or AWS API Use the following steps to create a CloudWatch metric stream. **To use the AWS CLI or AWS API to create a metric stream** 1. If you're streaming to Amazon S3, first create the bucket. For more information, see [ Creating a bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html). 1. Create the Firehose delivery stream. For more information, see [ Creating a Firehose stream](https://docs.aws.amazon.com/firehose/latest/dev/basic-create.html). 1. Create an IAM role that enables CloudWatch to write to the Firehose delivery stream. For more information about the contents of this role, see [Trust between CloudWatch and Firehose](CloudWatch-metric-streams-trustpolicy.md). 1. Use the `aws cloudwatch put-metric-stream` CLI command or the `PutMetricStream` API to create the CloudWatch metric stream. ## AWS CloudFormation You can use CloudFormation to set up a metric stream. For more information, see [ AWS::CloudWatch::MetricStream](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-cloudwatch-metricstream.html). **To use CloudFormation to create a metric stream** 1. If you're streaming to Amazon S3, first create the bucket. For more information, see [ Creating a bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html). 1. Create the Firehose delivery stream. For more information, see [ Creating a Firehose stream](https://docs.aws.amazon.com/firehose/latest/dev/basic-create.html). 1. Create an IAM role that enables CloudWatch to write to the Firehose delivery stream. For more information about the contents of this role, see [Trust between CloudWatch and Firehose](CloudWatch-metric-streams-trustpolicy.md). 1. Create the stream in CloudFormation. For more information, see [ AWS::CloudWatch::MetricStream](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-cloudwatch-metricstream.html). ## AWS Cloud Development Kit (AWS CDK) You can use AWS Cloud Development Kit (AWS CDK) to set up a metric stream. **To use the AWS CDK to create a metric stream** 1. If you're streaming to Amazon S3, first create the bucket. For more information, see [ Creating a bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html). 1. Create the Firehose delivery stream. For more information, see [ Creating an Amazon Data Firehose Delivery Stream](https://docs.aws.amazon.com/firehose/latest/dev/basic-create.html). 1. Create an IAM role that enables CloudWatch to write to the Firehose delivery stream. For more information about the contents of this role, see [Trust between CloudWatch and Firehose](CloudWatch-metric-streams-trustpolicy.md). 1. Create the metric stream. The metric stream resource is available in AWS CDK as a Level 1 (L1) Construct named `CfnMetricStream`. For more information, see [ Using L1 constructs](https://docs.aws.amazon.com/cdk/latest/guide/constructs.html#constructs_l1_using.html). # Use Quick Amazon S3 setup The **Quick S3 Setup** method works well if you want to quickly set up a stream to Amazon S3 and you don't need any formatting transformation beyond the supported JSON, OpenTelemetry 1.0.0, and OpenTelemetry 0.7.0 formats. CloudWatch will create all necessary resources including the Firehose delivery stream and the necessary IAM roles. The default format for this option is JSON, but you can change the format while you set up the stream. Alternatively, if you want the final format to be Parquet format or Optimized Row Columnar (ORC), you should instead follow the steps in [Custom setup with Firehose](CloudWatch-metric-streams-setup-datalake.md). ## CloudWatch console This section describes how to use the CloudWatch console to set up a metric stream Amazon S3 using Quick S3 setup. **To set up a metric stream using Quick S3 setup** 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the navigation pane, choose **Metrics**, **Streams**. Then choose **Create metric stream**. 1. (Optional) If you are signed in to an account that is set up as a monitoring account in CloudWatch cross-account observability, you can choose whether to include metrics from linked source accounts in this metric stream. To include metrics from source accounts, choose **Include source account metrics**. 1. Choose **Quick S3 setup**. CloudWatch will create all necessary resources including the Firehose delivery stream and the necessary IAM roles. The default format for this option is JSON, but you can change the format later in this procedure. 1. (Optional) Choose **Select existing resources** to use an existing S3 bucket or existing IAM roles instead of having CloudWatch create new ones for you. 1. (Optional) To change the output format from the default format for your scenario, choose **Change output format**. The supported formats are JSON, OpenTelemetry 1.0.0, and OpenTelemetry 0.7.0. 1. For **Metrics to be streamed**, choose either **All metrics** or **Select metrics**. If you choose **All metrics**, all metrics from this account will be included in the stream. Consider carefully whether to stream all metrics, because the more metrics that you stream the higher your metric stream charges will be. If you choose **Select metrics**, do one of the following: + To stream most metric namespaces, choose **Exclude** and select the namespaces or metrics to exclude. When you specify a namespace in **Exclude**, you can optionally select some specific metrics from that namespace to exclude. If you choose to exclude a namespace but don't then select metrics in that namespace, all metrics from that namespace are excluded. + To include only a few metric namespaces or metrics in the metric stream, choose **Include** and then select the namespaces or metrics to include. If you choose to include a namespace but don't then select metrics in that namespace, all metrics from that namespace are included. 1. (Optional) To stream additional statistics for some of these metrics beyond Minimum, Maximum, SampleCount, and Sum, choose **Add additional statistics**. Either choose **Add recommended metrics** to add some commonly used statistics, or manually select the namespace and metric name to stream additional statistics for. Next, select the additional statistics to stream. To then choose another group of metrics to stream a different set of additional statistics for, choose **Add additional statistics**. Each metric can include as many as 20 additional statistics, and as many as 100 metrics within a metric stream can include additional statistics. Streaming additional statistics incurs more charges. For more information, see [Statistics that can be streamed](CloudWatch-metric-streams-statistics.md). For definitions of the additional statistics, see [CloudWatch statistics definitions](Statistics-definitions.md). 1. (Optional) Customize the name of the new metric stream under **Metric stream name**. 1. Choose **Create metric stream**. # Quick partner setup CloudWatch provides a quick setup experience for the following third-party partners. To use this workflow, you need to provide only a destination URL and API key for your destination. CloudWatch handles the rest of setup including creating the Firehose delivery stream and the necessary IAM roles. **Important** Before you use quick partner setup to create a metric stream, we strongly recommend that you read that partner's documentation, linked in the following list. + [Datadog](https://docs.datadoghq.com/integrations/guide/aws-cloudwatch-metric-streams-with-kinesis-data-firehose/) + [Dynatrace](https://www.dynatrace.com/support/help/dynatrace-api/basics/dynatrace-api-authentication) + [Elastic](https://www.elastic.co/docs/current/integrations/awsfirehose) + [New Relic](https://docs.newrelic.com/install/aws-cloudwatch/) + [Splunk Observability Cloud](https://docs.splunk.com/observability/en/gdi/get-data-in/connect/aws/aws-console-ms.html) + [SumoLogic](https://www.sumologic.com) When you set up a metric stream to one of these partners, the stream is created with some default settings, as listed in the following sections. **Topics** + [ ## Set up a metric stream using quick partner setup ](#CloudWatch-metric-streams-QuickPartner-setup) + [ ## Datadog stream defaults ](#CloudWatch-metric-streams-QuickPartner-Datadog) + [ ## Dynatrace stream defaults ](#CloudWatch-metric-streams-QuickPartner-Dynatrace) + [ ## Elastic stream defaults ](#CloudWatch-metric-streams-QuickPartner-Elastic) + [ ## New Relic stream defaults ](#CloudWatch-metric-streams-QuickPartner-NewRelic) + [ ## Splunk Observability Cloud stream defaults ](#CloudWatch-metric-streams-QuickPartner-Splunk) + [ ## Sumo Logic stream defaults ](#CloudWatch-metric-streams-QuickPartner-Sumologic) ## Set up a metric stream using quick partner setup CloudWatch provides a quick setup option for some third-party partners. Before you start the steps in this section, you must have certain information for the partner. This information might include a destination URL and/or an API key for your partner destination. You should also read the documentation at the partner's website linked in the previous section, and the defaults for that partner listed in the following sections. To stream to a third-party destination not supported by quick setup, you can follow the instructions in Follow the instructions in [Custom setup with Firehose](CloudWatch-metric-streams-setup-datalake.md) to set up a stream using Firehose, and then send those metrics from Firehose to the final destination. **To use quick partner setup to create a metric stream to third-party provider** 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the navigation pane, choose **Metrics**, **Streams**. Then choose **Create metric stream**. 1. (Optional) If you are signed in to an account that is set up as a monitoring account in CloudWatch cross-account observability, you can choose whether to include metrics from linked source accounts in this metric stream. To include metrics from source accounts, choose **Include source account metrics**. 1. Choose **Quick Amazon Web Services partner setup** 1. Select the name of the partner that you want to stream metrics to. 1. For **Endpoint URL**, enter the destination URL. 1. For **Access Key** or **API Key**, enter the access key for the partner. Not all partners require an access key. 1. For **Metrics to be streamed**, choose either **All metrics** or **Select metrics**. If you choose **All metrics**, all metrics from this account will be included in the stream. Consider carefully whether to stream all metrics, because the more metrics that you stream the higher your metric stream charges will be. If you choose **Select metrics**, do one of the following: + To stream most metric namespaces, choose **Exclude** and select the namespaces or metrics to exclude. When you specify a namespace in **Exclude**, you can optionally select some specific metrics from that namespace to exclude. If you choose to exclude a namespace but don't then select metrics in that namespace, all metrics from that namespace are excluded. + To include only a few metric namespaces or metrics in the metric stream, choose **Include** and then select the namespaces or metrics to include. If you choose to include a namespace but don't then select metrics in that namespace, all metrics from that namespace are included. 1. (Optional) To stream additional statistics for some of these metrics beyond Minimum, Maximum, SampleCount, and Sum, choose **Add additional statistics**. Either choose **Add recommended metrics** to add some commonly used statistics, or manually select the namespace and metric name to stream additional statistics for. Next, select the additional statistics to stream. To then choose another group of metrics to stream a different set of additional statistics for, choose **Add additional statistics**. Each metric can include as many as 20 additional statistics, and as many as 100 metrics within a metric stream can include additional statistics. Streaming additional statistics incurs more charges. For more information, see [Statistics that can be streamed](CloudWatch-metric-streams-statistics.md). For definitions of the additional statistics, see [CloudWatch statistics definitions](Statistics-definitions.md). 1. (Optional) Customize the name of the new metric stream under **Metric stream name**. 1. Choose **Create metric stream**. ## Datadog stream defaults Quick partner setup streams to Datadog use the following defaults: + **Output format:** OpenTelemetry 0.7.0 + **Firehose stream content encoding** GZIP + **Firehose stream buffering options** Interval of 60 seconds, size of 4 MBs + **Firehose stream retry option** Duration of 60 seconds When you use quick partner setup to create a metric stream to Datadog and you stream certain metrics, by default those metrics include some additional statistics. Streaming additional statistics can incur additional charges. For more information about statistics and their charges, see [Statistics that can be streamed](CloudWatch-metric-streams-statistics.md). The following list shows the metrics that have additional statistics streamed by default, if you choose to stream those metrics. You can choose to de-select these additional statistics before you start the stream. + **`Duration` in `AWS/Lambda`:** p50, p80, p95, p99, p99.9 + **`PostRuntimeExtensionDuration` in `AWS/Lambda`:** p50, p99 + **`FirstByteLatency` and `TotalRequestLatency`in `AWS/S3`:** p50, p90, p95, p99, p99.9 + **`ResponseLatency` in `AWS/Polly` and `TargetResponseTime` in AWS/ApplicationELB:** p50, p90, p95, p99 + **`Latency` and `IntegrationLatency` in `AWS/ApiGateway`:** p90, p95, p99 + **`Latency` and `TargetResponseTime` in `AWS/ELB`:** p95, p99 + **`RequestLatency` in `AWS/AppRunner`:** p50, p95, p99 + **`ActivityTime`, `ExecutionTime`, `LambdaFunctionRunTime`, `LambdaFunctionScheduleTime`, `LambdaFunctionTime`, `ActivityRunTime`, and `ActivityScheduleTime` in `AWS/States`:** p95, p99 + **`EncoderBitRate`, `ConfiguredBitRate`, and `ConfiguredBitRateAvailable` in `AWS/MediaLive`:** p90 + **`Latency` in `AWS/AppSync`:** p90 ## Dynatrace stream defaults Quick partner setup streams to Dynatrace use the following defaults: + **Output format:** OpenTelemetry 0.7.0 + **Firehose stream content encoding** GZIP + **Firehose stream buffering options** Interval of 60 seconds, size of 5 MBs + **Firehose stream retry option** Duration of 600 seconds ## Elastic stream defaults Quick partner setup streams to Elastic use the following defaults: + **Output format:** OpenTelemetry 1.0.0 + **Firehose stream content encoding** GZIP + **Firehose stream buffering options** Interval of 60 seconds, size of 1 MB + **Firehose stream retry option** Duration of 60 seconds ## New Relic stream defaults Quick partner setup streams to New Relic use the following defaults: + **Output format:** OpenTelemetry 0.7.0 + **Firehose stream content encoding** GZIP + **Firehose stream buffering options** Interval of 60 seconds, size of 1 MB + **Firehose stream retry option** Duration of 60 seconds ## Splunk Observability Cloud stream defaults Quick partner setup streams to Splunk Observability Cloud use the following defaults: + **Output format:** OpenTelemetry 1.0.0 + **Firehose stream content encoding** GZIP + **Firehose stream buffering options** Interval of 60 seconds, size of 1 MB + **Firehose stream retry option** Duration of 300 seconds ## Sumo Logic stream defaults Quick partner setup streams to Sumo Logic use the following defaults: + **Output format:** OpenTelemetry 0.7.0 + **Firehose stream content encoding** GZIP + **Firehose stream buffering options** Interval of 60 seconds, size of 1 MB + **Firehose stream retry option** Duration of 60 seconds # Statistics that can be streamed Metric streams always include the following statistics: `Minimum`, `Maximum`, `SampleCount`, and `Sum`. You can also choose to include the following additional statistics in a metric stream. This choice is on a per-metric basis. For more information about these statistics, see [CloudWatch statistics definitions](Statistics-definitions.md). + Percentile values such as p95 or p99 (For streams with either JSON or OpenTelemetry format) + Trimmed mean (Only for streams with the JSON format) + Winsorized mean (Only for streams with the JSON format) + Trimmed count (Only for streams with the JSON format) + Trimmed sum (Only for streams with the JSON format) + Percentile rank (Only for streams with the JSON format) + Interquartile mean (Only for streams with the JSON format) Streaming additional statistics incurs additional charges. Streaming between one and five of these additional statistics for a particular metric is billed as an additional metric update. Thereafter, each additional set of up to five of these statistics is billed as another metric update. For example, suppose that for one metric you are streaming the following six additional statistics: p95, p99, p99.9, Trimmed mean, Winsorized mean, and Trimmed sum. Each update of this metric is billed as three metric updates: one for the metric update which includes the default statistics, one for the first five additional statistics, and one for the sixth additional statistic. Adding up to four more additional statistics for a total of ten would not increase the billing, but an eleventh additional statistic would do so. When you specify a metric name and namespace combination to stream additional statistics, all dimension combinations of that metric name and namespace are streamed with the additional statistics. CloudWatch metric streams publishes a new metric, `TotalMetricUpdate`, which reflects the base number of metric updates plus extra metric updates incurred by streaming additional statistics. For more information, see [Monitoring your metric streams with CloudWatch metrics](CloudWatch-metric-streams-monitoring.md). For more information, see [Amazon CloudWatch Pricing](https://aws.amazon.com/cloudwatch/pricing/). **Note** Some metrics do not support percentiles. Percentile statistics for these metrics are excluded from the stream and do not incur metric stream charges. An example of these statistics that do not support percentiles are some metrics in the `AWS/ECS` namespace. The additional statistics that you configure are streamed only if they match the filters for the stream. For example, if you create a stream that has only `EC2` and `RDS` in the include filters, and then your statistics configuration lists `EC2` and `Lambda`, then the stream includes `EC2` metrics with additional statistics, `RDS` metrics with only the default statistics, and doesn't include `Lambda` statistics at all. # Metric stream operation and maintenance Metric streams are always in one of two states, **Running** or **Stopped**. + **Running**— The metric stream is running correctly. There might not be any metric data streamed to the destination because of the filters on the stream. + **Stopped**— The metric stream has been explicitly stopped by someone, and not because of an error. It might be useful to stop your stream to temporarily pause the streaming of data without deleting the stream. If you stop and restart a metric stream, the metric data that was published to CloudWatch while the metric stream was stopped is not backfilled to the metric stream. If you change the output format of a metric stream, in certain cases you might see a small amount of metric data written to the destination in both the old format and the new format. To avoid this situation, you can create a new Firehose delivery stream with the same configuration as your current one, then change to the new Firehose delivery stream and change the output format at the same time. This way, the Kinesis records with different output format are stored on Amazon S3 in separate objects. Later, you can direct the traffic back to the original Firehose delivery stream and delete the second delivery stream. **To view, edit, stop, and start your metric streams** 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the navigation pane, choose **Metrics**, **Streams**. The list of streams appears, and the **Status** column displays whether each stream is running or stopped. 1. To stop or start a metric stream, select the stream and choose **Stop** or **Start**. 1. To see the details about a metric stream, select the stream and choose **View details**. 1. To change the stream's output format, filters, destination Firehose stream, or roles, choose **Edit** and make the changes that you want. If you change the filters, there might be some gaps in the metric data during the transition. # Monitoring your metric streams with CloudWatch metrics Monitoring your metric streams Metric streams emit CloudWatch metrics about their health and operation in the `AWS/CloudWatch/MetricStreams` namespace. The following metrics are emitted. These metrics are emitted with a `MetricStreamName` dimension and with no dimension. You can use the metrics with no dimensions to see aggregated metrics for all of your metric streams. You can use the metrics with the `MetricStreamName` dimension to see the metrics about only that metric stream. For all of these metrics, values are emitted only for metric streams that are in the **Running** state. | Metric | Description | | --- | --- | | `MetricUpdate` | The number of metric updates sent to the metric stream. If no metric updates are streamed during a time period, this metric is not emitted during that time period. If you stop the metric stream, this metric stops being emitted until the metric stream is started again. Valid Statistic: `Sum` Units: None | | `TotalMetricUpdate` | This is calculated as **MetricUpdate \$1 a number based on additional statistics that are being streamed**. For each unique namespace and metric name combination, streaming 1-5 additional statistics adds 1 to the `TotalMetricUpdate`, streaming 6-10 additional statistics adds 2 to `TotalMetricUpdate`, and so on. Valid Statistic: `Sum` Units: None | | `PublishErrorRate` | The number of unrecoverable errors that occur when putting data into the Firehose delivery stream. If no errors occur during a time period, this metric is not emitted during that time period. If you stop the metric stream, this metric stops being emitted until the metric stream is started again. Valid Statistic: `Average` to see the rate of metric updates unable to be written. This value will be between 0.0 and 1.0. Units: None | # Trust between CloudWatch and Firehose The Firehose delivery stream must trust CloudWatch through an IAM role that has write permissions to Firehose. These permissions can be limited to the single Firehose delivery stream that the CloudWatch metric stream uses. The IAM role must trust the `streams.metrics.cloudwatch.amazonaws.com` service principal. If you use the CloudWatch console to create a metric stream, you can have CloudWatch create the role with the correct permissions. If you use another method to create a metric stream, or you want to create the IAM role itself, it must contain the following permissions policy and trust policy. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Action": [ "firehose:PutRecord", "firehose:PutRecordBatch" ], "Effect": "Allow", "Resource": "arn:aws:firehose:us-east-1:123456789012:deliverystream/*" } ] } ``` ------ ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "streams.metrics.cloudwatch.amazonaws.com" }, "Action": "sts:AssumeRole" } ] } ``` ------ Metric data is streamed by CloudWatch to the destination Firehose delivery stream on behalf of the source that owns the metric stream resource. # CloudWatch metric stream output in JSON format JSON output format In a CloudWatch metric stream that uses the JSON format, each Firehose record contains multiple JSON objects separated by a newline character (\$1n). Each object includes a single data point of a single metric. The JSON format that is used is fully compatible with AWS Glue and with Amazon Athena. If you have a Firehose delivery stream and an AWS Glue table formatted correctly, the format can be automatically transformed into Parquet format or Optimized Row Columnar (ORC) format before being stored in S3. For more information about transforming the format, see [Converting Your Input Record Format in Firehose](https://docs.aws.amazon.com/firehose/latest/dev/record-format-conversion.html). For more information about the correct format for AWS Glue, see [Which AWS Glue schema should I use for JSON output format?](#CloudWatch-metric-streams-format-glue). In the JSON format, the valid values for `unit` are the same as for the value of `unit` in the `MetricDatum` API structure. For more information, see [ MetricDatum](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_MetricDatum.html). The value for the `timestamp` field is in epoch milliseconds, such as `1616004674229`. The following is an example of the format. In this example, the JSON is formatted for easy reading, but in practice the whole format is on a single line. ``` { "metric_stream_name": "MyMetricStream", "account_id": "1234567890", "region": "us-east-1", "namespace": "AWS/EC2", "metric_name": "DiskWriteOps", "dimensions": { "InstanceId": "i-123456789012" }, "timestamp": 1611929698000, "value": { "count": 3.0, "sum": 20.0, "max": 18.0, "min": 0.0, "p99": 17.56, "p99.9": 17.8764, "TM(25%:75%)": 16.43 }, "unit": "Seconds" } ``` ## Which AWS Glue schema should I use for JSON output format? The following is an example of a JSON representation of the `StorageDescriptor` for an AWS Glue table, which would then be used by Firehose. For more information about `StorageDescriptor`, see [ StorageDescriptor](https://docs.aws.amazon.com/glue/latest/webapi/API_StorageDescriptor.html). ``` { "Columns": [ { "Name": "metric_stream_name", "Type": "string" }, { "Name": "account_id", "Type": "string" }, { "Name": "region", "Type": "string" }, { "Name": "namespace", "Type": "string" }, { "Name": "metric_name", "Type": "string" }, { "Name": "timestamp", "Type": "timestamp" }, { "Name": "dimensions", "Type": "map" }, { "Name": "value", "Type": "struct" }, { "Name": "unit", "Type": "string" } ], "Location": "s3://amzn-s3-demo-bucket/", "InputFormat": "org.apache.hadoop.mapred.TextInputFormat", "OutputFormat": "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat", "SerdeInfo": { "SerializationLibrary": "org.apache.hive.hcatalog.data.JsonSerDe" }, "Parameters": { "classification": "json" } } ``` The preceding example is for data written on Amazon S3 in JSON format. Replace the values in the following fields with the indicated values to store the data in Parquet format or Optimized Row Columnar (ORC) format. + **Parquet:** + inputFormat: org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat + outputFormat: org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat + SerDeInfo.serializationLib: org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe + parameters.classification: parquet + **ORC:** + inputFormat: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat + outputFormat: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat + SerDeInfo.serializationLib: org.apache.hadoop.hive.ql.io.orc.OrcSerde + parameters.classification: orc # CloudWatch metric stream output in OpenTelemetry 1.0.0 format OpenTelemetry 1.0.0 output format **Note** With the OpenTelemetry 1.0.0 format, metric attributes are encoded as a list of `KeyValue` objects instead of the `StringKeyValue` type used in the 0.7.0 format. As a consumer, this is the only major change between the 0.7.0 and 1.0.0 formats. A parser generated from the 0.7.0 proto files won't parse metric attributes encoded in the 1.0.0 format. The same is true in reverse, a parser generated from the 1.0.0 proto files will not parse metric attributes encoded in the 0.7.0 format. OpenTelemetry is a collection of tools, APIs, and SDKs. You can use it to instrument, generate, collect, and export telemetry data (metrics, logs, and traces) for analysis. OpenTelemetry is part of the Cloud Native Computing Foundation. For more information, see [OpenTelemetry](https://opentelemetry.io/). For information about the full OpenTelemetry 1.0.0 specification, see [Release version 1.0.0](https://github.com/open-telemetry/opentelemetry-proto/releases/tag/v1.0.0). A Kinesis record can contain one or more `ExportMetricsServiceRequest` OpenTelemetry data structures. Each data structure starts with a header with an `UnsignedVarInt32` indicating the record length in bytes. Each `ExportMetricsServiceRequest` may contain data from multiple metrics at once. The following is a string representation of the message of the `ExportMetricsServiceRequest` OpenTelemetry data structure. OpenTelemetry serializes the Google Protocol Buffers binary protocol, and this is not human-readable. ``` resource_metrics { resource { attributes { key: "cloud.provider" value { string_value: "aws" } } attributes { key: "cloud.account.id" value { string_value: "123456789012" } } attributes { key: "cloud.region" value { string_value: "us-east-1" } } attributes { key: "aws.exporter.arn" value { string_value: "arn:aws:cloudwatch:us-east-1:123456789012:metric-stream/MyMetricStream" } } } scope_metrics { metrics { name: "amazonaws.com/AWS/DynamoDB/ConsumedReadCapacityUnits" unit: "NoneTranslated" summary { data_points { start_time_unix_nano: 60000000000 time_unix_nano: 120000000000 count: 1 sum: 1.0 quantile_values { value: 1.0 } quantile_values { quantile: 0.95 value: 1.0 } quantile_values { quantile: 0.99 value: 1.0 } quantile_values { quantile: 1.0 value: 1.0 } attributes { key: "Namespace" value { string_value: "AWS/DynamoDB" } } attributes { key: "MetricName" value { string_value: "ConsumedReadCapacityUnits" } } attributes { key: "Dimensions" value { kvlist_value { values { key: "TableName" value { string_value: "MyTable" } } } } } } data_points { start_time_unix_nano: 70000000000 time_unix_nano: 130000000000 count: 2 sum: 5.0 quantile_values { value: 2.0 } quantile_values { quantile: 1.0 value: 3.0 } attributes { key: "Namespace" value { string_value: "AWS/DynamoDB" } } attributes { key: "MetricName" value { string_value: "ConsumedReadCapacityUnits" } } attributes { key: "Dimensions" value { kvlist_value { values { key: "TableName" value { string_value: "MyTable" } } } } } } } } } } ``` **Top-level object to serialize OpenTelemetry metric data** `ExportMetricsServiceRequest` is the top-level wrapper to serialize an OpenTelemetry exporter payload. It contains one or more `ResourceMetrics`. ``` message ExportMetricsServiceRequest { // An array of ResourceMetrics. // For data coming from a single resource this array will typically contain one // element. Intermediary nodes (such as OpenTelemetry Collector) that receive // data from multiple origins typically batch the data before forwarding further and // in that case this array will contain multiple elements. repeated opentelemetry.proto.metrics.v1.ResourceMetrics resource_metrics = 1; } ``` `ResourceMetrics` is the top-level object to represent MetricData objects. ``` // A collection of ScopeMetrics from a Resource. message ResourceMetrics { reserved 1000; // The resource for the metrics in this message. // If this field is not set then no resource info is known. opentelemetry.proto.resource.v1.Resource resource = 1; // A list of metrics that originate from a resource. repeated ScopeMetrics scope_metrics = 2; // This schema_url applies to the data in the "resource" field. It does not apply // to the data in the "scope_metrics" field which have their own schema_url field. string schema_url = 3; } ``` **The Resource object** A `Resource` object is a value-pair object that contains some information about the resource that generated the metrics. For metrics created by AWS, the data structure contains the Amazon Resource Name (ARN) of the resource related to the metric, such as an EC2 instance or an S3 bucket. The `Resource` object contains an attribute called `attributes`, which store a list of key-value pairs. + `cloud.account.id` contains the account ID + `cloud.region` contains the Region + `aws.exporter.arn` contains the metric stream ARN + `cloud.provider` is always `aws`. ``` // Resource information. message Resource { // Set of attributes that describe the resource. // Attribute keys MUST be unique (it is not allowed to have more than one // attribute with the same key). repeated opentelemetry.proto.common.v1.KeyValue attributes = 1; // dropped_attributes_count is the number of dropped attributes. If the value is 0, then // no attributes were dropped. uint32 dropped_attributes_count = 2; } ``` **The ScopeMetrics object** The `scope` field will not be filled. We fill only the metrics field that we are exporting. ``` // A collection of Metrics produced by an Scope. message ScopeMetrics { // The instrumentation scope information for the metrics in this message. // Semantically when InstrumentationScope isn't set, it is equivalent with // an empty instrumentation scope name (unknown). opentelemetry.proto.common.v1.InstrumentationScope scope = 1; // A list of metrics that originate from an instrumentation library. repeated Metric metrics = 2; // This schema_url applies to all metrics in the "metrics" field. string schema_url = 3; } ``` **The Metric object** The metric object contains some metadata and a `Summary` data field that contains a list of `SummaryDataPoint`. For metric streams, the metadata is as follows: + `name` will be `amazonaws.com/metric_namespace/metric_name` + `description` will be blank + `unit` will be filled by mapping the metric datum's unit to the case-sensitive variant of the Unified code for Units of Measure. For more information, see [Translations with OpenTelemetry 1.0.0 format in CloudWatch](CloudWatch-metric-streams-formats-opentelemetry-translation-100.md) and [The Unified Code For Units of Measure](https://ucum.org/ucum.html). + `type` will be `SUMMARY` ``` message Metric { reserved 4, 6, 8; // name of the metric, including its DNS name prefix. It must be unique. string name = 1; // description of the metric, which can be used in documentation. string description = 2; // unit in which the metric value is reported. Follows the format // described by http://unitsofmeasure.org/ucum.html. string unit = 3; // Data determines the aggregation type (if any) of the metric, what is the // reported value type for the data points, as well as the relatationship to // the time interval over which they are reported. oneof data { Gauge gauge = 5; Sum sum = 7; Histogram histogram = 9; ExponentialHistogram exponential_histogram = 10; Summary summary = 11; } } message Summary { repeated SummaryDataPoint data_points = 1; } ``` **The SummaryDataPoint object** The SummaryDataPoint object contains the value of a single data point in a time series in a DoubleSummary metric. ``` // SummaryDataPoint is a single data point in a timeseries that describes the // time-varying values of a Summary metric. message SummaryDataPoint { reserved 1; // The set of key/value pairs that uniquely identify the timeseries from // where this point belongs. The list may be empty (may contain 0 elements). // Attribute keys MUST be unique (it is not allowed to have more than one // attribute with the same key). repeated opentelemetry.proto.common.v1.KeyValue attributes = 7; // StartTimeUnixNano is optional but strongly encouraged, see the // the detailed comments above Metric. // // Value is UNIX Epoch time in nanoseconds since 00:00:00 UTC on 1 January // 1970. fixed64 start_time_unix_nano = 2; // TimeUnixNano is required, see the detailed comments above Metric. // // Value is UNIX Epoch time in nanoseconds since 00:00:00 UTC on 1 January // 1970. fixed64 time_unix_nano = 3; // count is the number of values in the population. Must be non-negative. fixed64 count = 4; // sum of the values in the population. If count is zero then this field // must be zero. // // Note: Sum should only be filled out when measuring non-negative discrete // events, and is assumed to be monotonic over the values of these events. // Negative events *can* be recorded, but sum should not be filled out when // doing so. This is specifically to enforce compatibility w/ OpenMetrics, // see: https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#summary double sum = 5; // Represents the value at a given quantile of a distribution. // // To record Min and Max values following conventions are used: // - The 1.0 quantile is equivalent to the maximum value observed. // - The 0.0 quantile is equivalent to the minimum value observed. // // See the following issue for more context: // https://github.com/open-telemetry/opentelemetry-proto/issues/125 message ValueAtQuantile { // The quantile of a distribution. Must be in the interval // [0.0, 1.0]. double quantile = 1; // The value at the given quantile of a distribution. // // Quantile values must NOT be negative. double value = 2; } // (Optional) list of values at different quantiles of the distribution calculated // from the current snapshot. The quantiles must be strictly increasing. repeated ValueAtQuantile quantile_values = 6; // Flags that apply to this specific data point. See DataPointFlags // for the available flags and their meaning. uint32 flags = 8; } ``` For more information, see [Translations with OpenTelemetry 1.0.0 format in CloudWatch](CloudWatch-metric-streams-formats-opentelemetry-translation-100.md). # Translations with OpenTelemetry 1.0.0 format in CloudWatch CloudWatch performs some transformations to put CloudWatch data into OpenTelemetry format. **Translating namespace, metric name, and dimensions** These attributes are key-value pairs encoded in the mapping. + One attribute has the key `Namespace` and its value is the namespace of the metric + One attribute has the key `MetricName` and its value is the name of the metric + One pair has the key `Dimensions` and its value is a nested list of key-value pairs. Each pair in this list maps to a CloudWatch metric dimension, where the pair's key is the name of the dimension and its value is the value of the dimension. **Translating Average, Sum, SampleCount, Min and Max** The Summary datapoint enables CloudWatch to export all of these statistics using one datapoint. + `startTimeUnixNano` contains the CloudWatch `startTime` + `timeUnixNano` contains the CloudWatch `endTime` + `sum` contains the Sum statistic. + `count` contains the SampleCount statistic. + `quantile_values` contains two `valueAtQuantile.value` objects: + `valueAtQuantile.quantile = 0.0` with `valueAtQuantile.value = Min value` + `valueAtQuantile.quantile = 0.99` with `valueAtQuantile.value = p99 value` + `valueAtQuantile.quantile = 0.999` with `valueAtQuantile.value = p99.9 value` + `valueAtQuantile.quantile = 1.0` with `valueAtQuantile.value = Max value` Resources that consume the metric stream can calculate the Average statistic as **Sum/SampleCount**. **Translating units** CloudWatch units are mapped to the case-sensitive variant of the Unified code for Units of Measure, as shown in the following table. For more information, see [The Unified Code For Units of Measure](https://ucum.org/ucum.html). | CloudWatch | OpenTelemetry | | --- | --- | | Second | s | | Second or Seconds | s | | Microseconds | us | | Milliseconds | ms | | Bytes | By | | Kilobytes | kBy | | Megabytes | MBy | | Gigabytes | GBy | | Terabytes | TBy | | Bits | bit | | Kilobits | kbit | | Megabits | MBit | | Gigabits | GBit | | Terabits | Tbit | | Percent | % | | Count | \$1Count\$1 | | None | 1 | Units that are combined with a slash are mapped by applying the OpenTelemetry conversion of both the units. For example, Bytes/Second is mapped to By/s. # How to parse OpenTelemetry 1.0.0 messages This section provides information to help you get started with parsing OpenTelemetry 1.0.0. First, you should get language-specific bindings, which enable you to parse OpenTelemetry 1.0.0 messages in your preferred language. **To get language-specific bindings** + The steps depend on your preferred language. + To use Java, add the following Maven dependency to your Java project: [OpenTelemetry Java >> 0.14.1](https://mvnrepository.com/artifact/io.opentelemetry/opentelemetry-proto/0.14.1). + To use any other language, follow these steps: 1. Make sure that your language is supported by checking the list at [Generating Your Classes](https://developers.google.com/protocol-buffers/docs/proto3#generating). 1. Install the Protobuf compiler by following the steps at [Download Protocol Buffers](https://developers.google.com/protocol-buffers/docs/downloads). 1. Download the OpenTelemetry 1.0.0 ProtoBuf definitions at [Release version 1.0.0](https://github.com/open-telemetry/opentelemetry-proto/releases/tag/v1.0.0). 1. Confirm that you are in the root folder of the downloaded OpenTelemetry 1.0.0 ProtoBuf definitions. Then create a `src` folder and then run the command to generate language-specific bindings. For more information, see [Generating Your Classes](https://developers.google.com/protocol-buffers/docs/proto3#generating). The following is an example for how to generate Javascript bindings. ``` protoc --proto_path=./ --js_out=import_style=commonjs,binary:src \ opentelemetry/proto/common/v1/common.proto \ opentelemetry/proto/resource/v1/resource.proto \ opentelemetry/proto/metrics/v1/metrics.proto \ opentelemetry/proto/collector/metrics/v1/metrics_service.proto ``` The following section includes examples of using the language-specific bindings that you can build using the previous instructions. **Java** ``` package com.example; import io.opentelemetry.proto.collector.metrics.v1.ExportMetricsServiceRequest; import java.io.IOException; import java.io.InputStream; import java.util.ArrayList; import java.util.List; public class MyOpenTelemetryParser { public List parse(InputStream inputStream) throws IOException { List result = new ArrayList<>(); ExportMetricsServiceRequest request; /* A Kinesis record can contain multiple `ExportMetricsServiceRequest` records, each of them starting with a header with an UnsignedVarInt32 indicating the record length in bytes: ------ --------------------------- ------ ----------------------- |UINT32|ExportMetricsServiceRequest|UINT32|ExportMetricsService... ------ --------------------------- ------ ----------------------- */ while ((request = ExportMetricsServiceRequest.parseDelimitedFrom(inputStream)) != null) { // Do whatever we want with the parsed message result.add(request); } return result; } } ``` **Javascript** This example assumes that the root folder with the bindings generated is `./` The data argument of the function `parseRecord` can be one of the following types: + `Uint8Array` this is optimal + `Buffer` optimal under node + `Array.number` 8-bit integers ``` const pb = require('google-protobuf') const pbMetrics = require('./opentelemetry/proto/collector/metrics/v1/metrics_service_pb') function parseRecord(data) { const result = [] // Loop until we've read all the data from the buffer while (data.length) { /* A Kinesis record can contain multiple `ExportMetricsServiceRequest` records, each of them starting with a header with an UnsignedVarInt32 indicating the record length in bytes: ------ --------------------------- ------ ----------------------- |UINT32|ExportMetricsServiceRequest|UINT32|ExportMetricsService... ------ --------------------------- ------ ----------------------- */ const reader = new pb.BinaryReader(data) const messageLength = reader.decoder_.readUnsignedVarint32() const messageFrom = reader.decoder_.cursor_ const messageTo = messageFrom + messageLength // Extract the current `ExportMetricsServiceRequest` message to parse const message = data.subarray(messageFrom, messageTo) // Parse the current message using the ProtoBuf library const parsed = pbMetrics.ExportMetricsServiceRequest.deserializeBinary(message) // Do whatever we want with the parsed message result.push(parsed.toObject()) // Shrink the remaining buffer, removing the already parsed data data = data.subarray(messageTo) } return result } ``` **Python** You must read the `var-int` delimiters yourself or use the internal methods `_VarintBytes(size)` and `_DecodeVarint32(buffer, position)`. These return the position in the buffer just after the size bytes. The read-side constructs a new buffer that is limited to reading only the bytes of the message. ``` size = my_metric.ByteSize() f.write(_VarintBytes(size)) f.write(my_metric.SerializeToString()) msg_len, new_pos = _DecodeVarint32(buf, 0) msg_buf = buf[new_pos:new_pos+msg_len] request = metrics_service_pb.ExportMetricsServiceRequest() request.ParseFromString(msg_buf) ``` **Go** Use `Buffer.DecodeMessage()`. **C\$1** Use `CodedInputStream`. This class can read size-delimited messages. **C\$1\$1** The functions described in `google/protobuf/util/delimited_message_util.h` can read size-delimited messages. **Other languages** For other languages, see [Download Protocol Buffers](https://developers.google.com/protocol-buffers/docs/downloads). When implementing the parser, consider that a Kinesis record can contain multiple `ExportMetricsServiceRequest` Protocol Buffers messages, each of them starting with a header with an `UnsignedVarInt32` that indicates the record length in bytes. # CloudWatch metric stream output in OpenTelemetry 0.7.0 format OpenTelemetry 0.7.0 output format OpenTelemetry is a collection of tools, APIs, and SDKs. You can use it to instrument, generate, collect, and export telemetry data (metrics, logs, and traces) for analysis. OpenTelemetry is part of the Cloud Native Computing Foundation. For more information, see [OpenTelemetry](https://opentelemetry.io/). For information about the full OpenTelemetry 0.7.0 specification, see [v0.7.0 release](https://github.com/open-telemetry/opentelemetry-proto/releases/tag/v0.7.0). A Kinesis record can contain one or more `ExportMetricsServiceRequest` OpenTelemetry data structures. Each data structure starts with a header with an `UnsignedVarInt32` indicating the record length in bytes. Each `ExportMetricsServiceRequest` may contain data from multiple metrics at once. The following is a string representation of the message of the `ExportMetricsServiceRequest` OpenTelemetry data structure. OpenTelemetry serializes the Google Protocol Buffers binary protocol, and this is not human-readable. ``` resource_metrics { resource { attributes { key: "cloud.provider" value { string_value: "aws" } } attributes { key: "cloud.account.id" value { string_value: "2345678901" } } attributes { key: "cloud.region" value { string_value: "us-east-1" } } attributes { key: "aws.exporter.arn" value { string_value: "arn:aws:cloudwatch:us-east-1:123456789012:metric-stream/MyMetricStream" } } } instrumentation_library_metrics { metrics { name: "amazonaws.com/AWS/DynamoDB/ConsumedReadCapacityUnits" unit: "1" double_summary { data_points { labels { key: "Namespace" value: "AWS/DynamoDB" } labels { key: "MetricName" value: "ConsumedReadCapacityUnits" } labels { key: "TableName" value: "MyTable" } start_time_unix_nano: 1604948400000000000 time_unix_nano: 1604948460000000000 count: 1 sum: 1.0 quantile_values { quantile: 0.0 value: 1.0 } quantile_values { quantile: 0.95 value: 1.0 } quantile_values { quantile: 0.99 value: 1.0 } quantile_values { quantile: 1.0 value: 1.0 } } data_points { labels { key: "Namespace" value: "AWS/DynamoDB" } labels { key: "MetricName" value: "ConsumedReadCapacityUnits" } labels { key: "TableName" value: "MyTable" } start_time_unix_nano: 1604948460000000000 time_unix_nano: 1604948520000000000 count: 2 sum: 5.0 quantile_values { quantile: 0.0 value: 2.0 } quantile_values { quantile: 1.0 value: 3.0 } } } } } } ``` **Top-level object to serialize OpenTelemetry metric data** `ExportMetricsServiceRequest` is the top-level wrapper to serialize an OpenTelemetry exporter payload. It contains one or more `ResourceMetrics`. ``` message ExportMetricsServiceRequest { // An array of ResourceMetrics. // For data coming from a single resource this array will typically contain one // element. Intermediary nodes (such as OpenTelemetry Collector) that receive // data from multiple origins typically batch the data before forwarding further and // in that case this array will contain multiple elements. repeated opentelemetry.proto.metrics.v1.ResourceMetrics resource_metrics = 1; } ``` `ResourceMetrics` is the top-level object to represent MetricData objects. ``` // A collection of InstrumentationLibraryMetrics from a Resource. message ResourceMetrics { // The resource for the metrics in this message. // If this field is not set then no resource info is known. opentelemetry.proto.resource.v1.Resource resource = 1; // A list of metrics that originate from a resource. repeated InstrumentationLibraryMetrics instrumentation_library_metrics = 2; } ``` **The Resource object** A `Resource` object is a value-pair object that contains some information about the resource that generated the metrics. For metrics created by AWS, the data structure contains the Amazon Resource Name (ARN) of the resource related to the metric, such as an EC2 instance or an S3 bucket. The `Resource` object contains an attribute called `attributes`, which store a list of key-value pairs. + `cloud.account.id` contains the account ID + `cloud.region` contains the Region + `aws.exporter.arn` contains the metric stream ARN + `cloud.provider` is always `aws`. ``` // Resource information. message Resource { // Set of labels that describe the resource. repeated opentelemetry.proto.common.v1.KeyValue attributes = 1; // dropped_attributes_count is the number of dropped attributes. If the value is 0, // no attributes were dropped. uint32 dropped_attributes_count = 2; } ``` **The InstrumentationLibraryMetrics object** The instrumentation\$1library field will not be filled. We will fill only the metrics field that we are exporting. ``` // A collection of Metrics produced by an InstrumentationLibrary. message InstrumentationLibraryMetrics { // The instrumentation library information for the metrics in this message. // If this field is not set then no library info is known. opentelemetry.proto.common.v1.InstrumentationLibrary instrumentation_library = 1; // A list of metrics that originate from an instrumentation library. repeated Metric metrics = 2; } ``` **The Metric object** The metric object contains a `DoubleSummary` data field that contains a list of `DoubleSummaryDataPoint`. ``` message Metric { // name of the metric, including its DNS name prefix. It must be unique. string name = 1; // description of the metric, which can be used in documentation. string description = 2; // unit in which the metric value is reported. Follows the format // described by http://unitsofmeasure.org/ucum.html. string unit = 3; oneof data { IntGauge int_gauge = 4; DoubleGauge double_gauge = 5; IntSum int_sum = 6; DoubleSum double_sum = 7; IntHistogram int_histogram = 8; DoubleHistogram double_histogram = 9; DoubleSummary double_summary = 11; } } message DoubleSummary { repeated DoubleSummaryDataPoint data_points = 1; } ``` **The MetricDescriptor object** The MetricDescriptor object contains metadata. For more information, see [metrics.proto](https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/metrics/v1/metrics.proto#L110) on GitHub. For metric streams, the MetricDescriptor has the following contents: + `name` will be `amazonaws.com/metric_namespace/metric_name` + `description` will be blank. + `unit` will be filled by mapping the metric datum's unit to the case-sensitive variant of the Unified code for Units of Measure. For more information, see [Translations with OpenTelemetry 0.7.0 format in CloudWatch](CloudWatch-metric-streams-formats-opentelemetry-translation.md) and [The Unified Code For Units of Measure](https://ucum.org/ucum.html). + `type` will be `SUMMARY`. **The DoubleSummaryDataPoint object** The DoubleSummaryDataPoint object contains the value of a single data point in a time series in a DoubleSummary metric. ``` // DoubleSummaryDataPoint is a single data point in a timeseries that describes the // time-varying values of a Summary metric. message DoubleSummaryDataPoint { // The set of labels that uniquely identify this timeseries. repeated opentelemetry.proto.common.v1.StringKeyValue labels = 1; // start_time_unix_nano is the last time when the aggregation value was reset // to "zero". For some metric types this is ignored, see data types for more // details. // // The aggregation value is over the time interval (start_time_unix_nano, // time_unix_nano]. // // Value is UNIX Epoch time in nanoseconds since 00:00:00 UTC on 1 January // 1970. // // Value of 0 indicates that the timestamp is unspecified. In that case the // timestamp may be decided by the backend. fixed64 start_time_unix_nano = 2; // time_unix_nano is the moment when this aggregation value was reported. // // Value is UNIX Epoch time in nanoseconds since 00:00:00 UTC on 1 January // 1970. fixed64 time_unix_nano = 3; // count is the number of values in the population. Must be non-negative. fixed64 count = 4; // sum of the values in the population. If count is zero then this field // must be zero. double sum = 5; // Represents the value at a given quantile of a distribution. // // To record Min and Max values following conventions are used: // - The 1.0 quantile is equivalent to the maximum value observed. // - The 0.0 quantile is equivalent to the minimum value observed. message ValueAtQuantile { // The quantile of a distribution. Must be in the interval // [0.0, 1.0]. double quantile = 1; // The value at the given quantile of a distribution. double value = 2; } // (Optional) list of values at different quantiles of the distribution calculated // from the current snapshot. The quantiles must be strictly increasing. repeated ValueAtQuantile quantile_values = 6; } ``` For more information, see [Translations with OpenTelemetry 0.7.0 format in CloudWatch](CloudWatch-metric-streams-formats-opentelemetry-translation.md). # Translations with OpenTelemetry 0.7.0 format in CloudWatch CloudWatch performs some transformations to put CloudWatch data into OpenTelemetry format. **Translating namespace, metric name, and dimensions** These attributes are key-value pairs encoded in the mapping. + One pair contains the namespace of the metric + One pair contains the name of the metric + For each dimension, CloudWatch stores the following pair: `metricDatum.Dimensions[i].Name, metricDatum.Dimensions[i].Value` **Translating Average, Sum, SampleCount, Min and Max** The Summary datapoint enables CloudWatch to export all of these statistics using one datapoint. + `startTimeUnixNano` contains the CloudWatch `startTime` + `timeUnixNano` contains the CloudWatch `endTime` + `sum` contains the Sum statistic. + `count` contains the SampleCount statistic. + `quantile_values` contains two `valueAtQuantile.value` objects: + `valueAtQuantile.quantile = 0.0` with `valueAtQuantile.value = Min value` + `valueAtQuantile.quantile = 0.99` with `valueAtQuantile.value = p99 value` + `valueAtQuantile.quantile = 0.999` with `valueAtQuantile.value = p99.9 value` + `valueAtQuantile.quantile = 1.0` with `valueAtQuantile.value = Max value` Resources that consume the metric stream can calculate the Average statistic as **Sum/SampleCount**. **Translating units** CloudWatch units are mapped to the case-sensitive variant of the Unified code for Units of Measure, as shown in the following table. For more information, see [The Unified Code For Units of Measure](https://ucum.org/ucum.html). | CloudWatch | OpenTelemetry | | --- | --- | | Second | s | | Second or Seconds | s | | Microsecond | us | | Milliseconds | ms | | Bytes | By | | Kilobytes | kBy | | Megabytes | MBy | | Gigabytes | GBy | | Terabytes | TBy | | Bits | bit | | Kilobits | kbit | | Megabits | MBit | | Gigabits | GBit | | Terabits | Tbit | | Percent | % | | Count | \$1Count\$1 | | None | 1 | Units that are combined with a slash are mapped by applying the OpenTelemetry conversion of both the units. For example, Bytes/Second is mapped to By/s. # How to parse OpenTelemetry 0.7.0 messages This section provides information to help you get started with parsing OpenTelemetry 0.7.0. First, you should get language-specific bindings, which enable you to parse OpenTelemetry 0.7.0 messages in your preferred language. **To get language-specific bindings** + The steps depend on your preferred language. + To use Java, add the following Maven dependency to your Java project: [OpenTelemetry Java >> 0.14.1](https://mvnrepository.com/artifact/io.opentelemetry/opentelemetry-proto/0.14.1). + To use any other language, follow these steps: 1. Make sure that your language is supported by checking the list at [Generating Your Classes](https://developers.google.com/protocol-buffers/docs/proto3#generating). 1. Install the Protobuf compiler by following the steps at [Download Protocol Buffers](https://developers.google.com/protocol-buffers/docs/downloads). 1. Download the OpenTelemetry 0.7.0 ProtoBuf definitions at [v0.7.0 release](https://github.com/open-telemetry/opentelemetry-proto/releases/tag/v0.7.0). 1. Confirm that you are in the root folder of the downloaded OpenTelemetry 0.7.0 ProtoBuf definitions. Then create a `src` folder and then run the command to generate language-specific bindings. For more information, see [Generating Your Classes](https://developers.google.com/protocol-buffers/docs/proto3#generating). The following is an example for how to generate Javascript bindings. ``` protoc --proto_path=./ --js_out=import_style=commonjs,binary:src \ opentelemetry/proto/common/v1/common.proto \ opentelemetry/proto/resource/v1/resource.proto \ opentelemetry/proto/metrics/v1/metrics.proto \ opentelemetry/proto/collector/metrics/v1/metrics_service.proto ``` The following section includes examples of using the language-specific bindings that you can build using the previous instructions. **Java** ``` package com.example; import io.opentelemetry.proto.collector.metrics.v1.ExportMetricsServiceRequest; import java.io.IOException; import java.io.InputStream; import java.util.ArrayList; import java.util.List; public class MyOpenTelemetryParser { public List parse(InputStream inputStream) throws IOException { List result = new ArrayList<>(); ExportMetricsServiceRequest request; /* A Kinesis record can contain multiple `ExportMetricsServiceRequest` records, each of them starting with a header with an UnsignedVarInt32 indicating the record length in bytes: ------ --------------------------- ------ ----------------------- |UINT32|ExportMetricsServiceRequest|UINT32|pExportMetricsService... ------ --------------------------- ------ ----------------------- */ while ((request = ExportMetricsServiceRequest.parseDelimitedFrom(inputStream)) != null) { // Do whatever we want with the parsed message result.add(request); } return result; } } ``` **Javascript** This example assumes that the root folder with the bindings generated is `./` The data argument of the function `parseRecord` can be one of the following types: + `Uint8Array` this is optimal + `Buffer` optimal under node + `Array.number` 8-bit integers ``` const pb = require('google-protobuf') const pbMetrics = require('./opentelemetry/proto/collector/metrics/v1/metrics_service_pb') function parseRecord(data) { const result = [] // Loop until we've read all the data from the buffer while (data.length) { /* A Kinesis record can contain multiple `ExportMetricsServiceRequest` records, each of them starting with a header with an UnsignedVarInt32 indicating the record length in bytes: ------ --------------------------- ------ ----------------------- |UINT32|ExportMetricsServiceRequest|UINT32|ExportMetricsService... ------ --------------------------- ------ ----------------------- */ const reader = new pb.BinaryReader(data) const messageLength = reader.decoder_.readUnsignedVarint32() const messageFrom = reader.decoder_.cursor_ const messageTo = messageFrom + messageLength // Extract the current `ExportMetricsServiceRequest` message to parse const message = data.subarray(messageFrom, messageTo) // Parse the current message using the ProtoBuf library const parsed = pbMetrics.ExportMetricsServiceRequest.deserializeBinary(message) // Do whatever we want with the parsed message result.push(parsed.toObject()) // Shrink the remaining buffer, removing the already parsed data data = data.subarray(messageTo) } return result } ``` **Python** You must read the `var-int` delimiters yourself or use the internal methods `_VarintBytes(size)` and `_DecodeVarint32(buffer, position)`. These return the position in the buffer just after the size bytes. The read-side constructs a new buffer that is limited to reading only the bytes of the message. ``` size = my_metric.ByteSize() f.write(_VarintBytes(size)) f.write(my_metric.SerializeToString()) msg_len, new_pos = _DecodeVarint32(buf, 0) msg_buf = buf[new_pos:new_pos+msg_len] request = metrics_service_pb.ExportMetricsServiceRequest() request.ParseFromString(msg_buf) ``` **Go** Use `Buffer.DecodeMessage()`. **C\$1** Use `CodedInputStream`. This class can read size-delimited messages. **C\$1\$1** The functions described in `google/protobuf/util/delimited_message_util.h` can read size-delimited messages. **Other languages** For other languages, see [Download Protocol Buffers](https://developers.google.com/protocol-buffers/docs/downloads). When implementing the parser, consider that a Kinesis record can contain multiple `ExportMetricsServiceRequest` Protocol Buffers messages, each of them starting with a header with an `UnsignedVarInt32` that indicates the record length in bytes. # Troubleshooting metric streams in CloudWatch If you're not seeing metric data at your final destination, check the following: + Check that the metric stream is in the running state. For steps on how to use the CloudWatch console to do this, see [Metric stream operation and maintenance](CloudWatch-metric-streams-operation.md). + Metrics published more than two days in the past are not streamed. To determine whether a particular metric will be streamed, graph the metric in the CloudWatch console and check how old the last visible datapoint is. If it is more than two days in the past, then it won't be picked up by metric streams. + Check the metrics emitted by the metric stream. In the CloudWatch console, under **Metrics**, look at the **AWS/CloudWatch/MetricStreams ** namespace for the **MetricUpdate**, **TotalMetricUpdate**, and **PublishErrorRate** metrics. + If the **PublishErrorRate** metric is high, confirm that the destination that is used by the Firehose delivery stream exists and that the IAM role specified in the metric stream's configuration grants the `CloudWatch` service principal permissions to write to it. For more information, see [Trust between CloudWatch and Firehose](CloudWatch-metric-streams-trustpolicy.md). + Check that the Firehose delivery stream has permission to write to the final destination. + In the Firehose console, view the Firehose delivery stream that is used for the metric stream and check the **Monitoring** tab to see whether the Firehose delivery stream is receiving data. + Confirm that you have configured your Firehose delivery stream with the correct details. + Check any available logs or metrics for the final destination that the Firehose delivery stream writes to. + To get more detailed information, enable CloudWatch Logs error logging on the Firehose delivery stream. For more information, see [ Monitoring Amazon Data Firehose Using CloudWatch Logs](https://docs.aws.amazon.com/firehose/latest/dev/monitoring-with-cloudwatch-logs.html). **Note** After a data point for a specific metric and timestamp has been sent, it won’t be sent again even if the value of the data point changes later. # View available metrics Metrics are grouped first by namespace, and then by the various dimension combinations within each namespace. For example, you can view all EC2 metrics, EC2 metrics grouped by instance, or EC2 metrics grouped by Auto Scaling group. Only the AWS services that you're using send metrics to Amazon CloudWatch. For a list of AWS services that send metrics to CloudWatch, see [AWS services that publish CloudWatch metrics](aws-services-cloudwatch-metrics.md). From this page, you can also see the metrics and dimensions that are published by each of those services. **Note** Metrics that have not had any new data points in the past two weeks do not appear in the console. They also do not appear when you type their metric name or dimension names in the search box in the **All metrics** tab in the console, and they are not returned in the results of a [list-metrics](https://docs.aws.amazon.com/cli/latest/reference/cloudwatch/list-metrics.html) command. The best way to retrieve these metrics is with the [get-metric-data](https://docs.aws.amazon.com/cli/latest/reference/cloudwatch/get-metric-data.html) or [get-metric-statistics](https://docs.aws.amazon.com/cli/latest/reference/cloudwatch/get-metric-statistics.html) commands in the AWS CLI. If the old metric you want to view has a current metric with similar dimensions, you can view that current similar metric and then choose the **Source** tab, and change the metric name and dimension fields to the ones that you want, and also change the time range to a time when the metric was being reported. The following steps help you browse through the metric namespaces to find and view metrics. You can also search for metrics using targeted search terms. For more information, see [Search for available metrics](finding_metrics_with_cloudwatch.md). If you are browsing in an account set up as a monitoring account in CloudWatch cross-account observability, you can view metrics from the source accounts linked to this monitoring account. When metrics from source accounts are displayed, the ID or label of the account that they are from is also displayed. For more information, see [CloudWatch cross-account observability](CloudWatch-Unified-Cross-Account.md). **Note** After the source account stops sharing the metrics with the *Monitoring* account, the *Source* account metrics data is not accessible to the monitoring account. Source metric names can be visible to the monitoring account for upto 14 days. **To view available metrics by namespace and dimension using the console** 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the navigation pane, choose **Metrics**, **All metrics**. 1. Select a metric namespace (for example, **EC2** or **Lambda**). 1. Select a metric dimension (for example, **Per-Instance Metrics** or **By Function Name**). 1. The **Browse** tab displays all metrics for that dimension in the namespace. By each metric name is an information button you can choose to see a pop-up with the metric definition. If this is a monitoring account in CloudWatch cross-account observability, you also see the metrics from the source accounts linked to this monitoring account. The **Account label** and **Account id** columns in the table display which account each metric is from. You can do the following: 1. To sort the table, use the column heading. 1. To graph a metric, select the check box next to the metric. To select all metrics, select the check box in the heading row of the table. 1. To filter by account, choose the account label or account ID and then choose **Add to search**. 1. To filter by resource, choose the resource ID and then choose **Add to search**. 1. To filter by metric, choose the metric name and then choose **Add to search**. 1. (Optional) To add this graph to a CloudWatch dashboard, choose **Actions**, **Add to dashboard**. **To view available metrics by account namespace, dimension, or metric using the AWS CLI** Use the [list-metrics](https://docs.aws.amazon.com/cli/latest/reference/cloudwatch/list-metrics.html) command to list CloudWatch metrics. For a list of the namespaces, metrics, and dimensions for all services that publish metrics, see [AWS services that publish CloudWatch metrics](aws-services-cloudwatch-metrics.md). The following example command lists all the metrics for Amazon EC2. ``` aws cloudwatch list-metrics --namespace AWS/EC2 ``` The following is example output. ``` { "Metrics" : [ ... { "Namespace": "AWS/EC2", "Dimensions": [ { "Name": "InstanceId", "Value": "i-1234567890abcdef0" } ], "MetricName": "NetworkOut" }, { "Namespace": "AWS/EC2", "Dimensions": [ { "Name": "InstanceId", "Value": "i-1234567890abcdef0" } ], "MetricName": "CPUUtilization" }, { "Namespace": "AWS/EC2", "Dimensions": [ { "Name": "InstanceId", "Value": "i-1234567890abcdef0" } ], "MetricName": "NetworkIn" }, ... ] } ``` **To list all the available metrics for a specified resource** The following example specifies the `AWS/EC2` namespace and the `InstanceId` dimension to view the results for the specified instance only. ``` aws cloudwatch list-metrics --namespace AWS/EC2 --dimensions Name=InstanceId,Value=i-1234567890abcdef0 ``` **To list a metric for all resources** The following example specifies the `AWS/EC2` namespace and a metric name to view the results for the specified metric only. ``` aws cloudwatch list-metrics --namespace AWS/EC2 --metric-name CPUUtilization ``` **To retrieve metrics from linked source accounts in CloudWatch cross-account observability** The following example is run in a monitoring account to retrieve metrics from both the monitoring account and all linked source accounts. If you do not add `--include-linked-accounts`, the command returns only the monitoring account’s metrics. ``` aws cloudwatch list-metrics --include-linked-accounts ``` **To retrieve metrics from a source account in CloudWatch cross-account observability** The following example is run in a monitoring account to retrieve metrics from the source account with the ID 111122223333. ``` aws cloudwatch list-metrics --include-linked-accounts --owning-account "111122223333" ``` # Search for available metrics You can search within all of the metrics in your account using targeted search terms. Metrics are returned that have matching results within their namespace, metric name, or dimensions. If this is a monitoring account in CloudWatch cross-account observability, you also search for metrics from the source accounts linked to this monitoring account. **Note** Metrics that have not had any new data points in the past two weeks do not appear in the console. They also do not appear when you type their metric name or dimension names in the search box in the **All metrics** tab in the console, and they are not returned in the results of a [list-metrics](https://docs.aws.amazon.com/cli/latest/reference/cloudwatch/list-metrics.html) command. The best way to retrieve these metrics is with the [get-metric-data](https://docs.aws.amazon.com/cli/latest/reference/cloudwatch/get-metric-data.html) or [get-metric-statistics](https://docs.aws.amazon.com/cli/latest/reference/cloudwatch/get-metric-statistics.html) commands in the AWS CLI. **To search for available metrics in CloudWatch** 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the navigation pane, choose **Metrics**, **All metrics**. 1. In the search field, enter a search term, such as a metric name, namespace, account ID, account label, dimension name or value, or resource name. This shows you all of the namespaces with metrics with this search term. For example, if you search for **volume**, this shows the namespaces that contain metrics with this term in their name. For more information on search, see [Use search expressions in graphs](using-search-expressions.md) 1. To graph all the search results, choose **Graph search** or Select a namespace to view the metrics from that namespace. You can then do the following: 1. To graph one or more metrics, select the check box next to each metric. To select all metrics, select the check box in the heading row of the table. 1. To refine your search, hover over a metric name and choose **Add to search** or **Search for this only**. 1. To view one of the resources on its console, choose the resource ID and then choose **Jump to resource**. 1. To view help for a metric, select the metric name and choose **What is this?**. The selected metrics appear on the graph. ![\[View the resulting metrics for a search term\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/images/metrics_search_results.png) 1. (Optional) Select one of the buttons in the search bar to edit that part of the search term. # Graphing metrics Use the CloudWatch console to graph metric data generated by other AWS services. This makes it more efficient to see the metric activity on your services. The following procedures describe how to graph metrics in CloudWatch. **Topics** + [ # Graph a metric ](graph_a_metric.md) + [ # Merge two graphs into one ](merge_graphs.md) + [ # Use dynamic labels ](graph-dynamic-labels.md) + [ # Modify the time range or time zone format for a graph ](modify_graph_date_time.md) + [ # Zoom in on a line graph or stacked area graph ](zoom-graph.md) + [ # Modify the y-axis for a graph ](switch_graph_axes.md) + [ # Create an alarm from a metric on a graph ](create_alarm_metric_graph.md) # Graph a metric You can select metrics and create graphs of the metric data using the CloudWatch console. CloudWatch supports the following statistics on metrics: `Average`, `Minimum`, `Maximum`, `Sum`, and `SampleCount`. For more information, see [Statistics](cloudwatch_concepts.md#Statistic). You can view your data at different levels of detail. For example, you can choose a one-minute view, which can be useful when troubleshooting. Or, choose a less detailed, one-hour view. That can be useful when viewing a broader time range (for example, 3 days) so that you can see trends over time. For more information, see [Periods](cloudwatch_concepts.md#CloudWatchPeriods). If you are using an account that is set up as a monitoring account in CloudWatch cross-account observability, you can graph metrics from the source accounts linked to this monitoring account. For more information, see [CloudWatch cross-account observability](CloudWatch-Unified-Cross-Account.md). ## Creating a graph **To graph a metric** 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the navigation pane, choose **Metrics**, **All metrics**. 1. In the **Browse** tab, enter a search term in the search field, such as a metric name, account ID, or resource name. For example, if you search for the `CPUUtilization` metric, you see the namespaces and dimensions with this metric. 1. Select one of the results for your search to view the metrics. 1. To graph one or more metrics, select the check box next to each metric. To select all metrics, select the check box in the heading row of the table. 1. (Optional) To change the type of graph, choose the **Options** tab. You can then choose between a line graph, stacked area chart, number display, gauge, bar chart, or pie chart. 1. Choose the **Graphed metrics** tab. 1. (Optional) To change the statistic used in the graph, choose the new statistic in the **Statistic** column next to the metric name. For more information about CloudWatch statistics, see [CloudWatch statistics definitions](Statistics-definitions.md). For more information about the **p*xx* percentile statistics, see [Percentiles](cloudwatch_concepts.md#Percentiles).** 1. (Optional) To add an anomaly detection band that shows expected values for the metric, choose the anomaly detection icon under **Actions** next to the metric. For more information about anomaly detection, see [Using CloudWatch anomaly detection](CloudWatch_Anomaly_Detection.md). ![\[The metrics console, with the anomaly detection icon circled.\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/images/Anomaly_Detection_Icon.PNG) CloudWatch uses up to two weeks of the metric's recent historical data to calculate a model for expected values. It then displays this range of expected values as a band on the graph. CloudWatch adds a new row under the metric to display the anomaly detection band math expression, labeled **ANOMALY\$1DETECTION\$1BAND**. If recent historical data exists, you immediately see a preview anomaly detection band, which is an approximation of the anomaly detection band generated by the model. It takes up to 15 minutes for the actual anomaly detection band to appear. By default, CloudWatch creates the upper and lower bounds of the band of expected values with a default value of 2 for the band threshold. To change this number, change the value at the end of the formula under **Details** for the band. 1. (Optional) Choose **Edit model** to change how the anomaly detection model is calculated. You can exclude past and future time periods from being used in the training for calculating the model. It is critical to exclude unusual events system as system outage, deployments, and holidays from the training data. You can also specify the time zone to use for the model for daylight saving time changes. For more information, see [Editing an anomaly detection model](Create_Anomaly_Detection_Alarm.md#Modify_Anomaly_Detection_Model). For more information about anomaly detection, see [Using CloudWatch anomaly detection](CloudWatch_Anomaly_Detection.md). To hide the model from the graph, remove the check mark from the line with the ` ANOMALY_DETECTION_BAND` function or choose the `X` icon. To delete the model entirely, choose **Edit model**, **Delete model**. 1. (Optional) As you choose metrics to graph, specify a dynamic label to appear on the graph legend for each metric. Dynamic labels display a statistic about the metric, and automatically update when the dashboard or graph is refreshed. To add a dynamic label, choose **Graphed metrics**, **Add dynamic label**. By default, the dynamic values that you add to the label appear at the beginning of the label. You can then choose the **Label** value for the metric to edit the label. For more information, see [Use dynamic labels](graph-dynamic-labels.md). 1. To view more information about the metric being graphed, pause the mouse over the legend. 1. Horizontal annotations can help graph users more efficiently see when a metric has spiked to a certain level, or whether the metric is within a predefined range. To add a horizontal annotation, choose the **Options** tab and then **Add horizontal annotation**: 1. For **Label**, enter a label for the annotation. 1. For **Value**, enter the metric value where the horizontal annotation appears. 1. For **Fill**, specify whether to use fill shading with this annotation. For example, choose `Above` or `Below` for the corresponding area to be filled. If you specify `Between`, another `Value` field appears, and the area of the graph between the two values is filled. 1. For **Axis**, specify whether the numbers in `Value` refer to the metric associated with the left Y-axis or the right Y-axis, if the graph includes multiple metrics. You can change the fill color of an annotation by choosing the color square in the left column of the annotation. Repeat these steps to add multiple horizontal annotations to the same graph. To hide an annotation, clear the check box in the left column for that annotation. To delete an annotation, choose **x** in the **Actions** column. 1. To get a URL for your graph, choose **Actions**, **Share**. Copy the URL to save or share. 1. To add your graph to a dashboard, choose **Actions**, **Add to dashboard**. ## Creating a graph of metrics from another data source You can create a graph that displays resources from data sources other than CloudWatch. For more information about creating connections to these other data sources, see [Query metrics from other data sources](MultiDataSourceQuerying.md). **To graph a metric from another data source** 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the navigation pane, choose **Metrics**, **All metrics**. 1. Choose the **Multi source query** tab. 1. For **Data source**, select the data source that you want to use. If you haven't already created a connection to the data source that you want, select **Create and manage data sources**, then choose **Create and manage data sources**. For information about the rest of this data source creation process, see [Connect to a prebuilt data source with a wizard](CloudWatch_MultiDataSources-Connect.md). 1. The wizard or query editor prompts you for the information necessary for the query. The workflow is different for each data source, and is tailored to the data source. For example, for Amazon Managed Service for Prometheus; and Prometheus data sources, a PromQL query editor box with a query helper appear. 1. When you have finished constructing the query, choose **Graph query**. The graph is populated with metrics from the query. 1. (Optional) Horizontal annotations can help graph users more efficiently see when a metric has spiked to a certain level, or whether the metric is within a predefined range. To add a horizontal annotation, choose the **Options** tab and then **Add horizontal annotation**: 1. For **Label**, enter a label for the annotation. 1. For **Value**, enter the metric value where the horizontal annotation appears. 1. For **Fill**, specify whether to use fill shading with this annotation. For example, choose `Above` or `Below` for the corresponding area to be filled. If you specify `Between`, another `Value` field appears, and the area of the graph between the two values is filled. 1. For **Axis**, specify whether the numbers in `Value` refer to the metric associated with the left Y-axis or the right Y-axis, if the graph includes multiple metrics. You can change the fill color of an annotation by choosing the color square in the left column of the annotation. Repeat these steps to add multiple horizontal annotations to the same graph. To hide an annotation, clear the check box in the left column for that annotation. To delete an annotation, choose **x** in the **Actions** column. 1. (Optional) To add this graph to a dashboard, choose **Actions**, **Add to dashboard**. ## Updating a graph **To update your graph** 1. To change the name of the graph, choose the pencil icon. 1. To change the time range, select one of the predefined values or choose **custom**. For more information, see [Modify the time range or time zone format for a graph](modify_graph_date_time.md). 1. To change the statistic, choose the **Graphed metrics** tab. Choose the column heading or an individual value and then choose one of the statistics or predefined percentiles, or specify a custom percentile (for example, **p95.45**). 1. To change the period, choose the **Graphed metrics** tab. Choose the column heading or an individual value and then choose a different value. 1. To add a horizontal annotation, choose **Graph options** and then **Add horizontal annotation**: 1. For **Label**, enter a label for the annotation. 1. For **Value**, enter the metric value where the horizontal annotation appears. 1. For **Fill**, specify whether to use fill shading with this annotation. For example, choose `Above` or `Below` for the corresponding area to be filled. If you specify `Between`, another `Value` field appears, and the area of the graph between the two values is filled. 1. For **Axis**, specify whether the numbers in `Value` refer to the metric associated with the left y-axis or the right y-axis, if the graph includes multiple metrics. You can change the fill color of an annotation by choosing the color square in the left column of the annotation. Repeat these steps to add multiple horizontal annotations to the same graph. To hide an annotation, clear the check box in the left column for that annotation. To delete an annotation, choose **x** in the **Actions** column. 1. To change the refresh interval, choose **Refresh options** and then select **Auto refresh** or choose **1 Minute**, **2 Minutes**, **5 Minutes**, or **15 Minutes**. ## Duplicating a metric **To duplicate a metric** 1. Choose the **Graphed metrics** tab. 1. For **Actions**, choose the **Duplicate** icon. ![\[Duplicate a metric\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/images/metric_graph_duplicate.png) 1. Update the duplicate metric as needed. # Merge two graphs into one You can merge two different graphs into one, and then the resulting graph shows both metrics. This can be useful if you already have different metrics displayed in different graphs and want to combine them, or you want to easily create a single graph with metrics from different Regions. To merge a graph into another one, you use either the URL or JSON source of the graph that you want to merge in. **To merge two graphs into one** 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. Open the graph that you want to merge into another graph. To do so, you can choose **Metrics**, **All metrics**, and then choose a metric to graph. Or you can open a dashboard and then open one of the graphs on the dashboard by selecting the graph and choosing **Open in metrics** from the menu at the upper right of the graph. 1. After you have a graph open, do one of the following:. + Copy the URL from the browser bar. + Choose the **Source** tab and then choose **Copy**. 1. Open the graph that you want to merge the previous graph into. 1. When you have the second graph open in the **Metrics** view, choose **Actions**, **Merge graph**. 1. Enter the URL or JSON that you previously copied, and choose **Merge**. 1. The merged graphs appear. The y-axis on the left is for the original graph, and the y-axis on the right is for the graph that you merged into it. **Note** If the graph that you merged into uses the **METRICS()** function, the metrics in the graph that was merged in are not included in the **METRICS()** calculation in the merged graph. 1. To save the merged graph to a dashboard, choose **Actions**, **Add to dashboard**. # Use dynamic labels You can use dynamic labels with your graphs. Dynamic labels add a dynamically updated value to the label for the selected metric. You can add a wide range of values to the labels, as shown in the following tables. The dynamic value shown in the label is derived from the time range currently shown on the graph. The dynamic part of the label automatically updates when either the dashboard or the graph is refreshed. If you use a dynamic label with a search expression, the dynamic label applies to every metric returned by the search. You can use the CloudWatch console to add a dynamic value to a label, edit the label, change the position of the dynamic value within the label column, and make other customizations. ## Dynamic labels Within a dynamic label, you can use the following values relating to properties of the metric: | Dynamic label live value | Description | | --- | --- | | \$1\$1AVG\$1 | The average of the values in the time range currently shown in the graph. | | \$1\$1DATAPOINT\$1COUNT\$1 | The number of data points in the time range that is currently shown in the graph. | | \$1\$1FIRST\$1 | The oldest of the metric values in the time range that is currently shown in the graph. | | \$1\$1FIRST\$1LAST\$1RANGE\$1 | The difference between the metric values of the oldest and newest data points that are currently shown in the graph. | | \$1\$1FIRST\$1LAST\$1TIME\$1RANGE\$1 | The absolute time range between the oldest and newest data points that are currently shown in the graph. | | \$1\$1FIRST\$1TIME\$1 | The timestamp of the oldest data point in the time range that is currently shown in the graph. | | \$1\$1FIRST\$1TIME\$1RELATIVE\$1 | The absolute time difference between now and the timestamp of the oldest data point in the time range that is currently shown in the graph. | | \$1\$1LABEL\$1 | The representation of the default label for a metric. | | \$1\$1LAST\$1 | The most recent of the metric values in the time range that is currently shown in the graph. | | \$1\$1LAST\$1TIME\$1 | The timestamp of the newest data point in the time range that is currently shown in the graph. | | \$1\$1LAST\$1TIME\$1RELATIVE\$1 | The absolute time difference between now and the timestamp of the newest data point in the time range that is currently shown in the graph. | | \$1\$1MAX\$1 | The maximum of the values in the time range currently shown in the graph. | | \$1\$1MAX\$1TIME\$1 | The timestamp of the data point that has the highest metric value, of the data points that are currently shown in the graph. | | \$1\$1MAX\$1TIME\$1RELATIVE\$1 | The absolute time difference between now and the timestamp of the data point with the highest value, of those data points that are currently shown in the graph. | | \$1\$1MIN\$1 | The minimum of the values in the time range currently shown in the graph. | | \$1\$1MIN\$1MAX\$1RANGE\$1 | The difference in metric values between the data points with the highest and lowest metric values, of those data points that are currently shown in the graph. | | \$1\$1MIN\$1MAX\$1TIME\$1RANGE\$1 | The absolute time range between the data points with the highest and lowest metric values, of those data points that are currently shown in the graph. | | \$1\$1MIN\$1TIME\$1 | The timestamp of the data point that has the lowest metric value, of the data points that are currently shown in the graph. | | \$1\$1MIN\$1TIME\$1RELATIVE\$1 | The absolute time difference between now and the timestamp of the data point with the lowest value, of those data points that are currently shown in the graph. | | \$1\$1PROP('AccountId')\$1 | The AWS account ID of the metric. | | \$1\$1PROP('AccountLabel')\$1 | The label specified for the source account that owns this metric, in CloudWatch cross-account observability. | | \$1\$1PROP('Dim.*dimension\$1name*')\$1 | The value of the specified dimension. Replace *dimension\$1name* with the case-sensitive name of your dimension. | | \$1\$1PROP('MetricName')\$1 | The name of the metric. | | \$1\$1PROP('Namespace')\$1 | The namespace of the metric. | | \$1\$1PROP('Period')\$1 | The period of the metric, in seconds. | | \$1\$1PROP('Region')\$1 | The AWS Region where the metric is published. | | \$1\$1PROP('Stat')\$1 | The metric statistic that is being graphed. | | \$1\$1SUM\$1 | The sum of the values in the time range currently shown in the graph. | For example, suppose you have a search expression **SEARCH(' \$1AWS/Lambda, FunctionName\$1 Errors ', 'Sum')**, which finds the `Errors` for each of your Lambda functions. If you set the label to be `[max: ${MAX} Errors for Function Name ${LABEL}]`, the label for each metric is **[max: *number* Errors for Function Name *Name*]**. You can add as many as six dynamic values to a label. You can use the `${LABEL}` placeholder only once within each label. # Modify the time range or time zone format for a graph This section describes how you can modify the date, time, and time zone format on a CloudWatch metrics graph. It also describes how you can zoom in on a graph to apply a specific time range. For information about creating a graph, see [Graph a metric](graph_a_metric.md). **Note** If the time range of a dashboard is shorter than the period used for a graph on the dashboard, the following happens: The graph is modified to display the amount of data corresponding one complete period for that widget, even though this is longer than the dashboard time range. This ensures that there is at least one data point on the graph. The start time of the period for this data point is adjusted backwards to ensure that at least one data point can be displayed. ## Set a relative time range ------ #### [ New interface ] **To specify a relative time range for a graph** 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the navigation pane, choose **Metrics**, and then choose **All metrics**. In the upper right corner of the screen, you can select one of the predefined time ranges, which span from 1 hour to 1 week (**1h**, **3h**, **12h**, **1d**, **3d**, or **1w**). Alternatively, you can choose **Custom** to set your own time range. 1. Choose **Custom**, and then select the **Relative** tab in the upper left corner of the box. You can specify a time range in **Minutes**, **Hours**, **Days**, **Weeks**, **Months**. 1. After you specify a time range, choose **Apply**. ------ #### [ Original interface ] **To specify a relative time range for a graph** 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the navigation pane, choose **Metrics**, and then choose **All metrics**. In the upper right corner of the screen, you can select one of the predefined time ranges, which span from 1 hour to 1 week (**1h**, **3h**, **12h**, **1d**, **3d**, or **1w**). Alternatively, you can choose **custom** to set your own time range. 1. Choose **custom**, and then choose **Relative** in the upper left corner of the box. You can specify a time range in **Minutes**, **Hours**, **Days**, **Weeks**, or **Months**. ------ ## Set an absolute time range ------ #### [ New interface ] **To specify an absolute time range for a graph** 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the navigation pane, choose **Metrics**, and then choose **All metrics**. In the upper right corner of the screen, you can select one of the predefined time ranges, which span from 1 hour to 1 week (**1h**, **3h**, **12h**, **1d**, **3d**, or **1w**). Alternatively, you can choose **Custom** to set your own time range. 1. Choose **Custom**, and then select the **Absolute** tab in the upper left corner of the box. Use the calendar picker or text field boxes to specify a time range. 1. After you specify a time range, choose **Apply**. ------ #### [ Original interface ] **To specify an absolute time range for a graph** 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the navigation pane, choose **Metrics**, and then choose **All metrics**. In the upper right corner of the screen, you can select one of the predefined time ranges, which span from 1 hour to 1 week (**1h**, **3h**, **12h**, **1d**, **3d**, or **1w**). Alternatively, you can choose **custom** to set your own time range. 1. Choose **custom**, and then choose **Absolute** in the upper left corner of the box. Use the calendar picker or text field boxes to specify a time range. 1. After you specify a time range, choose **Apply**. ------ ## Set the time zone format ------ #### [ New interface ] **To specify the time zone for a graph** 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the navigation pane, choose **Metrics**, and then choose **All metrics**. In the upper right corner of the screen, you can select one of the predefined time ranges, which span from 1 hour to 1 week (**1h**, **3h**, **12h**, **1d**, **3d**, or **1w**). Alternatively, you can choose **Custom** to set your own time range. 1. Choose **Custom**, and then choose the dropdown in the upper right corner of the box. You can change the time zone to **UTC** or **Local time zone**. 1. After you make your changes, choose **Apply**. ------ #### [ Original interface ] **To specify the time zone for a graph** 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the navigation pane, choose **Metrics**, and then choose **All metrics**. In the upper right corner of the screen, you can select one of the predefined time ranges, which span from 1 hour to 1 week (**1h**, **3h**, **12h**, **1d**, **3d**, or **1w**). Alternatively, you can choose **custom** to set your own time range. 1. Choose **custom**, and then choose the dropdown in the upper right corner of the box. You can change the time zone to **UTC** or **Local timezone**. ------ # Zoom in on a line graph or stacked area graph Zooming in on a graph In the CloudWatch console, you can use the mini-map zoom feature to focus on sections of line graphs and stacked area graphs without changing between zoomed-in and zoomed-out views. For example, you can use the mini-map zoom feature to focus on a peak in a line graph, so that you can compare the spike against other metrics in your dashboard from the same timeline. The procedures in this section describe how to use the zoom feature. ![\[A screenshot of the zoom feature that shows comparisons.\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/images/widget_zoom.png) In the preceding image, the zoom feature focuses on a spike in a line graph that's related to the input bytes processing rate while also showing other line graphs in the dashboard that focus on sections from the same timeline. ------ #### [ New interface ] **To zoom in on a graph** 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the navigation pane, choose **Metrics**, and then choose **All metrics**. 1. Choose **Browse**, and then select a metric or metrics to graph. 1. Choose **Options**, and select **Line** under ***Widget type***. 1. Choose and drag on the area of the graph that you want to focus on, and then release the drag. 1. To reset the zoom, choose the **Reset zoom** icon, which looks like a magnifying glass with a minus (-) symbol inside. ------ #### [ Original interface ] **To zoom in on a graph** 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the navigation pane, choose **Metrics**, and then choose **All metrics**. . 1. Choose **All metrics**, and then select a metric to graph. 1. Choose **Graph options**. Under ***Widget type*** select **Line**. 1. Choose and drag on the area of the graph that you want to focus on, and then release the drag. 1. To reset the zoom, choose the **Reset zoom** icon, which looks like a magnifying glass with a minus (-) symbol inside. ------ **Tip** If you already created a dashboard that contains a line graph or stacked area graph, you can go to the dashboard and begin using the zoom feature. # Modify the y-axis for a graph You can set custom bounds for the y-axis on a graph to help you see the data better. For example, you can change the bounds on a `CPUUtilization` graph to 100 percent so that it's easy to see whether the CPU is low (the plotted line is near the bottom of the graph) or high (the plotted line is near the top of the graph). You can switch between two different y-axes for your graph. This is useful if the graph contains metrics that have different units or that differ greatly in their range of values. **To modify the y-axis on a graph** 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the navigation pane, choose **Metrics**, **All metrics**. 1. Select a metric namespace (for example, **EC2**) and then a metric dimension (for example, **Per-Instance Metrics**). 1. The **All metrics** tab displays all metrics for that dimension in that namespace. To graph a metric, select the check box next to the metric. 1. On the **Graph options** tab, specify the **Min** and **Max** values for **Left Y Axis**. The value of **Min** can't be greater than the value of **Max**. ![\[Set custom bounds for the y-axis\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/images/metric_graph_custom_bounds.png) 1. To create a second y-axis, specify the **Min** and **Max** values for **Right Y Axis**. 1. To switch between the two y-axes, choose the **Graphed metrics** tab. For **Y Axis**, choose **Left Y Axis** or **Right Y Axis**. ![\[Switch between the y-axes for a graph\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/images/metric_graph_switch_axis.png) # Create an alarm from a metric on a graph You can graph a metric and then create an alarm from the metric on the graph, which has the benefit of populating many of the alarm fields for you. **To create an alarm from a metric on a graph** 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the navigation pane, choose **Metrics**, **All metrics**. 1. Select a metric namespace (for example, **EC2**) and then a metric dimension (for example, **Per-Instance Metrics**). 1. The **All metrics** tab displays all metrics for that dimension in that namespace. To graph a metric, select the check box next to the metric. 1. To create an alarm for the metric, choose the **Graphed metrics** tab. For **Actions**, choose the alarm icon. ![\[Create an alarm from a graphed metric\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/images/metric_graph_alarm.png) 1. Under **Conditions**, choose **Static** or **Anomaly detection** to specify whether to use a static threshold or anomaly detection model for the alarm. Depending on your choice, enter the rest of the data for the alarm conditions. 1. Choose **Additional configuration**. For **Datapoints to alarm**, specify how many evaluation periods (data points) must be in the `ALARM` state to trigger the alarm. If the two values here match, you create an alarm that goes to `ALARM` state if that many consecutive periods are breaching. To create an M out of N alarm, specify a lower number for the first value than you specify for the second value. For more information, see [Alarm evaluation](alarm-evaluation.md). 1. For **Missing data treatment**, choose how to have the alarm behave when some data points are missing. For more information, see [Configuring how CloudWatch alarms treat missing data](alarms-and-missing-data.md). 1. Choose **Next**. 1. Under **Notification**, select an SNS topic to notify when the alarm is in `ALARM` state, `OK` state, or `INSUFFICIENT_DATA` state. To have the alarm send multiple notifications for the same alarm state or for different alarm states, choose **Add notification**. To have the alarm not send notifications, choose **Remove**. 1. To have the alarm perform Auto Scaling or EC2 actions, choose the appropriate button and choose the alarm state and action to perform. 1. When finished, choose **Next**. 1. Enter a name and description for the alarm. The name must contain only ASCII characters. Then choose **Next**. 1. Under **Preview and create**, confirm that the information and conditions are what you want, then choose **Create alarm**. # Using CloudWatch anomaly detection Using anomaly detection When you enable *anomaly detection* for a metric, CloudWatch applies statistical and machine learning algorithms. These algorithms continuously analyze metrics of systems and applications, determine normal baselines, and surface anomalies with minimal user intervention. The algorithms generate an anomaly detection model. The model generates a range of expected values that represent normal metric behavior. You can enable anomaly detection using the AWS Management Console, the AWS CLI, CloudFormation, or the AWS SDK. You can enable anomaly detection on metrics vended by AWS and also on custom metrics. In an account set up as a monitoring account for CloudWatch cross-account observability, you can create anomaly detectors on metrics in source accounts in addition to metrics in the monitoring account. You can use the model of expected values in two ways: + Create anomaly detection alarms based on a metric's expected value. These types of alarms don't have a static threshold for determining alarm state. Instead, they compare the metric's value to the expected value based on the anomaly detection model. You can choose whether the alarm is triggered when the metric value is above the band of expected values, below the band, or both. For more information, see [Create a CloudWatch alarm based on anomaly detection](Create_Anomaly_Detection_Alarm.md). + When viewing a graph of metric data, overlay the expected values onto the graph as a band. This makes it visually clear which values in the graph are out of the normal range. For more information, see [Creating a graph](graph_a_metric.md#create-metric-graph). You can also retrieve the upper and lower values of the model's band by using the `GetMetricData` API request with the `ANOMALY_DETECTION_BAND` metric math function. For more information, see [GetMetricData](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_GetMetricData.html). In a graph with anomaly detection, the expected range of values is shown as a gray band. If the metric's actual value goes beyond this band, it is shown as red during that time. Anomaly detection algorithms account for the seasonality and trend changes of metrics. The seasonality changes could be hourly, daily, or weekly, as shown in the following examples. ![\[The metrics console showing anomaly detection enabled for the CPUUtilization metric.\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/images/Anomaly_Detection_Graph2.PNG) ![\[The metrics console showing anomaly detection enabled for the CPUUtilization metric.\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/images/Anomaly_Detection_Graph5.png) ![\[The metrics console showing anomaly detection enabled for the CPUUtilization metric.\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/images/Anomaly_Detection_Graph6.png) The longer-range trends could be downward or upward. ![\[The metrics console showing anomaly detection enabled for the CPUUtilization metric.\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/images/Anomaly_Detection_Graph3.PNG) Anomaly detection also works well with metrics with flat patterns. ![\[The metrics console showing anomaly detection enabled for the CPUUtilization metric.\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/images/Anomaly_Detection_Graph7.png) ## How CloudWatch anomaly detection works How anomaly detection works When you enable anomaly detection for a metric, CloudWatch applies machine learning algorithms to the metric's past data to create a model of the metric's expected values. The model assesses both trends and hourly, daily, and weekly patterns of the metric. The algorithm trains on up to two weeks of metric data, but you can enable anomaly detection on a metric even if the metric does not have a full two weeks of data. You specify a value for the anomaly detection threshold that CloudWatch uses along with the model to determine the "normal" range of values for the metric. A higher value for the anomaly detection threshold produces a thicker band of "normal" values. The machine learning model is specific to a metric and a statistic. For example, if you enable anomaly detection for a metric using the `AVG` statistic, the model is specific to the `AVG` statistic. When CloudWatch creates a model for many common metrics from AWS services, it ensures that the band doesn’t extend outside of logical values. For example, the band for `MemoryUtilization` of an EC2 instance will stay between 0 and 100, and the bands tracking CloudFront `Requests`, which can't be negative, will never extend below zero. After you create a model, CloudWatch anomaly detection continually evaluates the model and makes adjustments to it to ensure that it is as accurate as possible. This includes re-training the model to adjust if the metric values evolve over time or have sudden changes, and also includes predictors to improve the models of metrics that are seasonal, spiky, or sparse. After you enable anomaly detection on a metric, you can choose to exclude specified time periods of the metric from being used to train the model. This way, you can exclude deployments or other unusual events from being used for model training, ensuring the most accurate model is created. Using anomaly detection models for alarms incurs charges on your AWS account. For more information, see [Amazon CloudWatch Pricing](http://aws.amazon.com/cloudwatch/pricing). ## Anomaly detection on metric math Anomaly detection on metric math Anomaly detection on metric math is a feature that you can use to create anomaly detection alarms on the output metric math expressions. You can use these expressions to create graphs that visualize anomaly detection bands. The feature supports basic arithmetic functions, comparison and logical operators, and most other functions. For information about functions that are not supported, see [Using metric math](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/using-metric-math.html#using-anomaly-detection-on-metric-math) in the *Amazon CloudWatch User Guide*. You can create anomaly detection models based on metric math expressions similar to how you already create anomaly detection models. From the CloudWatch console, you can apply anomaly detection to metric math expressions and select anomaly detection as a threshold type for these expressions. **Note** Anomaly detection on metric math only can be enabled and edited in the latest version of the metrics user interface. When you create anomaly detectors based on metric math expressions in the new version of the interface, you can view them in the old version, but not edit them. For information about how to create, edit, and delete alarms and models for anomaly detection and metric math, see the following sections: + [Create a CloudWatch alarm based on anomaly detection](Create_Anomaly_Detection_Alarm.md) + [Editing an anomaly detection model](Create_Anomaly_Detection_Alarm.md#Modify_Anomaly_Detection_Model) + [Deleting an anomaly detection model](Create_Anomaly_Detection_Alarm.md#Delete_Anomaly_Detection_Model) + [Creating a CloudWatch alarm based on a metric math expression](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Create-alarm-on-metric-math-expression.html) You also can create, delete, and discover anomaly detection models based on metric math expressions using the CloudWatch API with `PutAnomalyDetector`, `DeleteAnomalyDetector`, and `DescribeAnomalyDetectors`. For information about these API actions, see the following sections in the *Amazon CloudWatch API Reference*. + [PutAnomalyDetector](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_PutAnomalyDetector.html) + [DeleteAnomalyDetector](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_DeleteAnomalyDetector.html) + [DescribeAnomalyDetectors](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_DescribeAnomalyDetectors.html) For information about how anomaly detection alarms are priced, see [Amazon CloudWatch pricing](http://aws.amazon.com/cloudwatch/pricing). ## Anomaly detection using PromQL Anomaly detection using PromQL You can build anomaly detection bands for any Prometheus-compatible metric by using standard PromQL functions such as `quantile_over_time`, `stddev_over_time`, and `avg_over_time`. This approach computes a baseline and adds or subtracts a scaled standard deviation to define upper and lower bounds that adapt to your metric's natural patterns. This works for any metric that returns a float value, such as CPU usage, request latency, or error counts. For information about ingesting metrics using OpenTelemetry, see [OpenTelemetry](CloudWatch-OpenTelemetry-Sections.md). ### Defining upper and lower bounds To define an expected range for a metric, compute a baseline using the median or average over a time window, then add and subtract a multiple of the standard deviation. The multiplier controls sensitivity—higher values produce wider bands with fewer false positives, while lower values catch smaller deviations. The following example creates an upper bound for an ad request metric using a 60-minute window and a multiplier of 3: ``` quantile_over_time(0.5, {"app.ads.ad_requests"}[60m] offset 1m) + 3 * stddev_over_time({"app.ads.ad_requests"}[60m] offset 1m) ``` The following example creates the corresponding lower bound. The `clamp_min` function prevents the lower bound from going negative for metrics that can't have negative values: ``` clamp_min( quantile_over_time(0.5, {"app.ads.ad_requests"}[60m] offset 1m) - 3 * stddev_over_time({"app.ads.ad_requests"}[60m] offset 1m), 0) ``` You can graph both bounds together in CloudWatch Query Studio to visualize the expected range for your metric. For more information, see [Running PromQL queries in Query Studio (Preview)](CloudWatch-PromQL-QueryStudio.md). ### Detecting breaches To detect when a metric is outside its expected range, combine both bounds into a single query. The following expression returns only the data points where the metric value exceeds the upper bound or falls below the lower bound: ``` 1 * {"app.ads.ad_requests"} > quantile_over_time(0.5, {"app.ads.ad_requests"}[60m] offset 1m) + 3 * stddev_over_time({"app.ads.ad_requests"}[60m] offset 1m) or 1 * {"app.ads.ad_requests"} < clamp_min( quantile_over_time(0.5, {"app.ads.ad_requests"}[60m] offset 1m) - 3 * stddev_over_time({"app.ads.ad_requests"}[60m] offset 1m), 0) ``` This query works across multiple label values, so you can detect anomalies across your entire fleet in a single query. You can use this expression to create a PromQL alarm that triggers when any time series breaches the expected range. For more information, see [Creating a CloudWatch alarm using PromQL for anomaly detection](CloudWatch-PromQL-Alarms.md#promql_alarm_anomaly_detection). # Using math expressions with CloudWatch metrics Metric math enables you to query multiple CloudWatch metrics and use math expressions to create new time series based on these metrics. You can visualize the resulting time series on the CloudWatch console and add them to dashboards. Using AWS Lambda metrics as an example, you could divide the `Errors` metric by the `Invocations` metric to get an error rate. Then add the resulting time series to a graph on your CloudWatch dashboard. You can also perform metric math programmatically, using the `GetMetricData` API operation. For more information, see [GetMetricData](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_GetMetricData.html). ## Add a math expression to a CloudWatch graph You can add a math expression to a graph on your CloudWatch dashboard. Each graph is limited to using a maximum of 500 metrics and expressions, so you can add a math expression only if the graph has 499 or fewer metrics. This applies even if not all the metrics are displayed on the graph. **To add a math expression to a graph** 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. Create or edit a graph. There needs to be at least one metric in the graph. 1. Choose **Graphed metrics**. 1. Choose **Math expression**, **Start with empty expression**. A new line appears for the expression. 1. In the new line, under the **Details** column, enter the math expression. The tables in the **Metric Math Syntax and Functions** section list the functions that you can use in the expression. To use a metric or the result of another expression as part of the formula for this expression, use the value shown in the **Id** column: for example, **m1\$1m2** or **e1-MIN(e1)**. You can change the value of **Id**. It can include numbers, letters, an underscore, and must start with a lowercase letter. Changing the value of **Id** to a more meaningful name can also make a graph easier to understand;for example, changing **m1** and **m2** to **errors** and **requests**. **Tip** Choose the down arrow next to **Math Expression** to see a list of supported functions, which you can use when creating your expression. 1. For the **Label** column of the expression, enter a name that describes what the expression is calculating. If the result of an expression is an array of time series, each of those time series is displayed on the graph with a separate line, with different colors. Immediately under the graph is a legend for each line in the graph. For a single expression that produces multiple time series, the legend captions for those time series are in the format ***Expression-Label Metric-Label***. For example, if the graph includes a metric with a label of **Errors** and an expression **FILL(METRICS(), 0)** that has a label of **Filled With 0:**, one line in the legend would be **Filled With 0: Errors**. To have the legend show only the original metric labels, set *Expression-Label* to be empty. When one expression produces an array of time series on the graph, you can't change the colors used for each of those time series. 1. After you have added the desired expressions, you can simplify the graph by hiding some of the original metrics. To hide a metric or expression, clear the check box to the left of the **Id** field. ## Metric math syntax and functions The following sections explain the functions available for metric math. All functions must be written in uppercase letters (such as **AVG**), and the **Id** field for all metrics and math expressions must start with a lowercase letter. The final result of any math expression must be a single time series or an array of time series. Some functions produce a scalar number. You can use these functions within a larger function that ultimately produces a time series. For example, taking the **AVG** of a single time series produces a scalar number, so it can't be the final expression result. But you could use it in the function **m1-AVG(m1)** to display a time series of the difference between each individual data point and the average value in the time series. ### Data type abbreviations Some functions are valid for only certain types of data. The abbreviations in the following list are used in the tables of functions to represent the types of data supported for each function: + **S** represents a scalar number, such as 2, -5, or 50.25. + **TS** is a time series (a series of values for a single CloudWatch metric over time): for example, the `CPUUtilization` metric for instance `i-1234567890abcdef0` over the last three days. + **TS[]** is an array of time series, such as the time series for multiple metrics. + **String[]** is an array of strings. ### The METRICS() function The **METRICS()** function returns all the metrics in the request. Math expressions aren't included. You can use **METRICS()** within a larger expression that produces a single time series or an array of time series. For example, the expression **SUM(METRICS())** returns a time series (TS) that is the sum of the values of all the graphed metrics. **METRICS()/100** returns an array of time series, each of which is a time series showing each data point of one of the metrics divided by 100. You can use the **METRICS()** function with a string to return only the graphed metrics that contain that string in their **Id** field. For example, the expression **SUM(METRICS("errors"))** returns a time series that is the sum of the values of all the graphed metrics that have ‘errors’ in their **Id** field. You can also use **SUM([METRICS(“4xx”), METRICS(“5xx”)])** to match multiple strings. ### Basic arithmetic functions The following table lists the basic arithmetic functions that are supported. Missing values in a time series are treated as 0. If the value of a data point causes a function to attempt to divide by zero, the data point is dropped. | Operation | Arguments | Examples | | --- | --- | --- | | Arithmetic operators: \$1 - \$1 / ^ | S, S S, TS TS, TS S, TS[] TS, TS[] | PERIOD(m1)/60 **5 \$1 m1** **m1 - m2** **SUM(100/[m1, m2])** **AVG(METRICS())** **METRICS()\$1100** | | Unary subtraction - | S TS TS[] | **-5\$1m1** **-m1** **SUM(-[m1, m2])** | ### Comparison and logical operators You can use comparison and logical operators with either a pair of time series or a pair of single scalar values. When you use a comparison operator with a pair of time series, the operators return a time series where each data point is either 0 (false) or 1 (true). If you use a comparison operator on a pair of scalar values, a single scalar value is returned, either 0 or 1. When comparison operators are used between two time series, and only one of the time series has a value for a particular time stamp, the function treats the missing value in the other time series as **0**. You can use logical operators in conjunction with comparison operators, to create more complex functions. The following table lists the operators that are supported. | Type of operator | Supported operators | | --- | --- | | Comparison operators | == \$1= <= >= < > | | Logical operators | AND or && OR or \$1\$1 | To see how these operators are used, suppose we have two time series: **metric1** has values of `[30, 20, 0, 0]` and **metric2** has values of `[20, -, 20, -]` where `-` indicates that there is no value for that timestamp. | Expression | Output | | --- | --- | | **(metric1 < metric2)** | **0, 0, 1, 0** | | **(metric1 >= 30)** | **1, 0, 0, 0** | | **(metric1 > 15 AND metric2 > 15)** | **1, 0, 0, 0** | ### Functions supported for metric math The following table describes the functions that you can use in math expressions. Enter all functions in uppercase letters. The final result of any math expression must be a single time series or an array of time series. Some functions in tables in the following sections produce a scalar number. You can use these functions within a larger function that ultimately produces a time series. For example, taking the **AVG** of a single time series produces a scalar number, so it can't be the final expression result. But you could use it in the function **m1-AVG(m1)** to display a time series of the difference between each individual data point and the average value of that data point. In the following table, every example in the **Examples** column is an expression that results in a single time series or an array of time series. These examples show how functions that return scalar numbers can be used as part of a valid expression that produces a single time series. | Function | Arguments | Return type**\$1** | Description | Examples | Supported for cross-account? | | --- | --- | --- | --- | --- | --- | | **ABS** | TS TS[] | TS TS[] | Returns the absolute value of each data point. | **ABS(m1-m2)** **MIN(ABS([m1, m2]))** **ABS(METRICS())** | ✓ | | **ANOMALY\$1DETECTION\$1BAND** | TS TS, S | TS[] | Returns an anomaly detection band for the specified metric. The band consists of two time series, one representing the upper limit of the "normal" expected value of the metric, and the other representing the lower limit. The function can take two arguments. The first is the ID of the metric to create the band for. The second argument is the number of standard deviations to use for the band. If you don't specify this argument, the default of 2 is used. For more information, see [Using CloudWatch anomaly detection](CloudWatch_Anomaly_Detection.md). | **ANOMALY\$1DETECTION\$1BAND(m1)** **ANOMALY\$1DETECTION\$1BAND(m1,4)** | | | **AVG** | TS TS[] | S TS | The **AVG** of a single time series returns a scalar representing the average of all the data points in the metric. The **AVG** of an array of time series returns a single time series. Missing values are treated as 0. We recommend that you do not use this function in CloudWatch alarms if you want the function to return a scalar. For example, `AVG(m2)`. Whenever an alarm evaluates whether to change state, CloudWatch attempts to retrieve a higher number of data points than the number specified as Evaluation Periods. This function acts differently when extra data is requested. To use this function with alarms, especially alarms that have Auto Scaling actions, we recommend that you set the alarm to use M out of N datapoints, where M < N. | **SUM([m1,m2])/AVG(m2)** **AVG(METRICS())** | ✓ | | **CEIL** | TSTS[] | TS TS[] | Returns the ceiling of each metric. The ceiling is the smallest integer greater than or equal to each value. | **CEIL(m1)** **CEIL(METRICS())** **SUM(CEIL(METRICS()))** | ✓ | | **DATAPOINT\$1COUNT** | TS TS[] | S TS | Returns a count of the data points that reported values. This is useful for calculating averages of sparse metrics. We recommend that you do not use this function in CloudWatch alarms. Whenever an alarm evaluates whether to change state, CloudWatch attempts to retrieve a higher number of data points than the number specified as Evaluation Periods. This function acts differently when extra data is requested. | **SUM(m1) / DATAPOINT\$1COUNT(m1)** **DATAPOINT\$1COUNT(METRICS())** | ✓ | | **DB\$1PERF\$1INSIGHTS** | String, String, String String, String, String[] | TS (if given a single string) TS[] (if given an array of strings) | Returns Performance Insights Counter metrics for databases such as Amazon Relational Database Service and Amazon DocumentDB (with MongoDB compatibility). This function returns the same amount of data that you can get by directly querying the Performance Insights APIs. You can use these metrics in CloudWatch for graphing and creating alarms. When you use this function, you must specify the Unique Database Resource ID of the database. This is different than the database identifier. To find the database resource ID in the Amazon RDS console, choose the DB instance to see its details. Then choose the **Configuration** tab. The **Resource ID** is displayed in the **Configuration** section. **DB\$1PERF\$1INSIGHTS** also brings in the `DBLoad` metric at sub-minute intervals. Performance Insights metrics retrieved with this function are not stored in CloudWatch. Therefore, some CloudWatch features such as cross-account observability, anomaly detection, metric streams, metrics explorer, and Metric Insights don't work with Performance Insights metrics that you retrieve with **DB\$1PERF\$1INSIGHTS**. A single request using the **DB\$1PERF\$1INSIGHTS** function can retrieve the following numbers of data points. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/using-metric-math.html) The **DB\$1PERF\$1INSIGHTS** function supports only the following period lengths: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/using-metric-math.html) For more information about Amazon RDS Performance Insights counter metrics, see [ Performance Insights counter metrics](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_PerfInsights_Counters.html). For more information about Amazon DocumentDB Performance Insights counter metrics, see [ Performance Insights for counter metrics](https://docs.aws.amazon.com/documentdb/latest/developerguide/performance-insights-counter-metrics.html). High-resolution metrics with sub-minute granularity retrieved by **DB\$1PERF\$1INSIGHTS** are only applicable to the **DBLoad** metric, or for operating system metrics if you have enabled Enhanced Monitoring at a higher resolution. For more information about Amazon RDS enhanced monitoring, see [ Monitoring OS metrics with Enhanced Monitoring](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_Monitoring.OS.html). You can create a high-resolution alarm using the **DB\$1PERF\$1INSIGHTS** function for a maximum time range of three hours. You can use the CloudWatch console to graph metrics retrieved with the **DB\$1PERF\$1INSIGHTS** function for any time range. | **DB\$1PERF\$1INSIGHTS(‘RDS’, ‘db-ABCDEFGHIJKLMNOPQRSTUVWXY1’, ‘os.cpuUtilization.user.avg’)** **DB\$1PERF\$1INSIGHTS(‘DOCDB, ‘db-ABCDEFGHIJKLMNOPQRSTUVWXY1’, [‘os.cpuUtilization.idle.avg’, ‘os.cpuUtilization.user.max’])** | | | **DIFF** | TSTS[] | TS TS[] | Returns the difference between each value in the time series and the preceding value from that time series. | **DIFF(m1)** | ✓ | | **DIFF\$1TIME** | TSTS[] | TS TS[] | Returns the difference in seconds between the timestamp of each value in the time series and the timestamp of the preceding value from that time series. | **DIFF\$1TIME(METRICS())** | ✓ | | **FILL** | TS, [S \$1 REPEAT \$1 LINEAR] TS[], [TS \$1 S \$1 REPEAT \$1 LINEAR] | TS TS[] | Fills the missing values of a time series. There are several options for the values to use as the filler for missing values: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/using-metric-math.html) When you use this function in an alarm, you can encounter an issue if your metrics are being published with a slight delay, and the most recent minute never has data. In this case, **FILL** replaces that missing data point with the requested value. That causes the latest data point for the metric to always be the FILL value, which can result in the alarm being stuck in either OK state or ALARM state. You can work around this by using a M out of N alarm. For more information, see [Alarm evaluation](alarm-evaluation.md). | **FILL(m1,10)** **FILL(METRICS(), 0)** **FILL(METRICS(), m1)** **FILL(m1, MIN(m1))** **FILL(m1, REPEAT)** **FILL(METRICS(), LINEAR)** | ✓ | | **FIRST** **LAST** | TS[] | TS | Returns the first or last time series from an array of time series. This is useful when used with the **SORT** function. It can also be used to get the high and low thresholds from the **ANOMALY\$1DETECTION\$1BAND** function. | **IF(FIRST(SORT(METRICS(), AVG, DESC))>100, 1, 0)** Looks at the top metric from an array, which is sorted by AVG. It then returns a 1 or 0 for each data point, depending on whether that data point value is more than 100. **LAST(ANOMALY\$1DETECTION\$1BAND(m1))** returns the upper bound of the anomaly prediction band. | ✓ | | **FLOOR** | TSTS[] | TS TS[] | Returns the floor of each metric. The floor is the largest integer less than or equal to each value. | **FLOOR(m1)** **FLOOR(METRICS())** | ✓ | | **IF** | **IF** expression | TS | Use **IF** along with a comparison operator to filter out data points from a time series, or create a mixed time-series composed of multiple collated time series. For more information, see [Using IF expressions](#using-IF-expressions). | For examples, see [Using IF expressions](#using-IF-expressions). | ✓ | | **INSIGHT\$1RULE\$1METRIC** | **INSIGHT\$1RULE\$1METRIC(ruleName, metricName)** | TS | Use **INSIGHT\$1RULE\$1METRIC** to extract statistics from a rule in Contributor Insights. For more information, see [Graphing metrics generated by rules in CloudWatchSetting an alarm on Contributor Insights metric data](ContributorInsights-GraphReportData.md). | | | | **LAMBDA** | **LAMBDA(LambdaFunctionName [, optional-arg]\$1)** | TS TS[] | Calls a Lambda function to query metrics from a data source that is not CloudWatch. For more information, see [How to pass arguments to your Lambda function](CloudWatch_MultiDataSources-Custom-Use.md#MultiDataSources-Connect-Custom-Lambda-arguments). | | | | **LOG** | TS TS[] | TS TS[] | The **LOG** of a time series returns the natural logarithm value of each value in the time series. | **LOG(METRICS())** | ✓ | | **LOG10** | TS TS[] | TS TS[] | The **LOG10** of a time series returns the base-10 logarithm value of each value in the time series. | **LOG10(m1)** | ✓ | | **MAX** | TS TS[] | S TS | The **MAX** of a single time series returns a scalar representing the maximum value of all data points in the metric. If you input an array of time series, the **MAX** function creates and returns a time series that consists of the highest value for each data point, among the time series that were used as the input. We recommend that you do not use this function in CloudWatch alarms if you want the function to return a scalar. For example, `MAX(m2)` Whenever an alarm evaluates whether to change state, CloudWatch attempts to retrieve a higher number of data points than the number specified as Evaluation Periods. This function acts differently when extra data is requested. | **MAX(m1)/m1** **MAX(METRICS())** | ✓ | | **METRIC\$1COUNT** | TS[] | S | Returns the number of metrics in the time series array. | **m1/METRIC\$1COUNT(METRICS())** | ✓ | | **METRICS** | null string | TS[] | The **METRICS()** function returns all CloudWatch metrics in the request. Math expressions aren't included. You can use **METRICS()** within a larger expression that produces a single time series or an array of time series. You can use the **METRICS()** function with a string to return only the graphed metrics that contain that string in their **Id** field. For example, the expression **SUM(METRICS("errors"))** returns a time series that is the sum of the values of all the graphed metrics that have ‘errors’ in their **Id** field. You can also use **SUM([METRICS(“4xx”), METRICS(“5xx”)])** to match multiple strings. | **AVG(METRICS())** **SUM(METRICS("errors"))** | ✓ | | **MIN** | TS TS[] | S TS | The **MIN** of a single time series returns a scalar representing the minimum value of all data points in the metric. If you input an array of time series, the **MIN** function creates and returns a time series that consists of the lowest value for each data point, among the time series that were used as the input. If you input an array of time series, the **MIN** function creates and returns a time series that consists of the lowest value for each data point, among the time series that were used as the input. We recommend that you do not use this function in CloudWatch alarms if you want the function to return a scalar. For example, `MIN(m2)` Whenever an alarm evaluates whether to change state, CloudWatch attempts to retrieve a higher number of data points than the number specified as Evaluation Periods. This function acts differently when extra data is requested. | **m1-MIN(m1)** **MIN(METRICS())** | ✓ | | **MINUTE** **HOUR** **DAY** **DATE** **MONTH** **YEAR** **EPOCH** | TS | TS | These functions take the period and range of the time series and return a new non-sparse time series where each value is based on its timestamp. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/using-metric-math.html) | **MINUTE(m1)** **IF(DAY(m1)<6,m1)** returns metrics only from weekdays, Monday to Friday. **IF(MONTH(m1) == 4,m1)** returns only metrics published in April. | ✓ | | **PERIOD** | TS | S | Returns the period of the metric in seconds. Valid input is metrics, not the results of other expressions. | **m1/PERIOD(m1)** | ✓ | | **RATE** | TS TS[] | TS TS[] | Returns the rate of change of the metric per second. This is calculated as the difference between the latest data point value and the previous data point value, divided by the time difference in seconds between the two values. Setting alarms on expressions that use the RATE function on metrics with sparse data can behave unpredictably, because the range of data points fetched when evaluating the alarm can vary based on when the data points were last published. | **RATE(m1)** **RATE(METRICS())** | ✓ | | **REMOVE\$1EMPTY** | TS[] | TS[] | Removes any time series that have no data points from an array of time series. The result is an array of time series where each time series contains at least one data point. We recommend that you do not use this function in CloudWatch alarms. Whenever an alarm evaluates whether to change state, CloudWatch attempts to retrieve a higher number of data points than the number specified as Evaluation Periods. This function acts differently when extra data is requested. | **REMOVE\$1EMPTY(METRICS())** | ✓ | | **RUNNING\$1SUM** | TS TS[] | TS TS[] | Returns a time series with the running sum of the values in the original time series. We recommend that you do not use this function in CloudWatch alarms. Whenever an alarm evaluates whether to change state, CloudWatch attempts to retrieve a higher number of data points than the number specified as Evaluation Periods. This function acts differently when extra data is requested. | **RUNNING\$1SUM([m1,m2])** | ✓ | | **SEARCH** | Search expression | One or more TS | Returns one or more time series that match a search criteria that you specify. The **SEARCH** function enables you to add multiple related time series to a graph with one expression. The graph is dynamically updated to include new metrics that are added later and match the search criteria. For more information, see [Use search expressions in graphs](using-search-expressions.md). You can't create an alarm based on a **SEARCH** expression. This is because search expressions return multiple time series, and an alarm based on a math expression can watch only one time series. If you are signed in to a monitoring account in CloudWatch cross-account observability, the **SEARCH** function finds metrics in the source accounts and the monitoring account. | | ✓ | | **SERVICE\$1QUOTA** | TS that is a usage metric | TS | Returns the service quota for the given usage metric. You can use this to visualize how your current usage compares to the quota, and to set alarms that warn you when you approach the quota. For more information, see [AWS usage metrics](CloudWatch-Service-Quota-Integration.md). | | ✓ | | **SLICE** | (TS[], S, S) or (TS[], S) | TS[] TS | Retrieves part of an array of time series. This is especially useful when combined with **SORT**. For example, you can exclude the top result from an array of time series. You can use two scalar arguments to define the set of time series that you want returned. The two scalars define the start (inclusive) and end (exclusive) of the array to return. The array is zero-indexed, so the first time series in the array is time series 0. Alternatively, you can specify just one value, and CloudWatch returns all time series starting with that value. | **SLICE(SORT(METRICS(), SUM, DESC), 0, 10)** returns the 10 metrics from the array of metrics in the request that have the highest SUM value. **SLICE(SORT(METRICS(), AVG, ASC), 5)** sorts the array of metrics by the AVG statistic, then returns all the time series except for the 5 with the lowest AVG. | ✓ | | **SORT** | (TS[], FUNCTION, SORT\$1ORDER) (TS[], FUNCTION, SORT\$1ORDER, S) | TS[] | Sorts an array of time series according to the function you specify. The function you use can be **AVG**, **MIN**, **MAX**, or **SUM**. The sort order can be either **ASC** for ascending (lowest values first) or **DESC** to sort the higher values first. You can optionally specify a number after the sort order which acts as a limit. For example, specifying a limit of 5 returns only the top 5 time series from the sort. When this math function is displayed on a graph, the labels for each metric in the graph are also sorted and numbered. | **SORT(METRICS(), AVG, DESC, 10)** calculates the average value of each time series, sorts the time series with the highest values at the beginning of the sort, and returns only the 10 time series with the highest averages. **SORT(METRICS(), MAX, ASC)** sorts the array of metrics by the MAX statistic, then returns all of them in ascending order. | ✓ | | **STDDEV** | TS TS[] | S TS | The **STDDEV** of a single time series returns a scalar representing the standard deviation of all data points in the metric. The **STDDEV** of an array of time series returns a single time series. We recommend that you do not use this function in CloudWatch alarms if you want the function to return a scalar. For example, `STDDEV(m2)` Whenever an alarm evaluates whether to change state, CloudWatch attempts to retrieve a higher number of data points than the number specified as Evaluation Periods. This function acts differently when extra data is requested. | **m1/STDDEV(m1)** **STDDEV(METRICS())** | ✓ | | **SUM** | TS TS[] | S TS | The **SUM** of a single time series returns a scalar representing the sum of the values of all data points in the metric. The **SUM** of an array of time series returns a single time series. We recommend that you do not use this function in CloudWatch alarms if you want the function to return a scalar. For example, `SUM(m1)`. Whenever an alarm evaluates whether to change state, CloudWatch attempts to retrieve a higher number of data points than the number specified as Evaluation Periods. This function acts differently when extra data is requested. | **SUM(METRICS())/SUM(m1)** **SUM([m1,m2])** **SUM(METRICS("errors"))/SUM(METRICS("requests"))\$1100** | ✓ | | **TIME\$1SERIES** | S | TS | Returns a non-sparse time series where every value is set to a scalar argument. | **TIME\$1SERIES(MAX(m1))** **TIME\$1SERIES(5\$1AVG(m1))** **TIME\$1SERIES(10)** | ✓ | **\$1**Using a function that returns only a scalar number is not valid, as all final results of expressions must be a single time series or an array of time series. Instead, use these functions as part of a larger expression that returns a time series. ## Using IF expressions Use **IF** along with a comparison operator to filter out data points from a time series, or create a mixed time-series composed of multiple collated time series. **IF** uses the following arguments: ``` IF(condition, trueValue, falseValue) ``` The condition evaluates to FALSE if the value of the condition data point is 0, and to TRUE if the value of the condition is any other value, whether that value is positive or negative. If the condition is a time series, it is evaluated separately for every timestamp. The following lists the valid syntax’s. For each of these syntax’s, the output is a single time series. + **IF(TS *Comparison Operator* S, S \$1 TS, *S \$1 TS*)** **Note** If the `TS comparison operator S` is TRUE but `metric2` doesn't have a corresponding data point, the output will be 0. + **IF(TS, TS, *TS*)** + **IF(TS, S, *TS*)** + **IF(TS, TS, S)** + **IF(TS, S, S)** + **IF(S, TS, *TS*)** The following sections provide more details and examples for these syntax’s. **IF(TS *Comparison Operator* S, scalar2 \$1 metric2, scalar3 \$1 metric3)** The corresponding output time series value: + has the value of **scalar2** or **metric2**, if **TS *Comparison Operator* S** is TRUE + has the value of **scalar3** or **metric3**, if **TS *Comparison Operator* S** is FALSE + has the value of **0** if the **TS *Comparison Operator*** is TRUE and the corresponding data point in **metric2** doesn't exist. + has the value of **0** if the **TS *Comparison Operator*** is FALSE and the corresponding data point in **metric3** doesn't exist. + is an empty time series, if the corresponding data point of does not exist in **metric3**, or if **scalar3**/**metric3** is omitted from the expression **IF(metric1, metric2, *metric3*)** For each data point of **metric1**, the corresponding output time series value: + has the value of **metric2**, if the corresponding data point of **metric1** is TRUE. + has the value of **metric3**, if the corresponding data point of **metric1** is FALSE. + has the value of **0**, if the corresponding data point of **metric1** is TRUE and the corresponding data point does not exist in **metric2**. + has the value of **0**, if the corresponding data point of **metric1** is FALSE and the corresponding data point does not exist in **metric3** + is dropped, if the corresponding data point of **metric1** is FALSE and **metric3** is omitted from the expression. + is dropped, if the corresponding data point of **metric1** is missing. The following table shows an example for this syntax. | Metric or function | Values | | --- | --- | | **(metric1)** | **[1, 1, 0, 0, -]** | | **(metric2)** | **[30, -, 0, 0, 30]** | | **(metric3)** | **[0, 0, 20, -, 20]** | | **IF(metric1, metric2, metric3)** | **[30, 0, 20, 0, -]** | **IF(metric1, scalar2, metric3)** For each data point of **metric1**, the corresponding output time series value: + has the value of **scalar2**, if the corresponding data point of **metric1** is TRUE. + has the value of **metric3**, if the corresponding data point of **metric1** is FALSE. + is dropped, if the corresponding data point of **metric1** is FALSE and the corresponding data point does not exist on **metric3**, or if **metric3** is omitted from the expression. | Metric or function | Values | | --- | --- | | **(metric1)** | **[1, 1, 0, 0, -]** | | **scalar2** | **5** | | **(metric3)** | **[0, 0, 20, -, 20]** | | **IF(metric1, scalar2, metric3)** | **[5, 5, 20, -, -]** | **IF(metric1, metric2, scalar3)** For each data point of **metric1**, the corresponding output time series value: + has the value of **metric2**, if the corresponding data point of **metric1** is TRUE. + has the value of **scalar3**, if the corresponding data point of **metric1** is FALSE. + has the value of **0**, if the corresponding data point of **metric1** is TRUE and the corresponding data point does not exist in **metric2**. + is dropped if the corresponding data point in **metric1** does not exist. | Metric or function | Values | | --- | --- | | **(metric1)** | **[1, 1, 0, 0, -]** | | **(metric2)** | **[30, -, 0, 0, 30]** | | **scalar3** | **5** | | **IF(metric1, metric2, scalar3)** | **[30, 0, 5, 5, -]** | **IF(scalar1, metric2, *metric3*)** The corresponding output time series value: + has the value of **metric2**, if **scalar1** is TRUE. + has the value of **metric3**, if **scalar1** is FALSE. + is an empty time series, if **metric3** is omitted from the expression. ### Use case examples for IF expressions The following examples illustrate the possible uses of the **IF** function. + To display only the low values of a metric: **IF(metric1<400, metric1)** + To change each data point in a metric to one of two values, to show relative highs and lows of the original metric: **IF(metric1<400, 10, 2)** + To display a 1 for each timestamp where latency is over the threshold, and display a 0 for all other data points: **IF(latency>threshold, 1, 0)** ### Use metric math with the GetMetricData API operation You can use `GetMetricData` to perform calculations using math expressions, and also retrieve large batches of metric data in one API call. For more information, see [GetMetricData](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_GetMetricData.html). ## Anomaly detection on metric math Anomaly detection on metric math Anomaly detection on metric math is a feature that you can use to create anomaly detection alarms on single metrics and the outputs of metric math expressions. You can use these expressions to create graphs that visualize anomaly detection bands. The feature supports basic arithmetic functions, comparison and logical operators, and most other functions. **Anomaly detection on metric math doesn't support the following functions:** + Expressions that contain more than one **ANOMALY\$1DETECTION\$1BAND** in the same line. + Expressions that contain more than 10 metrics or math expressions. + Expressions that contain the **METRICS** expression. + Expressions that contain the **SEARCH** function. + Expressions that use the **DP\$1PERF\$1INSIGHTS** function. + Expressions that use metrics with different periods. + Expressions that use periods longer than one hour as input. + Expressions that use high-resolution metrics as input. For more information about this feature, see [Using CloudWatch anomaly detection](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Anomaly_Detection.html#anomaly_detection_on_metric_math) in the *Amazon CloudWatch User Guide*. # Use search expressions in graphs Search expressions are a type of math expression that you can add to CloudWatch graphs. Search expressions enable you to quickly add multiple related metrics to a graph. They also enable you to create dynamic graphs that automatically add appropriate metrics to their display, even if those metrics don't exist when you first create the graph. For example, you can create a search expression that displays the `AWS/EC2 CPUUtilization` metric for all instances in the Region. If you later launch a new instance, the `CPUUtilization` of the new instance is automatically added to the graph. When you use a search expression in a graph, the search finds the search expression in metric names, namespaces, dimension names, and dimension values. You can use Boolean operators for more complex and powerful searches. A search expression can find only metrics that have reported data within the past two weeks. You can't create an alarm based on the **SEARCH** expression. This is because search expressions return multiple time series, and an alarm based on a math expression can watch only one time series. If you are using a monitoring account in CloudWatch cross-account observability, your search expressions can find metrics in the source accounts linked to that monitoring account. **Topics** + [ # CloudWatch search expression syntax ](search-expression-syntax.md) + [ # CloudWatch search expression examples ](search-expression-examples.md) + [ # Create a CloudWatch graph with a search expression ](create-search-expression.md) # CloudWatch search expression syntax Search expression syntax A valid search expression has the following format. ``` SEARCH(' {Namespace, DimensionName1, DimensionName2, ...} SearchTerm', 'Statistic') ``` For example: ``` SEARCH('{AWS/EC2,InstanceId} MetricName="CPUUtilization"', 'Average') ``` + The first part of the query after the word `SEARCH`, enclosed in curly braces, is the *metric schema* to be searched. The metric schema contains a metric namespace and one or more dimension names. Including a metric schema in a search query is optional. If specified, the metric schema must contain a namespace and can optionally contain one or more dimension names that are valid in that namespace. You don't need to use quote marks inside the metric schema unless a namespace or dimension name includes spaces or non-alphanumeric characters. In that case, you must enclose the name that contains those characters with double quotes. + The `SearchTerm` is also optional, but a valid search must contain either the metric schema, the `SearchTerm`, or both. The `SearchTerm` usually contains one or more account IDs, metric names or dimension values. The `SearchTerm` can include multiple terms to search for, by both partial match and exact match. It can also contain Boolean operators. Using an account ID in a `SearchTerm` works only in accounts that are set up as monitoring accounts for CloudWatch cross-account observability. The syntax for an account ID in `SearchTerm` is `:aws.AccountId = 444455556666`. You can also use `'LOCAL'` to specify the monitoring account itself: `:aws.AccountId = 'LOCAL'` For more information, see [CloudWatch cross-account observability](CloudWatch-Unified-Cross-Account.md). The `SearchTerm` can include one or more designators, such as `MetricName=` as in this example, but using designators isn't required. The metric schema and `SearchTerm` must be enclosed together in a pair of single quote marks. + The `Statistic` is the name of any valid CloudWatch statistic. It must be enclosed by single quotes. For more information, see [Statistics](cloudwatch_concepts.md#Statistic). The preceding example searches the `AWS/EC2` namespace for any metrics that have `InstanceId` as a dimension name. It returns all `CPUUtilization` metrics that it finds, with the graph showing the `Average` statistic. A search expression can find only metrics that have reported data within the past two weeks. **Search expression limits** The maximum search expression query size is 1024 characters. You can have as many as 100 search expressions on one graph. A graph can display as many as 500 time series. ## CloudWatch search expressions: Tokenization Tokenization When you specify a `SearchTerm`, the search function searches for *tokens*, which are substrings that CloudWatch automatically generates from full metric names, dimension names, dimension values, and namespaces. CloudWatch generates tokens distinguished by the camel-case capitalization in the original string. Numeric characters also serve as the start of new tokens, and non-alphanumeric characters serve as delimiters, creating tokens before and after the non-alphanumeric characters. A continuous string of the same type of token delimiter character results in one token. All generated tokens are in lowercase. The following table shows some examples of tokens generated. | Original string | Tokens generated | | --- | --- | | CustomCount1 | `customcount1`, `custom`, `count`, `1` | | SDBFailure | `sdbfailure`, `sdb`, `failure` | | Project2-trial333 | `project2trial333`, `project`, `2`, `trial`, `333` | ## CloudWatch search expressions: Partial matches Partial matches When you specify a `SearchTerm`, the search term is also tokenized. CloudWatch finds metrics based on partial matches, which are matches of a single token generated from the search term to a single token generated from a metric name, namespace, dimension name, or dimension value. Partial match searches to match a single token are case insensitive. For example, using any of the following search terms can return the `CustomCount1` metric: + `count` + `Count` + `COUNT` However, using `couNT` as a search term doesn't find `CustomCount1` because the capitalization in the search term `couNT` is tokenized into `cou` and `NT`. Searches can also match composite tokens, which are multiple tokens that appear consecutively in the original name. To match a composite token, the search is case sensitive. For example, if the original term is `CustomCount1`, searches for `CustomCount` or `Count1` are successful, but searches for `customcount` or `count1` aren't. ## CloudWatch search expressions: Exact matches Exact matches You can define a search to find only exact matches of your search term by using double quotes around the part of the search term that requires an exact match. These double-quotes are enclosed in the single-quotes used around the entire search term. For example, **SEARCH(' \$1MyNamespace\$1, "CustomCount1" ', 'Maximum')** finds the exact string `CustomCount1` if it exists as a metric name, dimension name, or dimension value in the namespace named `MyNamespace`. However, the searches **SEARCH(' \$1MyNamespace\$1, "customcount1" ', 'Maximum')** or **SEARCH(' \$1MyNamespace\$1, "Custom" ', 'Maximum')** do not find this string. You can combine partial match terms and exact match terms in a single search expression. For example, **SEARCH(' \$1AWS/NetworkELB, LoadBalancer\$1 "ConsumedLCUs" OR flow ', 'Maximum')** returns the Elastic Load Balancing metric named `ConsumedLCUs` as well as all Elastic Load Balancing metrics or dimensions that contain the token `flow`. Using exact match is also a good way to find names with special characters, such as non-alphanumeric characters or spaces, as in the following example. ``` SEARCH(' {"My Namespace", "Dimension@Name"}, "Custom:Name[Special_Characters" ', 'Maximum') ``` ## CloudWatch search expressions: Excluding a metric schema Exclude a metric schema All examples shown so far include a metric schema, in curly braces. Searches that omit a metric schema are also valid. For example, **SEARCH(' "CPUUtilization" ', 'Average')** returns all metric names, dimension names, dimension values, and namespaces that are an exact match for the string `CPUUtilization`. In the AWS metric namespaces, this can include metrics from several services including Amazon EC2, Amazon ECS, SageMaker AI, and others. To narrow this search to only one AWS service, the best practice is to specify the namespace and any necessary dimensions in the metric schema, as in the following example. Although this narrows the search to the `AWS/EC2` namespace, it would still return results of other metrics if you have defined `CPUUtilization` as a dimension value for those metrics. ``` SEARCH(' {AWS/EC2, InstanceType} "CPUUtilization" ', 'Average') ``` Alternatively you could add the namespace in the `SearchTerm` as in the following example. But in this example, the search would match any `AWS/EC2` string, even if it was a custom dimension name or value. ``` SEARCH(' "AWS/EC2" MetricName="CPUUtilization" ', 'Average') ``` ## CloudWatch search expressions: Specifying property names in the search Specify property names in the search The following exact match search for `"CustomCount1"` returns all metrics with exactly that name. ``` SEARCH(' "CustomCount1" ', 'Maximum') ``` But it also returns metrics with dimension names, dimension values, or namespaces of `CustomCount1`. To structure your search further, you can specify the property name of the type of object that you want to find in your searches. The following example searches all namespaces and returns metrics named `CustomCount1`. ``` SEARCH(' MetricName="CustomCount1" ', 'Maximum') ``` You can also use namespaces and dimension name/value pairs as property names, as in the following examples. The first of these examples also illustrates that you can use property names with partial match searches as well. ``` SEARCH(' InstanceType=micro ', 'Average') ``` ``` SEARCH(' InstanceType="t2.micro" Namespace="AWS/EC2" ', 'Average') ``` ## CloudWatch search expressions: Non-alphanumeric characters Non-alphanumeric characters Non-alphanumeric characters serve as delimiters, and mark where the names of metrics, dimensions, namespaces, and search terms are to be separated into tokens. When terms are tokenized, non-alphanumeric characters are stripped out and don't appear in the tokens. For example, `Network-Errors_2` generates the tokens `network`, `errors`, and `2`. Your search term can include any non-alphanumeric characters. If these characters appear in your search term, they can specify composite tokens in a partial match. For example, all of the following searches would find metrics named either `Network-Errors-2` or `NetworkErrors2`. ``` network/errors network+errors network-errors Network_Errors ``` When you're doing an exact value search, any non-alphanumeric characters used in the exact search must be the correct characters that appear in the string being searched for. For example, if you want to find `Network-Errors-2`, searching for `"Network-Errors-2"` is successful, but a search for `"Network_Errors_2"` isn't. When you perform an exact match search, the following characters must be escaped with a backslash. ``` " \ ( ) ``` For example, to find the metric name `Europe\France Traffic(Network)` by exact match, use the search term **"Europe\$1\$1France Traffic\$1(Network\$1)"** ## CloudWatch search expressions: Boolean operators Boolean operators Search supports the use of the Boolean operators `AND`, `OR`, and `NOT` within the `SearchTerm`. Boolean operators are enclosed in the single quote marks that you use to enclose the entire search term. Boolean operators are case sensitive, so `and`, `or`, and `not` aren't valid as Boolean operators. You can use `AND` explicitly in your search, such as **SEARCH('\$1AWS/EC2,InstanceId\$1 network AND packets', 'Average')**. Not using any Boolean operator between search terms implicitly searches them as if there were an `AND` operator, so **SEARCH(' \$1AWS/EC2,InstanceId\$1 network packets ', 'Average')** yields the same search results. Use `NOT` to exclude subsets of data from the results. For example, **SEARCH(' \$1AWS/EC2,InstanceId\$1 MetricName="CPUUtilization" NOT i-1234567890123456 ', 'Average')** returns the `CPUUtilization` for all your instances, except for the instance `i-1234567890123456`. You can also use a `NOT` clause as the only search term. For example, **SEARCH( 'NOT Namespace=AWS ', 'Maximum')** yields all your custom metrics (metrics with namespaces that don't include `AWS`). You can use multiple `NOT` phrases in a query. For example, **SEARCH(' \$1AWS/EC2,InstanceId\$1 MetricName="CPUUtilization" NOT "ProjectA" NOT "ProjectB" ', 'Average')** returns the `CPUUtilization` of all instances in the Region, except for those with dimension values of `ProjectA` or `ProjectB`. You can combine Boolean operators for more powerful and detailed searches, as in the following examples. Use parentheses to group the operators. Both of the next two examples return all metric names containing `ReadOps` from both the EC2 and EBS namespaces. ``` SEARCH(' (EC2 OR EBS) AND MetricName=ReadOps ', 'Maximum') ``` ``` SEARCH(' (EC2 OR EBS) MetricName=ReadOps ', 'Maximum') ``` The following example narrows the previous search to only results that include `ProjectA`, which could be the value of a dimension. ``` SEARCH(' (EC2 OR EBS) AND ReadOps AND ProjectA ', 'Maximum') ``` The following example uses nested grouping. It returns Lambda metrics for `Errors` from all functions, and `Invocations` of functions with names that include the strings `ProjectA` or `ProjectB`. ``` SEARCH(' {AWS/Lambda,FunctionName} MetricName="Errors" OR (MetricName="Invocations" AND (ProjectA OR ProjectB)) ', 'Average') ``` ## CloudWatch search expressions: Using math expressions Using math expressions You can use a search expression within a math expressions in a graph. For example, **SUM(SEARCH(' \$1AWS/Lambda, FunctionName\$1 MetricName="Errors" ', 'Sum'))** returns the sum of the `Errors` metric of all your Lambda functions. Using separate lines for your search expression and math expression might yield more useful results. For example, suppose that you use the following two expressions in a graph. The first line displays separate `Errors` lines for each of your Lambda functions. The ID of this expression is `e1`. The second line adds another line showing the sum of the errors from all of the functions. ``` SEARCH(' {AWS/Lambda, FunctionName}, MetricName="Errors" ', 'Sum') SUM(e1) ``` # CloudWatch search expression examples Search expression examples The following examples illustrate more search expression uses and syntax. Let's start with a search for `CPUUtilization` across all instances in the Region and then look at variations. This example displays one line for each instance in the Region, showing the `CPUUtilization` metric from the `AWS/EC2` namespace. ``` SEARCH(' {AWS/EC2,InstanceId} MetricName="CPUUtilization" ', 'Average') ``` Changing `InstanceId` to `InstanceType` changes the graph to show one line for each instance type used in the Region. Data from all instances of each type is aggregated into one line for that instance type. ``` SEARCH(' {AWS/EC2,InstanceType} MetricName="CPUUtilization" ', 'Average') ``` The following example aggregates the `CPUUtilization` by instance type and displays one line for each instance type that includes the string `micro`. ``` SEARCH('{AWS/EC2,InstanceType} InstanceType=micro MetricName="CPUUtilization" ', 'Average') ``` This example narrows the previous example, changing the `InstanceType` to an exact search for t2.micro instances. ``` SEARCH('{AWS/EC2,InstanceType} InstanceType="t2.micro" MetricName="CPUUtilization" ', 'Average') ``` The following search removes the `{metric schema}` part of the query, so the `CPUUtilization` metric from all namespaces appears in the graph. This can return quite a few results because the graph includes multiple lines for the `CPUUtilization` metric from each AWS service, aggregated along different dimensions. ``` SEARCH('MetricName="CPUUtilization" ', 'Average') ``` To narrow these results a bit, you can specify two specific metric namespaces. ``` SEARCH('MetricName="CPUUtilization" AND ("AWS/ECS" OR "AWS/ES") ', 'Average') ``` The preceding example is the only way to do a search of specific multiple namespaces with one search query, as you can specify only one metric schema in each query. However, to add more structure, you could use two queries in the graph, as in the following example. This example also adds more structure by specifying a dimension to use to aggregate the data for Amazon ECS. ``` SEARCH('{AWS/ECS ClusterName}, MetricName="CPUUtilization" ', 'Average') SEARCH(' {AWS/EBS} MetricName="CPUUtilization" ', 'Average') ``` The following example returns the Elastic Load Balancing metric named `ConsumedLCUs` as well as all Elastic Load Balancing metrics or dimensions that contain the token `flow`. ``` SEARCH('{AWS/NetworkELB, LoadBalancer} "ConsumedLCUs" OR flow ', 'Maximum') ``` The following example uses nested grouping. It returns Lambda metrics for `Errors` from all functions and `Invocations` of functions with names that include the strings `ProjectA` or `ProjectB`. ``` SEARCH('{AWS/Lambda,FunctionName} MetricName="Errors" OR (MetricName="Invocations" AND (ProjectA OR ProjectB)) ', 'Average') ``` The following example displays all of your custom metrics, excluding metrics generated by AWS services. ``` SEARCH('NOT Namespace=AWS ', 'Average') ``` The following example displays metrics with metric names, namespaces, dimension names, and dimension values that contain the string `Errors` as part of their name. ``` SEARCH('Errors', 'Average') ``` The following example narrows that search to exact matches. For example, this search finds the metric name `Errors` but not metrics named `ConnectionErrors` or `errors`. ``` SEARCH(' "Errors" ', 'Average') ``` The following example shows how to specify names that contain spaces or special characters in the metric schema part of the search term. ``` SEARCH('{"Custom-Namespace", "Dimension Name With Spaces"}, ErrorCount ', 'Maximum') ``` ## CloudWatch cross-account observability search expression examples **CloudWatch cross-account observability examples** If you are signed in to an account that is set up as a monitoring account in CloudWatch cross-account observability, you can use the **SEARCH** function to return metrics from specified source accounts. For more information, see [CloudWatch cross-account observability](CloudWatch-Unified-Cross-Account.md). The following example retrieves all Lambda metrics from the account with the account ID 111122223333. ``` SEARCH(' AWS/Lambda :aws.AccountId = 111122223333 ', 'Average') ``` The following example retrieves all `AWS/EC2` metrics from two accounts: 111122223333 and 777788889999. ``` SEARCH(' AWS/EC2 :aws.AccountId = (111122223333 OR 777788889999) ', 'Average') ``` The following example retrieves all `AWS/EC2` metrics from the source account 111122223333 and from the monitoring account itself. ``` SEARCH(' AWS/EC2 :aws.AccountId = (111122223333 OR 'LOCAL') ', 'Average') ``` The following example retrieves the `SUM` of the `MetaDataToken` metric from the account `444455556666` with the `InstanceId` dimension. ``` SEARCH('{AWS/EC2,InstanceId} :aws.AccountId=444455556666 MetricName=\"MetadataNoToken\"','Sum') ``` # Create a CloudWatch graph with a search expression Creating a graph with a search expression On the CloudWatch console, you can access search capability when you add a graph to a dashboard, or by using the **Metrics** view. You can't create an alarm based on a **SEARCH** expression. This is because search expressions return multiple time series, and an alarm based on a math expression can watch only one time series. **To add a graph with a search expression to an existing dashboard** 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the navigation pane, choose **Dashboards** and select a dashboard. 1. Choose **Add widget**. 1. Choose either **Line** or **Stacked area** and choose **Configure**. 1. On the **Graphed metrics** tab, choose **Add a math expression**. 1. For **Details**, enter the search expression that you want. For example, **SEARCH('\$1AWS/EC2,InstanceId\$1 MetricName="CPUUtilization"', 'Average')** 1. (Optional) To add another search expression or math expression to the graph, choose **Add a math expression** 1. (Optional) After you add a search expression, you can specify a dynamic label to appear on the graph legend for each metric. Dynamic labels display a statistic about the metric and automatically update when the dashboard or graph is refreshed. To add a dynamic label, choose **Graphed metrics** and then **Dynamic labels**. By default, the dynamic values you add to the label appear at the beginning of the label. You can then click the **Label** value for the metric to edit the label. For more information, see [Use dynamic labels](graph-dynamic-labels.md). 1. (Optional) To add a single metric to the graph, choose the **All metrics** tab and drill down to the metric you want. 1. (Optional) To change the time range shown on the graph, choose either **custom** at the top of the graph or one of the time periods to the left of **custom**. 1. (Optional) Horizontal annotations help dashboard users quickly see when a metric has spiked to a certain level or whether the metric is within a predefined range. To add a horizontal annotation, choose **Graph options** and then **Add horizontal annotation**: 1. For **Label**, enter a label for the annotation. 1. For **Value**, enter the metric value where the horizontal annotation appears. 1. For **Fill**, specify whether to use fill shading with this annotation. For example, choose `Above` or `Below` for the corresponding area to be filled. If you specify `Between`, another `Value` field appears, and the area of the graph between the two values is filled. 1. For **Axis**, specify whether the numbers in `Value` refer to the metric associated with the left y-axis or the right y-axis if the graph includes multiple metrics. You can change the fill color of an annotation by choosing the color square in the left column of the annotation. Repeat these steps to add multiple horizontal annotations to the same graph. To hide an annotation, clear the check box in the left column for that annotation. To delete an annotation, choose **x** in the **Actions** column. 1. (Optional) Vertical annotations help you mark milestones in a graph, such as operational events or the beginning and end of a deployment. To add a vertical annotation, choose **Graph options** and then **Add vertical annotation**: 1. For **Label**, enter a label for the annotation. To show only the date and time on the annotation, keep the **Label** field blank. 1. For **Date**, specify the date and time where the vertical annotation appears. 1. For **Fill**, specify whether to use fill shading before or after a vertical annotation or between two vertical annotations. For example, choose `Before` or `After` for the corresponding area to be filled. If you specify `Between`, another `Date` field appears, and the area of the graph between the two values is filled. Repeat these steps to add multiple vertical annotations to the same graph. To hide an annotation, clear the check box in the left column for that annotation. To delete an annotation, choose **x** in the **Actions** column. 1. Choose **Create widget**. 1. Choose **Save dashboard**. **To use the Metrics view to graph searched metrics** 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the navigation pane, choose **Metrics**, **All metrics**. 1. In the search field, enter the tokens to search for: for example, **cpuutilization t2.small**. Results that match your search appear. 1. To graph all of the metrics that match your search, choose **Graph search**. or To refine your search, choose one of the namespaces that appeared in your search results. 1. If you selected a namespace to narrow your results, you can do the following: 1. To graph one or more metrics, select the check box next to each metric. To select all metrics, select the check box in the heading row of the table. 1. To refine your search, hover over a metric name and choose **Add to search** or **Search for this only**. 1. To view help for a metric, select the metric name and choose **What is this?**. The selected metrics appear on the graph. 1. (Optional) Select one of the buttons in the search bar to edit that part of the search term. 1. (Optional) To add the graph to a dashboard, choose **Actions** and then **Add to dashboard**. # Get statistics for a metric # CloudWatch statistics definitions Statistics are metric data aggregations over specified periods of time. When you graph or retrieve the statistics for a metric, you specify the *Period* of time, such as five minutes, to use to calculate each statistical value. For example, if the **Period** is five minutes, the **Sum** is the sum of all sample values collected during the five-minute period, while the **Minimum** is the lowest value collected during the five-minute period. CloudWatch supports the following statistics for metrics. + **SampleCount** is the number of data points during the period. + **Sum** is the sum of the values of the all data points collected during the period. + **Average** is the value of `Sum/SampleCount` during the specified period. + **Minimum** is the lowest value observed during the specified period. + **Maximum** is the highest value observed during the specified period. + **Percentile (p)** indicates the relative standing of a value in a dataset. For example, **p95** is the 95th percentile and means that 95 percent of the data within the period is lower than this value and 5 percent of the data is higher than this value. Percentiles help you get a better understanding of the distribution of your metric data. + **Trimmed mean (TM)** is the mean of all values that are between two specified boundaries. Values outside of the boundaries are ignored when the mean is calculated. You define the boundaries as one or two numbers between 0 and 100, up to 10 decimal places. The numbers can be absolute values or percentages. For example, **tm90** calculates the average after removing the 10% of data points with the highest values. **TM(2%:98%)** calculates the average after removing the 2% lowest data points and the 2% highest data points. **TM(150:1000)** calculates the average after removing all data points that are lower than or equal to 150, or higher than 1000. + **Interquartile mean (IQM)** is the trimmed mean of the *interquartile range*, or the middle 50% of values. It is equivalent to **TM(25%:75%)**. + **Winsorized mean (WM)** is similar to trimmed mean. However, with winsorized mean, the values that are outside the boundary are not ignored, but instead are considered to be equal to the value at the edge of the appropriate boundary. After this normalization, the average is calculated. You define the boundaries as one or two numbers between 0 and 100, up to 10 decimal places. For example, **wm98** calculates the average while treating the 2% of the highest values to be equal to the value at the 98th percentile. **WM(10%:90%)** calculates the average while treating the highest 10% of data points to be the value of the 90% boundary, and treating the lowest 10% of data points to be the value of the 10% boundary. + **Percentile rank (PR)** is the percentage of values that meet a fixed threshold. For example, **PR(:300)** returns the percentage of data points that have a value of 300 or less. **PR(100:2000)** returns the percentage of data points that have a value between 100 and 2000. Percentile rank is exclusive on the lower bound and inclusive on the upper bound. + **Trimmed count (TC)** is the number of data points in the chosen range for a trimmed mean statistic. For example, **tc90** returns the number of data points not including any data points that fall in the highest 10% of the values. **TC(0.005:0.030)** returns the number of data points with values between 0.005 (exclusive) and 0.030 (inclusive). Trimmed count can return a decimal value instead of an integer. This is because it is an interpolated, approximate value and can give fractional results. + **Trimmed sum (TS)** is the sum of the values of data points in a chosen range for a trimmed mean statistic. It is equivalent to (Trimmed Mean) \$1 (Trimmed count). For example, **ts90** returns the sum of the data points not including any data points that fall in the highest 10% of the values. **TS(80%:)** returns the sum of the data point values, not including any data points with values in the lowest 80% of the range of values. **Note** For Trimmed Mean, Trimmed Count, Trimmed Sum, and Winsorized Mean, if you define two boundaries as fixed values instead of percentages, the calculation includes values equal to the higher boundary, but does not include values equal to the lower boundary. ## Syntax For Trimmed Mean, Trimmed Count, Trimmed Sum, and Winsorized Mean, the following syntax rules apply: + Using parentheses with one or two numbers with percent signs defines the boundaries to use as the values in the data set that fall in between the two percentiles that you specify. For example, **TM(10%:90%)** uses only the values between the 10th and 90th percentiles. **TM(:95%)** uses the values from the lowest end of the data set up to the 95th percentile, ignoring the 5% of data points with the highest values. + Using parenthesis with one or two numbers without percent signs defines the boundaries to use as the values in the data set that fall in between the explicit values that you specify. For example, **TC(80:500)** uses only the values that are between 80 (exclusive) and 500 (inclusive). **TC(:0.5)** uses only the values that equal 0.5 or are lower. + Using one number without parentheses calculates using percentages, ignoring data points that are higher than the specified percentile. For example, **tm99** calculates the mean while ignoring the 1% of the data points with the highest value. It is the same as **TM(:99%)**. + Trimmed mean, Trimmed Count, Trimmed Sum, and Winsorized Mean can all be abbreviated using uppercase letters when specifying a range, such as **TM(5%:95%)**, **TM(100:200)**, or **TM(:95%)**. They can only be abbreviated using lowercase letters when you specifying only one number, such as **tm99**. ## Statistics use cases + **Trimmed mean** is most useful for metrics with a large sample size, such as webpage latency. For example, **tm99** disregards extreme high outliers that could result from network problems or human errors, to give a more accurate number for the average latency of typical requests. Similarly, **TM(10%:)** disregards the lowest 10% of latency values, such as those resulting from cache hits. And **TM(10%:99%)** excludes both of these types of outliers. We recommend that you use trimmed mean for monitoring latency. + It is a good idea to keep watch on trimmed count whenever you are using trimmed mean, to make sure that the number of values being used in your trimmed mean calculations are enough to be statistically significant. + Percentile rank enables you to put values into "bins" of ranges, and you can use this to manually create a histogram. To do this, break your values down into various bins, such as **PR(:1)**, **PR(1:5)**, **PR(5:10)**, and **PR(10:)**. Put each of these bins into a visualization as bar charts, and you have a histogram. Percentile rank is exclusive on the lower bound and inclusive on the upper bound. ## Percentiles versus trimmed mean A percentile such as **p99** and a trimmed mean such as **tm99** measure similar, but not identical values. Both **p99** and **tm99** ignore the 1% of the data points with the highest values, which are considered outliers. After that, **p99** is the **maximum value of the remaining 99%, while **tm99** is the *average* of the remaining 99%. If you are looking at the latency of web requests, **p99** tells you the worst customer experience, ignoring outliers, while **tm99** tells you the average customer experience, ignoring outliers. Trimmed mean is a good latency statistic to watch if you are looking to optimize your customer experience. ## Requirements to use percentiles, trimmed mean, and some other statistics CloudWatch needs raw data points to calculate the following statistics: + Percentiles + Trimmed mean + Interquartile mean + Winsorized mean + Trimmed sum + Trimmed count + Percentile rank If you publish data for a custom statistics using a statistic set instead of raw data, you can retrieve these types of statistics for this data only if one of the following conditions is true: + The SampleCount value of the statistic set is 1 and Min, Max, and Sum are all equal. + The Min and Max are equal, and Sum is equal to Min multiplied by SampleCount. The following AWS services include metrics that support these types of statistics. + API Gateway + Application Load Balancer + Amazon EC2 + Elastic Load Balancing + Kinesis + Amazon RDS Additionally, these type of statistics are not available for metrics when any of the metric values are negative numbers. ## Examples The following examples show you how to get statistics for the CloudWatch metrics for your resources, such as your EC2 instances. **Topics** # Get statistics for a specific resource The following example shows you how to determine the maximum CPU utilization of a specific EC2 instance. **Requirements** + You must have the ID of the instance. You can get the instance ID using the Amazon EC2 console or the [describe-instances](https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-instances.html) command. + By default, basic monitoring is enabled, but you can enable detailed monitoring. For more information, see [Enable or Disable Detailed Monitoring for Your Instances](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-cloudwatch-new.html) in the *Amazon EC2 User Guide*. **To display the average CPU utilization for a specific instance using the console** 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the navigation pane, choose **Metrics**, **All metrics**. 1. Select the **EC2** metric namespace. ![\[Select the EC2 metrics namespace\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/images/metric_view_categories.png) 1. Select the **Per-Instance Metrics** dimension. ![\[View the metric dimensions for Amazon EC2\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/images/metric_view_metric_category.png) 1. In the search field, enter **CPUUtilization** and press Enter. Select the row for the specific instance, which displays a graph for the `CPUUtilization` metric for the instance. To change the name of the graph, choose the pencil icon. To change the time range, select one of the predefined values or choose **custom**. ![\[Graph a single metric\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/images/metric_statistics_ec2_instance.png) 1. To change the statistic, choose the **Graphed metrics** tab. Choose the column heading or an individual value and then choose one of the statistics or predefined percentiles, or specify a custom percentile (for example, **p99.999**). ![\[Change the statistic for a metric\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/images/metric_statistics_ec2_instance_statistic_period.png) 1. To change the period, choose the **Graphed metrics** tab. Choose the column heading or an individual value, and then choose a different value. **To get the CPU utilization per EC2 instance using the AWS CLI** Use the [get-metric-statistics](https://docs.aws.amazon.com/cli/latest/reference/cloudwatch/get-metric-statistics.html) command as follows to get the `CPUUtilization` metric for the specified instance. ``` aws cloudwatch get-metric-statistics --namespace AWS/EC2 --metric-name CPUUtilization \ --dimensions Name=InstanceId,Value=i-1234567890abcdef0 --statistics Maximum \ --start-time 2016-10-18T23:18:00 --end-time 2016-10-19T23:18:00 --period 360 ``` The returned statistics are 6-minute values for the requested 24-hour time interval. Each value represents the maximum CPU utilization percentage for the specified instance for a particular 6-minute time period. The data points aren't returned in chronological order. The following shows the beginning of the example output (the full output includes data points for every 6 minutes of the 24-hour period). ``` { "Datapoints": [ { "Timestamp": "2016-10-19T00:18:00Z", "Maximum": 0.33000000000000002, "Unit": "Percent" }, { "Timestamp": "2016-10-19T03:18:00Z", "Maximum": 99.670000000000002, "Unit": "Percent" }, { "Timestamp": "2016-10-19T07:18:00Z", "Maximum": 0.34000000000000002, "Unit": "Percent" }, ... ], "Label": "CPUUtilization" } ``` # Aggregate statistics across resources You can aggregate the metrics for AWS resources across multiple resources. Metrics are completely separate between Regions, but you can use metric math to aggregate similar metrics across Regions. For more information, see [Using math expressions with CloudWatch metrics](using-metric-math.md). For example, you can aggregate statistics for your EC2 instances that have detailed monitoring enabled. Instances that use basic monitoring aren't included. Therefore, you must enable detailed monitoring (at an additional charge), which provides data in 1-minute periods. For more information, see [Enable or Disable Detailed Monitoring for Your Instances](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-cloudwatch-new.html) in the *Amazon EC2 User Guide*. This example shows you how to get the average CPU usage for your EC2 instances. Because no dimension is specified, CloudWatch returns statistics for all dimensions in the `AWS/EC2` namespace. To get statistics for other metrics, see [AWS services that publish CloudWatch metrics](aws-services-cloudwatch-metrics.md). **Important** This technique for retrieving all dimensions across an AWS namespace doesn't work for custom namespaces that you publish to CloudWatch. With custom namespaces, you must specify the complete set of dimensions that are associated with any given data point to retrieve statistics that include the data point. **To display average CPU utilization for your EC2 instances** 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the navigation pane, choose **Metrics**, **All metrics**. 1. Choose the **EC2** namespace and choose **Across All Instances**. 1. Select the row that contains `CPUUtilization`, which displays a graph for the metric for all your EC2 instances. To change the name of the graph, choose the pencil icon. To change the time range, select one of the predefined values or choose **custom**. ![\[Metrics aggregated across your EC2 instances\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/images/metric_aggregated_instances.png) 1. To change the statistic, choose the **Graphed metrics** tab. Choose the column heading or an individual value and then choose one of the statistics or predefined percentiles, or specify a custom percentile (for example, **p95.45**). 1. To change the period, choose the **Graphed metrics** tab. Choose the column heading or an individual value and then choose a different value. **To get average CPU utilization across your EC2 instances using the AWS CLI** Use the [get-metric-statistics](https://docs.aws.amazon.com/cli/latest/reference/cloudwatch/get-metric-staticstics.html) command as follows: ``` aws cloudwatch get-metric-statistics --namespace AWS/EC2 --metric-name CPUUtilization --statistics "Average" "SampleCount" \ --start-time 2016-10-11T23:18:00 --end-time 2016-10-12T23:18:00 --period 3600 ``` The following is example output: ``` { "Datapoints": [ { "SampleCount": 238.0, "Timestamp": "2016-10-12T07:18:00Z", "Average": 0.038235294117647062, "Unit": "Percent" }, { "SampleCount": 240.0, "Timestamp": "2016-10-12T09:18:00Z", "Average": 0.16670833333333332, "Unit": "Percent" }, { "SampleCount": 238.0, "Timestamp": "2016-10-11T23:18:00Z", "Average": 0.041596638655462197, "Unit": "Percent" }, ... ], "Label": "CPUUtilization" } ``` # Aggregate statistics by Auto Scaling group You can aggregate statistics for the EC2 instances in an Auto Scaling group. Metrics are completely separate between Regions, but you can use CloudWatch metric math to aggregate and transform metrics from multiple Regions. You can also use the cross-account dashboard to perform metric math on metrics from different accounts. This example shows you how to get the total bytes written to disk for one Auto Scaling group. The total is computed for 1-minute periods for a 24-hour interval across all EC2 instances in the specified Auto Scaling group. **To display DiskWriteBytes for the instances in an Auto Scaling group using the console** 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the navigation pane, choose **Metrics**, **All metrics**. 1. Choose the **EC2** namespace and then choose **By Auto Scaling Group**. 1. Select the row for the **DiskWriteBytes** metric and the specific Auto Scaling group, which displays a graph for the metric for the instances in the Auto Scaling group. To change the name of the graph, choose the pencil icon. To change the time range, select one of the predefined values or choose **custom**. ![\[Metrics aggregated across an Auto Scaling group\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/images/metric_aggregated_auto_scaling.png) 1. To change the statistic, choose the **Graphed metrics** tab. Choose the column heading or an individual value and then choose one of the statistics or predefined percentiles, or specify a custom percentile (for example, **p95.45**). 1. To change the period, choose the **Graphed metrics** tab. Choose the column heading or an individual value and then choose a different value. **To get DiskWriteBytes for the instances in an Auto Scaling group using the AWS CLI** Use the [get-metric-statistics](https://docs.aws.amazon.com/cli/latest/reference/cloudwatch/get-metric-statistics.html) command as follows. ``` aws cloudwatch get-metric-statistics --namespace AWS/EC2 --metric-name DiskWriteBytes --dimensions Name=AutoScalingGroupName,Value=my-asg --statistics "Sum" "SampleCount" \ --start-time 2016-10-16T23:18:00 --end-time 2016-10-18T23:18:00 --period 360 ``` The following is example output. ``` { "Datapoints": [ { "SampleCount": 18.0, "Timestamp": "2016-10-19T21:36:00Z", "Sum": 0.0, "Unit": "Bytes" }, { "SampleCount": 5.0, "Timestamp": "2016-10-19T21:42:00Z", "Sum": 0.0, "Unit": "Bytes" } ], "Label": "DiskWriteBytes" } ``` # Aggregate statistics by Amazon Machine Image (AMI) Aggregating statistics by AMI You can aggregate statistics for the EC2 instances that have detailed monitoring enabled. Instances that use basic monitoring aren't included. For more information, see [Enable or Disable Detailed Monitoring for Your Instances](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-cloudwatch-new.html) in the *Amazon EC2 User Guide*. This example shows you how to determine average CPU utilization for all instances that use the specified AMI. The average is over 60-second time intervals for a one-day period. **To display the average CPU utilization by AMI using the console** 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the navigation pane, choose **Metrics**, **All metrics**. 1. Choose the **EC2** namespace and then choose **By Image (AMI) Id**. 1. Select the row for the `CPUUtilization` metric and the specific AMI, which displays a graph for the metric for the specified AMI. To change the name of the graph, choose the pencil icon. To change the time range, select one of the predefined values or choose **custom**. ![\[Metrics aggregated by AMI\]](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/images/metric_aggregated_ami.png) 1. To change the statistic, choose the **Graphed metrics** tab. Choose the column heading or an individual value and then choose one of the statistics or predefined percentiles, or specify a custom percentile (for example, **p95.45**). 1. To change the period, choose the **Graphed metrics** tab. Choose the column heading or an individual value and then choose a different value. **To get the average CPU utilization by AMI using the AWS CLI** Use the [get-metric-statistics](https://docs.aws.amazon.com/cli/latest/reference/cloudwatch/get-metric-statistics.html) command as follows. ``` aws cloudwatch get-metric-statistics --namespace AWS/EC2 --metric-name CPUUtilization \ --dimensions Name=ImageId,Value=ami-3c47a355 --statistics Average \ --start-time 2016-10-10T00:00:00 --end-time 2016-10-11T00:00:00 --period 3600 ``` The operation returns statistics that are one-hour values for the one-day interval. Each value represents an average CPU utilization percentage for EC2 instances running the specified AMI. The following is example output. ``` { "Datapoints": [ { "Timestamp": "2016-10-10T07:00:00Z", "Average": 0.041000000000000009, "Unit": "Percent" }, { "Timestamp": "2016-10-10T14:00:00Z", "Average": 0.079579831932773085, "Unit": "Percent" }, { "Timestamp": "2016-10-10T06:00:00Z", "Average": 0.036000000000000011, "Unit": "Percent" }, ... ], "Label": "CPUUtilization" } ``` # Publish custom metrics You can publish your own metrics to CloudWatch using the OpenTelemetry Protocol (OTLP) or the CloudWatch API. ## Publish metrics using OpenTelemetry (recommended) For new implementations, we recommend using OpenTelemetry to publish custom metrics to CloudWatch. OpenTelemetry provides vendor-agnostic instrumentation with richer descriptive labels and support for metric types including gauge, sum, histogram, and exponential histogram. To publish metrics using OpenTelemetry, configure your OpenTelemetry Collector or SDK to send metrics to the CloudWatch OTLP endpoint. Metrics sent through OTLP are available to query using the Prometheus Query Language (PromQL) in CloudWatch Query Studio. You can also set PromQL-based CloudWatch Alarms on these metrics. Key differences from the CloudWatch API approach: | Feature | OpenTelemetry (OTLP) | CloudWatch API (PutMetricData) | | --- | --- | --- | | Labels per metric | Up to 150 | Up to 30 dimensions | | Metric types | Gauge, sum, histogram, exponential histogram | Single values, statistic sets | | Granularity | Ingested at the resolution sent | 1 second (high resolution) or 1 minute (standard) | | Query language | PromQL | GetMetricStatistics, Metrics Insights | | Alarms | PromQL-based CloudWatch Alarms | Standard CloudWatch Alarms | | Instrumentation | Vendor-agnostic OpenTelemetry SDKs and Collectors | AWS SDK or CLI | To get started, see [Sending metrics using OpenTelemetry](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-OpenTelemetry-Sections.html). ## Publish metrics using the CloudWatch API You can also publish custom metrics using the PutMetricData API or the AWS CLI. This approach uses CloudWatch namespaces, metric names, and dimensions. **Topics** + [ ## Publish metrics using OpenTelemetry (recommended) ](#publishMetricsOTLP) + [ ## Publish metrics using the CloudWatch API ](#publishMetricsCWAPI) + [ ## High-resolution metrics ](#high-resolution-metrics) + [ ## Use dimensions ](#usingDimensions) + [ ## Publish single data points ](#publishingDataPoints) + [ ## Publish statistic sets ](#publishingDataPoints1) + [ ## Publish the value zero ](#publishingZero) + [ ## Stop publishing metrics ](#CloudWatch-Stop-Publishing-Metrics) ## High-resolution metrics Each metric is one of the following: + Standard resolution, with data having a one-minute granularity + High resolution, with data at a granularity of one second Metrics produced by AWS services are standard resolution by default. When you publish a custom metric, you can define it as either standard resolution or high resolution. When you publish a high-resolution metric, CloudWatch stores it with a resolution of 1 second, and you can read and retrieve it with a period of 1 second, 5 seconds, 10 seconds, 30 seconds, or any multiple of 60 seconds. High-resolution metrics can give you more immediate insight into your application's sub-minute activity. Keep in mind that every `PutMetricData` call for a custom metric is charged, so calling `PutMetricData` more often on a high-resolution metric can lead to higher charges. For more information about CloudWatch pricing, see [Amazon CloudWatch Pricing](https://aws.amazon.com/cloudwatch/pricing/). If you set an alarm on a high-resolution metric, you can specify a high-resolution alarm with a period of 10 seconds or 30 seconds, or you can set a regular alarm with a period of any multiple of 60 seconds. There is a higher charge for high-resolution alarms with a period of 10 or 30 seconds. ## Use dimensions In custom metrics, the `--dimensions` parameter is common. A dimension further clarifies what the metric is and what data it stores. You can have up to 30 dimensions assigned to one metric, and each dimension is defined by a name and value pair. How you specify a dimension is different when you use different commands. With [put-metric-data](https://docs.aws.amazon.com/cli/latest/reference/cloudwatch/put-metric-data.html), you specify each dimension as *MyName*=*MyValue*, and with [get-metric-statistics](https://docs.aws.amazon.com/cli/latest/reference/cloudwatch/get-metric-statistics.html) or [put-metric-alarm](https://docs.aws.amazon.com/cli/latest/reference/cloudwatch/put-metric-alarm.html) you use the format `Name=`*MyName*, `Value=`*MyValue*. For example, the following command publishes a `Buffers` metric with two dimensions named `InstanceId` and `InstanceType`. ``` aws cloudwatch put-metric-data --metric-name Buffers --namespace MyNameSpace --unit Bytes --value 231434333 --dimensions InstanceId=1-23456789,InstanceType=m1.small ``` This command retrieves statistics for that same metric. Separate the Name and Value parts of a single dimension with commas, but if you have multiple dimensions, use a space between one dimension and the next. ``` aws cloudwatch get-metric-statistics --metric-name Buffers --namespace MyNameSpace --dimensions Name=InstanceId,Value=1-23456789 Name=InstanceType,Value=m1.small --start-time 2016-10-15T04:00:00Z --end-time 2016-10-19T07:00:00Z --statistics Average --period 60 ``` If a single metric includes multiple dimensions, you must specify a value for every defined dimension when you use [get-metric-statistics](https://docs.aws.amazon.com/cli/latest/reference/cloudwatch/get-metric-statistics.html). For example, the Amazon S3 metric `BucketSizeBytes` includes the dimensions `BucketName` and `StorageType`, so you must specify both dimensions with [get-metric-statistics](https://docs.aws.amazon.com/cli/latest/reference/cloudwatch/get-metric-statistics.html). ``` aws cloudwatch get-metric-statistics --metric-name BucketSizeBytes --start-time 2017-01-23T14:23:00Z --end-time 2017-01-26T19:30:00Z --period 3600 --namespace AWS/S3 --statistics Maximum --dimensions Name=BucketName,Value=amzn-s3-demo-bucket Name=StorageType,Value=StandardStorage --output table ``` To see what dimensions are defined for a metric, use the [list-metrics](https://docs.aws.amazon.com/cli/latest/reference/cloudwatch/list-metrics.html) command. ## Publish single data points To publish a single data point for a new or existing metric, use the [put-metric-data](https://docs.aws.amazon.com/cli/latest/reference/cloudwatch/put-metric-data.html) command with one value and time stamp. For example, the following actions each publish one data point. ``` aws cloudwatch put-metric-data --metric-name PageViewCount --namespace MyService --value 2 --timestamp 2016-10-20T12:00:00.000Z aws cloudwatch put-metric-data --metric-name PageViewCount --namespace MyService --value 4 --timestamp 2016-10-20T12:00:01.000Z aws cloudwatch put-metric-data --metric-name PageViewCount --namespace MyService --value 5 --timestamp 2016-10-20T12:00:02.000Z ``` If you call this command with a new metric name, CloudWatch creates a metric for you. Otherwise, CloudWatch associates your data with the existing metric that you specified. **Note** When you create a metric, it can take up to 2 minutes before you can retrieve statistics for the new metric using the [get-metric-statistics](https://docs.aws.amazon.com/cli/latest/reference/cloudwatch/get-metric-statistics.html) command. However, it can take up to 15 minutes before the new metric appears in the list of metrics retrieved using the [list-metrics](https://docs.aws.amazon.com/cli/latest/reference/cloudwatch/list-metrics.html) command. Although you can publish data points with time stamps as granular as one-thousandth of a second, CloudWatch aggregates the data to a minimum granularity of 1 second. CloudWatch records the average (sum of all items divided by number of items) of the values received for each period, as well as the number of samples, maximum value, and minimum value for the same time period. For example, the `PageViewCount` metric from the previous examples contains three data points with time stamps just seconds apart. If you have your period set to 1 minute, CloudWatch aggregates the three data points because they all have time stamps within a 1-minute period. You can use the **get-metric-statistics** command to retrieve statistics based on the data points that you published. ``` aws cloudwatch get-metric-statistics --namespace MyService --metric-name PageViewCount \ --statistics "Sum" "Maximum" "Minimum" "Average" "SampleCount" \ --start-time 2016-10-20T12:00:00.000Z --end-time 2016-10-20T12:05:00.000Z --period 60 ``` The following is example output. ``` { "Datapoints": [ { "SampleCount": 3.0, "Timestamp": "2016-10-20T12:00:00Z", "Average": 3.6666666666666665, "Maximum": 5.0, "Minimum": 2.0, "Sum": 11.0, "Unit": "None" } ], "Label": "PageViewCount" } ``` ## Publish statistic sets You can aggregate your data before you publish to CloudWatch. When you have multiple data points per minute, aggregating data minimizes the number of calls to **put-metric-data**. For example, instead of calling **put-metric-data** multiple times for three data points that are within 3 seconds of each other, you can aggregate the data into a statistic set that you publish with one call, using the `--statistic-values` parameter. ``` aws cloudwatch put-metric-data --metric-name PageViewCount --namespace MyService --statistic-values Sum=11,Minimum=2,Maximum=5,SampleCount=3 --timestamp 2016-10-14T12:00:00.000Z ``` CloudWatch needs raw data points to calculate percentiles. If you publish data using a statistic set instead, you can't retrieve percentile statistics for this data unless one of the following conditions is true: + The `SampleCount` of the statistic set is 1 + The `Minimum` and the `Maximum` of the statistic set are equal ## Publish the value zero When your data is more sporadic and you have periods that have no associated data, you can choose to publish the value zero (`0`) for that period or no value at all. If you use periodic calls to `PutMetricData` to monitor the health of your application, you might want to publish zero instead of no value. For example, you can set a CloudWatch alarm to notify you if your application fails to publish metrics every five minutes. You want such an application to publish zeros for periods with no associated data. You might also publish zeros if you want to track the total number of data points or if you want statistics such as minimum and average to include data points with the value 0. ## Stop publishing metrics To stop publishing custom metrics to CloudWatch, change your application's or service's code to stop using **PutMetricData**. CloudWatch doesn't pull metrics from applications, it only receives what is pushed to it, so to stop publishing your metrics you must stop them at the source.