PromQL querying

When you ingest OpenTelemetry metrics into CloudWatch via OpenTelemetry Protocol (OTLP), the hierarchical OTLP data model is flattened into PromQL-compatible labels. This section describes the label structure, the PromQL syntax for querying these labels, and the UTF-8 support in PromQL.

When using PromQL in CloudWatch, the @ prefix convention distinguishes OTLP-scoped labels from standard Prometheus labels. Fields within each scope use a double-@ prefix (for example, @resource.@schema_url), while attributes use a single-@ scope prefix, for example, @resource.service.name. Datapoint attributes also support bare (un-prefixed) access for backward compatibility with standard PromQL queries, for example, {"http.server.active_requests"} and {"@datapoint.@name"="http.server.active_requests"} are equivalent.

A PromQL expression is enclosed in curly braces, specifying the metric name and an optional set of label matchers. The following example selects all time series for the http.server.active_requests metric:

{"http.server.active_requests"}

The following example selects all time series for the metric http.server.active_requests where the OpenTelemetry resource attribute service.name equals myservice:

{"http.server.active_requests", "@resource.service.name"="myservice"}

You can combine multiple label matchers in a single query. The following example selects all time series for the http.server.active_requests metric where the OpenTelemetry resource attribute service.name equals myservice across all US regions:

{"http.server.active_requests",
 "@resource.service.name"="myservice",
 "@aws.region"=~"us-.*"}

The following example shows a range query. It calculates the average value of all datapoints within a specified time range for each time series:

avg_over_time(
  {"http.server.active_requests",
   "@resource.service.name"="myservice"}[5m]
)

The following table summarizes the prefix conventions for each OTLP scope:

Querying vended AWS metrics with PromQL

To be able to query vended AWS metrics in PromQL, you first need to enable OTel enrichment of vended metrics. See: Enabling vended metrics in PromQL.

After you enable OTel enrichment, vended AWS metrics become queryable via PromQL with additional labels. The metric name is the same as the original CloudWatch metric name, and the original CloudWatch dimensions are available as datapoint attributes. The following labels are available (the example below is for an EC2 instance):

The following example selects Invocations for a specific Lambda function:

histogram_sum({Invocations, FunctionName="my-api-handler"})

The following example selects Lambda Errors for all functions tagged with a specific team:

histogram_sum(
  {Errors, "@instrumentation.@name"="cloudwatch.aws/lambda", "@aws.tag.Team"="backend"}
)

The following example computes the total Lambda Invocations grouped by team:

sum by ("@aws.tag.Team")(
    {Invocations, "@instrumentation.@name"="cloudwatch.aws/lambda"}
)

The following example selects all time series for the EC2 CPUUtilization metric. The usage of "@instrumentation.@name"="cloudwatch.aws/ec2" is to exclusively match CPUUtilization from EC2 and not from other AWS services such as Amazon Relational Database Service:

histogram_avg({CPUUtilization, "@instrumentation.@name"="cloudwatch.aws/ec2"})