Troubleshoot Amazon Managed Service for Prometheus errors
Use the following sections to help troubleshoot issues with Amazon Managed Service for Prometheus.
Topics
429 or limit exceeded errors
If you see a 429 error similar to the following example, your requests have exceeded Amazon Managed Service for Prometheus ingestion quotas.
ts=2020-10-29T15:34:41.845Z caller=dedupe.go:112 component=remote level=error remote_name=e13b0c url=http://iamproxy-external.prometheus.uswest2-prod.eks:9090/workspaces/workspace_id/api/v1/remote_write msg="non-recoverable error" count=500 err="server returned HTTP status 429 Too Many Requests: ingestion rate limit (6666.666666666667) exceeded while adding 499 samples and 0 metadata
If you see a 429 error similar to the following example, your requests have exceeded the Amazon Managed Service for Prometheus quota for the number of active metrics in a workspace.
ts=2020-11-05T12:40:33.375Z caller=dedupe.go:112 component=remote level=error remote_name=aps url=http://iamproxy-external.prometheus.uswest2-prod.eks:9090/workspaces/workspace_id/api/v1/remote_write msg="non-recoverable error" count=500 err="server returned HTTP status 429 Too Many Requests: user=accountid_workspace_id: per-user series limit (local limit: 0 global limit: 3000000 actual local limit: 500000) exceeded
If you see a 429 error similar to the following example, your requests have exceeded
the Amazon Managed Service for Prometheus quota for the rate (transactions per second) that you can send data to
your workspace using the RemoteWrite Prometheus compatible API.
ts=2024-03-26T16:50:21.780708811Z caller=dedupe.go:112 component=remote level=error remote_name=ab123c url=https://aps-workspaces.us-east-1.amazonaws.com/workspaces/workspace_id/api/v1/remote_write msg="non-recoverable error" count=1000 exemplarCount=0 err="server returned HTTP status 429 Too Many Requests: {\"message\":\"Rate exceeded\"}"
If you see a 400 error similar to the following example, your requests have exceeded Amazon Managed Service for Prometheus quota for active time series. For details about how active time series quotas are handled, see Active series default quotas.
ts=2024-03-26T16:50:21.780708811Z caller=push.go:53 level=warn url=https://aps-workspaces.us-east-1.amazonaws.com/workspaces/workspace_id/api/v1/remote_write msg="non-recoverable error" count=500 exemplarCount=0 err="server returned HTTP status 400 Bad Request: maxFailure (quorum) on a given error family, rpc error: code = Code(400) desc = addr=10.1.41.23:9095 state=ACTIVE zone=us-east-1a, rpc error: code = Code(400) desc = user=accountid_workspace_id: per-user series limit of 10000000 exceeded, Capacity from 2,000,000 to 10,000,000 is automatically adjusted based on the last 30 min of usage. If throttled above 10,000,000 or in case of incoming surges, please contact administrator to raise it. (local limit: 0 global limit: 10000000 actual local limit: 92879)"
For more information about Amazon Managed Service for Prometheus service quotas and about how to request increases, see Amazon Managed Service for Prometheus service quotas
I see duplicate samples
If you are using a high-availability Prometheus group, you need to use external labels on your Prometheus instances to set up deduplication. For more information, see Deduplicating high availability metrics sent to Amazon Managed Service for Prometheus.
Other issues around duplicated data are discussed in the next section.
I see errors about sample timestamps
Amazon Managed Service for Prometheus ingests data in order, and expects each sample to have a timestamp later than the previous sample.
If your data does not arrive in order, you can see errors about out-of-order
samples, duplicate sample for timestamp, or samples with
different value but same timestamp. These issues are typically caused by
incorrect setup of the client that is sending data to Amazon Managed Service for Prometheus. If you are using a
Prometheus client running in agent mode, check the configuration for rules with
duplicate series name, or duplicated targets. If your metrics provide the timestamp
directly, check that they are not out of order.
You can configure an out-of-order time window for your workspace to accept out-of-order samples. For more information, see Configure your workspace.
For more details about how this works, or ways to check your setup, see the blog post Understanding Duplicate Samples and Out-of-order Timestamp Errors in Prometheus
I see an error message related to a limit
Note
Amazon Managed Service for Prometheus provides CloudWatch usage metrics to monitor Prometheus resource usage. Using the CloudWatch usage metrics alarm feature, you can monitor Prometheus resources and usage to prevent limit errors.
If you see one of the following error messages, you can request an increase in one of the Amazon Managed Service for Prometheus quotas to solve the issue. For more information, see Amazon Managed Service for Prometheus service quotas.
-
per-user series limit of<value>exceeded, please contact administrator to raise it -
per-metric series limit of<value>exceeded, please contact administrator to raise it -
ingestion rate limit (...) exceeded -
series has too many labels (...) series: '%s' -
the query time range exceeds the limit (query length: xxx, limit: yyy) -
the query hit the max number of chunks limit while fetching chunks from ingesters -
Limit exceeded. Maximum workspaces per account.
Your local Prometheus server output exceeds the limit.
Amazon Managed Service for Prometheus has service quotas for the amount of data that a workspace can receive from Prometheus servers. To find the amount of data that your Prometheus server is sending to Amazon Managed Service for Prometheus, you can run the following queries on your Prometheus server. If you find that your Prometheus output is exceeding a Amazon Managed Service for Prometheus limit, you can request an increase of the corresponding service quota. For more information, see Amazon Managed Service for Prometheus service quotas.
| Type of data | Query to use |
|---|---|
|
Current active series |
|
|
Current ingestion rate |
|
|
Most-to-least list of active series per metric name |
|
|
Number of labels per metric series |
|
Some of my data isn't appearing
Data that is sent to Amazon Managed Service for Prometheus can be discarded for various reasons. The following table shows reasons that data might be discarded rather than being ingested.
You can track the amount and reasons that data is discarded using Amazon CloudWatch. For more information, see Use CloudWatch metrics to monitor Amazon Managed Service for Prometheus resources.
|
Reason |
Meaning |
|---|---|
|
greater_than_max_sample_age |
Discarding log lines which are older than the current time |
|
new-value-for-timestamp |
Duplicate samples are sent with the same timestamp as the previous sample but with different values. |
|
per_metric_series_limit |
User has hit the active series per metric limit |
|
per_user_series_limit |
User has hit the total number of active series limit |
|
rate_limited |
Ingestion rate limited |
|
sample-out-of-order |
Samples are sent out of order and cannot be processed. This reason is only emitted when the out-of-order time window is set to 0. |
|
sample-too-old |
Sample is older than the configured out-of-order time window and cannot be processed. |
|
out-of-order-rate-limit |
Out-of-order ingestion rate limit was reached and the sample cannot be processed. For more information, see Amazon Managed Service for Prometheus service quotas. |
|
label_value_too_long |
Label value is longer than allowed character limit |
|
max_label_names_per_series |
User has hit the label names per metric |
|
missing_metric_name |
Metric name is not provided |
|
metric_name_invalid |
Invalid metric name provided |
|
label_invalid |
Invalid label provided |
|
duplicate_label_names |
Duplicate label names provided |
|
native_histogram_sample_size_bytes_exceeded |
Native histogram sample exceeds the maximum allowed sample size. Reduce the number of buckets or spans in the histogram on the client side. |
|
native_histogram_invalid_schema |
Native histogram has an invalid schema value. Valid schemas range
from -4 to 8. Check your Prometheus client configuration for the
|
|
native_histogram_invalid |
Native histogram fails validation (for example, negative bucket counts, mismatched bucket counts, or malformed bucket spans). This typically indicates a bug in the instrumentation library. |
|
native_histogram_buckets_exceeded |
Native histogram exceeds the maximum bucket count limit and
resolution cannot be automatically reduced. Reduce the number of
buckets on the client side using
|
|
native_histogram_rate_limited |
Native histogram sample was rejected because the native histogram ingestion rate limit was reached. Reduce the number of native histogram series or the scrape frequency, or request a limit increase. |
|
per_user_native_histogram_series_limit |
User has hit the native histogram active series limit per workspace. Reduce the number of native histogram series or request a limit increase. |
HTTP 422 error: query timeout
If your query runs for more than 1 minute, Amazon Managed Service for Prometheus returns an HTTP 422 status code. The following example shows this error:
{"status":"error","errorType":"execution","error":"query timed out - consider simplifying your query"}
To reduce evaluation time, use one or more of the following approaches:
-
Specify a shorter time window to evaluate fewer samples.
-
Add label matchers to reduce the number of series that your query evaluates.
-
Use recording rules to precompute expensive aggregations that you query frequently.
-
Avoid regex matchers on high-cardinality labels (labels with many unique values).
For more information, see Query your Prometheus metrics.
