# Log anomaly detection You can detect anomalies in your log data in two ways: by creating a *log anomaly detector* for continuous monitoring, or by using the [`anomaly detection`](CWL_QuerySyntax-Anomaly.md) command in CloudWatch Logs Insights queries for on-demand analysis. A log anomaly detector scans the log events ingested into a log group and finds anomalies in the log data automatically. Anomaly detection uses machine-learning and pattern recognition to establish baselines of typical log content. For on-demand analysis, you can use the `anomaly detection` command in CloudWatch Logs Insights queries to identify unusual patterns in time-series data. For more information about query-based anomaly detection, see [Using anomaly detection in CloudWatch Logs Insights](LogsAnomalyDetection-Insights.md). After you create an anomaly detector for a log group, it trains using the past two weeks of log events in the log group for training. The training period can take up to 15 minutes. After the training is complete, it begins to analyze incoming logs to identify anomalies, and the anomalies are displayed in the CloudWatch Logs console for you to examine. CloudWatch Logs pattern recognition extracts log patterns by identifying static and dynamic content in your logs. Patterns are useful for analyzing large log sets because a large number of log events can often be compressed into a few patterns. For example, see the following sample of three log events. ``` 2023-01-01 19:00:01 [INFO] Calling DynamoDB to store for ResourceID: 12342342k124-12345 2023-01-01 19:00:02 [INFO] Calling DynamoDB to store for ResourceID: 324892398123-1234R 2023-01-01 19:00:03 [INFO] Calling DynamoDB to store for ResourceID: 3ff231242342-12345 ``` In the previous sample, all three log events follow one pattern: ``` [INFO] Calling DynamoDB to store for resource id ``` Fields within a pattern are called *tokens*. Fields that vary within a pattern, such as a request ID or timestamp, are referred to as *dynamic tokens*. Each different value found for a dynamic token is called a *token value*. If CloudWatch Logs can infer the type of data that a dynamic token represents, it displays the token as ``. The *string* is a description of the type of data that the token represents. The *number* shows where in the pattern this token appears, compared to the other dynamic tokens. CloudWatch Logs assigns the string part of the name based on analyzing the content of the log events that contain it. If CloudWatch Logs can't infer the type of data that a dynamic token represents, it displays the token as , and *number* indicates where in the pattern this token appears, compared to the other dynamic tokens. Common examples of dynamic tokens include error codes, IP addresses, timestamps, and request IDs. Logs anomaly detection uses these patterns to find anomalies. After the anomaly detector model training period, logs are evaluated against known trends. The anomaly detector flags significant fluctuations as anomalies. This chapter describes how to enable anomaly detection, view anomalies, create alarms for log anomaly detectors, and metrics that log anomaly detectors publish. It also describes how to encrypt anomaly detector and its results with AWS Key Management Service. Creating log anomaly detectors doesn't incur charges. ## Severity and priority of anomalies and patterns Each anomaly found by a log anomaly detector is assigned a *priority*. Each pattern found is assigned a *severity*. + *Priority* is automatically computed, and is based on both the severity level of the pattern and the amount of deviation from expected values. For example, if a certain token value suddenly increases by 500%, that anomaly might be designated as `HIGH` priority even if its severity is `NONE`. + *Severity* is based only on keywords found in the patterns such as `FATAL`, `ERROR`, and `WARN`. If none of these keywords are found, the severity of a pattern is marked as `NONE`. ## Anomaly visibility time When you create an anomaly detector, you specify the maximum anomaly visibility period for it. This is the number of days that the anomaly is displayed in the console and is returned by the [ListAnomalies](https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_ListAnomalies.html) API operation. After this time period has elapsed for an anomaly, if it continues to happen, it's automatically accepted as regular behavior and the anomaly detector model stops flagging it as an anomaly. If you don't adjust the visibility time when you create an anomaly detector, 21 days is used as the default. ## Suppressing an anomaly After an anomaly has been found, you can choose to suppress it temporarily or permanently. Suppressing an anomaly causes the anomaly detector to stop flagging this occurrence as an anomaly for the amount of time that you specify. When you suppress an anomaly, you can choose to suppress only that specific anomaly, or suppress all anomalies related to the pattern that the anomaly was found in. You can still view suppressed anomalies in the console. You can also choose to stop suppressing them. ## Frequently asked questions **Does AWS use my data to train machine-learning algorithms for AWS use or for other customers? ** No. The anomaly detection model created by the training is based on the log events in a log group and is used only within that log group and that AWS account. **What types of log events work well with anomaly detection?** **Log anomaly detection is well-suited for:** Application logs and other types of logs where most log entries fit typical patterns. Log groups with events that contain a log level or severity keywords such as **INFO**, **ERROR**, and **DEBUG** are especially well-suited to log anomaly detection. **Log anomaly detection is not suited for:** Log events with extremely long JSON structures, such as CloudTrail Logs. Pattern analysis analyzes only up to the first 1500 characters of a log line, so any characters beyond that limit are skipped. Audit or access logs, such as VPC flow logs, will also have less success with anomaly detection. Anomaly detection is meant to find application issues, so it might not be well-suited for network or access anomalies. To help you determine whether an anomaly detector is suited to a certain log group, use CloudWatch Logs pattern analysis to find the number of patterns in the log events in the group. If the number of patterns is no more than about 300, anomaly detection might work well. For more information about pattern analysis, see [Pattern analysis](CWL_AnalyzeLogData_Patterns.md). **What gets flagged as an anomaly?** The following occurrences can cause a log event to be flagged as an anomaly: + A log event with a pattern not seen before in the log group. + A significant variation to a known pattern. + A new value for a dynamic token that has a discrete set of usual values. + A large change in the number of occurrences of a value for a dynamic token. While all the preceding items might be flagged as anomalies, they don't all mean that the application is performing poorly. For example, a higher-than-usual number of `200` success values might be flagged as an anomaly. In cases like this, you might consider suppressing these anomalies that don't indicate problems. **What happens with sensitive data that is being masked?** Any parts of log events that are masked as sensitive data are not scanned for anomalies. For more information about masking sensitive data, see [Help protect sensitive log data with masking](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/mask-sensitive-log-data.html). # Using anomaly detection in CloudWatch Logs Insights In addition to creating log anomaly detectors for continuous monitoring, you can also use the `anomaly` command in CloudWatch Logs Insights queries to identify unusual patterns in your log data on-demand. This command extends the existing `pattern` functionality and uses machine learning to detect five types of anomalies including pattern frequency changes, new patterns, and token variations. The `anomaly` command is particularly useful for: + Ad-hoc analysis of historical log data to identify unusual patterns + Investigating specific time periods for anomalous behavior + Monitoring applications like Lambda functions for execution issues For more information about using the `anomaly` command in your queries, see [anomaly](CWL_QuerySyntax-Anomaly.md). This query-based anomaly detection complements the continuous anomaly detectors described in the following sections, giving you both real-time monitoring and on-demand analysis capabilities. # Enable anomaly detection on a log group Use the following steps to use the CloudWatch console to create a log anomaly detector that scans a log group for anomalies. You can also create anomaly detectors programmatically. For more information, see [CreateLogAnomalyDetector](https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_CreateLogAnomalyDetector.html). **To create a log anomaly detector** 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. Choose **Logs**, **Log Anomalies**. 1. Choose **Create anomaly detector**. 1. Select the log group to create this anomaly detector for. 1. Enter a name for the detector in **Anomaly detector name**. 1. (Optional) Change the **Evaluation frequency** from the default of 5 minutes. Set this value according to the frequency that the log group receives new logs. For example, if the log group receives new log events in batches every 10 minutes, then setting the evaluation frequency to 15 minutes might be appropriate. 1. (Optional) To configure the anomaly detector to look for anomalies only in log events that contain certain words or strings, choose **Filter patterns**. Then, enter a pattern in **Anomaly detection filter pattern**. For more information about pattern syntax, [Filter pattern syntax for metric filters, subscription filters, filter log events, and Live Tail](FilterAndPatternSyntax.md). (Optional) To test your filter pattern, enter some log messages into **Log event messages** and then choose **Test Pattern**. 1. (Optional) To change the anomaly visibility period from the default or to associate an AWS KMS key with this anomaly detector, choose **Advanced configuration**. 1. To change the anomaly visibility period from the default, enter a new value in **Maximum anomaly visibility period (days)**. 1. To associate an AWS KMS key with this anomaly detector, enter the ARN in **KMS key ARN**. If you assign a key, the anomaly information found by this detector is encrypted at rest with the key. Users must have permissions for this key and for the anomaly detector to retrieve information about the anomalies that it finds. You must also ensure that the CloudWatch Logs service principal has permission to use the key. For more information, see [Encrypt an anomaly detector and its results with AWS KMS](LogsAnomalyDetection-KMS.md). 1. Choose **Enable Anomaly Detection**. The anomaly detector is created and starts training its model, based on the log events the log group is ingesting. After about 15 minutes, anomaly detection is active and begins to find and surface anomalies. # View anomalies that have been found After you create one or more log anomaly detectors, you can use the CloudWatch console to view the anomalies that they have found. You can view anomalies programmatically. For more information, see [ListAnomalies](https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_ListAnomalies.html). **To view the anomalies found by all of your log anomaly detectors** 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. Choose **Logs**, **Log Anomalies**. The **Logs anomalies** table appears. The number at the top next to **Log anomalies** displays how many log anomalies are listed in the table. Each row in the table displays the following information: + The **Anomaly** column displays a short summary of the anomaly. These summaries are generated by CloudWatch Logs. + The **Priority** of the anomaly. Priority is automatically computed based on the amount of change in the log events, key words such as `Exception` occurring in a log event, and more. + The **Log pattern** that the anomaly is based on. For more information about patterns, see [Log anomaly detection](LogsAnomalyDetection.md). + **Anomaly log trend** displays a histogram depicting the volume of logs matching the pattern. + **Last detection time** displays the most recent time that this anomaly was found. + **First detection time** displays the first time that this anomaly was found. + **Anomaly detector** displays the name of the log group containing the log events related to this anomaly. You can choose this name to see the log group details page. 1. To further inspect one anomaly, choose the radio button in its row. The **Pattern inspect** pane appears and displays the following: + The **Pattern** that this anomaly is based on. Select a token within the pattern to analyze that token's values. + A histogram showing the number of occurrences of the anomaly over the queried time range. + The **Log samples** tab displays a few of the log events that are part of the anomaly. + The **Token Values** tab displays the values of the selected dynamic token, if you have selected one. **Note** A maximum of 10 token values is captured for each token. Token counts might not be precise. CloudWatch Logs uses a probabilistic counter to generate the token count, not the absolute value. 1. To suppress an anomaly, choose the radio button in its row and then do the following: 1. Choose **Actions**, **Suppress Anomaly**. 1. Then specify how long you want the anomaly to be suppressed. 1. To suppress all anomalies related to this pattern, select **Suppress Pattern**. 1. Choose **Suppress anomaly**. **To view the anomalies found in a single log group** 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. Choose **Logs**, **Log groups**. 1. Choose the name of a log group, and then choose the **Anomaly detection** tab. The **Anomaly detection** table appears. The number at the top next to **Log anomalies** displays how many log anomalies are listed in the table. Each row in the table displays the following information: + The **Anomaly** column displays a short summary of the anomaly. These summaries are generated by CloudWatch Logs. + The **Priority** of the anomaly. Priority is automatically computed based on the amount of change in the log events, key words such as `Exception` occurring in a log event, and more. + The **Log pattern** that the anomaly is based on. For more information about patterns, see [Log anomaly detection](LogsAnomalyDetection.md). + **Anomaly log trend** displays a histogram depicting the volume of logs matching the pattern. + **Last detection time** displays the most recent time that this anomaly was found. + **First detection time** displays the first time that this anomaly was found. 1. To further inspect one anomaly, choose the radio button in its row. The **Pattern inspect** pane appears and displays the following: + The **Pattern** that this anomaly is based on. Select a token within the pattern to analyze that token's values. + A histogram showing the number of occurrences of the anomaly over the queried time range. + The **Log samples** tab displays a few of the log events that are part of the anomaly. + The **Token Values** tab displays the values of the selected dynamic token, if you have selected one. **Note** A maximum of 10 token values is captured for each token. Token counts might not be precise. CloudWatch Logs uses a probabilistic counter to generate the token count, not the absolute value. 1. To suppress an anomaly, choose the radio button in its row and then do the following: 1. Choose **Actions**, **Suppress Anomaly**. 1. Then specify how long you want the anomaly to be suppressed. 1. To suppress all anomalies related to this pattern, select **Suppress Pattern**. 1. Choose **Suppress anomaly**. # Create alarms on log anomaly detectors You can create an alarm for a log anomaly detector in a log group. You can specify for the alarm to go into `ALARM` state when a specified number of anomalies are found in the log group during a specified period of time. You can also use filters so that only anomalies of specified priorities are counted by the alarm. **To create an alarm for a log anomaly detector** 1. Open the CloudWatch console at [https://console.aws.amazon.com/cloudwatch/](https://console.aws.amazon.com/cloudwatch/). 1. In the navigation pane, choose **Logs**, **Log Anomalies**. The table of log anomaly detectors appears. 1. Choose the radio button for the anomaly detector that you want to set the alarm for, and choose **Create alarm**. The CloudWatch alarm creation wizard appears. The **LogAnomalyDetector** field displays the name of the anomaly detector that you chose. The **Metric name** field displays **AnomalyCount**. 1. (Optional) To filter this alarm for anomaly priority, do one of the following: + To have the alarm count only high-priority anomalies, enter **HIGH** for **LogAnomalyPriority**. + To have the alarm count only high- and medium-priority anomalies, enter **MEDIUM** for **LogAnomalyPriority**. For more information about priority levels, see [Severity and priority of anomalies and patterns](LogsAnomalyDetection.md#LogsAnomalyDetection-Severity-Priority). 1. Choose to use a static or metric anomaly detection threshold for the alarm. This selection determines how the alarm threshold is set. A **Static** threshold means that the alarm threshold is a static, constant number that you choose. An **Anomaly detection** threshold means that CloudWatch determines a range of usual values, and the alarm triggers if the actual count crosses the threshold of this band. You don't have to choose **Anomaly detection** for a log anomaly detection alarm. For more information about metric anomaly detection, see [ Using CloudWatch anomaly detection](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Anomaly_Detection.html). 1. For **Whenever *your-metric-name* is . . .**, choose **Greater**, **Greater/Equal**, **Lower/Equal**, or **Lower**. Then for **than . . .**, specify a number for your threshold value. The alarm goes into `ALARM` state if the anomaly detector finds more than this number of alarms during a time specified by **Period**. 1. Choose **Additional configuration**. For **Datapoints to alarm**, specify how many evaluation periods (data points) must be in the `ALARM` state to trigger the alarm. If the two values here match, you create an alarm that goes to `ALARM` state if that many consecutive periods are breaching. To create an M out of N alarm, specify a number for the first value that is lower than the number for the second value. For more information, see [Evaluating an alarm](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html#alarm-evaluation). 1. For **Missing data treatment**, choose how the alarm behaves when some data points are missing. For more information, see [ Configuring how CloudWatch alarms treat missing data](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html#alarms-and-missing-data). 1. Choose **Next**. 1. For **Notification**, choose **Add notification**, and then specify an Amazon SNS topic to notify when your alarm transitions to the `ALARM`, `OK`, or `INSUFFICIENT_DATA` state. 1. (Optional) To send multiple notifications for the same alarm state or for different alarm states, choose **Add notification**. **Note** We recommend that you set the alarm to take actions when it goes into **Insufficient data** state in addition to when it goes into **Alarm** state. This is because many issues with the Lambda function that connects to the data source can cause the alarm to transition to **Insufficient data**. 1. (Optional) To not send Amazon SNS notifications, choose **Remove**. 1. (Optional) If you want your alarm to perform actions for Amazon EC2 Auto Scaling, Amazon EC2, tickets, or AWS Systems Manager, choose the appropriate button, and specify the alarm state and action. **Note** Your alarm can perform Systems Manager actions only when it's in the `ALARM` state. For information about Systems Manager actions, see [Configuring CloudWatch to create OpsItems](https://docs.aws.amazon.com/systems-manager/latest/userguide/OpsCenter-create-OpsItems-from-CloudWatch-Alarms.html) and [Incident creation](https://docs.aws.amazon.com/incident-manager/latest/userguide/incident-creation.html). 1. Choose **Next**. 1. Under **Name and description**, enter a name and description for your alarm, and choose **Next**. The name must contain only UTF-8 characters, and can't contain ASCII control characters. The description can include markdown formatting, which is displayed only in the alarm **Details** tab in the CloudWatch console. The markdown can be useful to add links to runbooks or other internal resources. **Tip** The alarm name must contain only UTF-8 characters. It can't contain ASCII control characters. 1. Under **Preview and create**, confirm that your alarm's information and conditions are correct, and choose **Create alarm**. # Metrics published by log anomaly detectors CloudWatch Logs publishes the **AnomalyCount** metric to CloudWatch metrics. This metric is published to the `AWS/Logs` namespace. The **AnomalyCount** metric is published with the following dimensions: + **LogAnomalyDetector**– The name of the anomaly detector + **LogAnomalyPriority**– The priority level of the anomaly # Encrypt an anomaly detector and its results with AWS KMS Anomaly detector data is always encrypted in CloudWatch Logs. By default, CloudWatch Logs uses server-side encryption for the data at rest. As an alternative, you can use AWS Key Management Service for this encryption. If you do, the encryption is done using an AWS KMS key. Encryption using AWS KMS is enabled at the anomaly detector level, by associating a KMS key with an anomaly detector. **Important** CloudWatch Logs supports only symmetric KMS keys. Do not use an asymmetric key to encrypt the data in your log groups. For more information, see [Using Symmetric and Asymmetric Keys](https://docs.aws.amazon.com/kms/latest/developerguide/symmetric-asymmetric.html). ## Limits + To perform the following steps, you must have the following permissions: `kms:CreateKey`, `kms:GetKeyPolicy`, and `kms:PutKeyPolicy`. + After you associate or disassociate a key from an anomaly detector, it can take up to five minutes for the operation to take effect. + If you revoke CloudWatch Logs access to an associated key or delete an associated KMS key, your encrypted data in CloudWatch Logs can no longer be retrieved. ### Step 1: Create an AWS KMS key To create an KMS key, use the following [create-key](https://docs.aws.amazon.com/cli/latest/reference/kms/create-key.html) command: ``` aws kms create-key ``` The output contains the key ID and Amazon Resource Name (ARN) of the key. The following is example output: ``` { "KeyMetadata": { "Origin": "AWS_KMS", "KeyId": "key-default-1", "Description": "", "KeyManager": "CUSTOMER", "Enabled": true, "CustomerMasterKeySpec": "SYMMETRIC_DEFAULT", "KeyUsage": "ENCRYPT_DECRYPT", "KeyState": "Enabled", "CreationDate": 1478910250.94, "Arn": "arn:aws:kms:us-west-2:123456789012:key/key-default-1", "AWSAccountId": "123456789012", "EncryptionAlgorithms": [ "SYMMETRIC_DEFAULT" ] } } ``` ### Step 2: Set permissions on the KMS key By default, all AWS KMS keys are private. Only the resource owner can use it to encrypt and decrypt data. However, the resource owner can grant permissions to access the KMS key to other users and resources. With this step, you give the CloudWatch Logs service principal permission to use the key. This service principal must be in the same AWS Region where the KMS key is stored. As a best practice, we recommend that you restrict the use of the KMS key to only those AWS accounts or anomaly detectors that you specify. First, save the default policy for your KMS key as `policy.json` using the following [get-key-policy](https://docs.aws.amazon.com/cli/latest/reference/kms/get-key-policy.html) command: ``` aws kms get-key-policy --key-id key-id --policy-name default --output text > ./policy.json ``` Open the `policy.json` file in a text editor and add the section in bold from one of the following statements. Separate the existing statement from the new statement with a comma. These statements use `Condition` sections to enhance the security of the AWS KMS key. For more information, see [AWS KMS keys and encryption context](encrypt-log-data-kms.md#encrypt-log-data-kms-policy). The `Condition` section in this example limits the use of the AWS KMS key to the specified account, but it can be used for any anomaly detector. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Id": "key-default-1", "Statement": [ { "Sid": "EnableIAMUserPermissions", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::111122223333:root" }, "Action": "kms:*", "Resource": "*" }, { "Sid": "AllowCloudWatchLogsEncryption", "Effect": "Allow", "Principal": { "Service": "logs.us-east-1.amazonaws.com" }, "Action": [ "kms:Encrypt", "kms:Decrypt", "kms:GenerateDataKey*" ], "Resource": "*", "Condition": { "StringEquals": { "aws:SourceAccount": "123456789012" }, "StringLike": { "kms:EncryptionContext:aws:logs:arn": "arn:aws:logs:us-east-1:123456789012:anomaly-detector:*" }, "ArnLike": { "aws:SourceArn": "arn:aws:logs:us-east-1:123456789012:anomaly-detector:*" } } }, { "Sid": "AllowCloudWatchLogsDescribeKey", "Effect": "Allow", "Principal": { "Service": "logs.us-east-1.amazonaws.com" }, "Action": "kms:DescribeKey", "Resource": "*", "Condition": { "StringEquals": { "aws:SourceAccount": "123456789012" } } }, { "Sid": "AllowCloudWatchLogsReEncryption", "Effect": "Allow", "Principal": { "Service": "logs.us-east-1.amazonaws.com" }, "Action": [ "kms:Encrypt", "kms:Decrypt", "kms:ReEncrypt*", "kms:GenerateDataKey*" ], "Resource": "*", "Condition": { "StringEquals": { "aws:SourceAccount": "123456789012" }, "StringLike": { "kms:EncryptionContext:aws-crypto-ec:aws:logs:arn": "arn:aws:logs:us-east-1:123456789012:anomaly-detector:*" }, "ArnLike": { "aws:SourceArn": "arn:aws:logs:us-east-1:123456789012:anomaly-detector:*" } } }, { "Sid": "AllowCloudWatchLogsDescribeKeyForReEncryption", "Effect": "Allow", "Principal": { "Service": "logs.us-east-1.amazonaws.com" }, "Action": "kms:DescribeKey", "Resource": "*", "Condition": { "StringEquals": { "aws:SourceAccount": "123456789012" } } } ] } ``` ------ Finally, add the updated policy using the following [put-key-policy](https://docs.aws.amazon.com/cli/latest/reference/kms/put-key-policy.html) command: ``` aws kms put-key-policy --key-id key-id --policy-name default --policy file://policy.json ``` ### Step 3: Associate a KMS key with an anomaly detector You can associate a KMS key with an anomaly detector when you create it in the console or using the AWS CLI or APIs. #### Step 4: Disassociate key from an anomaly detector After a key has been associated with an anomaly detector, you can't update the key. The only way to remove the key is to delete the anomaly detector, and then re-create it.