Monitor application activity by using CloudWatch Logs Insights - AWS Prescriptive Guidance
SummaryPrerequisites and limitationsArchitectureToolsBest practicesEpicsTroubleshootingRelated resources

Monitor application activity by using CloudWatch Logs Insights

Ram Kandaswamy, Amazon Web Services

Summary

This pattern provides a solution for automatically detecting and alerting on application exceptions by using Amazon CloudWatch Logs Insights. By implementing automated log analysis and alerting, you can quickly identify and respond to application issues in your production environment.

Logs play a crucial role in monitoring system behavior, identifying issues, and ensuring optimal performance. During a migration process, log files are invaluable for validating the system's functioning in the new environment, detecting compatibility problems, and identifying any unexpected behaviors. Issues could be related to operations or security. For security-related issues, enabling the detection of unauthorized access attempts or suspicious activities early is essential for maintaining security and regulatory compliance. This capability is especially important when dealing with sensitive data or critical systems.

This pattern is particularly valuable for teams that need to do the following:

  • Maintain high application availability.

  • Respond to production issues quickly.

  • Analyze application-specific errors not captured by AWS service logs.

  • Perform on-demand log analysis without pre-built infrastructure.

CloudWatch Logs Insights is optimal for analyzing application-generated logs where the error context exists only within your application code. CloudWatch Logs Insights excels at the following tasks:

  • Query unstructured or semi-structured log data.

  • Perform on-demand analysis during incident response.

  • Correlate events across multiple log groups.

  • Create quick visualizations without external tools.

Prerequisites and limitations

Prerequisites

  • A production application deployed in active AWS account

  • Basic understanding of the production application's logging format and exception patterns

  • Application logs configured to stream to Amazon CloudWatch Logs

Limitations

Architecture

The following diagram shows how CloudWatch Logs Insights evaluates resource logs and sends a relevant data visualization to a CloudWatch dashboard.

CloudWatch Logs Insights evaluates resource logs and sends data visualization to dashboard.

The diagram shows the following workflow:

  1. The resources publish logs to CloudWatch Logs. Resources can include AWS resources such as Amazon Elastic Compute Cloud (Amazon EC2) instances or Amazon Simple Storage Service (Amazon S3) buckets. Another example includes on-premises systems with CloudWatch Agent installed that can publish logs to CloudWatch.

  2. CloudWatch Logs Insights filters for the relevant pattern string. Examples of search pattern strings include "error", "exception", or a specific regular expression.

  3. Typically, the production support team or developers add the pattern visualization to the CloudWatch dashboard.

Automation and scale

Developers can automate this pattern’s solution by using the AWS Cloud Development Kit (AWS CDK), AWS CloudFormation, or AWS SDKs to handle multiple string patterns. Teams can incorporate this automation into their continuous integration and deployment (CI/CD) DevOps processes.

Tools

AWS services

Best practices

Query efficiency

  • Define and configure log groups to analyze relevant log data.

  • Use field explorers to understand the structure and fields available in your log data.

  • Write efficient queries by using CloudWatch Logs Insights query syntax.

  • Adapt sample queries to your specific requirements for quicker analysis.

  • Limit query time ranges to reduce data scanned and improve performance.

  • Save queries for future use to save time and ensure consistent analysis.

Security

Cost optimization

  • CloudWatch Logs Insights charges per GB of data scanned per query. Narrow time ranges and target specific log groups to reduce costs.

  • Configure appropriate log retention policies to manage storage costs.

  • For frequent analysis of large historical datasets, consider exporting logs to Amazon S3 and using Amazon Athena.

  • Review CloudWatch pricing to understand cost implications for your use case.

Epics

TaskDescriptionSkills required

Configure IAM permissions.

To configure IAM permissions, do the following:

  1. Identify the IAM roles used by your application for writing logs and by users or services that will create dashboards, queries, and alarms.

  2. For the application role, add the following API actions and resource to a policy:

    {
         "Version": "2012-10-17",		 	 	 
         "Statement": [
             {
                 "Effect": "Allow",
                 "Action": [
                     "logs:CreateLogGroup",
                     "logs:CreateLogStream",
                     "logs:PutLogEvents"
                 ],
                 "Resource": [
                     "arn:aws:logs:*:*:*"
                 ]
             }
         ]
     }

  3. For managing CloudWatch resources, add the following to a policy:

    {
    "Version": "2012-10-17",		 	 	 
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "cloudwatch:PutDashboard",
                "cloudwatch:GetDashboard",
                "logs:StartQuery",
                "logs:StopQuery",
                "logs:GetQueryResults",
                "cloudwatch:PutMetricAlarm"
            ],
            "Resource": "*"
        }
    ]
}

For information about how to create IAM policies or to add permissions to existing policies, see Define custom IAM permissions with customer managed policies and Edit IAM policies in the IAM User Guide. For more information, see Identity and access management for Amazon CloudWatch Logs and CloudWatch Logs permissions reference in the Amazon CloudWatch Logs User Guide.

AWS administrator, AWS DevOps, AWS systems administrator, Cloud administrator, Cloud architect, DevOps engineer

Create a log group.

To create a log group, use any of the following options:

  • Create or open a CloudFormation template file (YAML or JSON format) with a .yaml or .json extension. (The following code uses YAML format.) Add the following resource definition to the Resources section of your template:

    MyLogGroup:
  Type: AWS::Logs::LogGroup
  Properties:
    LogGroupName: my-log-group

    For more information, see Quick Start: Use AWS CloudFormation to get started with CloudWatch Logs in the Amazon CloudWatch Logs User Guide.

AWS administrator, AWS DevOps, AWS systems administrator, Cloud administrator, Cloud architect, DevOps engineer

Generate a CloudWatch Logs Insights query.

To create and save a CloudWatch Logs Insights query, do the following:

  1. Open the CloudWatch console.

  2. Navigate to Logs Insights.

  3. Create your query by using one of the following methods:

    • Start with a sample query from the Queries section.

    • Write a custom query.

    • Modify the following example query:

      fields @timestamp, @message | filter @message like /(?i)exception/|stats count() by bin(30s)

      This query checks the log file, extracts the timestamp and message content, filters for the word "exception" (case-insensitive), and counts occurrences in 30-second intervals.

AWS administrator, AWS DevOps, AWS systems administrator, Cloud administrator, Cloud architect, DevOps engineer

Create visualization in a CloudWatch dashboard.

To use a CloudWatch dashboard to create a visualization, do the following:

  1. Open the CloudWatch console, and create a CloudWatch dashboard (if you don’t have one already) or open an existing dashboard.

  2. Add your exception monitoring visualization. Bar, line, and pie charts are available, depending on your business need. For more information, see Using widgets on CloudWatch dashboards in the Amazon CloudWatch Logs User Guide. For real-time data visualizations, you can customize a widget to meet your requirements.

For more information about dashboard options and capabilities, see Using Amazon CloudWatch dashboards and Creating flexible CloudWatch dashboards with dashboard variables in the Amazon CloudWatch Logs User Guide.

AWS administrator, AWS DevOps, AWS systems administrator, Cloud administrator, Cloud architect, DevOps engineer

Troubleshooting

IssueSolution

Unable to see query results or query seems broken

Start with a working query that was modified from a sample query. Perform small incremental changes to parts of the query (such as a filter or field), and take advantage of the CloudWatch Logs query generator feature.

Log groups not creating log streams

In the IAM policy, make sure that the resource for the CreateLogStream and the CreateLogGroup operations is provided with a wildcard character (*) value. The create operation will not succeed without this wildcard permission.

Query timeout or slow performance

Reduce the time range, target specific log groups, or simplify the query. Complex regular expression (regex) patterns and large time ranges increase query time.

No data returned for valid time range

Verify log group selection and check that logs are being ingested (review log streams), and confirm the filter pattern matches your log format.

Related resources