Publish Amazon EMR logs to CloudWatch Logs

Overview

Amazon EMR on EC2 provides native integration with Amazon CloudWatch Logs, enabling you to send cluster logs directly to CloudWatch This feature simplifies log management and provides centralized access to your EMR cluster logs for monitoring, troubleshooting, and analysis.

With CloudWatch logging enabled, you can automatically capture and stream logs from your EMR clusters to CloudWatch log groups. This includes step execution logs, Spark driver logs, and Spark executor logs, giving you comprehensive visibility into your cluster operations and application behavior.

The CloudWatch logging feature is available starting with Amazon EMR release 7.11.0 and is configured through the MonitoringConfiguration parameter when creating your cluster. Once enabled, logs are automatically streamed to CloudWatch as they are generated, providing near real-time access to log data through the CloudWatch console or API.

Prerequisites

Before enabling CloudWatch logging for your EMR cluster, ensure the following prerequisites are met:

Permissions

The CloudWatch Agent requires specific AWS Identity and Access Management(IAM) permissions to create log groups, create log streams, and write log events to CloudWatch Logs. These permissions must be attached to the Amazon EC2 instance profile used by your EMR cluster.

Required IAM policy

Add the following policy to your EC2 instance profile for Amazon EMR to grant the necessary permissions:

{
  "Version": "2012-10-17", 		 	 	 
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:PutLogEvents",
        "logs:PutRetentionPolicy",
        "logs:DescribeLogStreams",
        "logs:DescribeLogGroups",
        "logs:CreateLogStream",
        "logs:CreateLogGroup"
      ],
      "Resource": "*",
      "Sid": "AllowCWACloudWatchLogs"
    }
  ]
}

Attaching the Policy

To attach this policy to your EC2 instance profile for EMR:

For more information about IAM roles for Amazon EMR, see Configure IAM roles for Amazon EMR permissions to AWS services and resources in the Amazon EMR Management Guide.

Configuring CloudWatch Logging

You can enable CloudWatch logging when creating a new EMR cluster through the AWS Management Console, AWS CLI, or AWS SDKs. The configuration is specified through the MonitoringConfiguration parameter.

Using the AWS Management Console

To create a cluster with CloudWatch logging from the console:

After the cluster is created, you can access the CloudWatch Logs link from the EMR Cluster Details page under Cluster management → Log destination in Amazon CloudWatch.

Using the AWS CLI

You can enable CloudWatch logging using the AWS CLI with the create-cluster command. The CloudWatch Agent must be included in the --applications parameter, and logging is configured through the --monitoring-configuration parameter.

Example: Default Configuration

EMR will automatically capture step logs and Spark driver logs only and send them to the default log group.

aws emr create-cluster \
  --name "EMR cluster with CloudWatch Logs" \
  --release-label emr-7.11.0 \
  --applications Name=Spark Name=AmazonCloudWatchAgent \
  --instance-type m7g.2xlarge \
  --instance-count 3 \
  --use-default-roles \
  --monitoring-configuration '{
    "CloudWatchLogConfiguration": {
      "Enabled": true
    }
  }'

When using the default configuration:

Example: Custom Configuration

This example demonstrates a custom configuration with specific log group names, KMS encryption, and selective log types.

aws emr create-cluster \
  --name "EMR cluster with custom CloudWatch Logs" \
  --release-label emr-7.11.0 \
  --applications Name=Spark Name=AmazonCloudWatchAgent \
  --instance-type m7g.2xlarge \
  --instance-count 3 \
  --use-default-roles \
  --monitoring-configuration '{
    "CloudWatchLogConfiguration": {
      "Enabled": true,
      "LogGroupName": "/my-company/emr/production",
      "LogStreamNamePrefix": "cluster-prod",
      "EncryptionKeyArn": "arn:aws:kms:us-east-1:123456789012:key/12345678-1234-1234-1234-123456789012",
      "LogTypes": {
        "STEP_LOGS": ["STDOUT", "STDERR"],
        "SPARK_DRIVER": ["STDOUT", "STDERR"],
        "SPARK_EXECUTOR": ["STDERR", "STDOUT"]
      }
    }
  }'

This configuration:

For more information about using the AWS CLI with Amazon EMR, see the AWS CLI Command Reference for EMR.

Configuration Reference

CloudWatchLogConfiguration Parameters

The CloudWatchLogConfigurationv object supports the following parameters:

Log types

Amazon EMR supports three log types, each capturing both standard output and standard error streams:

Default log types configuration

When you don't specify the LogTypes parameter, EMR uses the following default configuration:

"LogTypes": {
  "STEP_LOGS": ["STDOUT", "STDERR"],
  "SPARK_DRIVER": ["STDOUT", "STDERR"]
}

Custom log types configuration

You can customize which log types to capture by explicitly specifying the LogTypes parameter. For example, to capture only step logs:

"LogTypes": {
  "STEP_LOGS": ["STDOUT", "STDERR"]
}

Or to capture only standard error from Spark drivers:

"LogTypes": {
  "SPARK_DRIVER": ["STDERR"]
}

Log group and stream naming

CloudWatch organizes logs into log groups and log streams:

Encrypting Logs with AWS KMS

You can encrypt your CloudWatch logs at rest using AWS Key Management Service (KMS). To enable encryption:

For detailed information about encrypting CloudWatch Logs data, see Encrypt log data in CloudWatch Logs using AWS Key Management Service.

Example with KMS Encryption

{
  "CloudWatchLogConfiguration": {
    "Enabled": true,
    "EncryptionKeyArn": "arn:aws:kms:us-east-1:123456789012:key/12345678-1234-1234-1234-123456789012"
  }
}

Viewing Logs in CloudWatch

After your cluster is running with CloudWatch logging enabled, you can view and analyze your logs through the CloudWatch console or API.

Accessing Logs from the EMR Console

The fastest way to access your cluster logs is directly from the EMR console:

This link takes you directly to the CloudWatch Logs console filtered to your cluster's log group.

Accessing Logs from the CloudWatch console

To manually navigate to your logs in CloudWatch:

For more information about working with CloudWatch Logs, see the Amazon CloudWatch Logs User Guide.

Considerations

CloudWatch Agent Behavior

The Amazon CloudWatch Agent provides both metrics and logging capabilities:

Enabling CloudWatch logging only (Disabling CloudWatch Metrics)

If you want to enable CloudWatch logging but disable the metrics collection feature, you can configure the CloudWatch Agent to stop exporting metrics. Add the following classification to your cluster configuration:

[
  {
    "Classification": "emr-metrics",
    "Properties": {},
    "Configurations": [
      {
        "Classification": "emr-system-metrics",
        "Properties": {},
        "Configurations": []
      }
    ]
  }
]

For more information about CloudWatch metrics, see Monitor metrics with Amazon CloudWatch.

Known Limitations

Private Subnet Requirements

To publish logs to CloudWatch Logs for an EMR cluster in a private subnet, create and associate the CloudWatch Logs VPC endpoint with your cluster's VPC.

For more information about CloudWatch Logs endpoints, see Amazon CloudWatch Logs endpoints and quotas in the AWS General Reference Guide.

Cost Considerations

CloudWatch Logs charges are based on:

To optimize costs:

For current pricing information, see Amazon CloudWatch Pricing.

Additional Resources