# Amazon CloudWatch agent The Amazon CloudWatch agent on Amazon EMR is a tool that can monitor the Amazon EC2 instances in your EMR cluster. You can store and view the metrics that you collect with the CloudWatch agent in CloudWatch. For more information about the CloudWatch agent, see the [https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Install-CloudWatch-Agent.html](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Install-CloudWatch-Agent.html). **Note** You incur additional charges if you use other AWS services to publish, query, or view Amazon CloudWatch agent metrics. See the following pages for more pricing information. [Amazon CloudWatch Pricing](https://aws.amazon.com/cloudwatch/pricing/) [Amazon Managed Service for Prometheus Pricing](https://aws.amazon.com/prometheus/pricing/) [Amazon Managed Grafana Pricing](https://aws.amazon.com/grafana/pricing/) These metrics are separate from the free metrics that Amazon EMR provides under the `AWS/ElasticMapReduce` namespace in CloudWatch. For more information about the metrics that the CloudWatch agent doesn't publish, see [Monitoring Amazon EMR metrics with CloudWatch](https://docs.aws.amazon.com/emr/latest/ManagementGuide/UsingEMR_ViewingMetrics.html). With Amazon EMR 7.0 and higher, you can install a custom version of the Amazon CloudWatch agent on your cluster to collect metrics from your EMR cluster. CloudWatchAgent is supported on Runtime Role Clusters for EMR 7.6 and above. The agent collects the metrics from all nodes in the cluster, gathers them on the primary node, and publishes the metrics from that node to the cloud. You can install the agent when you create a new cluster in the console, or when you use the `create-cluster` API. For more information, see [Create an EMR cluster that uses Amazon CloudWatch agent](AmazonCloudWatchAgent-create.md). By default, an Amazon EMR cluster that runs the release 7.x series publishes [34 system-level metrics](AmazonCloudWatchAgent-metrics.md) to CloudWatch at 60-second intervals, but you can configure the agent to publish different metrics. Another option is to publish metrics to Amazon Managed Service for Prometheus, and you can choose which metrics to publish as well. For different use cases and setups, you might configure the agent to view and query the metrics in the CloudWatch console, Amazon Managed Grafana, or through the APIs for CloudWatch or Amazon Managed Service for Prometheus. These AWS services incur charges when you use them to store and query metrics. The CloudWatch agent on Amazon EMR can publish system metrics as well as JMX metrics for the following services on your Amazon EMR cluster. + Hadoop DataNode + Hadoop NameNode + Yarn NodeManager + Yarn ResourceManager + HBase Master – Amazon EMR 7.1\$1 only + HBase RegionServer – Amazon EMR 7.1\$1 only + HBase ThriftServer – Amazon EMR 7.1\$1 only For more information about available metrics and how to configure the CloudWatch agent on Amazon EMR, see [Configuring CloudWatch agent for Amazon EMR](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/AmazonCloudWatchAgent-config.html). The following table lists the version of AmazonCloudWatchAgent included in the latest release of the Amazon EMR 7.x series, along with the components that Amazon EMR installs with AmazonCloudWatchAgent. For the version of components installed with AmazonCloudWatchAgent in this release, see [Release 7.12.0 Component Versions](emr-7120-release.md). **AmazonCloudWatchAgent version information for emr-7.12.0** | Amazon EMR Release Label | AmazonCloudWatchAgent Version | Components Installed With AmazonCloudWatchAgent | | --- | --- | --- | | emr-7.12.0 | AmazonCloudWatchAgent 1.300032.2-amzn-0 | adot-java-agent, emrfs, emr-amazon-cloudwatch-agent, emr-ddb, emr-goodies, emr-kinesis, emr-s3-dist-cp, hadoop-client, hadoop-hdfs-datanode, hadoop-hdfs-library, hadoop-hdfs-namenode, hadoop-hdfs-zkfc, hadoop-httpfs-server, hadoop-kms-server, hadoop-mapred, hadoop-yarn-nodemanager, hadoop-yarn-resourcemanager, hadoop-yarn-timeline-server | **Topics** + [Create an EMR cluster that uses Amazon CloudWatch agent](AmazonCloudWatchAgent-create.md) + [Default metrics for CloudWatch agent with Amazon EMR](AmazonCloudWatchAgent-metrics.md) + [Configuring CloudWatch agent for Amazon EMR](AmazonCloudWatchAgent-config.md) + [Considerations and limitations](AmazonCloudWatchAgent-considerations.md) + [CloudWatch agent release history](AmazonCloudWatchAgent-release-history.md) # Create an EMR cluster that uses Amazon CloudWatch agent The procedures in this section describe the steps to create a cluster in Amazon EMR with Amazon CloudWatch agent from the AWS Management Console and the AWS CLI. **Topics** + [Required IAM permissions for CloudWatch agent](#AmazonCloudWatchAgent-permissions) + [Required CloudWatch agent endpoint](#AmazonCloudWatchAgent-endpoints) + [Create an EMR cluster](#AmazonCloudWatchAgent-create-cluster) ## Required IAM permissions for CloudWatch agent The CloudWatch agent requires the AWS Identity and Access Management (IAM) `cloudwatch:PutMetricData` permission in the Amazon EC2 instance profile for Amazon EMR. The Amazon EMR default role already has this permission. You can create the default role from the AWS CLI with `aws emr create-default-roles`. For more information, see [Service role for cluster EC2 instances (EC2 instance profile)](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-iam-role-for-ec2.html) in the *Amazon EMR Management Guide*. The following example IAM policy includes the `cloudwatch:PutMetricData` permission: ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "cloudwatch:PutMetricData" ], "Resource": [ "*" ], "Sid": "AllowCLOUDWATCHPutmetricdata" } ] } ``` ------ ## Required CloudWatch agent endpoint To publish metrics to CloudWatch for an EMR cluster in a private subnet, create a CloudWatch agent endpoint and associate with the VPC that the private subnet is in. For more information about the CloudWatch endpoints for each AWS Region, see [Amazon CloudWatch endpoints and quotas](https://docs.aws.amazon.com/general/latest/gr/cw_region.html) in the *AWS General Reference Guide*. ## Create an EMR cluster Once you have set up the required permissions and endpoint for use with the CloudWatch agent, use the AWS Management Console or the AWS CLI to create a new cluster with the agent installed. ------ #### [ Console ] **To create a cluster with Amazon CloudWatch agent from the console** 1. Navigate to the Amazon EMR console. 1. Choose **Create cluster**. 1. Under **Name and applications**, choose an Amazon EMR release of 7.0.0 or higher. 1. Under **Application bundle**, select the bundle or apps that you want to install to your cluster, and include **CloudWatch agent** with your selections. 1. Proceed to create the cluster to serve your use case needs. ------ #### [ AWS CLI ] In the AWS CLI, you can add Amazon CloudWatch agent to a cluster with the `--applications` parameter for `create-cluster`. **To create a cluster with Amazon CloudWatch agent from the AWS CLI** + When you create a cluster, use a command similar to the following to include the Amazon CloudWatch agent. Replace `myKey` with the name of your EC2 key pair. ``` aws emr create-cluster --name "Spark cluster with CloudWatch agent" \ --release-label emr-7.0.0 \ --applications Name=Spark Name=AmazonCloudWatchAgent \ --ec2-attributes KeyName=myKey --instance-type m7g.2xlarge \ --instance-count 3 --use-default-roles ``` For more details on how to use Amazon EMR with the AWS CLI, see the [AWS CLI Command Reference](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/emr/index.html). ------ # Default metrics for CloudWatch agent with Amazon EMR When you install the Amazon CloudWatch agent on Amazon EMR, the default configuration publishes the following system metrics for all of the instances in your cluster unless you [configure the agent differently](AmazonCloudWatchAgent-config.md). For definitions of each metric, see [Metrics collected by the CloudWatch agent](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/metrics-collected-by-CloudWatch-agent.html) in the *Amazon CloudWatch User Guide*. ------ #### [ CPU ] **CPU metrics** + `cpu_usage_active` + `cpu_usage_guest` + `cpu_usage_guest_nice` + `cpu_usage_idle` + `cpu_usage_iowait` + `cpu_usage_irq` + `cpu_usage_nice` + `cpu_usage_softirq` + `cpu_usage_steal` + `cpu_usage_system` + `cpu_usage_user` ------ #### [ Disk ] **Disk metrics** + `disk_free` + `disk_total` + `disk_used` + `disk_used_percent` ------ #### [ Memory ] **Memory metrics** + `mem_active` + `mem_available` + `mem_available_percent` + `mem_free` + `mem_inactive` + `mem_total` + `mem_used` + `mem_used_percent` + `mem_buffered` + `mem_cached` ------ #### [ Network IO ] **Network IO metrics** + `net_bytes_recv` + `net_bytes_sent` + `net_packets_recv` + `net_packets_sent` ------ #### [ Process ] **Process metrics** + `processes_running` + `processes_total` ------ #### [ Swap ] **Swap metrics** + `swap_free` + `swap_used` + `swap_used_percent` ------ By default, the agent publishes all metrics to CloudWatch under the custom namespace `CWAgent`, and under the schema `cluster.id`, `instance.id`, `node.type`, `service.name`. Use the following steps to view these default metrics that the CloudWatch agent publishes for Amazon EMR: 1. Navigate to the Amazon CloudWatch console. 1. Choose the **Metrics** tab, and then ** All metrics**. 1. Under **Custom namespaces**, select **`CWAgent`**. Then, select the schema: **`cluster.id`, `instance.id`, `node.type`, `service.name`**. 1. Continue to query the metrics from the CloudWatch interface as your use case requires. Amazon EMR attaches the following labels to each metric: `["cluster.id", "instance.id', "node.type", "service.name"]`. Consider the following with regard to these labels: + The value of the `cluster.id` label is the ID of the EMR cluster that produced the metric. An example value for the `cluster.id` label is: `j-123456789ABC`. + The value of the `instance.id` label is the ID of the instance in the EMR cluster that produced the metric. An example value for the `instance.id` label is `i-01bcf5f140f3355777`. + The value of the `node.type` label represents the node type of the instance in the cluster that produced the metric. This value can be either `master`, `core`, or `task`. + The value of the `service.name` label is the name of the service that produced the metric. The default `service.name` value for the default system metrics is `system`. **Note** Amazon EMR release 7.0.0 uses a slightly different metric label schema. `cluster.id` is `jobflow.id`, and `node.type` is unavailable. `instance.id` and `service.name` are available. # Configuring CloudWatch agent for Amazon EMR Amazon EMR 7.0.0 and higher include the Amazon CloudWatch agent. The following sections describe how to configure the agent on your EMR cluster as described in [Create an EMR cluster that uses Amazon CloudWatch agent](AmazonCloudWatchAgent-create.md). **Topics** + [Configure CloudWatch agent for Amazon EMR 7.1.0](AmazonCloudWatchAgent-config-710.md) + [Configure CloudWatch agent for Amazon EMR 7.0.0](AmazonCloudWatchAgent-config-700.md) # Configure CloudWatch agent for Amazon EMR 7.1.0 Starting with Amazon EMR 7.1.0, you can configure the Amazon CloudWatch agent to use additional system metrics, add application metrics, and change metrics destination by using the Amazon EMR configuration API. For more information about how to use the EMR configuration API to configure your cluster’s applications, see [Configure applications](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-configure-apps.html). **Note** 7.1.0 only supports the reconfiguration type `OVERWRITE`. For more information about the reconfiguration types, see [Considerations when you reconfigure an instance group](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-configure-apps-running-cluster.html#emr-configure-apps-running-cluster-considerations). **Topics** + [Configuration schema](#AmazonCloudWatchAgent-config-710-config-schema) + [System metrics configurations examples](#AmazonCloudWatchAgent-config-710-system-examples) + [Application metrics configurations examples](#AmazonCloudWatchAgent-config-710-application-examples) + [Amazon Managed Service for Prometheus example](#AmazonCloudWatchAgent-config-710-prometheus-examples) ## Configuration schema `emr-metrics` has the following classifications: + `emr-system-metrics` — configure system metrics, such as CPU, disk, and memory. + `emr-hadoop-hdfs-datanode-metrics` — configure Hadoop DataNode JMX metrics + `emr-hadoop-hdfs-namenode-metrics` — configure Hadoop NameNode JMX metrics + `emr-hadoop-yarn-nodemanager-metrics` — configure Yarn NodeManager JMX metrics + `emr-hadoop-yarn-resourcemanager-metrics` — configure Yarn ResourceManager JMX metrics + `emr-hbase-master-metrics` — configure HBase Master JMX metrics + `emr-hbase-region-server-metrics` — configure HBase Region Server JMX metrics + `emr-hbase-rest-server-metrics` — configure HBase REST Server JMX metrics + `emr-hbase-thrift-server-metrics` — configure HBase Thrift Server JMX metrics The following tables describe the available properties and configurations for all of the classifications. **emr-metrics properties** | Property | Required | Description | Default value | Possible values | Notes | | --- | --- | --- | --- | --- | --- | | metrics\$1destination | Optional | Determines whether cluster metrics are published to Amazon CloudWatch or Amazon Managed Service for Prometheus. | "CLOUDWATCH" | "CLOUDWATCH", "PROMETHEUS" | This property is case-insensitive. For example, "Cloudwatch" is the same as "CLOUDWATCH". | | prometheus\$1endpoint | Optional | If metrics\$1destination is set to "PROMETHEUS", this property configures the CloudWatch agent to send metrics to the provided Amazon Managed Service for Prometheus remote write endpoint. | N/A | Any valid Amazon Managed Service for Prometheus remote write URL. The remote write URL format is 
https://aps-workspaces..amazonaws.com/workspaces//api/v1/remote_write
| This field is required if metrics\$1destination is set to "PROMETHEUS". Provisioning will fail if you don't provide a key or if the value is an empty string. | **emr-system-metrics properties** | Property | Required | Description | Default value | Possible values | Notes | | --- | --- | --- | --- | --- | --- | | metrics\$1collection\$1interval | Optional | How often in seconds metrics are collected and published from the CloudWatch agent. | "60" | A string specifying the number of seconds. Only accepts whole numbers. | You can override this property with the metrics\$1collection\$1interval property from individual metric groups. | **emr-system-metrics configurations** ------ #### [ cpu ] | Property | Required | Description | Default value | Possible values | Notes | | --- | --- | --- | --- | --- | --- | | metrics | Optional | The list of CPU metrics for the agent to collect. | See [Default metrics for CloudWatch agent with Amazon EMR](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/AmazonCloudWatchAgent-metrics.html) | A comma-separated list of valid CPU metric names with or without the cpu\$1 prefix, such as usage\$1active and cpu\$1time\$1idle. See [ Metrics collected by the CloudWatch agent](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/metrics-collected-by-CloudWatch-agent.html#linux-metrics-enabled-by-CloudWatch-agent) for valid metrics. | Specifying an empty string means to not publish any CPU metrics. | | metrics\$1collection\$1interval | Optional | How often in seconds the agent should collect and publish CPU metrics. | The value of the global metrics\$1collection\$1interval. | A string specifying the number of seconds. Accepts only whole numbers. | This value overrides the global metrics\$1collection\$1interval property only for CPU metrics. | | drop\$1original\$1metrics | Optional | List of CPU metrics for which to not publish unaggregated metrics. | No unaggregated CPU metrics published. | A comma-separated list of CPU metrics that are also specified in the metrics property. An empty string means to publish all CPU metrics. | The CloudWatch agent aggregates all metrics by cluster ID, instance ID, node type, and service name. By default, the CloudWatch agent doesn't publish the per-resource metrics for metrics with multiple resources. | | resources | Optional | Determines whether the agent will publish per-core metrics. | "\$1" | "\$1" enable per-core metrics. "" disable per-core metrics. | The CloudWatch agent only publishes per-core metrics for CPU metrics that aren't dropped in drop\$1original\$1metrics. | ------ #### [ disk ] | Property | Required | Description | Default value | Possible values | Notes | | --- | --- | --- | --- | --- | --- | | metrics | Optional | The list of disk metrics for the agent to collect. | See [Default metrics for CloudWatch agent with Amazon EMR](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/AmazonCloudWatchAgent-metrics.html) | A comma-separated list of valid disk metric names with or without the disk\$1 prefix, such as disk\$1total and used\$1percent. See [ Metrics collected by the CloudWatch agent](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/metrics-collected-by-CloudWatch-agent.html#linux-metrics-enabled-by-CloudWatch-agent) for valid metrics. | Specifying an empty string means to not publish any disk metrics. | | metrics\$1collection\$1interval | Optional | How often in seconds the agent should collect and publish disk metrics. | The value of the global metrics\$1collection\$1interval. | A string specifying the number of seconds. Accepts only whole numbers. | This value overrides the global metrics\$1collection\$1interval property only for disk metrics. | | drop\$1original\$1metrics | Optional | List of disk metrics for which to not publish unaggregated metrics. | No unaggregated disk metrics published. | A comma-separated list of disk metrics that are also specified in the metrics property. An empty string means to publish all disk metrics. | The CloudWatch agent aggregates all metrics by cluster ID, instance ID, node type, and service name. By default, the CloudWatch agent doesn't publish the per-resource metrics for metrics with multiple resources. | | resources | Optional | Determines whether the agent will publish per-mount-point metrics. | "\$1" | "\$1" means all mount points, "" means no mount points, or a comma-separated list of mount points. For example, "/,/emr". | The CloudWatch agent only publishes per-mount-point metrics for disk metrics that aren't dropped in drop\$1original\$1metrics. | ------ #### [ diskio ] | Property | Required | Description | Default value | Possible values | Notes | | --- | --- | --- | --- | --- | --- | | metrics | Optional | The list of disk IO metrics for the agent to collect. | See [Default metrics for CloudWatch agent with Amazon EMR](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/AmazonCloudWatchAgent-metrics.html) | A comma-separated list of valid disk IO metric names with or without the diskio\$1 prefix, such as diskio\$1reads and writes. See [ Metrics collected by the CloudWatch agent](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/metrics-collected-by-CloudWatch-agent.html#linux-metrics-enabled-by-CloudWatch-agent) for valid metrics. | Specifying an empty string means to not publish any disk IO metrics. | | metrics\$1collection\$1interval | Optional | How often in seconds the agent should collect and publish disk IO metrics. | The value of the global metrics\$1collection\$1interval. | A string specifying the number of seconds. Accepts only whole numbers. | This value overrides the global metrics\$1collection\$1interval property only for disk IO metrics. | | drop\$1original\$1metrics | Optional | List of disk IO metrics for which to not publish unaggregated metrics. | No unaggregated disk IO metrics published. | A comma-separated list of disk IO metrics that are also specified in the metrics property. An empty string means to publish all disk IO metrics. | The CloudWatch agent aggregates all metrics by cluster ID, instance ID, node type, and service name. By default, the CloudWatch agent doesn't publish the per-resource metrics for metrics with multiple resources. | | resources | Optional | Determines whether the agent will publish per-device metrics. | "\$1" | "\$1" means all storage devices, "" means no storage devices, or a comma-separated list of device names. For example, "nvme0n1,nvme1n1". | The CloudWatch agent only publishes per-device metrics for disk IO metrics that aren't dropped in drop\$1original\$1metrics. | ------ #### [ mem ] | Property | Required | Description | Default value | Possible values | Notes | | --- | --- | --- | --- | --- | --- | | metrics | Optional | The list of memory metrics for the agent to collect. | See [Default metrics for CloudWatch agent with Amazon EMR](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/AmazonCloudWatchAgent-metrics.html) | A comma-separated list of valid memory metric names with or without the mem\$1 prefix, such as mem\$1available and available\$1percent. See [ Metrics collected by the CloudWatch agent](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/metrics-collected-by-CloudWatch-agent.html#linux-metrics-enabled-by-CloudWatch-agent) for valid metrics. | Specifying an empty string means to not publish any memory metrics. | | metrics\$1collection\$1interval | Optional | How often in seconds the agent should collect and publish memory metrics. | The value of the global metrics\$1collection\$1interval. | A string specifying the number of seconds. Accepts only whole numbers. | This value overrides the global metrics\$1collection\$1interval property only for memory metrics. | ------ #### [ net ] | Property | Required | Description | Default value | Possible values | Notes | | --- | --- | --- | --- | --- | --- | | metrics | Optional | The list of network metrics for the agent to collect. | See [Default metrics for CloudWatch agent with Amazon EMR](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/AmazonCloudWatchAgent-metrics.html) | A comma-separated list of valid network metric names with or without the net\$1 prefix, such as net\$1packets\$1sent and packets\$1recv. See [ Metrics collected by the CloudWatch agent](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/metrics-collected-by-CloudWatch-agent.html#linux-metrics-enabled-by-CloudWatch-agent) for valid metrics. | Specifying an empty string means to not publish any network metrics. | | metrics\$1collection\$1interval | Optional | How often in seconds the agent should collect and publish network metrics. | The value of the global metrics\$1collection\$1interval. | A string specifying the number of seconds. Accepts only whole numbers. | This value overrides the global metrics\$1collection\$1interval property only for network metrics. | | drop\$1original\$1metrics | Optional | List of network metrics for which to not publish unaggregated metrics. | No unaggregated network metrics published. | A comma-separated list of network metrics that are also specified in the metrics property. An empty string means to publish all network metrics. | The CloudWatch agent aggregates all metrics by cluster ID, instance ID, node type, and service name. By default, the CloudWatch agent doesn't publish the per-resource metrics for metrics with multiple resources. | | resources | Optional | Determines whether the agent will publish per-interface metrics. | "\$1" | "\$1" means all network interfaces, "" means no network interfaces, or a comma-separated list of interfaces names. For example, "eth0,eth1". | The CloudWatch agent only publishes per-interface metrics for network metrics that aren't dropped in drop\$1original\$1metrics. | ------ #### [ netstat ] | Property | Required | Description | Default value | Possible values | Notes | | --- | --- | --- | --- | --- | --- | | metrics | Optional | The list of network statistics metrics for the agent to collect. | See [Default metrics for CloudWatch agent with Amazon EMR](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/AmazonCloudWatchAgent-metrics.html) | A comma-separated list of valid memory metric names with or without the netstat\$1 prefix, such as tcp\$1listen and netstat\$1udp\$1socket. See [ Metrics collected by the CloudWatch agent](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/metrics-collected-by-CloudWatch-agent.html#linux-metrics-enabled-by-CloudWatch-agent) for valid metrics. | Specifying an empty string means to not publish any network statistic metrics. | | metrics\$1collection\$1interval | Optional | How often in seconds the agent should collect and publish network statistic metrics. | The value of the global metrics\$1collection\$1interval. | A string specifying the number of seconds. Accepts only whole numbers. | This value overrides the global metrics\$1collection\$1interval property only for network statistic metrics. | ------ #### [ processes ] | Property | Required | Description | Default value | Possible values | Notes | | --- | --- | --- | --- | --- | --- | | metrics | Optional | The list of process metrics for the agent to collect. | See [Default metrics for CloudWatch agent with Amazon EMR](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/AmazonCloudWatchAgent-metrics.html) | A comma-separated list of valid memory metric names with or without the processes\$1 prefix, such as processes\$1running and total. See [ Metrics collected by the CloudWatch agent](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/metrics-collected-by-CloudWatch-agent.html#linux-metrics-enabled-by-CloudWatch-agent) for valid metrics. | Specifying an empty string means to not publish any process metrics. | | metrics\$1collection\$1interval | Optional | How often in seconds the agent should collect and publish system process metrics. | The value of the global metrics\$1collection\$1interval. | A string specifying the number of seconds. Accepts only whole numbers. | This value overrides the global metrics\$1collection\$1interval property only for system process metrics. | ------ #### [ swap ] | Property | Required | Description | Default value | Possible values | Notes | | --- | --- | --- | --- | --- | --- | | metrics | Optional | The list of swap metrics for the agent to collect. | See [Default metrics for CloudWatch agent with Amazon EMR](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/AmazonCloudWatchAgent-metrics.html) | A comma-separated list of valid memory metric names with or without the swap\$1 prefix, such as swap\$1free and used\$1percent. See [ Metrics collected by the CloudWatch agent](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/metrics-collected-by-CloudWatch-agent.html#linux-metrics-enabled-by-CloudWatch-agent) for valid metrics. | Specifying an empty string means to not publish any swap metrics. | | metrics\$1collection\$1interval | Optional | How often in seconds the agent should collect and publish swap metrics. | The value of the global metrics\$1collection\$1interval. | A string specifying the number of seconds. Accepts only whole numbers. | This value overrides the global metrics\$1collection\$1interval property only for swap metrics. | ------ **emr-hadoop-hdfs-datanode-metrics properties** | Property | Required | Description | Default value | Possible values | | --- | --- | --- | --- | --- | | | Optional | N/A | The MBean that CloudWatch agent should collect metrics from, such as Hadoop:service=DataNode,name=DataNodeActivity. You can find sample MBean names and their corresponding metrics in the [example JMX YAML files](https://github.com/aws-samples/aws-emr-utilities/tree/main/applications/cloudwatch-agent/configuration/7.0/application-metrics/mappings) for Amazon EMR release 7.0. | A string containing the comma-delimited list of metrics that are associated with the MBean. For example, BlocksCached,BlocksRead. | | otel.metric.export.interval | Optional | How often in milliseconds to collect Hadoop DataNode metrics. | "60000" | A string specifying the number of milliseconds. Accepts only whole numbers. | **emr-hadoop-hdfs-namenode-metrics properties** | Property | Required | Description | Default value | Possible values | | --- | --- | --- | --- | --- | | | Optional | N/A | The MBean that CloudWatch agent should collect metrics from, such as Hadoop:service=NameNode,name=FSNamesystem. You can find sample MBean names and their corresponding metrics in the [example JMX YAML files](https://github.com/aws-samples/aws-emr-utilities/tree/main/applications/cloudwatch-agent/configuration/7.0/application-metrics/mappings) for Amazon EMR release 7.0. | A string containing the comma-delimited list of metrics that are associated with the MBean. For example, BlockCapacity,CapacityUsedGB. | | otel.metric.export.interval | Optional | How often in milliseconds to collect Hadoop NameNode metrics. | "60000" | A string specifying the number of milliseconds. Accepts only whole numbers. | **emr-hadoop-yarn-nodemanager-metrics properties** | Property | Required | Description | Default value | Possible values | | --- | --- | --- | --- | --- | | | Optional | N/A | The MBean that CloudWatch agent should collect metrics from, such as Hadoop:service=NodeManager,name=NodeManagerMetrics. You can find sample MBean names and their corresponding metrics in the [example JMX YAML files](https://github.com/aws-samples/aws-emr-utilities/tree/main/applications/cloudwatch-agent/configuration/7.0/application-metrics/mappings) for Amazon EMR release 7.0. | A string containing the comma-delimited list of metrics that are associated with the MBean. For example, MaxCapacity,AllocatedGB. | | otel.metric.export.interval | Optional | How often in milliseconds to collect Hadoop YARN NodeManager metrics. | "60000" | A string specifying the number of milliseconds. Accepts only whole numbers. | **emr-hadoop-yarn-resourcemanager-metrics properties** | Property | Required | Description | Default value | Possible values | | --- | --- | --- | --- | --- | | | Optional | N/A | The MBean that CloudWatch agent should collect metrics from, such as Hadoop:service=ResourceManager,name=PartitionQueueMetrics. You can find sample MBean names and their corresponding metrics in the [example JMX YAML files](https://github.com/aws-samples/aws-emr-utilities/tree/main/applications/cloudwatch-agent/configuration/7.0/application-metrics/mappings) for Amazon EMR release 7.0. | A string containing the comma-delimited list of metrics that are associated with the MBean. For example, MaxCapacity,MaxCapacityVCores. | | otel.metric.export.interval | Optional | How often in milliseconds to collect Hadoop YARN ResourceManager metrics. | "60000" | A string specifying the number of milliseconds. Accepts only whole numbers. | **emr-hbase-master-metrics properties** | Property | Required | Description | Default value | Possible values | | --- | --- | --- | --- | --- | | | Optional | N/A | The MBean that CloudWatch agent should collect metrics from, such as Hadoop:service=HBase,name=Master,sub=AssignmentManager. You can find sample MBean names and their corresponding metrics in the [example JMX YAML files](https://github.com/aws-samples/aws-emr-utilities/tree/main/applications/cloudwatch-agent/configuration/7.0/application-metrics/mappings) for Amazon EMR release 7.0. | A string containing the comma-delimited list of metrics that are associated with the MBean. For example, AssignFailedCount,AssignSubmittedCount. | | otel.metric.export.interval | Optional | How often in milliseconds to collect HBase Master metrics. | "60000" | A string specifying the number of milliseconds. Accepts only whole numbers. | **emr-hbase-region-server-metrics properties** | Property | Required | Description | Default value | Possible values | | --- | --- | --- | --- | --- | | | Optional | N/A | The MBean that CloudWatch agent should collect metrics from, such as Hadoop:service=HBase,name=RegionServer,sub=IPC. You can find sample MBean names and their corresponding metrics in the [example JMX YAML files](https://github.com/aws-samples/aws-emr-utilities/tree/main/applications/cloudwatch-agent/configuration/7.0/application-metrics/mappings) for Amazon EMR release 7.0. | A string containing the comma-delimited list of metrics that are associated with the MBean. For example, numActiveHandler,numActivePriorityHandler. | | otel.metric.export.interval | Optional | How often in milliseconds to collect HBase Region Server metrics. | "60000" | A string specifying the number of milliseconds. Accepts only whole numbers. | **emr-hbase-rest-server-metrics properties** | Property | Required | Description | Default value | Possible values | | --- | --- | --- | --- | --- | | | Optional | N/A | The MBean that CloudWatch agent should collect metrics from, such as Hadoop:service=HBase,name=REST. You can find sample MBean names and their corresponding metrics in the [example JMX YAML files](https://github.com/aws-samples/aws-emr-utilities/tree/main/applications/cloudwatch-agent/configuration/7.0/application-metrics/mappings) for Amazon EMR release 7.0. | A string containing the comma-delimited list of metrics that are associated with the MBean. For example, successfulPut,successfulScanCount. | | otel.metric.export.interval | Optional | How often in milliseconds to collect HBase Rest Server metrics. | "60000" | A string specifying the number of milliseconds. Accepts only whole numbers. | **emr-hbase-thrift-server-metrics properties** | Property | Required | Description | Default value | Possible values | | --- | --- | --- | --- | --- | | | Optional | N/A | The MBean that CloudWatch agent should collect metrics from, such as Hadoop:service=HBase,name=Thrift,sub=ThriftOne. You can find sample MBean names and their corresponding metrics in the [example JMX YAML files](https://github.com/aws-samples/aws-emr-utilities/tree/main/applications/cloudwatch-agent/configuration/7.0/application-metrics/mappings) for Amazon EMR release 7.0. | A string containing the comma-delimited list of metrics that are associated with the MBean. For example, BatchGet\$1max,BatchGet\$1mean. | | otel.metric.export.interval | Optional | How often in milliseconds to collect HBase Thrift server metrics. | "60000" | A string specifying the number of milliseconds. Accepts only whole numbers. | ## System metrics configurations examples The following example demonstrates how to configure the CloudWatch agent to stop exporting all system metrics. ``` [ { "Classification": "emr-metrics", "Properties": {}, "Configurations": [ { "Classification": "emr-system-metrics", "Properties": {}, "Configurations": [] } ] } ] ``` The following example configures the CloudWatch agent to export the default system metrics. Doing so is a quick way to reset the agent back to only exporting the default system metrics if you've already reconfigured the system metrics at least once. This reset also removes any application metrics that were reconfigured before. ``` [ { "Classification": "emr-metrics", "Properties": {}, "Configurations": [] } ] ``` The following example configures the cluster to export the `cpu`, `mem`, and the `disk` metrics. ``` [ { "Classification": "emr-metrics", "Properties": {}, "Configurations": [ { "Classification": "emr-system-metrics", "Properties": { "metrics_collection_interval": "20" }, "Configurations": [ { "Classification": "cpu", "Properties": { "metrics": "cpu_usage_guest,cpu_usage_idle", "metrics_collection_interval": "30", "drop_original_metrics": "cpu_usage_guest" } }, { "Classification": "mem", "Properties": { "metrics": "mem_active" } }, { "Classification": "disk", "Properties": { "metrics": "disk_used_percent", "resources": "/,/mnt", "drop_original_metrics": "" } } ] } ] } ] ``` The previous example configuration has the following properties: + Every 30 seconds, the agent collects the `cpu_guest` metric for all CPUs. You can find the aggregated metric under the CloudWatch namespace `CWAgent > cluster.id, instance.id, node.type, service.name`. + Every 30 seconds, the agent collects the `cpu_idle` metric for all CPUs. You can find the aggregated metric under the CloudWatch namespace `CWAgent > cluster.id, instance.id, node.type, service.name`. The agent also collects the per-cpu metrics. You can find them in the same namespace. The agent collects this metric because the `drop_original_metrics` property doesn't contain `cpu_idle`, so the agent doesn't ignore the metric. + Every 20 seconds, the agent collects the `mem_active` metric. You can find the aggregated metric under the CloudWatch namespace `CWAgent > cluster.id, instance.id, node.type, service.name`. + Every 20 seconds, the agent collects the `disk_used_percent` metrics for the `/` and `/mnt` disk mounts. You can find the aggregated metrics under the CloudWatch namespace `CWAgent > cluster.id, instance.id, node.type, service.name`. The agent also collects the per-mount metrics. You can find them in the same namespace. The agent collects this metric because the `drop_original_metrics` property doesn't contain `disk_used_percent`, so the agent doesn't ignore the metric. ## Application metrics configurations examples The following example configures the CloudWatch agent to stop exporting metrics for the Hadoop Namenode service. ``` [ { "Classification": "emr-metrics", "Properties": {}, "Configurations": [ { "Classification": "emr-hadoop-hdfs-namenode-metrics", "Properties": {}, "Configurations": [] } ] } ] ``` The following example configures a cluster to export Hadoop application metrics. ``` [ { "Classification": "emr-metrics", "Properties": {}, "Configurations": [ { "Classification": "emr-hadoop-hdfs-namenode-metrics", "Properties": { "Hadoop:service=NameNode,name=FSNamesystem": "BlockCapacity,CapacityUsedGB", "otel.metric.export.interval": "20000" }, "Configurations": [] }, { "Classification": "emr-hadoop-hdfs-datanode-metrics", "Properties": { "Hadoop:service=DataNode,name=JvmMetrics": "MemNonHeapUsedM", "otel.metric.export.interval": "30000" }, "Configurations": [] }, { "Classification": "emr-hadoop-yarn-resourcemanager-metrics", "Properties": { "Hadoop:service=ResourceManager,name=CapacitySchedulerMetrics": "AllocateNumOps,NodeUpdateNumOps" }, "Configurations": [] } ] } ] ``` The previous example has the following properties: + Every 20 seconds, the agent collects the `BlockCapacity` and `CapacityUsedGB` metrics from instances running the Hadoop Namenode service. + Every 30 seconds, the agent collects `MemNonHeapUsedM` metrics from instances running the Hadoop Datanode service. + Every 30 seconds, the agent collects the `AllocateNumOps` and `NodeUpdateNumOps` metrics from instances that run the Hadoop YARN ResourceManaager. ## Amazon Managed Service for Prometheus example The following example demonstrates how to configure the CloudWatch agent to export metrics to Amazon Managed Service for Prometheus. If you are currently exporting metrics to Amazon Managed Service for Prometheus and want to reconfigure the metrics for the cluster and continue exporting metrics to Amazon Managed Service for Prometheus, you must include the properties `metrics_destination` and `prometheus_endpoint`. ``` [ { "Classification": "emr-metrics", "Properties": { "metrics_destination": "prometheus", "prometheus_endpoint": "http://amp-workspace/api/v1/remote_write" }, "Configurations": [] } ] ``` To use the CloudWatch agent to export metrics to CloudWatch, use the following example. ``` [ { "Classification": "emr-metrics", "Properties": { "metrics_destination": "cloudwatch" }, "Configurations": [] } ] ``` **Note** The CloudWatch agent has a Prometheus exporter that renames certain attributes. For the default metrics labels, Amazon Managed Service for Prometheus uses underscore characters in place of the periods that Amazon CloudWatch uses. If you use Amazon Managed Grafana to visualize the default metrics in Amazon Managed Service for Prometheus, the labels appear as `cluster_id`, `instance_id`, `node_type`, and `service_name`. # Configure CloudWatch agent for Amazon EMR 7.0.0 You can configure the Amazon CloudWatch agent to use additional system metrics beyond those that [the default CloudWatch agent configuration](AmazonCloudWatchAgent-metrics.md) provides. The configuration for 7.0.0 requires the use of bootstrap actions, which we've provided examples for in the following sections. In an upcoming release, Amazon EMR will provide additional configuration options through the Amazon EMR API. **Topics** + [Configure additional system metrics with Amazon EMR 7.0.0](#AmazonCloudWatchAgent-config-700-add-metrics) + [Configure application metrics with Amazon EMR 7.0.0](#AmazonCloudWatchAgent-config-700-app-metrics) + [Configure Amazon Managed Service for Prometheus as cloud storage for metrics with Amazon EMR 7.0.0](#AmazonCloudWatchAgent-config-700-prometheus) ## Configure additional system metrics with Amazon EMR 7.0.0 Use the following steps to configure the agent to use a different set of system metrics in Amazon EMR 7.0.0: 1. Create or choose a bucket in your Amazon S3 account where you want to store the configuration files that specify the CloudWatch agent metrics. 1. Create the `emr-amazon-cloudwatch-agent.json` configuration file with your preferred metrics specified. To do this, use one of the methods explained in [Create the CloudWatch agent configuration file](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/create-cloudwatch-agent-configuration-file.html). For more information about the structure of the CloudWatch agent configuration file, see [Manually create or edit the CloudWatch agent configuration file](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-Configuration-File-Details.html) in the *Amazon CloudWatch User Guide*. 1. Next, navigate to the **aws-emr-utilities** repo on GitHub and download the following system metrics scripts: + **[https://github.com/aws-samples/aws-emr-utilities/blob/main/applications/cloudwatch-agent/configuration/7.0/system-metrics/install_system_metrics_launcher.sh](https://github.com/aws-samples/aws-emr-utilities/blob/main/applications/cloudwatch-agent/configuration/7.0/system-metrics/install_system_metrics_launcher.sh)** – A script that downloads and then runs `install_system_metrics.sh` in the background so that the node can finish bootstrapping. + **[https://github.com/aws-samples/aws-emr-utilities/blob/main/applications/cloudwatch-agent/configuration/7.0/system-metrics/install_system_metrics.sh](https://github.com/aws-samples/aws-emr-utilities/blob/main/applications/cloudwatch-agent/configuration/7.0/system-metrics/install_system_metrics.sh)** – A script that waits for the instance it runs on to finish bootstrapping, then downloads and applies the configuration in the JSON file. 1. Open each SH file and replace `amzn-s3-demo-bucket` with the name of your bucket from Step 1. 1. Upload the one JSON and two SH files to your S3 bucket. 1. Now, you can navigate to the Amazon EMR console and create a new cluster with the CloudWatch agent. Under **EMR on EC2** in the left navigation, select **Clusters** and then **Create cluster**. 1. In the **Name and applications** section, choose an Amazon EMR release of 7.0.0 or higher. 1. Under **Application bundle**, select the bundle or apps that you want to install to your cluster, and include **Amazon CloudWatch Agent** with your selections. 1. In the **Bootstrap actions** section, select **Add**. + For the **Name**, insert **install\$1system\$1metrics\$1launcher.sh**. + For the **Script location**, insert **s3://*amzn-s3-demo-bucket*/install\$1system\$1metrics\$1launcher.sh**. Replace `amzn-s3-demo-bucket` with the path to your S3 bucket. + Leave the **Arguments** block empty. 1. Select **Add bootstrap action**. 1. Continue to create the cluster to serve your workload needs. When your cluster launches, the CloudWatch agent publishes the system metrics that you specified in the configuration file to CloudWatch. ## Configure application metrics with Amazon EMR 7.0.0 You can configure the Amazon CloudWatch agent to publish application metrics for HDFS and YARN in addition to system metrics. Use the following steps to configure the agent to publish application metrics: 1. Create or choose a bucket in your Amazon S3 account where you want to store the configuration files that specify the CloudWatch agent metrics. 1. Next, navigate to the **aws-emr-utilities** repo on GitHub and download the following scripts: + **[https://github.com/aws-samples/aws-emr-utilities/blob/main/applications/cloudwatch-agent/configuration/7.0/application-metrics/install_app_metrics_launcher.sh](https://github.com/aws-samples/aws-emr-utilities/blob/main/applications/cloudwatch-agent/configuration/7.0/application-metrics/install_app_metrics_launcher.sh)** – A script that downloads and then runs `install_app_metrics.sh` in the background so that the node can finish bootstrapping. + **[https://github.com/aws-samples/aws-emr-utilities/blob/main/applications/cloudwatch-agent/configuration/7.0/application-metrics/install_app_metrics.sh](https://github.com/aws-samples/aws-emr-utilities/blob/main/applications/cloudwatch-agent/configuration/7.0/application-metrics/install_app_metrics.sh)** – A script that waits for the instance it runs on to finish bootstrapping, then downloads and applies the configuration in the YAML files that you'll download in an upcoming step. 1. Open each file and replace `amzn-s3-demo-bucket` with the name of your bucket from Step 1. 1. Next, download the following YAML mapping files. For information about how these YAML files are structured, see [https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/jmx-metrics/javaagent](https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/jmx-metrics/javaagent) in the *OpenTelemetry Instrumentation for Java* GitHub repo. + **[https://github.com/aws-samples/aws-emr-utilities/blob/main/applications/cloudwatch-agent/configuration/7.0/application-metrics/mappings/datanode-metrics.yaml](https://github.com/aws-samples/aws-emr-utilities/blob/main/applications/cloudwatch-agent/configuration/7.0/application-metrics/mappings/datanode-metrics.yaml)** – The configuration for Hadoop DataNode metrics. + **[https://github.com/aws-samples/aws-emr-utilities/blob/main/applications/cloudwatch-agent/configuration/7.0/application-metrics/mappings/namenode-metrics.yaml](https://github.com/aws-samples/aws-emr-utilities/blob/main/applications/cloudwatch-agent/configuration/7.0/application-metrics/mappings/namenode-metrics.yaml)** – The configuration for Hadoop NameNode metrics. + **[https://github.com/aws-samples/aws-emr-utilities/blob/main/applications/cloudwatch-agent/configuration/7.0/application-metrics/mappings/nodemanager-metrics.yaml](https://github.com/aws-samples/aws-emr-utilities/blob/main/applications/cloudwatch-agent/configuration/7.0/application-metrics/mappings/nodemanager-metrics.yaml)** – The configuration for Yarn NodeManager metrics. + **[https://github.com/aws-samples/aws-emr-utilities/blob/main/applications/cloudwatch-agent/configuration/7.0/application-metrics/mappings/resourcemanager-metrics.yaml](https://github.com/aws-samples/aws-emr-utilities/blob/main/applications/cloudwatch-agent/configuration/7.0/application-metrics/mappings/resourcemanager-metrics.yaml)** – The configuration for Yarn ResourceManager metrics. 1. Upload the two SH and four YAML files to your S3 bucket. 1. Now, you can navigate to the Amazon EMR console and create a new cluster with the CloudWatch agent. Under **EMR on EC2** in the left navigation, select **Clusters** and then **Create cluster**. 1. In the **Name and applications** section, choose an Amazon EMR release of 7.0.0 or higher. 1. Under **Application bundle**, select the bundle or custom group of apps that you want to install to your cluster, and include **CloudWatch agent** with your selections. 1. In the **Bootstrap actions** section, select **Add**. + For the **Name**, insert **install\$1app\$1metrics\$1launcher.sh**. + For the **Script location**, insert **s3://*amzn-s3-demo-bucket*/install\$1app\$1metrics\$1launcher.sh**. Replace `amzn-s3-demo-bucket` with the path to your S3 bucket. + Leave the **Arguments** block empty. 1. Select **Add bootstrap action**. 1. Continue to create the cluster to serve your workload needs. When your cluster launches, The CloudWatch agent publishes the application metrics that you specified along with the system metrics to CloudWatch. ## Configure Amazon Managed Service for Prometheus as cloud storage for metrics with Amazon EMR 7.0.0 You can configure the Amazon CloudWatch agent to publish metrics to Amazon Managed Service for Prometheus instead of CloudWatch. **Note** You can publish metrics from the Amazon CloudWatch agent to either Amazon Managed Service for Prometheus or to Amazon CloudWatch, but you can't publish the metrics to both services for the same cluster. To configure the agent to publish metrics to Amazon Managed Service for Prometheus, you must add the `aps:RemoteWrite` AWS Identity and Access Management (IAM) permission to the Amazon EC2 instance profile for Amazon EMR. The following example policy contains the required permission: ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "aps:RemoteWrite" ], "Resource": [ "*" ], "Sid": "AllowAPSRemotewrite" } ] } ``` ------ **Use the CloudWatch agent on an EMR cluster to publish metrics to Amazon Managed Service for Prometheus** Once the service policy has the correct permissions, use the following steps to launch a cluster that uses the CloudWatch agent to publish metrics to Amazon Managed Service for Prometheus. 1. Use the AWS Management Console or AWS CLI to create an Amazon Managed Service for Prometheus workspace. For more information, see [Create a workspace](https://docs.aws.amazon.com/prometheus/latest/userguide/AMP-onboard-create-workspace.html) in the *Amazon Managed Service for Prometheus User Guide*. 1. Create or choose a bucket in your Amazon S3 account where you want to store the launch files that specify Amazon Managed Service for Prometheus as cloud storage. 1. Next, navigate to the **aws-emr-utilities** repo on GitHub and download the following scripts: + **[https://github.com/aws-samples/aws-emr-utilities/blob/main/applications/cloudwatch-agent/configuration/7.0/prometheus/add_prometheus_endpoint_launcher.sh](https://github.com/aws-samples/aws-emr-utilities/blob/main/applications/cloudwatch-agent/configuration/7.0/prometheus/add_prometheus_endpoint_launcher.sh)** – A script that downloads and then runs `add_prometheus_endpoint.sh` in the background so that the node can finish bootstrapping. + **[https://github.com/aws-samples/aws-emr-utilities/blob/main/applications/cloudwatch-agent/configuration/7.0/prometheus/add_prometheus_endpoint.sh](https://github.com/aws-samples/aws-emr-utilities/blob/main/applications/cloudwatch-agent/configuration/7.0/prometheus/add_prometheus_endpoint.sh)** – A script that waits for the instance it runs on to finish bootstrapping, then configures CloudWatch agent to publish to the Amazon Managed Service for Prometheus endpoint that you provide as an argument when you launch your cluster. 1. Open each file and replace `amzn-s3-demo-bucket` with the name of your bucket from Step 2. 1. Use the AWS CLI to create an EMR cluster with the `add_prometheus_endpoint_launcher.sh` bootstrap action. In the following command, replace `amzn-s3-demo-bucket` with the bucket that holds the bootstrap action, and replace `managedpro-remote-write-workspace-url` with the remote write endpoint for your Amazon Managed Service for Prometheus workspace. Be sure to specify an Amazon EMR release label of `emr-7.0.0` or higher. ``` aws emr create-cluster --name managedpro-cluster \ --release-label emr-7.0.0 \ --applications Name=Hadoop Name=AmazonCloudWatchAgent \ --ec2-attributes KeyName=myKey --instance-type m7g.2xlarge \ --instance-count 3 --use-default-roles --bootstrap-actions Name='Add Prometheus Endpoint',Path=s3://amzn-s3-demo-bucket/add_prometheus_endpoint_launcher.sh,Args='managedpro-remote-write-workspace-url' ``` When your cluster launches, the CloudWatch agent publishes the metrics it collects to Amazon Managed Service for Prometheus. **Use Amazon Managed Service for Prometheus as a data source for Amazon Managed Grafana** Once Amazon EMR has published the cluster metrics to Amazon Managed Service for Prometheus, you can use the following steps to visualize the metrics with Amazon Managed Grafana: 1. Use the AWS Management Console to create an Amazon Managed Grafana workspace and user with appropriate permissions. For more information, see [Create a workspace](https://docs.aws.amazon.com/grafana/latest/userguide/AMG-create-workspace.html) in the *Amazon Managed Grafana User Guide*. 1. Add your Amazon Managed Service for Prometheus workspace as a data source to Amazon Managed Grafana. For more information, see [Use AWS data source configuration to add Amazon Managed Service for Prometheus as a data source](https://docs.aws.amazon.com/grafana/latest/userguide/AMP-adding-AWS-config.html) in the *Amazon Managed Grafana User Guide*. **Note** The CloudWatch agent has a Prometheus exporter that renames certain attributes. For the default metrics labels, Amazon Managed Service for Prometheus uses underscore characters in place of the periods that Amazon CloudWatch uses. So if you use Amazon Managed Grafana to visualize the default metrics in Amazon Managed Service for Prometheus, the labels appear as `jobflow_id`, `instance_id`, and `service_name`. Also, any **application** metrics that the CloudWatch agent publishes to Amazon Managed Service for Prometheus use the label `job` instead of `service_name`. However, **system** metrics continue to use the `service_name` label. # Considerations and limitations + The native Amazon CloudWatch agent is available for clusters that you create with Amazon EMR releases 7.0.0 and higher. + The configuration for CloudWatch agent in Amazon EMR 7.0.0 requires the use of [bootstrap actions](AmazonCloudWatchAgent-config-700.md). In an upcoming release, Amazon EMR will provide additional configuration options through the Amazon EMR API. + You can't install the Amazon EMR CloudWatch agent if you've already deployed the CloudWatch agent by another method such as AWS Systems Manager Agent (SSM Agent) in the Region where you create your cluster. Doing so will result in your cluster terminating with errors. + The CloudWatch GetMetricData API supports up to 500 metrics per request. If your Amazon EMR cluster contains more than 250 nodes in an instance group or fleet, the corresponding graphs in the CloudWatch embedded dashboard in EMR will appear blank with the error **Too many metrics** because these metrics require two data points per metric in the **Cluster Overview** dashboard. However, by filtering the **Core** or **Task Instance Group** (or fleet) dashboards, you will be able to view the graphs for up to 500 nodes per instance group or fleet. This is because these do not require two data points per metric. Beyond 500 nodes per instance group or fleet, the **Too many metrics** error also occurs for the metrics in these dashboards. # CloudWatch agent release history The following table lists the version of AmazonCloudWatchAgent included in each release version of Amazon EMR, along with the components installed with the application. For component versions in each release, see the Component Version section for your release in [Amazon EMR 7.x release versions](emr-release-7x.md), [Amazon EMR 6.x release versions](emr-release-6x.md), or [Amazon EMR 5.x release versions](emr-release-5x.md). **AmazonCloudWatchAgent version information** | Amazon EMR Release Label | AmazonCloudWatchAgent Version | Components Installed With AmazonCloudWatchAgent | | --- | --- | --- | | emr-7.12.0 | 1.300032.2-amzn-0 | adot-java-agent, emrfs, emr-amazon-cloudwatch-agent, emr-ddb, emr-goodies, emr-kinesis, emr-s3-dist-cp, hadoop-client, hadoop-hdfs-datanode, hadoop-hdfs-library, hadoop-hdfs-namenode, hadoop-hdfs-zkfc, hadoop-httpfs-server, hadoop-kms-server, hadoop-mapred, hadoop-yarn-nodemanager, hadoop-yarn-resourcemanager, hadoop-yarn-timeline-server | | emr-7.11.0 | 1.300032.2-amzn-0 | adot-java-agent, emrfs, emr-amazon-cloudwatch-agent, emr-ddb, emr-goodies, emr-kinesis, emr-s3-dist-cp, hadoop-client, hadoop-hdfs-datanode, hadoop-hdfs-library, hadoop-hdfs-namenode, hadoop-hdfs-zkfc, hadoop-httpfs-server, hadoop-kms-server, hadoop-mapred, hadoop-yarn-nodemanager, hadoop-yarn-resourcemanager, hadoop-yarn-timeline-server | | emr-7.10.0 | 1.300032.2-amzn-0 | adot-java-agent, emrfs, emr-amazon-cloudwatch-agent, emr-ddb, emr-goodies, emr-kinesis, emr-s3-dist-cp, hadoop-client, hadoop-hdfs-datanode, hadoop-hdfs-library, hadoop-hdfs-namenode, hadoop-httpfs-server, hadoop-kms-server, hadoop-mapred, hadoop-yarn-nodemanager, hadoop-yarn-resourcemanager, hadoop-yarn-timeline-server | | emr-7.9.0 | 1.300032.2-amzn-0 | adot-java-agent, emrfs, emr-amazon-cloudwatch-agent, emr-ddb, emr-goodies, emr-kinesis, emr-s3-dist-cp, hadoop-client, hadoop-hdfs-datanode, hadoop-hdfs-library, hadoop-hdfs-namenode, hadoop-httpfs-server, hadoop-kms-server, hadoop-mapred, hadoop-yarn-nodemanager, hadoop-yarn-resourcemanager, hadoop-yarn-timeline-server | | emr-7.8.0 | 1.300032.2-amzn-0 | adot-java-agent, emrfs, emr-amazon-cloudwatch-agent, emr-ddb, emr-goodies, emr-kinesis, emr-s3-dist-cp, hadoop-client, hadoop-hdfs-datanode, hadoop-hdfs-library, hadoop-hdfs-namenode, hadoop-httpfs-server, hadoop-kms-server, hadoop-mapred, hadoop-yarn-nodemanager, hadoop-yarn-resourcemanager, hadoop-yarn-timeline-server | | emr-7.7.0 | 1.300032.2-amzn-0 | adot-java-agent, emrfs, emr-amazon-cloudwatch-agent, emr-ddb, emr-goodies, emr-kinesis, emr-s3-dist-cp, hadoop-client, hadoop-hdfs-datanode, hadoop-hdfs-library, hadoop-hdfs-namenode, hadoop-httpfs-server, hadoop-kms-server, hadoop-mapred, hadoop-yarn-nodemanager, hadoop-yarn-resourcemanager, hadoop-yarn-timeline-server | | emr-7.6.0 | 1.300032.2-amzn-0 | adot-java-agent, emrfs, emr-amazon-cloudwatch-agent, emr-ddb, emr-goodies, emr-kinesis, emr-s3-dist-cp, hadoop-client, hadoop-hdfs-datanode, hadoop-hdfs-library, hadoop-hdfs-namenode, hadoop-httpfs-server, hadoop-kms-server, hadoop-mapred, hadoop-yarn-nodemanager, hadoop-yarn-resourcemanager, hadoop-yarn-timeline-server | | emr-7.5.0 | 1.300032.2-amzn-0 | adot-java-agent, emrfs, emr-amazon-cloudwatch-agent, emr-ddb, emr-goodies, emr-kinesis, emr-s3-dist-cp, hadoop-client, hadoop-hdfs-datanode, hadoop-hdfs-library, hadoop-hdfs-namenode, hadoop-httpfs-server, hadoop-kms-server, hadoop-mapred, hadoop-yarn-nodemanager, hadoop-yarn-resourcemanager, hadoop-yarn-timeline-server | | emr-7.4.0 | 1.300032.2-amzn-0 | adot-java-agent, emrfs, emr-amazon-cloudwatch-agent, emr-ddb, emr-goodies, emr-kinesis, emr-s3-dist-cp, hadoop-client, hadoop-hdfs-datanode, hadoop-hdfs-library, hadoop-hdfs-namenode, hadoop-httpfs-server, hadoop-kms-server, hadoop-mapred, hadoop-yarn-nodemanager, hadoop-yarn-resourcemanager, hadoop-yarn-timeline-server | | emr-7.3.0 | 1.300032.2-amzn-0 | adot-java-agent, emrfs, emr-amazon-cloudwatch-agent, emr-ddb, emr-goodies, emr-kinesis, emr-s3-dist-cp, hadoop-client, hadoop-hdfs-datanode, hadoop-hdfs-library, hadoop-hdfs-namenode, hadoop-httpfs-server, hadoop-kms-server, hadoop-mapred, hadoop-yarn-nodemanager, hadoop-yarn-resourcemanager, hadoop-yarn-timeline-server | | emr-7.2.0 | 1.300032.2-amzn-0 | adot-java-agent, emrfs, emr-amazon-cloudwatch-agent, emr-ddb, emr-goodies, emr-kinesis, emr-s3-dist-cp, hadoop-client, hadoop-hdfs-datanode, hadoop-hdfs-library, hadoop-hdfs-namenode, hadoop-httpfs-server, hadoop-kms-server, hadoop-mapred, hadoop-yarn-nodemanager, hadoop-yarn-resourcemanager, hadoop-yarn-timeline-server | | emr-7.1.0 | 1.300032.2-amzn-0 | adot-java-agent, emrfs, emr-amazon-cloudwatch-agent, emr-ddb, emr-goodies, emr-kinesis, emr-s3-dist-cp, hadoop-client, hadoop-hdfs-datanode, hadoop-hdfs-library, hadoop-hdfs-namenode, hadoop-httpfs-server, hadoop-kms-server, hadoop-mapred, hadoop-yarn-nodemanager, hadoop-yarn-resourcemanager, hadoop-yarn-timeline-server | | emr-7.0.0 | 1.300031.1-amzn-0 | adot-java-agent, emr-amazon-cloudwatch-agent |