# Configuring CloudWatch agent for Amazon EMR Amazon EMR 7.0.0 and higher include the Amazon CloudWatch agent. The following sections describe how to configure the agent on your EMR cluster as described in [Create an EMR cluster that uses Amazon CloudWatch agent](AmazonCloudWatchAgent-create.md). **Topics** + [Configure CloudWatch agent for Amazon EMR 7.1.0](AmazonCloudWatchAgent-config-710.md) + [Configure CloudWatch agent for Amazon EMR 7.0.0](AmazonCloudWatchAgent-config-700.md) # Configure CloudWatch agent for Amazon EMR 7.1.0 Starting with Amazon EMR 7.1.0, you can configure the Amazon CloudWatch agent to use additional system metrics, add application metrics, and change metrics destination by using the Amazon EMR configuration API. For more information about how to use the EMR configuration API to configure your cluster’s applications, see [Configure applications](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-configure-apps.html). **Note** 7.1.0 only supports the reconfiguration type `OVERWRITE`. For more information about the reconfiguration types, see [Considerations when you reconfigure an instance group](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-configure-apps-running-cluster.html#emr-configure-apps-running-cluster-considerations). **Topics** + [Configuration schema](#AmazonCloudWatchAgent-config-710-config-schema) + [System metrics configurations examples](#AmazonCloudWatchAgent-config-710-system-examples) + [Application metrics configurations examples](#AmazonCloudWatchAgent-config-710-application-examples) + [Amazon Managed Service for Prometheus example](#AmazonCloudWatchAgent-config-710-prometheus-examples) ## Configuration schema `emr-metrics` has the following classifications: + `emr-system-metrics` — configure system metrics, such as CPU, disk, and memory. + `emr-hadoop-hdfs-datanode-metrics` — configure Hadoop DataNode JMX metrics + `emr-hadoop-hdfs-namenode-metrics` — configure Hadoop NameNode JMX metrics + `emr-hadoop-yarn-nodemanager-metrics` — configure Yarn NodeManager JMX metrics + `emr-hadoop-yarn-resourcemanager-metrics` — configure Yarn ResourceManager JMX metrics + `emr-hbase-master-metrics` — configure HBase Master JMX metrics + `emr-hbase-region-server-metrics` — configure HBase Region Server JMX metrics + `emr-hbase-rest-server-metrics` — configure HBase REST Server JMX metrics + `emr-hbase-thrift-server-metrics` — configure HBase Thrift Server JMX metrics The following tables describe the available properties and configurations for all of the classifications. **emr-metrics properties** | Property | Required | Description | Default value | Possible values | Notes | | --- | --- | --- | --- | --- | --- | | metrics\$1destination | Optional | Determines whether cluster metrics are published to Amazon CloudWatch or Amazon Managed Service for Prometheus. | "CLOUDWATCH" | "CLOUDWATCH", "PROMETHEUS" | This property is case-insensitive. For example, "Cloudwatch" is the same as "CLOUDWATCH". | | prometheus\$1endpoint | Optional | If metrics\$1destination is set to "PROMETHEUS", this property configures the CloudWatch agent to send metrics to the provided Amazon Managed Service for Prometheus remote write endpoint. | N/A | Any valid Amazon Managed Service for Prometheus remote write URL. The remote write URL format is 
https://aps-workspaces..amazonaws.com/workspaces//api/v1/remote_write
| This field is required if metrics\$1destination is set to "PROMETHEUS". Provisioning will fail if you don't provide a key or if the value is an empty string. | **emr-system-metrics properties** | Property | Required | Description | Default value | Possible values | Notes | | --- | --- | --- | --- | --- | --- | | metrics\$1collection\$1interval | Optional | How often in seconds metrics are collected and published from the CloudWatch agent. | "60" | A string specifying the number of seconds. Only accepts whole numbers. | You can override this property with the metrics\$1collection\$1interval property from individual metric groups. | **emr-system-metrics configurations** ------ #### [ cpu ] | Property | Required | Description | Default value | Possible values | Notes | | --- | --- | --- | --- | --- | --- | | metrics | Optional | The list of CPU metrics for the agent to collect. | See [Default metrics for CloudWatch agent with Amazon EMR](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/AmazonCloudWatchAgent-metrics.html) | A comma-separated list of valid CPU metric names with or without the cpu\$1 prefix, such as usage\$1active and cpu\$1time\$1idle. See [ Metrics collected by the CloudWatch agent](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/metrics-collected-by-CloudWatch-agent.html#linux-metrics-enabled-by-CloudWatch-agent) for valid metrics. | Specifying an empty string means to not publish any CPU metrics. | | metrics\$1collection\$1interval | Optional | How often in seconds the agent should collect and publish CPU metrics. | The value of the global metrics\$1collection\$1interval. | A string specifying the number of seconds. Accepts only whole numbers. | This value overrides the global metrics\$1collection\$1interval property only for CPU metrics. | | drop\$1original\$1metrics | Optional | List of CPU metrics for which to not publish unaggregated metrics. | No unaggregated CPU metrics published. | A comma-separated list of CPU metrics that are also specified in the metrics property. An empty string means to publish all CPU metrics. | The CloudWatch agent aggregates all metrics by cluster ID, instance ID, node type, and service name. By default, the CloudWatch agent doesn't publish the per-resource metrics for metrics with multiple resources. | | resources | Optional | Determines whether the agent will publish per-core metrics. | "\$1" | "\$1" enable per-core metrics. "" disable per-core metrics. | The CloudWatch agent only publishes per-core metrics for CPU metrics that aren't dropped in drop\$1original\$1metrics. | ------ #### [ disk ] | Property | Required | Description | Default value | Possible values | Notes | | --- | --- | --- | --- | --- | --- | | metrics | Optional | The list of disk metrics for the agent to collect. | See [Default metrics for CloudWatch agent with Amazon EMR](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/AmazonCloudWatchAgent-metrics.html) | A comma-separated list of valid disk metric names with or without the disk\$1 prefix, such as disk\$1total and used\$1percent. See [ Metrics collected by the CloudWatch agent](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/metrics-collected-by-CloudWatch-agent.html#linux-metrics-enabled-by-CloudWatch-agent) for valid metrics. | Specifying an empty string means to not publish any disk metrics. | | metrics\$1collection\$1interval | Optional | How often in seconds the agent should collect and publish disk metrics. | The value of the global metrics\$1collection\$1interval. | A string specifying the number of seconds. Accepts only whole numbers. | This value overrides the global metrics\$1collection\$1interval property only for disk metrics. | | drop\$1original\$1metrics | Optional | List of disk metrics for which to not publish unaggregated metrics. | No unaggregated disk metrics published. | A comma-separated list of disk metrics that are also specified in the metrics property. An empty string means to publish all disk metrics. | The CloudWatch agent aggregates all metrics by cluster ID, instance ID, node type, and service name. By default, the CloudWatch agent doesn't publish the per-resource metrics for metrics with multiple resources. | | resources | Optional | Determines whether the agent will publish per-mount-point metrics. | "\$1" | "\$1" means all mount points, "" means no mount points, or a comma-separated list of mount points. For example, "/,/emr". | The CloudWatch agent only publishes per-mount-point metrics for disk metrics that aren't dropped in drop\$1original\$1metrics. | ------ #### [ diskio ] | Property | Required | Description | Default value | Possible values | Notes | | --- | --- | --- | --- | --- | --- | | metrics | Optional | The list of disk IO metrics for the agent to collect. | See [Default metrics for CloudWatch agent with Amazon EMR](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/AmazonCloudWatchAgent-metrics.html) | A comma-separated list of valid disk IO metric names with or without the diskio\$1 prefix, such as diskio\$1reads and writes. See [ Metrics collected by the CloudWatch agent](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/metrics-collected-by-CloudWatch-agent.html#linux-metrics-enabled-by-CloudWatch-agent) for valid metrics. | Specifying an empty string means to not publish any disk IO metrics. | | metrics\$1collection\$1interval | Optional | How often in seconds the agent should collect and publish disk IO metrics. | The value of the global metrics\$1collection\$1interval. | A string specifying the number of seconds. Accepts only whole numbers. | This value overrides the global metrics\$1collection\$1interval property only for disk IO metrics. | | drop\$1original\$1metrics | Optional | List of disk IO metrics for which to not publish unaggregated metrics. | No unaggregated disk IO metrics published. | A comma-separated list of disk IO metrics that are also specified in the metrics property. An empty string means to publish all disk IO metrics. | The CloudWatch agent aggregates all metrics by cluster ID, instance ID, node type, and service name. By default, the CloudWatch agent doesn't publish the per-resource metrics for metrics with multiple resources. | | resources | Optional | Determines whether the agent will publish per-device metrics. | "\$1" | "\$1" means all storage devices, "" means no storage devices, or a comma-separated list of device names. For example, "nvme0n1,nvme1n1". | The CloudWatch agent only publishes per-device metrics for disk IO metrics that aren't dropped in drop\$1original\$1metrics. | ------ #### [ mem ] | Property | Required | Description | Default value | Possible values | Notes | | --- | --- | --- | --- | --- | --- | | metrics | Optional | The list of memory metrics for the agent to collect. | See [Default metrics for CloudWatch agent with Amazon EMR](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/AmazonCloudWatchAgent-metrics.html) | A comma-separated list of valid memory metric names with or without the mem\$1 prefix, such as mem\$1available and available\$1percent. See [ Metrics collected by the CloudWatch agent](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/metrics-collected-by-CloudWatch-agent.html#linux-metrics-enabled-by-CloudWatch-agent) for valid metrics. | Specifying an empty string means to not publish any memory metrics. | | metrics\$1collection\$1interval | Optional | How often in seconds the agent should collect and publish memory metrics. | The value of the global metrics\$1collection\$1interval. | A string specifying the number of seconds. Accepts only whole numbers. | This value overrides the global metrics\$1collection\$1interval property only for memory metrics. | ------ #### [ net ] | Property | Required | Description | Default value | Possible values | Notes | | --- | --- | --- | --- | --- | --- | | metrics | Optional | The list of network metrics for the agent to collect. | See [Default metrics for CloudWatch agent with Amazon EMR](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/AmazonCloudWatchAgent-metrics.html) | A comma-separated list of valid network metric names with or without the net\$1 prefix, such as net\$1packets\$1sent and packets\$1recv. See [ Metrics collected by the CloudWatch agent](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/metrics-collected-by-CloudWatch-agent.html#linux-metrics-enabled-by-CloudWatch-agent) for valid metrics. | Specifying an empty string means to not publish any network metrics. | | metrics\$1collection\$1interval | Optional | How often in seconds the agent should collect and publish network metrics. | The value of the global metrics\$1collection\$1interval. | A string specifying the number of seconds. Accepts only whole numbers. | This value overrides the global metrics\$1collection\$1interval property only for network metrics. | | drop\$1original\$1metrics | Optional | List of network metrics for which to not publish unaggregated metrics. | No unaggregated network metrics published. | A comma-separated list of network metrics that are also specified in the metrics property. An empty string means to publish all network metrics. | The CloudWatch agent aggregates all metrics by cluster ID, instance ID, node type, and service name. By default, the CloudWatch agent doesn't publish the per-resource metrics for metrics with multiple resources. | | resources | Optional | Determines whether the agent will publish per-interface metrics. | "\$1" | "\$1" means all network interfaces, "" means no network interfaces, or a comma-separated list of interfaces names. For example, "eth0,eth1". | The CloudWatch agent only publishes per-interface metrics for network metrics that aren't dropped in drop\$1original\$1metrics. | ------ #### [ netstat ] | Property | Required | Description | Default value | Possible values | Notes | | --- | --- | --- | --- | --- | --- | | metrics | Optional | The list of network statistics metrics for the agent to collect. | See [Default metrics for CloudWatch agent with Amazon EMR](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/AmazonCloudWatchAgent-metrics.html) | A comma-separated list of valid memory metric names with or without the netstat\$1 prefix, such as tcp\$1listen and netstat\$1udp\$1socket. See [ Metrics collected by the CloudWatch agent](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/metrics-collected-by-CloudWatch-agent.html#linux-metrics-enabled-by-CloudWatch-agent) for valid metrics. | Specifying an empty string means to not publish any network statistic metrics. | | metrics\$1collection\$1interval | Optional | How often in seconds the agent should collect and publish network statistic metrics. | The value of the global metrics\$1collection\$1interval. | A string specifying the number of seconds. Accepts only whole numbers. | This value overrides the global metrics\$1collection\$1interval property only for network statistic metrics. | ------ #### [ processes ] | Property | Required | Description | Default value | Possible values | Notes | | --- | --- | --- | --- | --- | --- | | metrics | Optional | The list of process metrics for the agent to collect. | See [Default metrics for CloudWatch agent with Amazon EMR](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/AmazonCloudWatchAgent-metrics.html) | A comma-separated list of valid memory metric names with or without the processes\$1 prefix, such as processes\$1running and total. See [ Metrics collected by the CloudWatch agent](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/metrics-collected-by-CloudWatch-agent.html#linux-metrics-enabled-by-CloudWatch-agent) for valid metrics. | Specifying an empty string means to not publish any process metrics. | | metrics\$1collection\$1interval | Optional | How often in seconds the agent should collect and publish system process metrics. | The value of the global metrics\$1collection\$1interval. | A string specifying the number of seconds. Accepts only whole numbers. | This value overrides the global metrics\$1collection\$1interval property only for system process metrics. | ------ #### [ swap ] | Property | Required | Description | Default value | Possible values | Notes | | --- | --- | --- | --- | --- | --- | | metrics | Optional | The list of swap metrics for the agent to collect. | See [Default metrics for CloudWatch agent with Amazon EMR](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/AmazonCloudWatchAgent-metrics.html) | A comma-separated list of valid memory metric names with or without the swap\$1 prefix, such as swap\$1free and used\$1percent. See [ Metrics collected by the CloudWatch agent](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/metrics-collected-by-CloudWatch-agent.html#linux-metrics-enabled-by-CloudWatch-agent) for valid metrics. | Specifying an empty string means to not publish any swap metrics. | | metrics\$1collection\$1interval | Optional | How often in seconds the agent should collect and publish swap metrics. | The value of the global metrics\$1collection\$1interval. | A string specifying the number of seconds. Accepts only whole numbers. | This value overrides the global metrics\$1collection\$1interval property only for swap metrics. | ------ **emr-hadoop-hdfs-datanode-metrics properties** | Property | Required | Description | Default value | Possible values | | --- | --- | --- | --- | --- | | | Optional | N/A | The MBean that CloudWatch agent should collect metrics from, such as Hadoop:service=DataNode,name=DataNodeActivity. You can find sample MBean names and their corresponding metrics in the [example JMX YAML files](https://github.com/aws-samples/aws-emr-utilities/tree/main/applications/cloudwatch-agent/configuration/7.0/application-metrics/mappings) for Amazon EMR release 7.0. | A string containing the comma-delimited list of metrics that are associated with the MBean. For example, BlocksCached,BlocksRead. | | otel.metric.export.interval | Optional | How often in milliseconds to collect Hadoop DataNode metrics. | "60000" | A string specifying the number of milliseconds. Accepts only whole numbers. | **emr-hadoop-hdfs-namenode-metrics properties** | Property | Required | Description | Default value | Possible values | | --- | --- | --- | --- | --- | | | Optional | N/A | The MBean that CloudWatch agent should collect metrics from, such as Hadoop:service=NameNode,name=FSNamesystem. You can find sample MBean names and their corresponding metrics in the [example JMX YAML files](https://github.com/aws-samples/aws-emr-utilities/tree/main/applications/cloudwatch-agent/configuration/7.0/application-metrics/mappings) for Amazon EMR release 7.0. | A string containing the comma-delimited list of metrics that are associated with the MBean. For example, BlockCapacity,CapacityUsedGB. | | otel.metric.export.interval | Optional | How often in milliseconds to collect Hadoop NameNode metrics. | "60000" | A string specifying the number of milliseconds. Accepts only whole numbers. | **emr-hadoop-yarn-nodemanager-metrics properties** | Property | Required | Description | Default value | Possible values | | --- | --- | --- | --- | --- | | | Optional | N/A | The MBean that CloudWatch agent should collect metrics from, such as Hadoop:service=NodeManager,name=NodeManagerMetrics. You can find sample MBean names and their corresponding metrics in the [example JMX YAML files](https://github.com/aws-samples/aws-emr-utilities/tree/main/applications/cloudwatch-agent/configuration/7.0/application-metrics/mappings) for Amazon EMR release 7.0. | A string containing the comma-delimited list of metrics that are associated with the MBean. For example, MaxCapacity,AllocatedGB. | | otel.metric.export.interval | Optional | How often in milliseconds to collect Hadoop YARN NodeManager metrics. | "60000" | A string specifying the number of milliseconds. Accepts only whole numbers. | **emr-hadoop-yarn-resourcemanager-metrics properties** | Property | Required | Description | Default value | Possible values | | --- | --- | --- | --- | --- | | | Optional | N/A | The MBean that CloudWatch agent should collect metrics from, such as Hadoop:service=ResourceManager,name=PartitionQueueMetrics. You can find sample MBean names and their corresponding metrics in the [example JMX YAML files](https://github.com/aws-samples/aws-emr-utilities/tree/main/applications/cloudwatch-agent/configuration/7.0/application-metrics/mappings) for Amazon EMR release 7.0. | A string containing the comma-delimited list of metrics that are associated with the MBean. For example, MaxCapacity,MaxCapacityVCores. | | otel.metric.export.interval | Optional | How often in milliseconds to collect Hadoop YARN ResourceManager metrics. | "60000" | A string specifying the number of milliseconds. Accepts only whole numbers. | **emr-hbase-master-metrics properties** | Property | Required | Description | Default value | Possible values | | --- | --- | --- | --- | --- | | | Optional | N/A | The MBean that CloudWatch agent should collect metrics from, such as Hadoop:service=HBase,name=Master,sub=AssignmentManager. You can find sample MBean names and their corresponding metrics in the [example JMX YAML files](https://github.com/aws-samples/aws-emr-utilities/tree/main/applications/cloudwatch-agent/configuration/7.0/application-metrics/mappings) for Amazon EMR release 7.0. | A string containing the comma-delimited list of metrics that are associated with the MBean. For example, AssignFailedCount,AssignSubmittedCount. | | otel.metric.export.interval | Optional | How often in milliseconds to collect HBase Master metrics. | "60000" | A string specifying the number of milliseconds. Accepts only whole numbers. | **emr-hbase-region-server-metrics properties** | Property | Required | Description | Default value | Possible values | | --- | --- | --- | --- | --- | | | Optional | N/A | The MBean that CloudWatch agent should collect metrics from, such as Hadoop:service=HBase,name=RegionServer,sub=IPC. You can find sample MBean names and their corresponding metrics in the [example JMX YAML files](https://github.com/aws-samples/aws-emr-utilities/tree/main/applications/cloudwatch-agent/configuration/7.0/application-metrics/mappings) for Amazon EMR release 7.0. | A string containing the comma-delimited list of metrics that are associated with the MBean. For example, numActiveHandler,numActivePriorityHandler. | | otel.metric.export.interval | Optional | How often in milliseconds to collect HBase Region Server metrics. | "60000" | A string specifying the number of milliseconds. Accepts only whole numbers. | **emr-hbase-rest-server-metrics properties** | Property | Required | Description | Default value | Possible values | | --- | --- | --- | --- | --- | | | Optional | N/A | The MBean that CloudWatch agent should collect metrics from, such as Hadoop:service=HBase,name=REST. You can find sample MBean names and their corresponding metrics in the [example JMX YAML files](https://github.com/aws-samples/aws-emr-utilities/tree/main/applications/cloudwatch-agent/configuration/7.0/application-metrics/mappings) for Amazon EMR release 7.0. | A string containing the comma-delimited list of metrics that are associated with the MBean. For example, successfulPut,successfulScanCount. | | otel.metric.export.interval | Optional | How often in milliseconds to collect HBase Rest Server metrics. | "60000" | A string specifying the number of milliseconds. Accepts only whole numbers. | **emr-hbase-thrift-server-metrics properties** | Property | Required | Description | Default value | Possible values | | --- | --- | --- | --- | --- | | | Optional | N/A | The MBean that CloudWatch agent should collect metrics from, such as Hadoop:service=HBase,name=Thrift,sub=ThriftOne. You can find sample MBean names and their corresponding metrics in the [example JMX YAML files](https://github.com/aws-samples/aws-emr-utilities/tree/main/applications/cloudwatch-agent/configuration/7.0/application-metrics/mappings) for Amazon EMR release 7.0. | A string containing the comma-delimited list of metrics that are associated with the MBean. For example, BatchGet\$1max,BatchGet\$1mean. | | otel.metric.export.interval | Optional | How often in milliseconds to collect HBase Thrift server metrics. | "60000" | A string specifying the number of milliseconds. Accepts only whole numbers. | ## System metrics configurations examples The following example demonstrates how to configure the CloudWatch agent to stop exporting all system metrics. ``` [ { "Classification": "emr-metrics", "Properties": {}, "Configurations": [ { "Classification": "emr-system-metrics", "Properties": {}, "Configurations": [] } ] } ] ``` The following example configures the CloudWatch agent to export the default system metrics. Doing so is a quick way to reset the agent back to only exporting the default system metrics if you've already reconfigured the system metrics at least once. This reset also removes any application metrics that were reconfigured before. ``` [ { "Classification": "emr-metrics", "Properties": {}, "Configurations": [] } ] ``` The following example configures the cluster to export the `cpu`, `mem`, and the `disk` metrics. ``` [ { "Classification": "emr-metrics", "Properties": {}, "Configurations": [ { "Classification": "emr-system-metrics", "Properties": { "metrics_collection_interval": "20" }, "Configurations": [ { "Classification": "cpu", "Properties": { "metrics": "cpu_usage_guest,cpu_usage_idle", "metrics_collection_interval": "30", "drop_original_metrics": "cpu_usage_guest" } }, { "Classification": "mem", "Properties": { "metrics": "mem_active" } }, { "Classification": "disk", "Properties": { "metrics": "disk_used_percent", "resources": "/,/mnt", "drop_original_metrics": "" } } ] } ] } ] ``` The previous example configuration has the following properties: + Every 30 seconds, the agent collects the `cpu_guest` metric for all CPUs. You can find the aggregated metric under the CloudWatch namespace `CWAgent > cluster.id, instance.id, node.type, service.name`. + Every 30 seconds, the agent collects the `cpu_idle` metric for all CPUs. You can find the aggregated metric under the CloudWatch namespace `CWAgent > cluster.id, instance.id, node.type, service.name`. The agent also collects the per-cpu metrics. You can find them in the same namespace. The agent collects this metric because the `drop_original_metrics` property doesn't contain `cpu_idle`, so the agent doesn't ignore the metric. + Every 20 seconds, the agent collects the `mem_active` metric. You can find the aggregated metric under the CloudWatch namespace `CWAgent > cluster.id, instance.id, node.type, service.name`. + Every 20 seconds, the agent collects the `disk_used_percent` metrics for the `/` and `/mnt` disk mounts. You can find the aggregated metrics under the CloudWatch namespace `CWAgent > cluster.id, instance.id, node.type, service.name`. The agent also collects the per-mount metrics. You can find them in the same namespace. The agent collects this metric because the `drop_original_metrics` property doesn't contain `disk_used_percent`, so the agent doesn't ignore the metric. ## Application metrics configurations examples The following example configures the CloudWatch agent to stop exporting metrics for the Hadoop Namenode service. ``` [ { "Classification": "emr-metrics", "Properties": {}, "Configurations": [ { "Classification": "emr-hadoop-hdfs-namenode-metrics", "Properties": {}, "Configurations": [] } ] } ] ``` The following example configures a cluster to export Hadoop application metrics. ``` [ { "Classification": "emr-metrics", "Properties": {}, "Configurations": [ { "Classification": "emr-hadoop-hdfs-namenode-metrics", "Properties": { "Hadoop:service=NameNode,name=FSNamesystem": "BlockCapacity,CapacityUsedGB", "otel.metric.export.interval": "20000" }, "Configurations": [] }, { "Classification": "emr-hadoop-hdfs-datanode-metrics", "Properties": { "Hadoop:service=DataNode,name=JvmMetrics": "MemNonHeapUsedM", "otel.metric.export.interval": "30000" }, "Configurations": [] }, { "Classification": "emr-hadoop-yarn-resourcemanager-metrics", "Properties": { "Hadoop:service=ResourceManager,name=CapacitySchedulerMetrics": "AllocateNumOps,NodeUpdateNumOps" }, "Configurations": [] } ] } ] ``` The previous example has the following properties: + Every 20 seconds, the agent collects the `BlockCapacity` and `CapacityUsedGB` metrics from instances running the Hadoop Namenode service. + Every 30 seconds, the agent collects `MemNonHeapUsedM` metrics from instances running the Hadoop Datanode service. + Every 30 seconds, the agent collects the `AllocateNumOps` and `NodeUpdateNumOps` metrics from instances that run the Hadoop YARN ResourceManaager. ## Amazon Managed Service for Prometheus example The following example demonstrates how to configure the CloudWatch agent to export metrics to Amazon Managed Service for Prometheus. If you are currently exporting metrics to Amazon Managed Service for Prometheus and want to reconfigure the metrics for the cluster and continue exporting metrics to Amazon Managed Service for Prometheus, you must include the properties `metrics_destination` and `prometheus_endpoint`. ``` [ { "Classification": "emr-metrics", "Properties": { "metrics_destination": "prometheus", "prometheus_endpoint": "http://amp-workspace/api/v1/remote_write" }, "Configurations": [] } ] ``` To use the CloudWatch agent to export metrics to CloudWatch, use the following example. ``` [ { "Classification": "emr-metrics", "Properties": { "metrics_destination": "cloudwatch" }, "Configurations": [] } ] ``` **Note** The CloudWatch agent has a Prometheus exporter that renames certain attributes. For the default metrics labels, Amazon Managed Service for Prometheus uses underscore characters in place of the periods that Amazon CloudWatch uses. If you use Amazon Managed Grafana to visualize the default metrics in Amazon Managed Service for Prometheus, the labels appear as `cluster_id`, `instance_id`, `node_type`, and `service_name`. # Configure CloudWatch agent for Amazon EMR 7.0.0 You can configure the Amazon CloudWatch agent to use additional system metrics beyond those that [the default CloudWatch agent configuration](AmazonCloudWatchAgent-metrics.md) provides. The configuration for 7.0.0 requires the use of bootstrap actions, which we've provided examples for in the following sections. In an upcoming release, Amazon EMR will provide additional configuration options through the Amazon EMR API. **Topics** + [Configure additional system metrics with Amazon EMR 7.0.0](#AmazonCloudWatchAgent-config-700-add-metrics) + [Configure application metrics with Amazon EMR 7.0.0](#AmazonCloudWatchAgent-config-700-app-metrics) + [Configure Amazon Managed Service for Prometheus as cloud storage for metrics with Amazon EMR 7.0.0](#AmazonCloudWatchAgent-config-700-prometheus) ## Configure additional system metrics with Amazon EMR 7.0.0 Use the following steps to configure the agent to use a different set of system metrics in Amazon EMR 7.0.0: 1. Create or choose a bucket in your Amazon S3 account where you want to store the configuration files that specify the CloudWatch agent metrics. 1. Create the `emr-amazon-cloudwatch-agent.json` configuration file with your preferred metrics specified. To do this, use one of the methods explained in [Create the CloudWatch agent configuration file](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/create-cloudwatch-agent-configuration-file.html). For more information about the structure of the CloudWatch agent configuration file, see [Manually create or edit the CloudWatch agent configuration file](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-Configuration-File-Details.html) in the *Amazon CloudWatch User Guide*. 1. Next, navigate to the **aws-emr-utilities** repo on GitHub and download the following system metrics scripts: + **[https://github.com/aws-samples/aws-emr-utilities/blob/main/applications/cloudwatch-agent/configuration/7.0/system-metrics/install_system_metrics_launcher.sh](https://github.com/aws-samples/aws-emr-utilities/blob/main/applications/cloudwatch-agent/configuration/7.0/system-metrics/install_system_metrics_launcher.sh)** – A script that downloads and then runs `install_system_metrics.sh` in the background so that the node can finish bootstrapping. + **[https://github.com/aws-samples/aws-emr-utilities/blob/main/applications/cloudwatch-agent/configuration/7.0/system-metrics/install_system_metrics.sh](https://github.com/aws-samples/aws-emr-utilities/blob/main/applications/cloudwatch-agent/configuration/7.0/system-metrics/install_system_metrics.sh)** – A script that waits for the instance it runs on to finish bootstrapping, then downloads and applies the configuration in the JSON file. 1. Open each SH file and replace `amzn-s3-demo-bucket` with the name of your bucket from Step 1. 1. Upload the one JSON and two SH files to your S3 bucket. 1. Now, you can navigate to the Amazon EMR console and create a new cluster with the CloudWatch agent. Under **EMR on EC2** in the left navigation, select **Clusters** and then **Create cluster**. 1. In the **Name and applications** section, choose an Amazon EMR release of 7.0.0 or higher. 1. Under **Application bundle**, select the bundle or apps that you want to install to your cluster, and include **Amazon CloudWatch Agent** with your selections. 1. In the **Bootstrap actions** section, select **Add**. + For the **Name**, insert **install\$1system\$1metrics\$1launcher.sh**. + For the **Script location**, insert **s3://*amzn-s3-demo-bucket*/install\$1system\$1metrics\$1launcher.sh**. Replace `amzn-s3-demo-bucket` with the path to your S3 bucket. + Leave the **Arguments** block empty. 1. Select **Add bootstrap action**. 1. Continue to create the cluster to serve your workload needs. When your cluster launches, the CloudWatch agent publishes the system metrics that you specified in the configuration file to CloudWatch. ## Configure application metrics with Amazon EMR 7.0.0 You can configure the Amazon CloudWatch agent to publish application metrics for HDFS and YARN in addition to system metrics. Use the following steps to configure the agent to publish application metrics: 1. Create or choose a bucket in your Amazon S3 account where you want to store the configuration files that specify the CloudWatch agent metrics. 1. Next, navigate to the **aws-emr-utilities** repo on GitHub and download the following scripts: + **[https://github.com/aws-samples/aws-emr-utilities/blob/main/applications/cloudwatch-agent/configuration/7.0/application-metrics/install_app_metrics_launcher.sh](https://github.com/aws-samples/aws-emr-utilities/blob/main/applications/cloudwatch-agent/configuration/7.0/application-metrics/install_app_metrics_launcher.sh)** – A script that downloads and then runs `install_app_metrics.sh` in the background so that the node can finish bootstrapping. + **[https://github.com/aws-samples/aws-emr-utilities/blob/main/applications/cloudwatch-agent/configuration/7.0/application-metrics/install_app_metrics.sh](https://github.com/aws-samples/aws-emr-utilities/blob/main/applications/cloudwatch-agent/configuration/7.0/application-metrics/install_app_metrics.sh)** – A script that waits for the instance it runs on to finish bootstrapping, then downloads and applies the configuration in the YAML files that you'll download in an upcoming step. 1. Open each file and replace `amzn-s3-demo-bucket` with the name of your bucket from Step 1. 1. Next, download the following YAML mapping files. For information about how these YAML files are structured, see [https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/jmx-metrics/javaagent](https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/jmx-metrics/javaagent) in the *OpenTelemetry Instrumentation for Java* GitHub repo. + **[https://github.com/aws-samples/aws-emr-utilities/blob/main/applications/cloudwatch-agent/configuration/7.0/application-metrics/mappings/datanode-metrics.yaml](https://github.com/aws-samples/aws-emr-utilities/blob/main/applications/cloudwatch-agent/configuration/7.0/application-metrics/mappings/datanode-metrics.yaml)** – The configuration for Hadoop DataNode metrics. + **[https://github.com/aws-samples/aws-emr-utilities/blob/main/applications/cloudwatch-agent/configuration/7.0/application-metrics/mappings/namenode-metrics.yaml](https://github.com/aws-samples/aws-emr-utilities/blob/main/applications/cloudwatch-agent/configuration/7.0/application-metrics/mappings/namenode-metrics.yaml)** – The configuration for Hadoop NameNode metrics. + **[https://github.com/aws-samples/aws-emr-utilities/blob/main/applications/cloudwatch-agent/configuration/7.0/application-metrics/mappings/nodemanager-metrics.yaml](https://github.com/aws-samples/aws-emr-utilities/blob/main/applications/cloudwatch-agent/configuration/7.0/application-metrics/mappings/nodemanager-metrics.yaml)** – The configuration for Yarn NodeManager metrics. + **[https://github.com/aws-samples/aws-emr-utilities/blob/main/applications/cloudwatch-agent/configuration/7.0/application-metrics/mappings/resourcemanager-metrics.yaml](https://github.com/aws-samples/aws-emr-utilities/blob/main/applications/cloudwatch-agent/configuration/7.0/application-metrics/mappings/resourcemanager-metrics.yaml)** – The configuration for Yarn ResourceManager metrics. 1. Upload the two SH and four YAML files to your S3 bucket. 1. Now, you can navigate to the Amazon EMR console and create a new cluster with the CloudWatch agent. Under **EMR on EC2** in the left navigation, select **Clusters** and then **Create cluster**. 1. In the **Name and applications** section, choose an Amazon EMR release of 7.0.0 or higher. 1. Under **Application bundle**, select the bundle or custom group of apps that you want to install to your cluster, and include **CloudWatch agent** with your selections. 1. In the **Bootstrap actions** section, select **Add**. + For the **Name**, insert **install\$1app\$1metrics\$1launcher.sh**. + For the **Script location**, insert **s3://*amzn-s3-demo-bucket*/install\$1app\$1metrics\$1launcher.sh**. Replace `amzn-s3-demo-bucket` with the path to your S3 bucket. + Leave the **Arguments** block empty. 1. Select **Add bootstrap action**. 1. Continue to create the cluster to serve your workload needs. When your cluster launches, The CloudWatch agent publishes the application metrics that you specified along with the system metrics to CloudWatch. ## Configure Amazon Managed Service for Prometheus as cloud storage for metrics with Amazon EMR 7.0.0 You can configure the Amazon CloudWatch agent to publish metrics to Amazon Managed Service for Prometheus instead of CloudWatch. **Note** You can publish metrics from the Amazon CloudWatch agent to either Amazon Managed Service for Prometheus or to Amazon CloudWatch, but you can't publish the metrics to both services for the same cluster. To configure the agent to publish metrics to Amazon Managed Service for Prometheus, you must add the `aps:RemoteWrite` AWS Identity and Access Management (IAM) permission to the Amazon EC2 instance profile for Amazon EMR. The following example policy contains the required permission: ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "aps:RemoteWrite" ], "Resource": [ "*" ], "Sid": "AllowAPSRemotewrite" } ] } ``` ------ **Use the CloudWatch agent on an EMR cluster to publish metrics to Amazon Managed Service for Prometheus** Once the service policy has the correct permissions, use the following steps to launch a cluster that uses the CloudWatch agent to publish metrics to Amazon Managed Service for Prometheus. 1. Use the AWS Management Console or AWS CLI to create an Amazon Managed Service for Prometheus workspace. For more information, see [Create a workspace](https://docs.aws.amazon.com/prometheus/latest/userguide/AMP-onboard-create-workspace.html) in the *Amazon Managed Service for Prometheus User Guide*. 1. Create or choose a bucket in your Amazon S3 account where you want to store the launch files that specify Amazon Managed Service for Prometheus as cloud storage. 1. Next, navigate to the **aws-emr-utilities** repo on GitHub and download the following scripts: + **[https://github.com/aws-samples/aws-emr-utilities/blob/main/applications/cloudwatch-agent/configuration/7.0/prometheus/add_prometheus_endpoint_launcher.sh](https://github.com/aws-samples/aws-emr-utilities/blob/main/applications/cloudwatch-agent/configuration/7.0/prometheus/add_prometheus_endpoint_launcher.sh)** – A script that downloads and then runs `add_prometheus_endpoint.sh` in the background so that the node can finish bootstrapping. + **[https://github.com/aws-samples/aws-emr-utilities/blob/main/applications/cloudwatch-agent/configuration/7.0/prometheus/add_prometheus_endpoint.sh](https://github.com/aws-samples/aws-emr-utilities/blob/main/applications/cloudwatch-agent/configuration/7.0/prometheus/add_prometheus_endpoint.sh)** – A script that waits for the instance it runs on to finish bootstrapping, then configures CloudWatch agent to publish to the Amazon Managed Service for Prometheus endpoint that you provide as an argument when you launch your cluster. 1. Open each file and replace `amzn-s3-demo-bucket` with the name of your bucket from Step 2. 1. Use the AWS CLI to create an EMR cluster with the `add_prometheus_endpoint_launcher.sh` bootstrap action. In the following command, replace `amzn-s3-demo-bucket` with the bucket that holds the bootstrap action, and replace `managedpro-remote-write-workspace-url` with the remote write endpoint for your Amazon Managed Service for Prometheus workspace. Be sure to specify an Amazon EMR release label of `emr-7.0.0` or higher. ``` aws emr create-cluster --name managedpro-cluster \ --release-label emr-7.0.0 \ --applications Name=Hadoop Name=AmazonCloudWatchAgent \ --ec2-attributes KeyName=myKey --instance-type m7g.2xlarge \ --instance-count 3 --use-default-roles --bootstrap-actions Name='Add Prometheus Endpoint',Path=s3://amzn-s3-demo-bucket/add_prometheus_endpoint_launcher.sh,Args='managedpro-remote-write-workspace-url' ``` When your cluster launches, the CloudWatch agent publishes the metrics it collects to Amazon Managed Service for Prometheus. **Use Amazon Managed Service for Prometheus as a data source for Amazon Managed Grafana** Once Amazon EMR has published the cluster metrics to Amazon Managed Service for Prometheus, you can use the following steps to visualize the metrics with Amazon Managed Grafana: 1. Use the AWS Management Console to create an Amazon Managed Grafana workspace and user with appropriate permissions. For more information, see [Create a workspace](https://docs.aws.amazon.com/grafana/latest/userguide/AMG-create-workspace.html) in the *Amazon Managed Grafana User Guide*. 1. Add your Amazon Managed Service for Prometheus workspace as a data source to Amazon Managed Grafana. For more information, see [Use AWS data source configuration to add Amazon Managed Service for Prometheus as a data source](https://docs.aws.amazon.com/grafana/latest/userguide/AMP-adding-AWS-config.html) in the *Amazon Managed Grafana User Guide*. **Note** The CloudWatch agent has a Prometheus exporter that renames certain attributes. For the default metrics labels, Amazon Managed Service for Prometheus uses underscore characters in place of the periods that Amazon CloudWatch uses. So if you use Amazon Managed Grafana to visualize the default metrics in Amazon Managed Service for Prometheus, the labels appear as `jobflow_id`, `instance_id`, and `service_name`. Also, any **application** metrics that the CloudWatch agent publishes to Amazon Managed Service for Prometheus use the label `job` instead of `service_name`. However, **system** metrics continue to use the `service_name` label.