# Plan and configure primary nodes in your Amazon EMR cluster When you launch an Amazon EMR cluster, you can choose to have one or three primary nodes in your cluster. High availability for *instance fleets* is supported with Amazon EMR releases 5.36.1, 5.36.2, 6.8.1, 6.9.1, 6.10.1, 6.11.1, 6.12.0, and higher. For *instance groups*, high availability is supported with Amazon EMR releases 5.23.0 and higher. To further improve cluster availability, Amazon EMR can use Amazon EC2 placement groups to ensure that primary nodes are placed on distinct underlying hardware. For more information, see [Amazon EMR integration with EC2 placement groups](emr-plan-ha-placementgroup.md). An Amazon EMR cluster with multiple primary nodes provides the following benefits: + The primary node is no longer a single point of failure. If one of the primary nodes fails, the cluster uses the other two primary nodes and runs without interruption. In the meantime, Amazon EMR automatically replaces the failed primary node with a new one that is provisioned with the same configuration and bootstrap actions. + Amazon EMR enables the Hadoop high-availability features of HDFS NameNode and YARN ResourceManager and supports high availability for a few other open source applications. For more information about how an Amazon EMR cluster with multiple primary nodes supports open source applications and other Amazon EMR features, see [Features that support high availability in an Amazon EMR cluster and how they work with open-source applications](emr-plan-ha-applications.md). **Note** The cluster can reside only in one Availability Zone or subnet. This section provides information about supported applications and features of an Amazon EMR cluster with multiple primary nodes as well as the configuration details, best practices, and considerations for launching the cluster. **Topics** + [Features that support high availability in an Amazon EMR cluster and how they work with open-source applications](emr-plan-ha-applications.md) + [Launch an Amazon EMR Cluster with multiple primary nodes](emr-plan-ha-launch.md) + [Amazon EMR integration with EC2 placement groups](emr-plan-ha-placementgroup.md) + [Considerations and best practices when you create an Amazon EMR cluster with multiple primary nodes](emr-plan-ha-considerations.md) # Features that support high availability in an Amazon EMR cluster and how they work with open-source applications This topic provides information about the Hadoop high-availability features of HDFS NameNode and YARN ResourceManager in an Amazon EMR cluster, and how the high-availability features work with open source applications and other Amazon EMR features. ## High-availability HDFS An Amazon EMR cluster with multiple primary nodes enables the HDFS NameNode high availability feature in Hadoop. For more information, see [HDFS high availability](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithNFS.html). In an Amazon EMR cluster, two or more separate nodes are configured as NameNodes. One NameNode is in an `active` state and the others are in a `standby` state. If the node with `active` NameNode fails, Amazon EMR starts an automatic HDFS failover process. A node with `standby` NameNode becomes `active` and takes over all client operations in the cluster. Amazon EMR replaces the failed node with a new one, which then rejoins as a `standby`. **Note** In Amazon EMR versions 5.23.0 upto 5.36.2, only two of the three primary nodes run HDFS NameNode. In Amazon EMR versions 6.x and higher, all three of the primary nodes run HDFS NameNode. If you need to find out which NameNode is `active`, you can use SSH to connect to any primary node in the cluster and run the following command: ``` hdfs haadmin -getAllServiceState ``` The output lists the nodes where NameNode is installed and their status. For example, ``` ip-##-#-#-##1.ec2.internal:8020 active ip-##-#-#-##2.ec2.internal:8020 standby ip-##-#-#-##3.ec2.internal:8020 standby ``` ## High-availability YARN ResourceManager An Amazon EMR cluster with multiple primary nodes enables the YARN ResourceManager high availability feature in Hadoop. For more information, see [ResourceManager high availability](https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerHA.html). In an Amazon EMR cluster with multiple primary nodes, YARN ResourceManager runs on all three primary nodes. One ResourceManager is in `active` state, and the other two are in `standby` state. If the primary node with `active` ResourceManager fails, Amazon EMR starts an automatic failover process. A primary node with a `standby` ResourceManager takes over all operations. Amazon EMR replaces the failed primary node with a new one, which then rejoins the ResourceManager quorum as a `standby`. You can connect to "http://*master-public-dns-name*:8088/cluster" for any primary node, which automatically directs you to the `active` resource manager. To find out which resource manager is `active`, use SSH to connect to any primary node in the cluster. Then run the following command to get a list of the three primary nodes and their status: ``` yarn rmadmin -getAllServiceState ``` ## Supported applications in an Amazon EMR Cluster with multiple primary nodes You can install and run the following applications on an Amazon EMR cluster with multiple primary nodes. For each application, the primary node failover process varies. | Application | Availability during primary node failover | Notes | | --- | --- | --- | | Flink | Availability not affected by primary node failover | Flink jobs on Amazon EMR run as YARN applications. Flink's JobManagers run as YARN's ApplicationMasters on core nodes. The JobManager is not affected by the primary node failover process. If you use Amazon EMR version 5.27.0 or earlier, the JobManager is a single point of failure. When the JobManager fails, it loses all job states and will not resume the running jobs. You can enable JobManager high availability by configuring application attempt count, checkpointing, and enabling ZooKeeper as state storage for Flink. For more information, see [Configuring Flink on an Amazon EMR Cluster with multiple primary nodes](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/flink-configure.html#flink-multi-master). Beginning with Amazon EMR version 5.28.0, no manual configuration is needed to enable JobManager high availability. | | Ganglia | Availability not affected by primary node failover | Ganglia is available on all primary nodes, so Ganglia can continue to run during the primary node failover process. | | Hadoop | High availability | HDFS NameNode and YARN ResourceManager automatically fail over to the standby node when the active primary node fails. | | HBase | High availability | HBase automatically fails over to the standby node when the active primary node fails. If you are connecting to HBase through a REST or Thrift server, you must switch to a different primary node when the active primary node fails. | | HCatalog | Availability not affected by primary node failover | HCatalog is built upon Hive metastore, which exists outside of the cluster. HCatalog remains available during the primary node failover process. | | JupyterHub | High availability | JupyterHub is installed on all three primary instances. It is highly recommended to configure notebook persistence to prevent notebook loss upon primary node failure. For more information, see [Configuring persistence for notebooks in Amazon S3](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-jupyterhub-s3.html). | | Livy | High availability | Livy is installed on all three primary nodes. When the active primary node fails, you lose access to the current Livy session and need to create a new Livy session on a different primary node or on the new replacement node. | | Mahout | Availability not affected by primary node failover | Since Mahout has no daemon, it is not affected by the primary node failover process. | | MXNet | Availability not affected by primary node failover | Since MXNet has no daemon, it is not affected by the primary node failover process. | | Phoenix | High Availability | Phoenix' QueryServer runs only on one of the three primary nodes. Phoenix on all three masters is configured to connect the Phoenix QueryServer. You can find the private IP of Phoenix's Query server by using `/etc/phoenix/conf/phoenix-env.sh` file | | Pig | Availability not affected by primary node failover | Since Pig has no daemon, it is not affected by the primary node failover process. | | Spark | High availability | All Spark applications run in YARN containers and can react to primary node failover in the same way as high-availability YARN features. | | Sqoop | High availability | By default, sqoop-job and sqoop-metastore store data(job descriptions) on local disk of master that runs the command, if you want to save metastore data on external Database, please refer to apache Sqoop documentation | | Tez | High availability | Since Tez containers run on YARN, Tez behaves the same way as YARN during the primary node failover process. | | TensorFlow | Availability not affected by primary node failover | Since TensorFlow has no daemon, it is not affected by the primary node failover process. | | Zeppelin | High availability | Zeppelin is installed on all three primary nodes. Zeppelin stores notes and interpreter configurations in HDFS by default to prevent data loss. Interpreter sessions are completely isolated across all three primary instances. Session data will be lost upon master failure. It is recommended to not modify the same note concurrently on different primary instances. | | ZooKeeper | High availability | ZooKeeper is the foundation of the HDFS automatic failover feature. ZooKeeper provides a highly available service for maintaining coordination data, notifying clients of changes in that data, and monitoring clients for failures. For more information, see [HDFS automatic failover](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithNFS.html#Automatic_Failover). | To run the following applications in an Amazon EMR cluster with multiple primary nodes, you must configure an external database. The external database exists outside the cluster and makes data persistent during the primary node failover process. For the following applications, the service components will automatically recover during the primary node failover process, but active jobs may fail and need to be retried. | Application | Availability during primary node failover | Notes | | --- | --- | --- | | Hive | High availability for service components only | An external metastore for Hive is required. This must be a MySQL external metastore, as PostgreSQL is not supported for multi-master clusters. For more information, see [Configuring an external metastore for Hive](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-metastore-external-hive.html). | | Hue | High availability for service components only | An external database for Hue is required. For more information, see [Using Hue with a remote database in Amazon RDS](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/hue-rds.html). | | Oozie | High availability for service components only | An external database for Oozie is required. For more information, see [Using Oozie with a remote database in Amazon RDS](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/oozie-rds.html). Oozie-server and oozie-client are installed on all three primary nodes. The oozie-clients are configured to connect to the correct oozie-server by default. | | PrestoDB or PrestoSQL/Trino | High availability for service components only | An external Hive metastore for PrestoDB (PrestoSQL on Amazon EMR 6.1.0-6.3.0 or Trino on Amazon EMR 6.4.0 and later) is required. You can use [Presto with the AWS Glue Data Catalog](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-presto-glue.html) or [use an external MySQL database for Hive](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-hive-metastore-external.html). The Presto CLI is installed on all three primary nodes so you can use it to access the Presto Coordinator from any of the primary nodes. The Presto Coordinator is installed on only one primary node. You can find the DNS name of the primary node where the Presto Coordinator is installed by calling the Amazon EMR `describe-cluster` API and reading the returned value of the `MasterPublicDnsName` field in the response. | **Note** When a primary node fails, your Java Database Connectivity (JDBC) or Open Database Connectivity (ODBC) terminates its connection to the primary node. You can connect to any of the remaining primary nodes to continue your work because the Hive metastore daemon runs on all primary nodes. Or you can wait for the failed primary node to be replaced. ## How Amazon EMR features work in a cluster with multiple primary nodes ### Connecting to primary nodes using SSH You can connect to any of the three primary nodes in an Amazon EMR cluster using SSH in the same way you connect to a single primary node. For more information, see [Connect to the primary node using SSH](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-connect-master-node-ssh.html). If a primary node fails, your SSH connection to that primary node ends. To continue your work, you can connect to one of the other two primary nodes. Alternatively, you can access the new primary node after Amazon EMR replaces the failed one with a new one. **Note** The private IP address for the replacement primary node remains the same as the previous one. The public IP address for the replacement primary node may change. You can retrieve the new IP addresses in the console or by using the `describe-cluster` command in the AWS CLI. NameNode only runs on two or three of the primary nodes. However, you can run `hdfs` CLI commands and operate jobs to access HDFS on all three primary nodes. ### Working with steps in an Amazon EMR Cluster with multiple primary nodes You can submit steps to an Amazon EMR cluster with multiple primary nodes in the same way you work with steps in a cluster with a single primary node. For more information, see [Submit work to a cluster](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-work-with-steps.html). The following are considerations for working with steps in an Amazon EMR cluster with multiple primary nodes: + If a primary node fails, the steps that are running on the primary node are marked as FAILED. Any data that were written locally are lost. However, the status FAILED may not reflect the real state of the steps. + If a running step has started a YARN application when the primary node fails, the step can continue and succeed due to the automatic failover of the primary node. + It is recommended that you check the status of steps by referring to the output of the jobs. For example, MapReduce jobs use a `_SUCCESS` file to determine if the job completes successfully. + It is recommended that you set ActionOnFailure parameter to CONTINUE, or CANCEL\$1AND\$1WAIT, instead of TERMINATE\$1JOB\$1FLOW, or TERMINATE\$1CLUSTER. ### Automatic termination protection Amazon EMR automatically enables termination protection for all clusters with multiple primary nodes, and overrides any step execution settings that you supply when you create the cluster. You can disable termination protection after the cluster has been launched. See [Configuring termination protection for running clusters](UsingEMR_TerminationProtection.md#emr-termination-protection-running-cluster). To shut down a cluster with multiple primary nodes, you must first modify the cluster attributes to disable termination protection. For instructions, see [Terminate an Amazon EMR Cluster with multiple primary nodes](emr-plan-ha-launch.md#emr-plan-ha-launch-terminate). For more information about termination protection, see [Using termination protection to protect your Amazon EMR clusters from accidental shut down](UsingEMR_TerminationProtection.md). ### Unsupported features in an Amazon EMR Cluster with multiple primary nodes The following Amazon EMR features are currently not available in an Amazon EMR cluster with multiple primary nodes: + EMR Notebooks + One-click access to persistent Spark history server + Persistent application user interfaces + One-click access to persistent application user interfaces is currently not available for Amazon EMR clusters with multiple primary nodes or for Amazon EMR clusters integrated with AWS Lake Formation. + Runtime role-based access control. For more information, see [Additional considerations](emr-steps-runtime-roles.md#emr-steps-runtime-roles-considerations) in [Runtime roles for Amazon EMR steps](emr-steps-runtime-roles.md). + Amazon EMR integration with AWS IAM Identity Center (trusted identity propagation). For more information, see [Integrate Amazon EMR with AWS IAM Identity Center](emr-idc.md). **Note** To use Kerberos authentication in your cluster, you must configure an external KDC. Beginning with Amazon EMR version 5.27.0, you can configure HDFS Transparent encryption on an Amazon EMR cluster with multiple primary nodes. For more information, see [Transparent encryption in HDFS on Amazon EMR](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-encryption-tdehdfs.html). # Launch an Amazon EMR Cluster with multiple primary nodes This topic provides configuration details and examples for launching an Amazon EMR cluster with multiple primary nodes. **Note** Amazon EMR automatically enables termination protection for all clusters that have multiple primary nodes, and overrides any auto-termination settings that you supply when you create the cluster. To shut down a cluster with multiple primary nodes, you must first modify the cluster attributes to disable termination protection. For instructions, see [Terminate an Amazon EMR Cluster with multiple primary nodes](#emr-plan-ha-launch-terminate). ## Prerequisites + You can launch an Amazon EMR cluster with multiple primary nodes in both public and private VPC subnets. **EC2-Classic** is not supported. To launch an Amazon EMR cluster with multiple primary nodes in a public subnet, you must enable the instances in this subnet to receive a public IP address by selecting **Auto-assign IPv4** in the console or running the following command. Replace *22XXXX01* with your subnet ID. ``` aws ec2 modify-subnet-attribute --subnet-id subnet-22XXXX01 --map-public-ip-on-launch ``` + To run Hive, Hue, or Oozie on an Amazon EMR cluster with multiple primary nodes, you must create an external metastore. For more information, see [Configuring an external metastore for Hive](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-metastore-external-hive.html), [Using Hue with a remote database in Amazon RDS](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/hue-rds.html), or [Apache Oozie](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-oozie.html). + To use Kerberos authentication in your cluster, you must configure an external KDC. For more information, see [Configuring Kerberos on Amazon Amazon EMR](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-kerberos-configure.html). ## Launch an Amazon EMR Cluster with multiple primary nodes You can launch a cluster with multiple primary nodes when you use instance groups or instance fleets. When you use *instance groups* with multiple primary nodes, you must specify an instance count value of `3` for the primary node instance group. When you use *instance fleets* with multiple primary nodes, you must specify the `TargetOnDemandCapacity` of `3`, `TargetSpotCapacity` of `0` for the primary instance fleet, and `WeightedCapacity` of `1` for each instance type that you configure for the primary fleet. The following examples demonstrate how to launch the cluster using the default AMI or a custom AMI with both instance groups and instance fleets: **Note** You must specify the subnet ID when you launch an Amazon EMR cluster with multiple primary nodes using the AWS CLI. Replace *22XXXX01* and *22XXXX02 *with your subnet ID in the following examples. ------ #### [ Default AMI, instance groups ] **Example – Launching an Amazon EMR instance group cluster with multiple primary nodes using a default AMI** ``` aws emr create-cluster \ --name "ha-cluster" \ --release-label emr-6.15.0 \ --instance-groups InstanceGroupType=MASTER,InstanceCount=3,InstanceType=m5.xlarge InstanceGroupType=CORE,InstanceCount=4,InstanceType=m5.xlarge \ --ec2-attributes KeyName=ec2_key_pair_name,InstanceProfile=EMR_EC2_DefaultRole,SubnetId=subnet-22XXXX01 \ --service-role EMR_DefaultRole \ --applications Name=Hadoop Name=Spark ``` ------ #### [ Default AMI, instance fleets ] **Example – Launching an Amazon EMR instance fleet cluster with multiple primary nodes using a default AMI** ``` aws emr create-cluster \ --name "ha-cluster" \ --release-label emr-6.15.0 \ --instance-fleets '[ { "InstanceFleetType": "MASTER", "TargetOnDemandCapacity": 3, "TargetSpotCapacity": 0, "LaunchSpecifications": { "OnDemandSpecification": { "AllocationStrategy": "lowest-price" } }, "InstanceTypeConfigs": [ { "WeightedCapacity": 1, "BidPriceAsPercentageOfOnDemandPrice": 100, "InstanceType": "m5.xlarge" }, { "WeightedCapacity": 1, "BidPriceAsPercentageOfOnDemandPrice": 100, "InstanceType": "m5.2xlarge" }, { "WeightedCapacity": 1, "BidPriceAsPercentageOfOnDemandPrice": 100, "InstanceType": "m5.4xlarge" } ], "Name": "Master - 1" }, { "InstanceFleetType": "CORE", "TargetOnDemandCapacity": 5, "TargetSpotCapacity": 0, "LaunchSpecifications": { "OnDemandSpecification": { "AllocationStrategy": "lowest-price" } }, "InstanceTypeConfigs": [ { "WeightedCapacity": 1, "BidPriceAsPercentageOfOnDemandPrice": 100, "InstanceType": "m5.xlarge" }, { "WeightedCapacity": 2, "BidPriceAsPercentageOfOnDemandPrice": 100, "InstanceType": "m5.2xlarge" }, { "WeightedCapacity": 4, "BidPriceAsPercentageOfOnDemandPrice": 100, "InstanceType": "m5.4xlarge" } ], "Name": "Core - 2" } ]' \ --ec2-attributes '{"InstanceProfile":"EMR_EC2_DefaultRole","SubnetIds":["subnet-22XXXX01", "subnet-22XXXX02"]}' \ --service-role EMR_DefaultRole \ --applications Name=Hadoop Name=Spark ``` ------ #### [ Custom AMI, instance groups ] **Example – Launching an Amazon EMR instance group cluster with multiple primary nodes using a custom AMI** ``` aws emr create-cluster \ --name "custom-ami-ha-cluster" \ --release-label emr-6.15.0 \ --instance-groups InstanceGroupType=MASTER,InstanceCount=3,InstanceType=m5.xlarge InstanceGroupType=CORE,InstanceCount=4,InstanceType=m5.xlarge \ --ec2-attributes KeyName=ec2_key_pair_name,InstanceProfile=EMR_EC2_DefaultRole,SubnetId=subnet-22XXXX01 \ --service-role EMR_DefaultRole \ --applications Name=Hadoop Name=Spark \ --custom-ami-id ami-MyAmiID ``` ------ #### [ Custom AMI, instance fleets ] **Example – Launching an Amazon EMR instance fleet cluster with multiple primary nodes using a custom AMI** ``` aws emr create-cluster \ --name "ha-cluster" \ --release-label emr-6.15.0 \ --instance-fleets '[ { "InstanceFleetType": "MASTER", "TargetOnDemandCapacity": 3, "TargetSpotCapacity": 0, "LaunchSpecifications": { "OnDemandSpecification": { "AllocationStrategy": "lowest-price" } }, "InstanceTypeConfigs": [ { "WeightedCapacity": 1, "BidPriceAsPercentageOfOnDemandPrice": 100, "InstanceType": "m5.xlarge" }, { "WeightedCapacity": 1, "BidPriceAsPercentageOfOnDemandPrice": 100, "InstanceType": "m5.2xlarge" }, { "WeightedCapacity": 1, "BidPriceAsPercentageOfOnDemandPrice": 100, "InstanceType": "m5.4xlarge" } ], "Name": "Master - 1" }, { "InstanceFleetType": "CORE", "TargetOnDemandCapacity": 5, "TargetSpotCapacity": 0, "LaunchSpecifications": { "OnDemandSpecification": { "AllocationStrategy": "lowest-price" } }, "InstanceTypeConfigs": [ { "WeightedCapacity": 1, "BidPriceAsPercentageOfOnDemandPrice": 100, "InstanceType": "m5.xlarge" }, { "WeightedCapacity": 2, "BidPriceAsPercentageOfOnDemandPrice": 100, "InstanceType": "m5.2xlarge" }, { "WeightedCapacity": 4, "BidPriceAsPercentageOfOnDemandPrice": 100, "InstanceType": "m5.4xlarge" } ], "Name": "Core - 2" } ]' \ --ec2-attributes '{"InstanceProfile":"EMR_EC2_DefaultRole","SubnetIds":["subnet-22XXXX01", "subnet-22XXXX02"]}' \ --service-role EMR_DefaultRole \ --applications Name=Hadoop Name=Spark \ --custom-ami-id ami-MyAmiID ``` ------ ## Terminate an Amazon EMR Cluster with multiple primary nodes To terminate an Amazon EMR cluster with multiple primary nodes, you must disable termination protection before terminating the cluster, as the following example demonstrates. Replace *j-3KVTXXXXXX7UG* with your cluster ID. ``` aws emr modify-cluster-attributes --cluster-id j-3KVTXXXXXX7UG --no-termination-protected aws emr terminate-clusters --cluster-id j-3KVTXXXXXX7UG ``` # Amazon EMR integration with EC2 placement groups When you launch an Amazon EMR multiple primary node cluster on Amazon EC2, you have the option to use placement group strategies to specify how you want the primary node instances deployed to protect against hardware failure. Placement group strategies are supported starting with Amazon EMR version 5.23.0 as an option for multiple primary node clusters. Currently, only primary node types are supported by the placement group strategy, and the `SPREAD` strategy is applied to those primary nodes. The `SPREAD` strategy places a small group of instances across separate underlying hardware to guard against the loss of multiple primary nodes in the event of a hardware failure. Note that an instance launch request could fail if there is insufficient unique hardware to fulfill the request. For more information about EC2 placement strategies and limitations, see [Placement groups](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/placement-groups.html) in the *EC2 User Guide for Linux Instances*. There is an initial limit from Amazon EC2 of 500 placement group strategy-enabled clusters that can be launched per AWS region. Contact AWS support to request an increase in the number of allowed placement groups. You can identify EC2 placement groups Amazon EMR creates by tracking the key-value pair that Amazon EMR associates with the Amazon EMR placement group strategy. For more information about EC2 cluster instance tags, see [View cluster instances in Amazon EC2](UsingEMR_Tagging.md). ## Attach the placement group managed policy to the Amazon EMRrole The placement group strategy requires a managed policy called `AmazonElasticMapReducePlacementGroupPolicy`, which allows Amazon EMR to create, delete, and describe placement groups on Amazon EC2. You must attach `AmazonElasticMapReducePlacementGroupPolicy` to the service role for Amazon EMR before you launch an Amazon EMR cluster with multiple primary nodes. You can alternatively attach the `AmazonEMRServicePolicy_v2` managed policy to the Amazon EMR service role instead of the placement group managed policy. `AmazonEMRServicePolicy_v2` allows the same access to placement groups on Amazon EC2 as the `AmazonElasticMapReducePlacementGroupPolicy`. For more information, see [Service role for Amazon EMR (EMR role)](emr-iam-role.md). The `AmazonElasticMapReducePlacementGroupPolicy` managed policy is the following JSON text that is created and administered by Amazon EMR. **Note** Because the `AmazonElasticMapReducePlacementGroupPolicy` managed policy is updated automatically, the policy shown here may be out-of-date. Use the AWS Management Console to view the current policy. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Resource": [ "*" ], "Effect": "Allow", "Action": [ "ec2:DeletePlacementGroup", "ec2:DescribePlacementGroups" ], "Sid": "AllowEC2Deleteplacementgroup" }, { "Resource": [ "arn:aws:ec2:*:*:placement-group/pg-*" ], "Effect": "Allow", "Action": [ "ec2:CreatePlacementGroup" ], "Sid": "AllowEC2Createplacementgroup" } ] } ``` ------ ## Launch an Amazon EMR cluster with multiple primary nodes using placement group strategy To launch an Amazon EMR cluster that has multiple primary nodes with a placement group strategy, attach the placement group managed policy `AmazonElasticMapReducePlacementGroupPolicy` to the Amazon EMR role. For more information, see [Attach the placement group managed policy to the Amazon EMRrole](#emr-plan-ha-launch-pg-policy). Every time you use this role to start an Amazon EMR cluster with multiple primary nodes, Amazon EMR attempts to launch a cluster with `SPREAD` strategy applied to its primary nodes. If you use a role that does not have the placement group managed policy `AmazonElasticMapReducePlacementGroupPolicy` attached to it, Amazon EMR attempts to launch an Amazon EMR cluster that has multiple primary nodes without a placement group strategy. If you launch an Amazon EMR cluster that has multiple primary nodes with the `placement-group-configs` parameter using the Amazon EMRAPI or CLI, Amazon EMR only launches the cluster if the Amazon EMRrole has the placement group managed policy `AmazonElasticMapReducePlacementGroupPolicy` attached. If the Amazon EMRrole does not have the policy attached, the Amazon EMR cluster with multiple primary nodes start fails. ------ #### [ Amazon EMR API ] **Example – Use a placement group strategy to launch an instance group cluster with multiple primary nodes from the Amazon EMR API** When you use the RunJobFlow action to create an Amazon EMR cluster with multiple primary nodes, set the `PlacementGroupConfigs` property to the following. Currently, the `MASTER` instance role automatically uses `SPREAD` as the placement group strategy. ``` { "Name":"ha-cluster", "PlacementGroupConfigs":[ { "InstanceRole":"MASTER" } ], "ReleaseLabel": emr-6.15.0, "Instances":{ "ec2SubnetId":"subnet-22XXXX01", "ec2KeyName":"ec2_key_pair_name", "InstanceGroups":[ { "InstanceCount":3, "InstanceRole":"MASTER", "InstanceType":"m5.xlarge" }, { "InstanceCount":4, "InstanceRole":"CORE", "InstanceType":"m5.xlarge" } ] }, "JobFlowRole":"EMR_EC2_DefaultRole", "ServiceRole":"EMR_DefaultRole" } ``` + Replace *ha-cluster* with the name of your high-availability cluster. + Replace *subnet-22XXXX01* with your subnet ID. + Replace the *ec2\$1key\$1pair\$1name* with the name of your EC2 key pair for this cluster. EC2 key pair is optional and only required if you want to use SSH to access your cluster. ------ #### [ AWS CLI ] **Example – Use a placement group strategy to launch an instance fleet cluster with multiple primary nodes from the AWS Command Line Interface** When you use the RunJobFlow action to create an Amazon EMR cluster with multiple primary nodes, set the `PlacementGroupConfigs` property to the following. Currently, the `MASTER` instance role automatically uses `SPREAD` as the placement group strategy. ``` aws emr create-cluster \ --name "ha-cluster" \ --placement-group-configs InstanceRole=MASTER \ --release-label emr-6.15.0 \ --instance-fleets '[ { "InstanceFleetType": "MASTER", "TargetOnDemandCapacity": 3, "TargetSpotCapacity": 0, "LaunchSpecifications": { "OnDemandSpecification": { "AllocationStrategy": "lowest-price" } }, "InstanceTypeConfigs": [ { "WeightedCapacity": 1, "BidPriceAsPercentageOfOnDemandPrice": 100, "InstanceType": "m5.xlarge" }, { "WeightedCapacity": 1, "BidPriceAsPercentageOfOnDemandPrice": 100, "InstanceType": "m5.2xlarge" }, { "WeightedCapacity": 1, "BidPriceAsPercentageOfOnDemandPrice": 100, "InstanceType": "m5.4xlarge" } ], "Name": "Master - 1" }, { "InstanceFleetType": "CORE", "TargetOnDemandCapacity": 5, "TargetSpotCapacity": 0, "LaunchSpecifications": { "OnDemandSpecification": { "AllocationStrategy": "lowest-price" } }, "InstanceTypeConfigs": [ { "WeightedCapacity": 1, "BidPriceAsPercentageOfOnDemandPrice": 100, "InstanceType": "m5.xlarge" }, { "WeightedCapacity": 2, "BidPriceAsPercentageOfOnDemandPrice": 100, "InstanceType": "m5.2xlarge" }, { "WeightedCapacity": 4, "BidPriceAsPercentageOfOnDemandPrice": 100, "InstanceType": "m5.4xlarge" } ], "Name": "Core - 2" } ]' \ --ec2-attributes '{ "KeyName": "ec2_key_pair_name", "InstanceProfile": "EMR_EC2_DefaultRole", "SubnetIds": [ "subnet-22XXXX01", "subnet-22XXXX02" ] }' \ --service-role EMR_DefaultRole \ --applications Name=Hadoop Name=Spark ``` + Replace *ha-cluster* with the name of your high-availability cluster. + Replace the *ec2\$1key\$1pair\$1name* with the name of your EC2 key pair for this cluster. EC2 key pair is optional and only required if you want to use SSH to access your cluster. + Replace *subnet-22XXXX01* and *subnet-22XXXX02*with your subnet IDs. ------ ## Launch a cluster with multiple primary nodes without a placement group strategy For a cluster with multiple primary nodes to launch primary nodes without the placement group strategy, you need to do one of the following: + Remove the placement group managed policy `AmazonElasticMapReducePlacementGroupPolicy`from the Amazon EMRrole, or + Launch a cluster with multiple primary nodes with the `placement-group-configs` parameter using the Amazon EMRAPI or CLI choosing `NONE` as the placement group strategy. ------ #### [ Amazon EMR API ] **Example – Launching a cluster with multiple primary nodes without placement group strategy using the Amazon EMRAPI.** When using the RunJobFlow action to create a cluster with multiple primary nodes, set the `PlacementGroupConfigs` property to the following. ``` { "Name":"ha-cluster", "PlacementGroupConfigs":[ { "InstanceRole":"MASTER", "PlacementStrategy":"NONE" } ], "ReleaseLabel":"emr-5.30.1", "Instances":{ "ec2SubnetId":"subnet-22XXXX01", "ec2KeyName":"ec2_key_pair_name", "InstanceGroups":[ { "InstanceCount":3, "InstanceRole":"MASTER", "InstanceType":"m5.xlarge" }, { "InstanceCount":4, "InstanceRole":"CORE", "InstanceType":"m5.xlarge" } ] }, "JobFlowRole":"EMR_EC2_DefaultRole", "ServiceRole":"EMR_DefaultRole" } ``` + Replace *ha-cluster* with the name of your high-availability cluster. + Replace *subnet-22XXXX01* with your subnet ID. + Replace the *ec2\$1key\$1pair\$1name* with the name of your EC2 key pair for this cluster. EC2 key pair is optional and only required if you want to use SSH to access your cluster. ------ #### [ Amazon EMR CLI ] **Example – Launching a cluster with multiple primary nodes without a placement group strategy using the Amazon EMRCLI.** When using the RunJobFlow action to create a cluster with multiple primary nodes, set the `PlacementGroupConfigs` property to the following. ``` aws emr create-cluster \ --name "ha-cluster" \ --placement-group-configs InstanceRole=MASTER,PlacementStrategy=NONE \ --release-label emr-5.30.1 \ --instance-groups InstanceGroupType=MASTER,InstanceCount=3,InstanceType=m5.xlarge InstanceGroupType=CORE,InstanceCount=4,InstanceType=m5.xlarge \ --ec2-attributes KeyName=ec2_key_pair_name,InstanceProfile=EMR_EC2_DefaultRole,SubnetId=subnet-22XXXX01 \ --service-role EMR_DefaultRole \ --applications Name=Hadoop Name=Spark ``` + Replace *ha-cluster* with the name of your high-availability cluster. + Replace *subnet-22XXXX01* with your subnet ID. + Replace the *ec2\$1key\$1pair\$1name* with the name of your EC2 key pair for this cluster. EC2 key pair is optional and only required if you want to use SSH to access your cluster. ------ ## Checking placement group strategy configuration attached to the cluster with multiple primary nodes You can use the Amazon EMR describe cluster API to see the placement group strategy configuration attached to the cluster with multiple primary nodes. **Example** ``` aws emr describe-cluster --cluster-id "j-xxxxx" { "Cluster":{ "Id":"j-xxxxx", ... ... "PlacementGroups":[ { "InstanceRole":"MASTER", "PlacementStrategy":"SPREAD" } ] } } ``` # Considerations and best practices when you create an Amazon EMR cluster with multiple primary nodes Consider the following when you create an Amazon EMR cluster with multiple primary nodes: **Important** To launch high-availability EMR clusters with multiple primary nodes, we strongly recommend that you use the latest Amazon EMR release. This ensures that you get the highest level of resiliency and stability for your high-availability clusters. + High availability for *instance fleets* is supported with Amazon EMR releases 5.36.1, 5.36.2, 6.8.1, 6.9.1, 6.10.1, 6.11.1, 6.12.0, and higher. For *instance groups*, high availability is supported with Amazon EMR releases 5.23.0 and higher. To learn more, see [About Amazon EMR Releases](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-components.html). + On high-availability clusters, Amazon EMR only supports the launch of primary nodes with On Demand instances. This ensures the highest availability for your cluster. + You can still specify multiple instance types for primary fleet but all the primary nodes of high-availability clusters are launched with the same instance type, including replacements for unhealthy primary nodes. + To continue operations, a high-availability cluster with multiple primary nodes requires two out of three primary nodes to be healthy. As a result, if any two primary nodes fail simultaneously, your EMR cluster will fail. + All EMR clusters, including high-availability clusters, are launched in a single Availability Zone. Therefore, they can't tolerate Availability Zone failures. In the case of an Availability Zone outage, you lose access to the cluster. + If you use If you’re using a custom service role or policy when you launch a cluster inside an instance fleet, you can add the `ec2:DescribeInstanceTypeOfferings` permission so Amazon EMR can filter out unsupported Availability Zones (AZ). When Amazon EMR filters out the AZs that don’t support any instance types of primary nodes, Amazon EMR prevents cluster launches from failing because of unsupported primary instance types. For more information, see [Instance type not supported](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-INSTANCE_TYPE_NOT_SUPPORTED-error.html). + Amazon EMR doesn't guarantee high availability for open-source applications other than the ones that are specified in [Supported applications in an Amazon EMR Cluster with multiple primary nodes](emr-plan-ha-applications.md#emr-plan-ha-applications-list). + In Amazon EMR releases 5.23.0 through 5.36.2, only two of the three primary nodes for an instance group cluster run HDFS NameNode. + In Amazon EMR releases 6.x and higher, all three of the primary nodes for an instance group run HDFS NameNode. Considerations for configuring subnet: + An Amazon EMR cluster with multiple primary nodes can reside only in one Availability Zone or subnet. Amazon EMR cannot replace a failed primary node if the subnet is fully utilized or oversubscribed in the event of a failover. To avoid this scenario, it is recommended that you dedicate an entire subnet to an Amazon EMR cluster. In addition, make sure that there are enough private IP addresses available in the subnet. Considerations for configuring core nodes: + To ensure the core nodes are also highly available, we recommend that you launch at least four core nodes. If you decide to launch a smaller cluster with three or fewer core nodes, set `dfs.replication parameter` to at least `2` for HDFS to have sufficient DFS replication. For more information, see [HDFS configuration](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-hdfs-config.html). **Warning** Setting `dfs.replication` to 1 on clusters with fewer than four nodes can lead to HDFS data loss if a single node goes down. We recommend you use a cluster with at least four core nodes for production workloads. Amazon EMR will not allow clusters to scale core nodes below `dfs.replication`. For example, if `dfs.replication = 2`, the minimum number of core nodes is 2. When you use Managed Scaling, Auto-scaling, or choose to manually resize your cluster, we recommend that you to set `dfs.replication` to 2 or higher. Considerations for Setting Alarms on Metrics: + Amazon EMR doesn't provide application-specific metrics about HDFS or YARN. We reccommend that you set up alarms to monitor the primary node instance count. Configure the alarms using the following Amazon CloudWatch metrics: `MultiMasterInstanceGroupNodesRunning`, `MultiMasterInstanceGroupNodesRunningPercentage`, or `MultiMasterInstanceGroupNodesRequested`. CloudWatch will notify you in the case of primary node failure and replacement. + If the `MultiMasterInstanceGroupNodesRunningPercentage` is lower than 100% and greater than 50%, the cluster may have lost a primary node. In this situation, Amazon EMR attempts to replace a primary node. + If the `MultiMasterInstanceGroupNodesRunningPercentage` drops below 50%, two primary nodes may have failed. In this situation, the quorum is lost and the cluster can't be recovered. You must manually migrate data off of this cluster. For more information, see [Setting alarms on metrics](https://docs.aws.amazon.com/emr/latest/ManagementGuide/UsingEMR_ViewingMetrics.html#UsingEMR_ViewingMetrics_Alarm).