# Running interactive workloads on Amazon EMR on EKS An *interactive endpoint* is a gateway that connects Amazon EMR Studio to Amazon EMR on EKS so that you can run interactive workloads. You can use interactive endpoints with EMR Studio to run interactive analytics with datasets in data stores like [Amazon S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/) and [Amazon DynamoDB](https://docs.aws.amazon.com/amazondynamodb/latest/gettingstartedguide/). **Use cases** + Create an ETL script with the EMR Studio IDE experience. The IDE ingests on-premises data and stores it in Amazon S3 after transformations for subsequent analysis. + Use notebooks to explore datasets and train a machine-learning model to detect anomalies in the datasets. + Create scripts that generate daily reports for analytic applications like business dashboards. **Topics** + [Overview of interactive endpoints](how-it-works.md) + [Prerequisites to create an interactive endpoint on Amazon EMR on EKS](prereqs-for-studio.md) + [Creating an interactive endpoint for your virtual cluster](create-managed-endpoint.md) + [Configuring settings for interactive endpoints](managed-endpoint-parameters.md) + [Monitoring interactive endpoints](managed-endpoints-customer-metrics.md) + [Using self-hosted Jupyter notebooks](managed-endpoints-self-hosted.md) + [Getting information about interactive endpoints with CLI commands](other-operations.md) # Overview of interactive endpoints An *interactive endpoint* provides the capability for interactive clients like Amazon EMR Studio to connect to Amazon EMR on EKS clusters to run interactive workloads. The interactive endpoint is backed by a Jupyter Enterprise Gateway that provides the remote kernel lifecycle management capability that interactive clients need. *Kernels* are language-specific processes that interact with the Jupyter-based Amazon EMR Studio client to run interactive workloads. Interactive endpoints support the following kernels: + Python 3 + PySpark on Kubernetes + Apache Spark with Scala **Note** Amazon EMR on EKS pricing applies for the interactive endpoints and kernels. For more information, see the [Amazon EMR on EKS pricing page](https://aws.amazon.com/emr/pricing/#Amazon_EMR_on_Amazon_EKS). The following entities are required for EMR Studio to connect with Amazon EMR on EKS. + **Amazon EMR on EKS virtual cluster** – A *virtual cluster* is a Kubernetes namespace that you register Amazon EMR with. Amazon EMR uses virtual clusters to run jobs and host endpoints. You can back multiple virtual clusters with the same physical cluster. However, each virtual cluster maps to one namespace on an Amazon EKS cluster. Virtual clusters don't create any active resources that contribute to your bill or that require lifecycle management outside the service. + **Amazon EMR on EKS interactive endpoint** – An *interactive endpoint* is an HTTPS endpoint to which EMR Studio users can connect a workspace. You can only access the HTTPS endpoints from your EMR Studio, and you create them in a private subnet of the Amazon Virtual Private Cloud (Amazon VPC) for your Amazon EKS cluster. The Python, PySpark, and Spark Scala kernels use the permissions defined in your Amazon EMR on EKS job execution role to invoke other AWS services. All kernels and users that connect to the interactive endpoint utilize the role that you specified when you created the endpoint. We recommend that you create separate endpoints for different users, and that the users have different AWS Identity and Access Management (IAM) roles. + **AWS Application Load Balancer controller** – The *AWS Application Load Balancer controller* manages Elastic Load Balancing for an Amazon EKS Kubernetes cluster. The controller provisions an Application Load Balancer (ALB) when you create a Kubernetes Ingress resource. An ALB exposes a Kubernetes service, such as an interactive endpoint, outside of the Amazon EKS cluster but within the same Amazon VPC. When you create an interactive endpoint, an Ingress resource is also deployed that exposes the interactive endpoint by means of the ALB for interactive clients to connect to. You only need to install one AWS Application Load Balancer controller for each Amazon EKS cluster. The following diagram depicts the interactive endpoints architecture in Amazon EMR on EKS. An Amazon EKS cluster comprises the *compute* to run the analytic workloads, and the *interactive endpoint*. The Application Load Balancer controller runs in the `kube-system` namespace; the workloads and interactive endpoints run in the namespace that you specify when you create the virtual cluster. When you create an interactive endpoint, the Amazon EMR on EKS control plane creates the interactive endpoint deployment in the Amazon EKS cluster. Additionally, an instance of the application load balancer ingress is created by the AWS load balancer controller. The application load balancer provides the external interface for clients like EMR Studio to connect to the Amazon EMR cluster and run interactive workloads. ![\[Interactive endpoints architecture diagram\]](http://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/images/emr-on-eks-managed-endpoints-architecture.png) # Prerequisites to create an interactive endpoint on Amazon EMR on EKS This section describes prerequisites to set up an interactive endpoint that EMR Studio can use to connect to an Amazon EMR on EKS cluster and run interactive workloads. ## AWS CLI Follow the steps in [Install or update to the latest version of the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) to install the latest version of the AWS Command Line Interface (AWS CLI). ## Installing eksctl Follow the steps in [Install kubectl](https://docs.aws.amazon.com/eks/latest/userguide/install-kubectl.html) to install the latest version of eksctl. If you are using Kubernetes version 1.22 or later for your Amazon EKS cluster, use an eksctl version greater than 0.117.0. ## Amazon EKS cluster Create an Amazon EKS cluster. Register the cluster as a virtual cluster with Amazon EMR on EKS. The following are requirements and considerations for this cluster: + The cluster must be in the same Amazon Virtual Private Cloud (VPC) as your EMR Studio. + The cluster must have at least one private subnet to activate interactive endpoints, to link Git-based repositories, and to launch the Application Load Balancer in private mode. + There must be at least one private subnet in common between your EMR Studio and the Amazon EKS cluster that you use to register your virtual cluster. This ensures that your interactive endpoint appears as an option in your Studio workspaces, and activates connectivity from Studio to the Application Load Balancer. There are two methods that you can choose from to connect your Studio and your Amazon EKS cluster: + Create an Amazon EKS cluster and associate it with the subnets that belong to your EMR Studio. + Alternatively, create an EMR Studio and specify the private subnets for your Amazon EKS cluster. + Amazon EKS optimized ARM Amazon Linux AMIs are not supported for Amazon EMR on EKS interactive endpoints. + Only [Amazon EKS managed node groups](https://docs.aws.amazon.com/eks/latest/userguide/managed-node-groups.html) and Karpenter provisioned nodes are supported. ## Grant Cluster access for Amazon EMR on EKS Use the the steps in [Grant Cluster Access for Amazon EMR on EKS](https://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/setting-up-cluster-access.html) to grant Amazon EMR on EKS access to a specific namespace in your cluster. ## Activate IRSA on the Amazon EKS cluster To activate IAM roles for Service Accounts (IRSA) on the Amazon EKS cluster, follow the steps in [Enable IAM Roles for Service Accounts (IRSA)](https://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/setting-up-enable-IAM.html). ## Create IAM job execution role You must create an IAM role to run workloads on Amazon EMR on EKS interactive endpoints. We refer to this IAM role as the *job execution role* in this documentation. This IAM role gets assigned to both the interactive endpoint container and the actual execution containers that are created when you submit jobs with EMR Studio. You'll need the Amazon Resource Name (ARN) of your job execution role for Amazon EMR on EKS. There are two steps required for this: + [Create a IAM role for job execution.](https://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/creating-job-execution-role.html) + [Update the trust policy of the job execution role.](https://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/setting-up-trust-policy.html) ## Grant users access to Amazon EMR on EKS The IAM entity (user or role) that makes the request to create an interactive endpoint must also have the following Amazon EC2 and `emr-containers` permissions. Follow the steps described in [Grant users access to Amazon EMR on EKS](setting-up-iam.md) to grant these permissions that allow Amazon EMR on EKS to create, manage, and delete the security groups that limit inbound traffic to the load balancer of your interactive endpoint. The following `emr-containers` permissions allow the user to perform basic interactive endpoint operations: ``` "ec2:CreateSecurityGroup", "ec2:DeleteSecurityGroup", "ec2:AuthorizeSecurityGroupEgress", "ec2:AuthorizeSecurityGroupIngress", "ec2:RevokeSecurityGroupEgress", "ec2:RevokeSecurityGroupIngress" "emr-containers:CreateManagedEndpoint", "emr-containers:ListManagedEndpoints", "emr-containers:DescribeManagedEndpoint", "emr-containers:DeleteManagedEndpoint" ``` ## Register the Amazon EKS cluster with Amazon EMR Set up a virtual cluster and map it to the namespace in the Amazon EKS cluster where you want to run your jobs. For AWS Fargate-only clusters, use the same namespace for both the Amazon EMR on EKS virtual cluster and Fargate profile. For information on setting up an Amazon EMR on EKS virtual cluster, see [Register the Amazon EKS cluster with Amazon EMR](setting-up-registration.md). ## Deploy AWS Load Balancer Controller to Amazon EKS cluster An AWS Application Load Balancer is required for your Amazon EKS cluster. You only need to set up one Application Load Balancer controller per Amazon EKS cluster. For information on setting up the AWS Application Load Balancer controller, see [Installing the AWS Load Balancer Controller add-on](https://docs.aws.amazon.com/eks/latest/userguide/aws-load-balancer-controller.html) in the *Amazon EKS User Guide*. # Creating an interactive endpoint for your virtual cluster This topic describes a couple ways to create an interactive endpoint using the AWS Command Line Interface (AWS CLI) and includes details on available configuration parameters. ## Create an interactive endpoint with the `create-managed-endpoint` command Specify the parameters in the `create-managed-endpoint` command as follows. Amazon EMR on EKS supports creating interactive endpoints with Amazon EMR releases 6.7.0 and higher. ``` aws emr-containers create-managed-endpoint \ ‐‐type JUPYTER_ENTERPRISE_GATEWAY \ ‐‐virtual‐cluster‐id 1234567890abcdef0xxxxxxxx \ ‐‐name example-endpoint-name \ ‐‐execution-role-arn arn:aws:iam::444455556666:role/JobExecutionRole \ ‐‐release-label emr-6.9.0-latest \ ‐‐configuration-overrides '{ "applicationConfiguration": [{ "classification": "spark-defaults", "properties": { "spark.driver.memory": "2G" } }], "monitoringConfiguration": { "cloudWatchMonitoringConfiguration": { "logGroupName": "log_group_name", "logStreamNamePrefix": "log_stream_prefix" }, "persistentAppUI": "ENABLED", "s3MonitoringConfiguration": { "logUri": "s3://my_s3_log_location" } } }' ``` For more information, see [Parameters for creating an interactive endpoint](#parameters-for-creating). ## Create an interactive endpoint with specified parameters in a JSON file 1. Create a `create-managed-endpoint-request.json` file and specify the required parameters for your endpoint, as shown in the following JSON file: ``` { "name": "MY_TEST_ENDPOINT", "virtualClusterId": "MY_CLUSTER_ID", "type": "JUPYTER_ENTERPRISE_GATEWAY", "releaseLabel": "emr-6.9.0-latest", "executionRoleArn": "arn:aws:iam::444455556666:role/JobExecutionRole", "configurationOverrides": { "applicationConfiguration": [ { "classification": "spark-defaults", "properties": { "spark.driver.memory": "8G" } } ], "monitoringConfiguration": { "persistentAppUI": "ENABLED", "cloudWatchMonitoringConfiguration": { "logGroupName": "my_log_group", "logStreamNamePrefix": "log_stream_prefix" }, "s3MonitoringConfiguration": { "logUri": "s3://my_s3_log_location" } } } } ``` 1. Use the `create-managed-endpoint` command with a path to the `create-managed-endpoint-request.json` file that is stored locally or in Amazon S3. ``` aws emr-containers create-managed-endpoint \ ‐‐cli-input-json file://./create-managed-endpoint-request.json ‐‐region AWS-Region ``` ## Output of create interactive endpoint You should see the following output in the terminal. The output includes the name and identifier of your new interactive endpoint: ``` { "id": "1234567890abcdef0", "name": "example-endpoint-name", "arn": "arn:aws:emr-containers:us-west-2:111122223333:/virtualclusters/444455556666/endpoints/444455556666", "virtualClusterId": "111122223333xxxxxxxx" } ``` Running `aws emr-containers create-managed-endpoint` creates a self-signed certificate that allows HTTPS communication between EMR Studio and the interactive endpoint server. If you run `create-managed-endpoint` and haven't completed the prerequisites, Amazon EMR returns an error message with the actions that you must take to continue. ## Parameters for creating an interactive endpoint **Topics** + [Required parameters for interactive endpoints](#parameters-for-creating-required) + [Optional parameters for interactive endpoints](#parameters-for-creating-optional) ### Required parameters for interactive endpoints You must specify the following parameters when you create an interactive endpoint: **`‐‐type`** Use `JUPYTER_ENTERPRISE_GATEWAY`. This is the only supported type. **`‐‐virtual-cluster-id`** The identifier of the virtual cluster that you registered with Amazon EMR on EKS. **`‐‐name`** A descriptive name for the interactive endpoint that helps EMR Studio users select it from the dropdown list. **`‐‐execution-role-arn`** The Amazon Resource Name (ARN) of your IAM job execution role for Amazon EMR on EKS that was created as part of the prerequisites. **`‐‐release-label`** The release label of the Amazon EMR release to use for the endpoint. For example, `emr-6.9.0-latest`. Amazon EMR on EKS supports interactive endpoints with Amazon EMR releases 6.7.0 and higher. ### Optional parameters for interactive endpoints Optionally, you can also specify the following parameters when you create an interactive endpoint: **`‐‐configuration-overrides`** To override the default configurations for applications, supply a coonfiguration object. You can use a shorthand syntax to provide the configuration, or you can reference the configuration object in a JSON file. Configuration objects consist of a classification, properties, and optional nested configurations. Properties consist of the settings that you want to override in that file. You can specify multiple classifications for multiple applications in a single JSON object. The configuration classifications that are available vary by Amazon EMR on EKS release. For a list of configuration classifications that are available for each release of Amazon EMR on EKS, see [Amazon EMR on EKS releases](emr-eks-releases.md). In addition to the configuration classifications listed for each release, interactive endpoints bring in the additional classification `jeg-config`. For more information, see [Jupyter Enterprise Gateway (JEG) configuration options](jeg-config-options.md). # Configuring settings for interactive endpoints This section contains a series of topics that cover various configurations for interactive endpoints and pod settings. These give you the ability to monitor and troubleshoot failures, send log information to Amazon S3 or to Amazon CloudWatch Logs, or to create interactive endpoints where you specify custom pod templates. **Topics** + [Monitoring Spark jobs](monitoring-spark-jobs.md) + [Specifying custom pod templates with interactive endpoints](custom-pod-templates.md) + [Deploying a JEG pod to a node group](managed-endpoint-nodegroups-setup.md) + [Jupyter Enterprise Gateway (JEG) configuration options](jeg-config-options.md) + [Modifying PySpark session parameters](modify-pyspark-parameters.md) + [Custom kernel image with interactive endpoint](custom-kernel.md) # Monitoring Spark jobs So that you can monitor and troubleshoot failures, configure your interactive endpoints so that the jobs initiated with the endpoint can send log information to Amazon S3, Amazon CloudWatch Logs, or both. The following sections describe how to send Spark application logs to Amazon S3 for the Spark jobs that you launch with Amazon EMR on EKS interactive endpoints. **Configure IAM policy for Amazon S3 logs** Before your kernels can send log data to Amazon S3, the permissions policy for the job execution role must include the following permissions. Replace *amzn-s3-demo-destination-bucket* with the name of your logging bucket. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:PutObject", "s3:GetObject", "s3:ListBucket" ], "Resource": [ "arn:aws:s3:::amzn-s3-demo-bucket", "arn:aws:s3:::amzn-s3-demo-bucket/*" ], "Sid": "AllowS3Putobject" } ] } ``` ------ **Note** Amazon EMR on EKS can also create an S3 bucket. If an S3 bucket is not available, include the `s3:CreateBucket` permission in the IAM policy. After you've given your execution role the permissions it needs to send logs to the S3 bucket, your log data is sent to the following Amazon S3 locations. This happens when `s3MonitoringConfiguration` is passed in the `monitoringConfiguration` section of a `create-managed-endpoint` request. + **Driver logs** – `logUri/virtual-cluster-id/endpoints/endpoint-id/containers/spark-application-id/spark-application-id-driver/(stderr.gz/stdout.gz)` + **Executor logs** – `logUri/virtual-cluster-id/endpoints/endpoint-id/containers/spark-application-id/executor-pod-name-exec-/(stderr.gz/stdout.gz)` **Note** Amazon EMR on EKS doesn't upload the endpoint logs to your S3 bucket. # Specifying custom pod templates with interactive endpoints You can create interactive endpoints where you specify custom pod templates for drivers and executors. *Pod templates* are specifications that determine how to run each pod. You can use pod template files to define the configurations of driver or executor pods that Spark configurations don't support. Pod templates are currently supported in Amazon EMR releases 6.3.0 and greater. For more information about pod templates, see [Using pod templates](https://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/pod-templates.html) in the *Amazon EMR on EKS Development Guide*. The following example shows how to create an interactive endpoint with pod templates: ``` aws emr-containers create-managed-endpoint \ --type JUPYTER_ENTERPRISE_GATEWAY \ --virtual-cluster-id virtual-cluster-id \ --name example-endpoint-name \ --execution-role-arn arn:aws:iam::aws-account-id:role/EKSClusterRole \ --release-label emr-6.9.0-latest \ --configuration-overrides '{ "applicationConfiguration": [ { "classification": "spark-defaults", "properties": { "spark.kubernetes.driver.podTemplateFile": "path/to/driver/template.yaml", "spark.kubernetes.executor.podTemplateFile": "path/to/executor/template.yaml" } }] }' ``` # Deploying a JEG pod to a node group JEG (Jupyter Enterprise Gateway) pod placement is a feature that allows you to deploy an interactive endpoint on a specific node group. With this feature, you can configure settings such as `instance type` for the interactive endpoint. ## Associating a JEG pod to a managed node group The following configuration property allows you to specify the name of a managed node group on your Amazon EKS cluster where the JEG pod will be deployed. ``` //payload --configuration-overrides '{ "applicationConfiguration": [ { "classification": "endpoint-configuration", "properties": { "managed-nodegroup-name": NodeGroupName } } ] }' ``` A node group must have the Kubernetes label `for-use-with-emr-containers-managed-endpoint-ng=NodeGroupName` attached to all nodes that are part of the node group. To list all nodes of a node group that have this tag, use the following command: ``` kubectl get nodes --show-labels | grep for-use-with-emr-containers-managed-endpoint-ng=NodeGroupName ``` If the output of the command above doesn't return nodes that are part of your managed node group, then there are no nodes in the node group that have the `for-use-with-emr-containers-managed-endpoint-ng=NodeGroupName` Kubernetes label attached. In this case, follow the steps below to attach that label to the nodes in your node group. 1. Use the following command to add the `for-use-with-emr-containers-managed-endpoint-ng=NodeGroupName` Kubernetes label to all nodes in a managed node group `NodeGroupName`: ``` kubectl label nodes --selector eks:nodegroup-name=NodeGroupName for-use-with-emr-containers-managed-endpoint-ng=NodeGroupName ``` 1. Verify that the nodes were labeled correctly using the following command: ``` kubectl get nodes --show-labels | grep for-use-with-emr-containers-managed-endpoint-ng=NodeGroupName ``` A managed node group must be associated with an Amazon EKS cluster’s security group, which is usually the case if you created your cluster and managed node group using `eksctl`. You can verify this in the AWS console using the following steps. 1. Go to your cluster in the Amazon EKS console. 1. Go to the networking tab of your cluster and note down the cluster security group. 1. Go to the compute tab of your cluster and click on the managed node group name. 1. Under the **Details** tab of the managed node group, verify that the cluster security group that you noted previously is listed under **Security groups**. If the managed node group is not attached to the Amazon EKS cluster security group, you need to attach the `for-use-with-emr-containers-managed-endpoint-sg=ClusterName/NodeGroupName` tag to the node group security group. Use the steps below to attach this tag. 1. Go to the Amazon EC2 console and click on security groups on the left navigation pane. 1. Select your managed node group’s security group by clicking the checkbox. 1. Under the **Tags** tab, add the tag `for-use-with-emr-containers-managed-endpoint-sg=ClusterName/NodeGroupName` using the **Manage tags** button. ## Associating a JEG pod to a self-managed node group The following configuration property allows you to specify the name of a self-managed or unmanaged node group on the Amazon EKS cluster where the JEG pod will be deployed. ``` //payload --configuration-overrides '{ "applicationConfiguration": [ { "classification": "endpoint-configuration", "properties": { "self-managed-nodegroup-name": NodeGroupName } } ] }' ``` The node group must have `for-use-with-emr-containers-managed-endpoint-ng=NodeGroupName` Kubernetes label attached to all nodes that are part of the node group. To list all the nodes of a node group that have this tag, use the following command: ``` kubectl get nodes --show-labels | grep for-use-with-emr-containers-managed-endpoint-ng=NodeGroupName ``` If the output of the command above doesn't return nodes that are part of your self-managed node group, then there are no nodes in the node group that have the `for-use-with-emr-containers-managed-endpoint-ng=NodeGroupName` Kubernetes label attached. In this case, follow the steps below to attach that label to the nodes in your node group. 1. If you created the self-managed node group using `eksctl`, then use the following command to add the `for-use-with-emr-containers-managed-endpoint-ng=NodeGroupName` Kubernetes label to all nodes in the self-managed node group `NodeGroupName` at once. ``` kubectl label nodes --selector alpha.eksctl.io/nodegroup-name=NodeGroupName for-use-with-emr-containers-managed-endpoint-ng=NodeGroupName ``` If you didn’t use `eksctl` to create the self-managed node group, then you will need to replace the selector in the above command to a different Kubernetes label that is attached to all the nodes of the node group. 1. Use the following command to verify that the nodes were labeled correctly: ``` kubectl get nodes --show-labels | grep for-use-with-emr-containers-managed-endpoint-ng=NodeGroupName ``` The security group for the self-managed node group must have the `for-use-with-emr-containers-managed-endpoint-sg=ClusterName/NodeGroupName` tag attached. Use the following steps to attach the tag to the security group from the AWS Management Console. 1. Navigate to the Amazon EC2 console. Select **Security groups** on the left navigation pane. 1. Select the checkbox next to the security group for your self-managed node group. 1. Under the **Tags** tab, use the **Manage tags** button to add the tag `for-use-with-emr-containers-managed-endpoint-sg=ClusterName/NodeGroupName`. Replace `ClusterName` and `NodeGroupName` with appropriate values. ## Associating a JEG pod to a managed node group with On-Demand instances You can also define additional labels, known as *Kubernetes label selectors*, to specify additional constraints or restrictions to run an interactive endpoint on a given node or node group. The following example shows how to use On-Demand Amazon EC2 instances for a JEG pod. ``` --configuration-overrides '{ "applicationConfiguration": [ { "classification": "endpoint-configuration", "properties": { "managed-nodegroup-name": NodeGroupName, "node-labels": "eks.amazonaws.com/capacityType:ON_DEMAND" } } ] }' ``` **Note** You can only use the `node-labels` property with either with a `managed-nodegroup-name` or `self-managed-nodegroup-name` property. # Jupyter Enterprise Gateway (JEG) configuration options Amazon EMR on EKS uses Jupyter Enterprise Gateway (JEG) to turn on interactive endpoints. You can set the following values for the allow-listed JEG configurations when you create the endpoint. + **`RemoteMappingKernelManager.cull_idle_timeout`** – Timeout in seconds (integer), after which a kernel is considered idle and ready to be culled. Values of `0` or lower deactivate culling. Short timeouts might result in kernels being culled for users with poor network connections. + **`RemoteMappingKernelManager.cull_interval`** – The interval in seconds (integer) on which to check for idle kernels that exceed the cull timeout value. # Modifying PySpark session parameters Starting with Amazon EMR on EKS release 6.9.0, in Amazon EMR Studio you can adjust the Spark configuration associated with a PySpark session by executing the `%%configure` magic command in the EMR notebook cell. The following example shows a sample payload that you can use to modify memory, cores, and other properties for the Spark driver and executor. For the `conf` settings, you can configure any Spark configuration mentioned in the [Apache Spark configuration documentation](https://spark.apache.org/docs/latest/configuration.html). ``` %%configure -f { "driverMemory": "16G", "driverCores": 4, "executorMemory" : "32G", "executorCores": 2, "conf": { "spark.dynamicAllocation.maxExecutors" : 10, "spark.dynamicAllocation.minExecutors": 1 } } ``` The following example shows a sample payload that you can use to add files, pyFiles, and jar dependencies to a Spark runtime. ``` %%configure -f { "files": "s3://amzn-s3-demo-bucket-emr-eks/sample_file.txt", "pyFiles": : "path-to-python-files", "jars" : "path-to-jars } ``` # Custom kernel image with interactive endpoint To ensure that you have the correct dependencies for your application when you run interactive workloads from Amazon EMR Studio, you can customize Docker images for interactive endpoints and run customized base kernel images. To create an interactive endpoint and connect it with a custom Docker image, perform the following steps. **Note** You can only override base images. You can't add new kernel image types. 1. **Create and publish a customized Docker image.** The base image contains the Spark runtime and the notebook kernels that run with it. To create the image, you can follow steps 1 through 4 in [How to customize Docker images](docker-custom-images-steps.md). In step 1, the base image URI in your Docker file must use `notebook-spark` in place of `spark`. ``` ECR-registry-account.dkr.ecr.Region.amazonaws.com/notebook-spark/container-image-tag ``` For more information on how to select AWS Regions and container image tags, see [Details for selecting a base image URI](docker-custom-images-tag.md). 1. **Create an interactive endpoint that can be used with the custom image.** 1. Create a JSON file `custom-image-managed-endpoint.json` with the following contents. This example uses Amazon EMR release 6.9.0. **Example** ``` { "name": "endpoint-name", "virtualClusterId": "virtual-cluster-id", "type": "JUPYTER_ENTERPRISE_GATEWAY", "releaseLabel": "emr-6.9.0-latest", "executionRoleArn": "execution-role-arn", "configurationOverrides": { "applicationConfiguration": [ { "classification": "jupyter-kernel-overrides", "configurations": [ { "classification": "python3", "properties": { "container-image": "123456789012.dkr.ecr.us-west-2.amazonaws.com/custom-notebook-python:latest" } }, { "classification": "spark-python-kubernetes", "properties": { "container-image": "123456789012.dkr.ecr.us-west-2.amazonaws.com/custom-notebook-spark:latest" } } ] } ] } } ``` 1. Create an interactive endpoint with the configurations specified in the JSON file as shown in the following example. For more information, see [Create an interactive endpoint with the `create-managed-endpoint` command](create-managed-endpoint.md#create-using-json-file). ``` aws emr-containers create-managed-endpoint --cli-input-json custom-image-managed-endpoint.json ``` 1. **Connect to the interactive endpoint via EMR Studio.** For more information and steps to complete, see [Connecting from Studio](https://emr-on-eks.workshop.aws/advanced/emr-studio/connecting-from-studio.html) in the Amazon EMR on EKS section of the AWS Workshop Studio docs. # Monitoring interactive endpoints With Amazon EMR on EKS version 6.10 and later, interactive endpoints emit Amazon CloudWatch metrics for monitoring and troubleshooting kernel lifecycle operations. Metrics are triggered by interactive clients, such as EMR Studio or self-hosted Jupyter notebooks. Each of the operations supported by interactive endpoints have metrics associated with them. The operations are modeled as dimensions to each metric, as shown in the table below. Metrics emitted by interactive endpoints are visible under a custom namespace, EMRContainers, in your account. | Metric | Description | Unit | | --- | --- | --- | | RequestCount | Cumulative number of requests of an operation processed by the interactive endpoint. | Count | | RequestLatency | The time from when a request arrived at the interactive endpoint and a response was sent by the interactive endpoint. | Millisecond | | 4XXError | Emitted when a request for an operation results in a 4xx error during processing. | Count | | 5XXError | Emitted when a request for an operation results in a 5Xxx server side error. | Count | | KernelLaunchSuccess | Applicable only for the CreateKernel operation. It indicates the cumulative number of kernel launches that were successful up to and including this request. | Count | | KernelLaunchFailure | Applicable only for the CreateKernel operation. It indicates the cumulative number of kernel launch failures up until and including this request. | Count | Each interactive endpoint metric has the following dimensions attached to it: + **`ManagedEndpointId`** – Identifier for the interactive endpoint + **`OperationName`** – The operation triggered by the interactive client Possible values for the **`OperationName`** dimension are shown in the following table: | `operationName` | Operation description | | --- | --- | | `CreateKernel` | Request that the interactive endpoint start a kernel. | | `ListKernels` | Request that the interactive endpoint list the kernels that have been previously started using the same session token. | | `GetKernel` | Request that the interactive endpoint get details about a specific kernel that has been previously started. | | `ConnectKernel` | Request that the interactive endpoint establish connectivity between the notebook client and the kernel. | | `ConfigureKernel` | Publish `%%configure magic request` on a pyspark kernel. | | `ListKernelSpecs` | Request that the interactive endpoint list the available kernel specs. | | `GetKernelSpec` | Request that the interactive endpoint get the kernel specs of a kernel that has been previously launched. | | `GetKernelSpecResource` | Request that the interactive endpoint get specific resources associated with the kernel specs that have been previously launched. | ## Examples ### To access the total number of kernels launched for an interactive endpoint on a given day: 1. Select the custom namespace: `EMRContainers` 1. Select your `ManagedEndpointId`, `OperationName – CreateKernel` 1. `RequestCount` metric with the statistic `SUM` and period `1 day` will provide all the kernel launch requests made in the last 24 hours. 1. KernelLaunchSuccess metric with statistic `SUM` and period `1 day` will provide all the successful kernel launch requests made in the last 24 hours. ### To access the number of kernel failures for an interactive endpoint on a given day: 1. Select the custom namespace: EMRContainers 1. Select your `ManagedEndpointId`, `OperationName – CreateKernel` 1. `KernelLaunchFailure` metric with statistic `SUM` and period `1 day` will provide all the failed kernel launch requests made in the last 24 hours. You can also select the `4XXError` and `5XXError` metric to know what kind of kernel launch failure happened. # Using self-hosted Jupyter notebooks You can host and manage Jupyter or JupyterLab notebooks on an Amazon EC2 instance or on your own Amazon EKS cluster as a *self-hosted Jupyter notebook*. You can then run interactive workloads with your self-hosted Jupyter notebooks. The following sections walk through the process to set up and deploy a self-hosted Jupyter notebook on an Amazon EKS cluster. **Topics** + [Create a security group](#managed-endpoints-self-hosted-security) + [Create an Amazon EMR on EKS interactive endpoint](#managed-endpoints-self-hosted-create-me) + [Retrieve the gateway server URL of your interactive endpoint](#managed-endpoints-self-hosted-gateway) + [Retrieve an auth token to connect to the interactive endpoint](#managed-endpoints-self-hosted-auth) + [Example: Deploy a JupyterLab notebook](#managed-endpoints-self-hosted-example) + [Delete a self-hosted Jupyter notebook](#managed-endpoints-self-hosted-cleanup) ## Create a security group Before you can create an interactive endpoint and run a self-hosted Jupyter or JupyterLab notebook, you must create a security group to control the traffic between your notebook and the interactive endpoint. To use the Amazon EC2 console or Amazon EC2 SDK to create the security group, refer to the steps in [Create a security group](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/working-with-security-groups.html#creating-security-group) in the *Amazon EC2 User Guide*. You should create the security group in the VPC where you want to deploy your notebook server. To follow the example in this guide, use the same VPC as your Amazon EKS cluster. If you want to host your notebook in a VPC that is different from the VPC for your Amazon EKS cluster, you might need to create a peering connection between those two VPCs. For steps to create a peering connection between two VPCs, see [Create a VPC peering connection](https://docs.aws.amazon.com/vpc/latest/peering/create-vpc-peering-connection.html) in the Amazon VPC Getting Started Guide. You need the ID for the security group to [create an Amazon EMR on EKS interactive endpoint](https://docs.aws.amazon.com/) in the next step. ## Create an Amazon EMR on EKS interactive endpoint After you create security group for your notebook, use the steps provided in [Creating an interactive endpoint for your virtual cluster](create-managed-endpoint.md) to create an interactive endpoint. You must provide the security group ID that you created for your notebook in [Create a security group](#managed-endpoints-self-hosted-security). Insert the security ID in place of *your-notebook-security-group-id* in the following configuration override settings: ``` --configuration-overrides '{ "applicationConfiguration": [ { "classification": "endpoint-configuration", "properties": { "notebook-security-group-id": "your-notebook-security-group-id" } } ], "monitoringConfiguration": { ...' ``` ## Retrieve the gateway server URL of your interactive endpoint After you create an interactive endpoint, retrieve the gateway server URL with the `describe-managed-endpoint` command in the AWS CLI. You need this URL to connect your notebook to the endpoint. The gateway server URL is a private endpoint. ``` aws emr-containers describe-managed-endpoint \ --region region \ --virtual-cluster-id virtualClusterId \ --id endpointId ``` Initially, your endpoint is in the **CREATING** state. After a few minutes, it transitions to the **ACTIVE** state. When the endpoint is **ACTIVE**, it's ready to use. Take note of the `serverUrl` attribute that the `aws emr-containers describe-managed-endpoint` command returns from the active endpoint. You need this URL to connect your notebook to the endpoint when you [deploy your self-hosted Jupyter or JupyterLab notebook](https://docs.aws.amazon.com/). ## Retrieve an auth token to connect to the interactive endpoint To connect to an interactive endpoint from a Jupyter or JupyterLab notebook, you must generate a session token with the `GetManagedEndpointSessionCredentials` API. The token acts as proof of authentication to connect to the interactive endpoint server. The following command is explained in more detail with an output example below. ``` aws emr-containers get-managed-endpoint-session-credentials \ --endpoint-identifier endpointArn \ --virtual-cluster-identifier virtualClusterArn \ --execution-role-arn executionRoleArn \ --credential-type "TOKEN" \ --duration-in-seconds durationInSeconds \ --region region ``` **`endpointArn`** The ARN of your endpoint. You can find the ARN in the result of a `describe-managed-endpoint` call. **`virtualClusterArn`** The ARN of the virtual cluster. **`executionRoleArn`** The ARN of the execution role. **`durationInSeconds`** The duration in seconds for which the token is valid. The default duration is 15 minutes (`900`), and the maximum is 12 hours (`43200`). **`region` ** The same region as your endpoint. Your output should resemble the following example. Take note of the `session-token` value that you will use when you [deploy your self-hosted Jupyter or JupyterLab notebook](https://docs.aws.amazon.com/). ``` { "id": "credentialsId", "credentials": { "token": "session-token" }, "expiresAt": "2022-07-05T17:49:38Z" } ``` ## Example: Deploy a JupyterLab notebook Once you've completed the steps above, you can try this example procedure to deploy a JupyterLab notebook into the Amazon EKS cluster with your interactive endpoint. 1. Create a namespace to run the notebook server. 1. Create a file locally, `notebook.yaml`, with the following contents. The file contents are described below. ``` apiVersion: v1 kind: Pod metadata: name: jupyter-notebook namespace: namespace spec: containers: - name: minimal-notebook image: jupyter/all-spark-notebook:lab-3.1.4 # open source image ports: - containerPort: 8888 command: ["start-notebook.sh"] args: ["--LabApp.token=''"] env: - name: JUPYTER_ENABLE_LAB value: "yes" - name: KERNEL_LAUNCH_TIMEOUT value: "400" - name: JUPYTER_GATEWAY_URL value: "serverUrl" - name: JUPYTER_GATEWAY_VALIDATE_CERT value: "false" - name: JUPYTER_GATEWAY_AUTH_TOKEN value: "session-token" ``` If you are deploying Jupyter notebook to a Fargate-only cluster, label the Jupyter pod with a `role` label as shown in the following example: ``` ... metadata: name: jupyter-notebook namespace: default labels: role: example-role-name-label spec: ... ``` **`namespace`** The Kubernetes namespace that the notebook deploys into. **`serverUrl`** The `serverUrl` attribute that the `describe-managed-endpoint` command returned in [Retrieve the gateway server URL of your interactive endpoint](#managed-endpoints-self-hosted-gateway). **`session-token`** The `session-token` attribute that the `get-managed-endpoint-session-credentials` command returned in [Retrieve an auth token to connect to the interactive endpoint](#managed-endpoints-self-hosted-auth). **`KERNEL_LAUNCH_TIMEOUT`** The amount of time in seconds that the interactive endpoint waits for the kernel to come to **RUNNING** state. Ensure sufficient time for kernel launch to complete by setting the kernel launch timeout to an appropriate value (maximum 400 seconds). **`KERNEL_EXTRA_SPARK_OPTS`** Optionally, you can pass additional Spark configurations for the Spark kernels. Set this environment variable with the values as the Spark configuration property as shown in the following example: ``` - name: KERNEL_EXTRA_SPARK_OPTS value: "--conf spark.driver.cores=2 --conf spark.driver.memory=2G --conf spark.executor.instances=2 --conf spark.executor.cores=2 --conf spark.executor.memory=2G --conf spark.dynamicAllocation.enabled=true --conf spark.dynamicAllocation.shuffleTracking.enabled=true --conf spark.dynamicAllocation.minExecutors=1 --conf spark.dynamicAllocation.maxExecutors=5 --conf spark.dynamicAllocation.initialExecutors=1 " ``` 1. Deploy the pod spec to your Amazon EKS cluster: ``` kubectl apply -f notebook.yaml -n namespace ``` This will start up a minimal JupyterLab notebook connected to your Amazon EMR on EKS interactive endpoint. Wait until the pod is **RUNNING**. You can check its status with the following command: ``` kubectl get pod jupyter-notebook -n namespace ``` When the pod is ready, the `get pod` command returns output similar to this: ``` NAME READY STATUS RESTARTS AGE jupyter-notebook 1/1 Running 0 46s ``` 1. Attach the notebook security group to the node where the notebook is scheduled. 1. First, identify the node where `jupyter-notebook` pod is scheduled with the `describe pod` command. ``` kubectl describe pod jupyter-notebook -n namespace ``` 1. Open the Amazon EKS console at [https://console.aws.amazon.com/eks/home\$1/clusters](https://console.aws.amazon.com/eks/home#/clusters). 1. Navigate to the **Compute** tab for your Amazon EKS cluster and select the node identified by the `describe pod` command. Select the instance ID for the node. 1. From the **Actions** menu, select **Security** > **Change security groups** to attach the security group that you created in [Create a security group](#managed-endpoints-self-hosted-security). 1. If you are deploying Jupyter notebook pod on AWS Fargate, create a []() to apply to the Jupyter notebook pod with the role label: ``` cat >my-security-group-policy.yaml < val x = random * 2 - 1 val y = random * 2 - 1 if (x*x + y*y <= 1) 1 else 0 }.reduce(_ + _) println(s"Pi is roughly ${4.0 * count / (n - 1)}") session.stop() ``` ![\[Screenshot of example Scala notebook code in JupyterLab.\]](http://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/images/emr-on-eks-Jupyter-notebook-scala-program.png) ## Delete a self-hosted Jupyter notebook When you're ready to delete your self-hosted notebook, you can also delete the interactive endpoint and security group, too. Perform the actions in the following order: 1. Use the following command to delete the `jupyter-notebook` pod: ``` kubectl delete pod jupyter-notebook -n namespace ``` 1. Then, delete your interactive endpoint with the `delete-managed-endpoint` command. For steps to delete an interactive endpoint, see [Delete an interactive endpoint](delete-managed-endpoint.md). Initially, your endpoint will be in the **TERMINATING** state. Once all resources have been cleaned up, it transitions to the **TERMINATED** state. 1. If you don’t plan to use the notebook security group that you created in [Create a security group](#managed-endpoints-self-hosted-security) for other Jupyter notebook deployments, you can delete it. See [Delete a security group](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/working-with-security-groups.html#deleting-security-group) in the Amazon EC2 User Guide for more information. # Getting information about interactive endpoints with CLI commands This topic covers the supported operations on an interactive endpoint other than [`create-managed-endpoint`](create-managed-endpoint.md). ## Fetch interactive endpoint details After you create an interactive endpoint, you can retrieve its details using the `describe-managed-endpoint` AWS CLI command. Insert your own values for *managed-endpoint-id*, *virtual-cluster-id*, and *region*: ``` aws emr-containers describe-managed-endpoint ‐‐id managed-endpoint-id \ ‐‐virtual-cluster-id virtual-cluster-id ‐‐region region ``` The output looks similar to the following, with the specified endpoint, such as ARN, ID, and name. ``` { "id": "as3ys2xxxxxxx", "name": "endpoint-name", "arn": "arn:aws:emr-containers:us-east-1:1828xxxxxxxx:/virtualclusters/lbhl6kwwyoxxxxxxxxxxxxxxx/endpoints/as3ysxxxxxxxx", "virtualClusterId": "lbhl6kwwyoxxxxxxxxxxxxxxx", "type": "JUPYTER_ENTERPRISE_GATEWAY", "state": "ACTIVE", "releaseLabel": "emr-6.9.0-latest", "executionRoleArn": "arn:aws:iam::1828xxxxxxxx:role/RoleName", "certificateAuthority": { "certificateArn": "arn:aws:acm:us-east-1:1828xxxxxxxx:certificate/zzzzzzzz-e59b-4ed0-aaaa-bbbbbbbbbbbb", "certificateData": "certificate-data" }, "configurationOverrides": { "applicationConfiguration": [ { "classification": "spark-defaults", "properties": { "spark.driver.memory": "8G" } } ], "monitoringConfiguration": { "persistentAppUI": "ENABLED", "cloudWatchMonitoringConfiguration": { "logGroupName": "log-group-name", "logStreamNamePrefix": "log-stream-name-prefix" }, "s3MonitoringConfiguration": { "logUri": "s3-bucket-name" } } }, "serverUrl": "https://internal-k8s-namespace-ingressa-aaaaaaaaaa-zzzzzzzzzz.us-east-1.elb.amazonaws.com:18888 (https://internal-k8s-nspluto-ingressa-51e860abbd-1620715833.us-east-1.elb.amazonaws.com:18888/)", "createdAt": "2022-09-19T12:37:49+00:00", "securityGroup": "sg-aaaaaaaaaaaaaa", "subnetIds": [ "subnet-11111111111", "subnet-22222222222", "subnet-33333333333" ], "stateDetails": "Endpoint created successfully. It took 3 Minutes 15 Seconds", "tags": {} } ``` ## List all interactive endpoints associated with a virtual cluster Use the `list-managed-endpoints` AWS CLI command to fetch a list of all the interactive endpoints associated with a specified virtual cluster. Replace `virtual-cluster-id` with the ID of your virtual cluster. ``` aws emr-containers list-managed-endpoints ‐‐virtual-cluster-id virtual-cluster-id ``` The output of the `list-managed-endpoint` command is shown below: ``` { "endpoints": [{ "id": "as3ys2xxxxxxx", "name": "endpoint-name", "arn": "arn:aws:emr-containers:us-east-1:1828xxxxxxxx:/virtualclusters/lbhl6kwwyoxxxxxxxxxxxxxxx/endpoints/as3ysxxxxxxxx", "virtualClusterId": "lbhl6kwwyoxxxxxxxxxxxxxxx", "type": "JUPYTER_ENTERPRISE_GATEWAY", "state": "ACTIVE", "releaseLabel": "emr-6.9.0-latest", "executionRoleArn": "arn:aws:iam::1828xxxxxxxx:role/RoleName", "certificateAuthority": { "certificateArn": "arn:aws:acm:us-east-1:1828xxxxxxxx:certificate/zzzzzzzz-e59b-4ed0-aaaa-bbbbbbbbbbbb", "certificateData": "certificate-data" }, "configurationOverrides": { "applicationConfiguration": [{ "classification": "spark-defaults", "properties": { "spark.driver.memory": "8G" } }], "monitoringConfiguration": { "persistentAppUI": "ENABLED", "cloudWatchMonitoringConfiguration": { "logGroupName": "log-group-name", "logStreamNamePrefix": "log-stream-name-prefix" }, "s3MonitoringConfiguration": { "logUri": "s3-bucket-name" } } }, "serverUrl": "https://internal-k8s-namespace-ingressa-aaaaaaaaaa-zzzzzzzzzz.us-east-1.elb.amazonaws.com:18888 (https://internal-k8s-nspluto-ingressa-51e860abbd-1620715833.us-east-1.elb.amazonaws.com:18888/)", "createdAt": "2022-09-19T12:37:49+00:00", "securityGroup": "sg-aaaaaaaaaaaaaa", "subnetIds": [ "subnet-11111111111", "subnet-22222222222", "subnet-33333333333" ], "stateDetails": "Endpoint created successfully. It took 3 Minutes 15 Seconds", "tags": {} }] } ``` # Delete an interactive endpoint To delete an interactive endpoint associated with an Amazon EMR on EKS virtual cluster, use the `delete-managed-endpoint` AWS CLI command. When you delete an interactive endpoint, Amazon EMR on EKS removes the default security groups that were created for that endpoint. Specify values for the following parameters to the command: + **‐‐id:** The identifier of the interactive endpoint that you want to delete. + **‐‐virtual-cluster-id** – The identifier of the virtual cluster associated with the interactive endpoint that you want to delete. This is the same virtual cluster ID that was specified when the interactive endpoint was created. ``` aws emr-containers delete-managed-endpoint ‐‐id managed-endpoint-id ‐‐virtual-cluster-id virtual-cluster-id ``` The command returns output similar to the following to confirm that you deleted the interactive endpoint: ``` { "id":"8gai4l4exxxxx", "virtualClusterId":"0b0qvauoy3ch1nqodxxxxxxxx" } ```