Setup for Upgrade Agent - Amazon EMR
PrerequisitesSetup Resources

Setup for Upgrade Agent

Note

The Apache Spark Upgrade Agent uses cross-region inference to process natural language requests and generate responses. For more details please refer to this page on Cross-region processing for the Apache Spark Upgrade Agent. The Amazon SageMaker Unified Studio MCP server is in preview and is subject to change.

Prerequisites

Before we begin our setup process for integration with Kiro CLI, make sure you have the following installed on your workstation:

Setup Resources

You can use an CloudFormation template to setup the resource for the MCP server. These templates are samples that you should modify to meet your requirements. The template creates the following resources for the upgrade process:

  • IAM role which has permissions to call MCP Server and required permissions for upgrade process for the underlying EMR platform.

  • Amazon S3 staging bucket used to upload upgrade artifacts and optional KMS key for Amazon S3 encryption.

Choose one of the Launch Stack buttons in the following table. This launches the stack on the CloudFormation console in the respective region.

Region Launch
US East (N. Virginia)
US East (Ohio)
US West (Oregon)
Europe (Ireland)
Asia Pacific (Tokyo)
Asia Pacific (Singapore)
Asia Pacific (Sydney)
Canada (Central)
South America (São Paulo)
Europe (Frankfurt)
Europe (Stockholm)
Europe (London)
Europe (Paris)
Asia Pacific (Seoul)
Asia Pacific (Mumbai)

Proceed to the Specify stack details page, enter the Stack name. Enter additional information under Parameters. Provide the following information and proceed to create the stack.

  • SparkUpgradeIAMRoleName - Name of the IAM role to use for Spark upgrade process

  • EnableEMREC2 - Enable EMR-EC2 upgrade permissions (default: true)

  • EnableEMRServerless - Enable EMR-Serverless upgrade permissions (default: true)

  • StagingBucketPath - Amazon S3 path for staging artifacts (e.g., s3://my-bucket/spark-upgrade or my-bucket/spark-upgrade). Leave empty to auto-generate a new bucket

  • UseS3Encryption - Enable KMS encryption for Amazon S3 staging bucket (default: false, set to true to use KMS encryption instead of default S3 encryption)

  • S3KmsKeyArn - (Optional) ARN of existing KMS key for Amazon S3 bucket encryption. Only used if UseS3Encryption is true and you have an existing bucket with a KMS key

  • CloudWatchKmsKeyArn - (Optional) ARN of existing KMS key for CloudWatch Logs encryption (EMR Serverless only, leave empty for default encryption)

  • EMRServerlessS3LogPath - (Optional) S3 path where EMR-Serverless application logs are stored (e.g., s3://my-bucket/emr-serverless-logs or my-bucket/emr-serverless-logs). When provided, grants the IAM role read access to these logs for analysis. Only used when EnableEMRServerless is true

  • ExecutionRoleToGrantS3Access - (Optional) IAM Role Name or ARN of existing EMR-EC2/EMR-Serverless execution role to grant Amazon S3 staging bucket access. Only applies when a new staging bucket is created. Useful for granting EMR job execution roles access to the staging bucket. Supports both simple role names and ARNs with paths.

Open the Outputs tab and copy the 1-line instruction upgrade_local_env_setup_instruction and execute it in your local environment. Example 1-line instruction:

export SMUS_MCP_REGION=us-east-1 && export IAM_ROLE=arn:aws:iam::111122223333:role/spark-upgrade-role-xxxxxx && export STAGING_BUCKET=amzn-s3-spark-upgrade-demo-bucket

Then run the following command locally to setup the IAM profile and MCP server configuration

# Step 1: Configure AWS CLI Profile  
aws configure set profile.spark-upgrade-profile.role_arn ${IAM_ROLE}  
aws configure set profile.spark-upgrade-profile.source_profile <AWS CLI Profile to assume the IAM role - ex: default>  
aws configure set profile.spark-upgrade-profile.region ${SMUS_MCP_REGION}  
  
# Step 2: if you are using Kiro CLI, use the following command to add the MCP configuration  
kiro-cli-chat mcp add \  
    --name "spark-upgrade" \  
    --command "uvx" \  
    --args "[\"mcp-proxy-for-aws@latest\",\"https://sagemaker-unified-studio-mcp.${SMUS_MCP_REGION}.api.aws/spark-upgrade/mcp\", \"--service\", \"sagemaker-unified-studio-mcp\", \"--profile\", \"spark-upgrade-profile\", \"--region\", \"${SMUS_MCP_REGION}\", \"--read-timeout\", \"180\"]" \  
    --timeout 180000\  
    --scope global

This should update ~/.kiro/settings/mcp.json to include the MCP server configuration as below.

{  
  "mcpServers": {  
    "spark-upgrade": {  
      "type": "stdio",  
      "command": "uvx",  
      "args": [  
        "mcp-proxy-for-aws@latest",  
        "https://sagemaker-unified-studio-mcp.us-east-1.api.aws/spark-upgrade/mcp",  
        "--service",  
        "sagemaker-unified-studio-mcp",  
        "--profile",  
        "spark-upgrade-profile",  
        "--region",  
        "us-east-1",  
        "--read-timeout",  
        "180"  
      ],  
      "timeout": 180000,  
      "disabled": false  
    }  
  }  
}

See Using the Upgrade Agent for the configuration guidance for different MCP clients like Kiro, Cline and GitHub CoPilot.