What is Apache Spark Upgrade Agent for Amazon EMR
Introduction
The Apache Spark Upgrade Agent for Amazon EMR is a conversational AI capability that accelerates Apache Spark version upgrades for your EMR applications. Traditional Spark upgrades require months of engineering effort to analyze API changes, resolve dependency conflicts, and validate functional correctness. The agent simplifies the upgrade process through natural language prompts, automated code transformation, and data quality validation.
You can use the agent to upgrade PySpark and Scala applications running on Amazon EMR on EC2 and Amazon EMR Serverless. The agent analyzes your code, identifies required changes, and performs automated transformations while maintaining your approval control over all modifications.
Architecture Overview
The upgrade agent has three main components: any MCP-compatible AI Assistant in your development environment for interaction, the
MCP Proxy for AWS
The AI assistant will orchestrate the upgrade using specialized tools provided by the MCP server following these steps:
-
Planning: The agent analyzes your project structure and generates or revises an upgrade plan that guides the end-to-end Spark upgrade process.
-
Compile and Build: The agent updates the build environment and dependencies, compiles the project, and iteratively fixes build and test failures.
-
Spark code edit tools: The agent applies targeted code updates to resolve Spark version incompatibilities, fixing both build-time and runtime errors.
-
Execute & Validation: The agent submits remote validation jobs to EMR, monitors execution and logs, and iteratively fixes runtime and data-quality issues.
-
Observability: The agent tracks upgrade progress using EMR observability tools and allows users to view upgrade analyses and status at any time.
Please refer to Using Spark Upgrade Tools for a list of major tools for each steps.
Topics
