Using Python virtual environments with AWS Glue

Starting with AWS Glue 5.0, you can run your ETL jobs in a Python virtual environment (venv). Virtual environments remove runtime dependency resolution from your job runs, ensure that each run uses the same packages, and prevent failures caused by upstream package changes.

AWS Glue supports two ways to use a virtual environment:

This topic describes how to migrate jobs that use --additional-python-modules to either approach. For information about other methods of managing Python dependencies, see Using Python libraries with AWS Glue.

Key differences from --additional-python-modules

The following table compares --additional-python-modules with a manually built virtual environment.

Choosing an approach

Use the following table to decide which approach fits your job.

Using a service-generated virtual environment with Amazon S3 caching

Starting with AWS Glue 6.0, you can use the --python-virtual-env-storage-prefix parameter to have AWS Glue build the virtual environment and cache it in Amazon S3. This approach combines the simplicity of --additional-python-modules with the performance benefit of a cached virtual environment.

How it works

When you provide --python-virtual-env-storage-prefix, AWS Glue does the following:

Differences from a manually built virtual environment

The following table compares the service-generated virtual environment with the manually built approach.

Configuring a service-generated virtual environment

The --python-virtual-env-storage-prefix parameter specifies the Amazon S3 location where AWS Glue stores the virtual environment that it builds, in the form s3://path/. AWS Glue caches the virtual environment at this prefix on the first job run and reuses it on later runs.

To enable a service-generated virtual environment, add the --python-virtual-env-storage-prefix parameter to your job and keep your existing --additional-python-modules parameter.

"--additional-python-modules": "requests==2.32.3,scikit-learn==1.5.0"
"--python-virtual-env-storage-prefix": "s3://amzn-s3-demo-bucket/venv-cache/"

You can also use the following optional parameters:

To enable caching for an existing job, add the storage prefix parameter. The first run takes longer because AWS Glue builds and uploads the virtual environment, but later runs use the cached virtual environment and perform no pip resolution.

# Before
"--additional-python-modules": "requests==2.32.3"

# After
"--additional-python-modules": "requests==2.32.3"
"--python-virtual-env-storage-prefix": "s3://amzn-s3-demo-bucket/venv-cache/"

How AWS Glue caches the virtual environment

AWS Glue keys the cache by your job configuration. The configuration includes the modules from --additional-python-modules, the value of --python-modules-installer-option, the AWS Glue version, and the value of --python-virtual-env-version.

An unchanged configuration results in a cache hit. If you change any of these values, AWS Glue builds a new virtual environment and creates a new cache entry.

AWS Glue stores each cached virtual environment under a separate key in your storage prefix. Jobs that use the same modules and installer options share the same cache entry.

Limitations

Building your own virtual environment

In AWS Glue 5.0 and later, you can build a virtual environment yourself and reference it with the --python-virtual-env parameter. Use this approach when you need full reproducibility, exact versions of container packages, or a build that runs in a CI/CD pipeline.

Prerequisites

Before you begin, make sure that you have the following:

For Python version and platform compatibility details for each AWS Glue version, see Appendix B: AWS Glue environment details.

Step 1: Create your requirements files

Create two requirements files that define the packages for your virtual environment.

Step 2: Create a Dockerfile

Create a Dockerfile that matches the environment of your target AWS Glue version. For platform and Python version details, see Appendix B: AWS Glue environment details.

AWS Glue 5.0 and 5.1 use Python 3.11 on Amazon Linux 2023.

FROM --platform=linux/amd64 public.ecr.aws/amazonlinux/amazonlinux:2023-minimal

RUN dnf install -y python3.11 zip && \
    dnf clean all

WORKDIR /build

AWS Glue 6.0 uses Python 3.13 on Amazon Linux 2023.

FROM --platform=linux/amd64 public.ecr.aws/amazonlinux/amazonlinux:2023-minimal

RUN dnf install -y python3.13 zip && \
    dnf clean all

WORKDIR /build

Step 3: Build and start the container

Build the Docker image. Then start a container with your requirements files and job script mounted.

docker build --platform linux/amd64 -t glue-venv-builder .

docker run --platform linux/amd64 \
  -v $(pwd)/base-requirements.txt:/working_dir/base-requirements.txt:ro \
  -v $(pwd)/additional-requirements.txt:/working_dir/additional-requirements.txt:ro \
  -v $(pwd)/my_glue_script/:/working_dir/my_glue_script/:ro \
  -v $(pwd):/output \
  -w /working_dir \
  -it glue-venv-builder bash

This command mounts your requirements files and your AWS Glue job script directory. The following step uses the script directory for import analysis.

Step 4: Build a temporary venv and discover required packages

Inside the container, build a temporary venv that mirrors the AWS Glue runtime. Then use static analysis to find the minimal set of packages your job needs.

For AWS Glue 5.0 and 5.1, which use Python 3.11, run the following commands.

# Create a temporary venv to reproduce the AWS Glue runtime environment
python3.11 -m venv temp_venv
source temp_venv/bin/activate

python3.11 -m pip install --upgrade pip

# Install base container libraries (mirrors what the AWS Glue container provides)
python3.11 -m pip install -r base-requirements.txt

# Install additional Python modules on top (mirrors how AWS Glue installs them at runtime)
python3.11 -m pip install -r additional-requirements.txt

# Freeze the full resolved environment
pip freeze > full-requirements.txt

# Install analysis tools
python3.11 -m pip install pipreqs pip-tools

# Use pipreqs to discover what the script actually imports
# --mode no-pin outputs package names without versions
pipreqs --mode no-pin --savepath discovered-requirements.txt /working_dir/my_glue_script

# Remove packages provided by the Spark runtime
sed -i '/pyspark/d' discovered-requirements.txt
sed -i '/py4j/d' discovered-requirements.txt

# Remove awsglue - install AWSGlueDataplanePython in Step 5 instead
sed -i '/awsglue/d' discovered-requirements.txt

# Use pip-compile to resolve the full dependency tree of the discovered packages,
# constrained to the versions from the temporary venv
pip-compile discovered-requirements.txt -c full-requirements.txt -o final-requirements.txt

echo "=== Final requirements.txt ==="
cat final-requirements.txt

# Deactivate and discard the temporary venv
deactivate
rm -rf temp_venv

For AWS Glue 6.0, which uses Python 3.13, run the following commands.

# Create a temporary venv to reproduce the AWS Glue runtime environment
python3.13 -m venv temp_venv
source temp_venv/bin/activate

python3.13 -m pip install --upgrade pip

# Install base container libraries (mirrors what the AWS Glue container provides)
python3.13 -m pip install -r base-requirements.txt

# Install additional Python modules on top (mirrors how AWS Glue installs them at runtime)
python3.13 -m pip install -r additional-requirements.txt

# Freeze the full resolved environment
pip freeze > full-requirements.txt

# Install analysis tools
python3.13 -m pip install pipreqs pip-tools

# Use pipreqs to discover what the script actually imports
# --mode no-pin outputs package names without versions
pipreqs --mode no-pin --savepath discovered-requirements.txt /working_dir/my_glue_script

# Remove packages provided by the Spark runtime
sed -i '/pyspark/d' discovered-requirements.txt
sed -i '/py4j/d' discovered-requirements.txt

# Remove awsglue - install AWSGlueDataplanePython in Step 5 instead
sed -i '/awsglue/d' discovered-requirements.txt

# Use pip-compile to resolve the full dependency tree of the discovered packages,
# constrained to the versions from the temporary venv
pip-compile discovered-requirements.txt -c full-requirements.txt -o final-requirements.txt

echo "=== Final requirements.txt ==="
cat final-requirements.txt

# Deactivate and discard the temporary venv
deactivate
rm -rf temp_venv

Step 5: Build the production venv

Create the final venv with only the packages your job needs. Then package it as a tarball.

For AWS Glue 5.0 and 5.1, which use Python 3.11, run the following commands.

python3.11 -m venv pyspark_venv
source pyspark_venv/bin/activate

python3.11 -m pip install --upgrade pip
python3.11 -m pip install -r final-requirements.txt

# Install the AWS Glue Python library that matches your AWS Glue version (see the version
# table in Step 1). Use 5.0.0 for AWS Glue 5.0, or 5.1.0 for AWS Glue 5.1.
python3.11 -m pip install AWSGlueDataplanePython==5.0.0

python3.11 -m pip install venv-pack
venv-pack -f -o pyspark_venv.tar.gz

cp pyspark_venv.tar.gz /output/
exit

For AWS Glue 6.0, which uses Python 3.13, run the following commands.

python3.13 -m venv pyspark_venv
source pyspark_venv/bin/activate

python3.13 -m pip install --upgrade pip
python3.13 -m pip install -r final-requirements.txt

# Install the AWS Glue Python library (see version table in Step 1)
python3.13 -m pip install AWSGlueDataplanePython==6.0.0

python3.13 -m pip install venv-pack
venv-pack -f -o pyspark_venv.tar.gz

cp pyspark_venv.tar.gz /output/
exit

Step 6: Upload to Amazon S3

Upload the packaged virtual environment to your Amazon S3 bucket.

aws s3 cp pyspark_venv.tar.gz s3://amzn-s3-demo-bucket/path/pyspark_venv.tar.gz

Step 7: Update job parameters

Update your AWS Glue job configuration to use --python-virtual-env instead of --additional-python-modules.

Remove the --additional-python-modules parameter and add the --python-virtual-env parameter pointing to your uploaded tarball.

# Before
"--additional-python-modules": "cryptography"

# After (remove --additional-python-modules entirely)
"--python-virtual-env": "s3://amzn-s3-demo-bucket/path/pyspark_venv.tar.gz"

Automating the migration with Kiro

If you prefer an automated approach, you can use Kiro, available on the Kiro website, to run the migration described in Building your own virtual environment from the command line. With a Kiro skill, Kiro analyzes your AWS Glue job configuration, builds the virtual environment in Docker, and produces the packaged tarball.

How it works

When you ask Kiro to migrate your AWS Glue job from --additional-python-modules to --python-virtual-env, Kiro does the following:

Example request

Provide your AWS Glue version, your additional Python modules, and your job script. For example:

I have a Glue 5.1 job with the following:
--additional-python-modules: ephem, awscli

Glue job script:
import awscli
import ephem

Help me migrate to using --python-virtual-env.

Getting the Kiro skill

The venv-migration skill file is maintained in the aws-glue-libs repository rather than in this guide. For the skill file and installation instructions, see venv-migration skill on the GitHub website.

Limitations

Troubleshooting

Use the following sections to resolve common issues when using Python virtual environments with AWS Glue.

Resolving pip version conflicts

A pip version conflict means that two packages require incompatible versions of the same dependency. To find and fix the conflict, do the following:

Resolving ModuleNotFoundError

This error indicates that your virtual environment does not include a required package. Common causes include the following:

To resolve this issue, add the missing package and rebuild the virtual environment. The steps depend on which approach your job uses.

Reducing venv tarball size

If your packaged virtual environment is too large, reduce its size with the following approaches:

Resolving platform compatibility errors

These errors occur when packages in the venv were built for a different operating system or architecture. To avoid these errors: