Amazon S3 Files volumes

S3 Files provides direct file system access to data stored in Amazon Simple Storage Service (Amazon S3) buckets. With AWS Batch, you can define S3 Files volumes in your job definitions so that your containers can read and write Amazon S3 data using standard file operations.

To use S3 Files volumes, you need an S3 file system and mount target configured in the same VPC as your AWS Batch compute environment. For complete setup instructions including bucket configuration, IAM roles, file system creation, and mount targets, see S3 Files prerequisites in the Amazon S3 User Guide and Configuring S3 Files for Amazon ECS in the Amazon Elastic Container Service Developer Guide.

Amazon S3 Files volume considerations

Consider the following when using S3 Files volumes:

Use Amazon S3 Files access points

S3 Files access points are application-specific entry points into a file system that enforce a POSIX user identity and root directory for all file system requests. You can use access points to isolate tenants so that each job can only access its own directory within a shared file system.

For more information about creating and managing access points, see Creating access points for an S3 file system in the Amazon S3 User Guide. For more information about using file system policies to enforce access point isolation, see How S3 Files works with IAM in the Amazon S3 User Guide.

Specify an Amazon S3 Files file system in your job definition

To use S3 Files volumes for your containers, you must specify the volume and mount point configurations in your job definition. The following job definition JSON snippet shows the syntax for the volumes and mountPoints objects for a container:

{
    "ecsProperties": {
        "taskProperties": [
            {
                ...,
                "taskRoleArn": "arn:aws:iam::<account>:role/<job-role-name>",
                "containers": [
                    {
                        ...,
                        "mountPoints": [
                            {
                                "sourceVolume": "myS3FilesVolume",
                                "containerPath": "/mnt/s3data",
                                "readOnly": false
                            }
                        ]
                    }
                ],
                "volumes": [
                    {
                        "name": "myS3FilesVolume",
                        "s3filesVolumeConfiguration": {
                            "fileSystemArn": "arn:aws:s3files:<region>:<account>:file-system/<fs-id>",
                            "rootDirectory": "/keypath/in/s3"
                        }
                    }
                ]
            }
        ]
    }
}

Use S3 Files volumes with AWS Batch and Amazon EKS

For jobs that use Amazon EKS resources, AWS Batch supports S3 Files volumes through a persistentVolumeClaim in the EKS job definition volume configuration. You must pre-create the persistent volume and persistent volume claim in your Amazon EKS cluster before referencing it in your job definition.

The following job definition snippet shows how to reference an S3 Files persistent volume claim:

{
    "eksProperties": {
        "podProperties": {
            ...,
            "containers": [
                {
                    ...,
                    "volumeMounts": [
                        {
                            "name": "s3files-vol",
                            "mountPath": "/mnt/s3data"
                        }
                    ]
                }
            ],
            "volumes": [
                {
                    "name": "s3files-vol",
                    "persistentVolumeClaim": {
                        "claimName": "<s3files-pvc-name>"
                    }
                }
            ]
        }
    }
}

For more information about setting up S3 Files with Amazon EKS, see Mounting S3 file systems in Amazon EKS in the Amazon S3 User Guide. For the full volume parameter reference, see EksVolume in the AWS Batch API Reference.