Package software.amazon.awscdk.services.glue.alpha

@Stability(Experimental) package software.amazon.awscdk.services.glue.alpha

AWS Glue Construct Library

---

cdk-constructs: Experimental

The APIs of higher level constructs in this module are experimental and under active development. They are subject to non-backward compatible changes or removal in any future version. These are not subject to the Semantic Versioning model and breaking changes will be announced in the release notes. This means that while you may use them, you may need to update your source code when upgrading to a newer version of this package.

This module is part of the AWS Cloud Development Kit project.

README

AWS Glue is a serverless data integration service that makes it easier to discover, prepare, move, and integrate data from multiple sources for analytics, machine learning (ML), and application development.

The Glue L2 construct has convenience methods working backwards from common use cases and sets required parameters to defaults that align with recommended best practices for each job type. It also provides customers with a balance between flexibility via optional parameter overrides, and opinionated interfaces that discouraging anti-patterns, resulting in reduced time to develop and deploy new resources.

References

Create a Glue Job

A Job encapsulates a script that connects to data sources, processes them, and then writes output to a data target. There are four types of Glue Jobs: Spark (ETL and Streaming), Python Shell, and Flex Jobs. Most of the required parameters for these jobs are common across all types, but there are a few differences depending on the languages supported and features provided by each type. For all job types, the L2 defaults to AWS best practice recommendations, such as:

  • Use of Secrets Manager for Connection JDBC strings
  • Glue job autoscaling
  • Default parameter values for Glue job creation

This iteration of the L2 construct introduces breaking changes to the existing glue-alpha-module, but these changes streamline the developer experience, introduce new constants for defaults, and replacing synth-time validations with interface contracts for enforcement of the parameter combinations that Glue supports. As an opinionated construct, the Glue L2 construct does not allow developers to create resources that use non-current versions of Glue or deprecated language dependencies (e.g. deprecated versions of Python). As always, L1s allow you to specify a wider range of parameters if you need or want to use alternative configurations.

Optional and required parameters for each job are enforced via interface rather than validation; see Glue's public documentation for more granular details.

Spark Jobs

ETL Jobs

ETL jobs support pySpark and Scala languages, for which there are separate but similar constructors. ETL jobs default to the G2 worker type, but you can override this default with other supported worker type values (G1, G2, G4 and G8). ETL jobs defaults to Glue version 4.0, which you can override to 3.0. The following ETL features are enabled by default: —enable-metrics, —enable-continuous-cloudwatch-log. The Spark UI (—enable-spark-ui) is off by default; enable it by setting the sparkUI prop. You can find more details about version, worker type and other features in Glue's public documentation.

Reference the pyspark-etl-jobs.test.ts and scalaspark-etl-jobs.test.ts unit tests for examples of required-only and optional job parameters when creating these types of jobs.

For the sake of brevity, examples are shown using the pySpark job variety.

Example with only required parameters: 

 import software.amazon.awscdk.*;
 import software.amazon.awscdk.services.iam.*;
 Stack stack;
 IRole role;
 Code script;
 
 PySparkEtlJob.Builder.create(stack, "PySparkETLJob")
         .role(role)
         .script(script)
         .jobName("PySparkETLJob")
         .build();

Example with optional override parameters: 

 import software.amazon.awscdk.*;
 import software.amazon.awscdk.services.iam.*;
 Stack stack;
 IRole role;
 Code script;
 
 PySparkEtlJob.Builder.create(stack, "PySparkETLJob")
         .jobName("PySparkETLJobCustomName")
         .description("This is a description")
         .role(role)
         .script(script)
         .glueVersion(GlueVersion.V5_1)
         .continuousLogging(ContinuousLoggingProps.builder().enabled(false).build())
         .workerConfiguration(WorkerConfiguration.builder()
                 .workerType(WorkerType.G_2X)
                 .numberOfWorkers(2)
                 .build())
         .maxConcurrentRuns(100)
         .timeout(Duration.hours(2))
         .connections(List.of(Connection.fromConnectionName(stack, "Connection", "connectionName")))
         .securityConfiguration(SecurityConfiguration.fromSecurityConfigurationName(stack, "SecurityConfig", "securityConfigName"))
         .tags(Map.of(
                 "FirstTagName", "FirstTagValue",
                 "SecondTagName", "SecondTagValue",
                 "XTagName", "XTagValue"))
         .maxRetries(2)
         .build();

Streaming Jobs

Streaming jobs are similar to ETL jobs, except that they perform ETL on data streams using the Apache Spark Structured Streaming framework. Some Spark job features are not available to Streaming ETL jobs. They support Scala and pySpark languages. PySpark streaming jobs run on Python 3. It defaults to the G2 worker type and Glue 4.0, both of which you can override. The following best practice features are enabled by default: —enable-metrics, —enable-continuous-cloudwatch-log. The Spark UI (—enable-spark-ui) is off by default; enable it by setting the sparkUI prop.

Reference the pyspark-streaming-jobs.test.ts and scalaspark-streaming-jobs.test.ts unit tests for examples of required-only and optional job parameters when creating these types of jobs.

Example with only required parameters: 

 import software.amazon.awscdk.*;
 import software.amazon.awscdk.services.iam.*;
 Stack stack;
 IRole role;
 Code script;
 
 PySparkStreamingJob.Builder.create(stack, "ImportedJob").role(role).script(script).build();

Example with optional override parameters: 

 import software.amazon.awscdk.*;
 import software.amazon.awscdk.services.iam.*;
 Stack stack;
 IRole role;
 Code script;
 
 PySparkStreamingJob.Builder.create(stack, "PySparkStreamingJob")
         .jobName("PySparkStreamingJobCustomName")
         .description("This is a description")
         .role(role)
         .script(script)
         .glueVersion(GlueVersion.V5_1)
         .continuousLogging(ContinuousLoggingProps.builder().enabled(false).build())
         .workerConfiguration(WorkerConfiguration.builder()
                 .workerType(WorkerType.G_2X)
                 .numberOfWorkers(2)
                 .build())
         .maxConcurrentRuns(100)
         .timeout(Duration.hours(2))
         .connections(List.of(Connection.fromConnectionName(stack, "Connection", "connectionName")))
         .securityConfiguration(SecurityConfiguration.fromSecurityConfigurationName(stack, "SecurityConfig", "securityConfigName"))
         .tags(Map.of(
                 "FirstTagName", "FirstTagValue",
                 "SecondTagName", "SecondTagValue",
                 "XTagName", "XTagValue"))
         .maxRetries(2)
         .build();

Flex Jobs

The flexible execution class is appropriate for non-urgent jobs such as pre-production jobs, testing, and one-time data loads. Flexible jobs default to Glue version 5.0 and worker type G_2X. The following best practice features are enabled by default: —enable-metrics, —enable-continuous-cloudwatch-log The Spark UI (—enable-spark-ui) is off by default; enable it by setting the sparkUI prop.

Reference the pyspark-flex-etl-jobs.test.ts and scalaspark-flex-etl-jobs.test.ts unit tests for examples of required-only and optional job parameters when creating these types of jobs.

Example with only required parameters: 

 import software.amazon.awscdk.*;
 import software.amazon.awscdk.services.iam.*;
 Stack stack;
 IRole role;
 Code script;
 
 PySparkFlexEtlJob.Builder.create(stack, "ImportedJob").role(role).script(script).build();

Example with optional override parameters: 

 import software.amazon.awscdk.*;
 import software.amazon.awscdk.services.iam.*;
 Stack stack;
 IRole role;
 Code script;
 
 PySparkFlexEtlJob.Builder.create(stack, "pySparkFlexEtlJob")
         .jobName("pySparkFlexEtlJob")
         .description("This is a description")
         .role(role)
         .script(script)
         .glueVersion(GlueVersion.V5_1)
         .continuousLogging(ContinuousLoggingProps.builder().enabled(false).build())
         .workerConfiguration(WorkerConfiguration.builder()
                 .workerType(WorkerType.G_2X)
                 .numberOfWorkers(2)
                 .build())
         .maxConcurrentRuns(100)
         .timeout(Duration.hours(2))
         .connections(List.of(Connection.fromConnectionName(stack, "Connection", "connectionName")))
         .securityConfiguration(SecurityConfiguration.fromSecurityConfigurationName(stack, "SecurityConfig", "securityConfigName"))
         .tags(Map.of(
                 "FirstTagName", "FirstTagValue",
                 "SecondTagName", "SecondTagValue",
                 "XTagName", "XTagValue"))
         .maxRetries(2)
         .build();

Python Shell Jobs

Python shell jobs support a Python version that depends on the AWS Glue version you use. These can be used to schedule and run tasks that don't require an Apache Spark environment. Python shell jobs default to Python 3.9 and a MaxCapacity of 0.0625. Python 3.9 supports pre-loaded analytics libraries using the library-set=analytics flag, which is enabled by default.

Reference the pyspark-shell-job.test.ts unit tests for examples of required-only and optional job parameters when creating these types of jobs.

Example with only required parameters: 

 import software.amazon.awscdk.*;
 import software.amazon.awscdk.services.iam.*;
 Stack stack;
 IRole role;
 Code script;
 
 PythonShellJob.Builder.create(stack, "ImportedJob").role(role).script(script).build();

Example with optional override parameters: 

 import software.amazon.awscdk.*;
 import software.amazon.awscdk.services.iam.*;
 Stack stack;
 IRole role;
 Code script;
 Code extraPythonFile;
 
 PythonShellJob.Builder.create(stack, "PythonShellJob")
         .jobName("PythonShellJobCustomName")
         .description("This is a description")
         .pythonVersion(PythonVersion.THREE_NINE)
         .maxCapacity(MaxCapacity.DPU_1)
         .role(role)
         .script(script)
         .extraPythonFiles(List.of(extraPythonFile))
         .glueVersion(GlueVersion.V3_0)
         .continuousLogging(ContinuousLoggingProps.builder().enabled(false).build())
         .maxConcurrentRuns(100)
         .timeout(Duration.hours(2))
         .connections(List.of(Connection.fromConnectionName(stack, "Connection", "connectionName")))
         .securityConfiguration(SecurityConfiguration.fromSecurityConfigurationName(stack, "SecurityConfig", "securityConfigName"))
         .tags(Map.of(
                 "FirstTagName", "FirstTagValue",
                 "SecondTagName", "SecondTagValue",
                 "XTagName", "XTagValue"))
         .maxRetries(2)
         .build();

Ray Jobs

⚠️ DEPRECATED: AWS Glue for Ray is closed to new customers as of April 30, 2026 and is in maintenance mode. Migrate to Amazon EKS with KubeRay Operator.

The RayJob construct, Runtime.RAY_TWO_FOUR, and JobType.RAY are deprecated and will be removed in a future release.

Metrics Control

By default, Glue jobs enable CloudWatch metrics (--enable-metrics) and observability metrics (--enable-observability-metrics) for monitoring and debugging. You can disable these metrics to reduce CloudWatch costs: 

 import software.amazon.awscdk.*;
 import software.amazon.awscdk.services.iam.*;
 Stack stack;
 IRole role;
 Code script;
 
 
 // Disable both metrics for cost optimization
 // Disable both metrics for cost optimization
 PySparkEtlJob.Builder.create(stack, "CostOptimizedJob")
         .role(role)
         .script(script)
         .enableMetrics(false)
         .enableObservabilityMetrics(false)
         .build();
 
 // Selective control - keep observability, disable profiling
 // Selective control - keep observability, disable profiling
 PySparkEtlJob.Builder.create(stack, "SelectiveJob")
         .role(role)
         .script(script)
         .enableMetrics(false)
         .build();

This feature is available for all Spark job types (ETL, Streaming, Flex).

Enable Job Run Queuing

AWS Glue job queuing monitors your account level quotas and limits. If quotas or limits are insufficient to start a Glue job run, AWS Glue will automatically queue the job and wait for limits to free up. Once limits become available, AWS Glue will retry the job run. Glue jobs will queue for limits like max concurrent job runs per account, max concurrent Data Processing Units (DPU), and resource unavailable due to IP address exhaustion in Amazon Virtual Private Cloud (Amazon VPC).

Enable job run queuing by setting the jobRunQueuingEnabled property to true. 

 import software.amazon.awscdk.*;
 import software.amazon.awscdk.services.iam.*;
 Stack stack;
 IRole role;
 Code script;
 
 PySparkEtlJob.Builder.create(stack, "PySparkETLJob")
         .role(role)
         .script(script)
         .jobName("PySparkETLJob")
         .jobRunQueuingEnabled(true)
         .build();

Uploading scripts from the CDK app repository to S3

Similar to other L2 constructs, the Glue L2 automates uploading local scripts to S3. Use glue.Code.fromAsset(path) to point at a script in your local file structure; it is uploaded to the CDK-managed asset bucket. To reference a script that already exists in S3, use glue.Code.fromBucket(bucket, key), which performs no upload. A script is required for every job.

Reference the unit tests for examples of repo and S3 code target examples.

Workflow Triggers

You can use Glue workflows to create and visualize complex extract, transform, and load (ETL) activities involving multiple crawlers, jobs, and triggers. Standalone triggers are an anti-pattern, so you must create triggers from within a workflow using the L2 construct.

Within a workflow object, there are functions to create different types of triggers with actions and predicates. You add triggers to the workflow, and each trigger references the jobs or crawlers it runs as its actions.

startOnCreation applies to scheduled triggers (and, via ConditionalTriggerOptions, conditional triggers) only. It defaults to false, but you can override it if you prefer for your trigger to start on creation.

Reference the workflow-triggers.test.ts unit tests for examples of creating workflows and triggers. 

 import software.amazon.awscdk.*;
 import software.amazon.awscdk.services.iam.*;
 Stack stack;
 IRole role;
 Code script;
 
 
 // Create a job to run from the workflow
 PySparkEtlJob job = PySparkEtlJob.Builder.create(stack, "Job").role(role).script(script).build();
 
 // Create a workflow and add a trigger that runs the job
 Workflow workflow = new Workflow(stack, "Workflow");
 workflow.addOnDemandTrigger("OnDemandTrigger", OnDemandTriggerOptions.builder()
         .actions(List.of(Action.builder().job(job).build()))
         .build());

1. On-Demand Triggers

On-demand triggers can start glue jobs or crawlers. This construct provides convenience functions to create on-demand crawler or job triggers. The constructor takes an optional description parameter, but abstracts the requirement of an actions list using the job or crawler objects using conditional types.

2. Scheduled Triggers

You can create scheduled triggers using cron expressions. This construct provides daily and weekly convenience functions, as well as a custom function that allows you to create your own custom timing using the existing event Schedule class without having to build your own cron expressions. The L2 extracts the expression that Glue requires from the Schedule object. The constructor takes an optional description and a list of jobs or crawlers as actions.

3. Notify Event Triggers

There are two types of notify event triggers: batching and non-batching. For batching triggers, you must specify BatchSize. For non-batching triggers, BatchSize defaults to 1. For both triggers, BatchWindow defaults to 900 seconds, but you can override the window to align with your workload's requirements.

4. Conditional Triggers

Conditional triggers have a predicate and actions associated with them. The trigger actions are executed when the predicateCondition is true.

Connection Properties

A Connection allows Glue jobs, crawlers and development endpoints to access certain types of data stores.

  • Secrets Management Manage JDBC connection credentials in Secrets Manager and pass the secret to the connection via the secret property (see the example below), rather than embedding credentials in properties.
  • Networking - the CDK determines the best fit subnet for Glue connection configuration You can specify the exact subnet of the Connection when it's defined, but you are not required to. Instead, you can provide a vpc and, optionally, a vpcSubnets selection, and the L2 leverages the existing EC2 Subnet Selection library to make the best choice selection for the subnet. A Glue connection targets a single subnet, so the first subnet of the selection is used. subnet and vpc are mutually exclusive.

Pin the connection to a specific subnet: 

 SecurityGroup securityGroup;
 Subnet subnet;
 
 Connection.Builder.create(this, "MyConnection")
         .type(ConnectionType.NETWORK)
         // The security groups granting AWS Glue inbound access to the data source within the VPC
         .securityGroups(List.of(securityGroup))
         // The VPC subnet which contains the data source
         .subnet(subnet)
         .build();

Or let the CDK select a subnet from a VPC: 

 SecurityGroup securityGroup;
 Vpc vpc;
 
 Connection.Builder.create(this, "MyConnection")
         .type(ConnectionType.NETWORK)
         .securityGroups(List.of(securityGroup))
         .vpc(vpc)
         // Optional - defaults to private subnets
         .vpcSubnets(SubnetSelection.builder().subnetType(SubnetType.PRIVATE_WITH_EGRESS).build())
         .build();

For RDS Connection by JDBC, it is recommended to manage credentials using AWS Secrets Manager. Pass the secret via the secret property: Glue reads the credentials at runtime through the connection's SECRET_ID, so the secret value never enters the template. Note that in this case, the subnet must have a route to the AWS Secrets Manager VPC endpoint or to the AWS Secrets Manager endpoint through a NAT gateway. 

 SecurityGroup securityGroup;
 Subnet subnet;
 DatabaseCluster db;
 
 Connection.Builder.create(this, "RdsConnection")
         .type(ConnectionType.JDBC)
         .securityGroups(List.of(securityGroup))
         .subnet(subnet)
         .secret(db.getSecret())
         .properties(Map.of(
                 "JDBC_CONNECTION_URL", String.format("jdbc:mysql://%s/databasename", db.getClusterEndpoint().getSocketAddress()),
                 "JDBC_ENFORCE_SSL", "false"))
         .build();

Prefer the secret property over placing credentials in properties. Connection properties are emitted verbatim into the CloudFormation template, so any credential placed there in plaintext is stored in plaintext in the template, cdk.out, and source control. If a property key looks like a credential (for example PASSWORD, SECRET, or TOKEN) and holds a plaintext literal, the construct emits a synthesis-time warning.

If you need to use a connection type that doesn't exist as a static member on ConnectionType, you can instantiate a ConnectionType object, e.g: new glue.ConnectionType('NEW_TYPE').

See Adding a Connection to Your Data Store and Connection Structure documentation for more information on the supported data stores and their configurations.

SecurityConfiguration

A SecurityConfiguration is a set of security properties that can be used by AWS Glue to encrypt data at rest.

Each encryption config is built with a factory that pairs the encryption mode with its key, so illegal combinations (such as an S3-managed encryption carrying a KMS key) cannot be expressed: 

 SecurityConfiguration.Builder.create(this, "MySecurityConfiguration")
         .cloudWatchEncryption(CloudWatchEncryption.kms())
         .jobBookmarksEncryption(JobBookmarksEncryption.clientSideKms())
         .s3Encryption(S3Encryption.kms())
         .build();

By default, a shared KMS key is created for use with the encryption configurations that require one. You can also supply your own key to any factory, for example, for CloudWatch encryption: 

 Key key;
 
 SecurityConfiguration.Builder.create(this, "MySecurityConfiguration")
         .cloudWatchEncryption(CloudWatchEncryption.kms(key))
         .build();

Use glue.S3Encryption.s3Managed() for S3-managed (SSE-S3) encryption, which takes no key.

See documentation for more info for Glue encrypting data written by Crawlers, Jobs, and Development Endpoints.

Catalog

The Glue Data Catalog is a persistent metadata store for your data assets. Every account has an implicit, account-wide catalog that always exists, and you can also create additional catalogs as AWS::Glue::Catalog resources (for example, to federate to another metastore).

A catalog's encryption is fixed when the catalog is created: a catalog either carries encryption settings or it does not. This keeps its configuration easy to reason about — there are no mutation methods that change encryption after the fact.

The account-wide catalog

Use Catalog.forAccount(scope) to obtain the implicit account catalog. It is not a CloudFormation resource — it always exists. Repeated calls within the same stack return the same instance: 

 ICatalog catalog = Catalog.forAccount(this);

To configure Data Catalog encryption for the account, use Catalog.encryptAccount(scope, options): 

 Key key;
 
 Catalog.encryptAccount(this, CatalogEncryptionOptions.builder()
         .encryptionAtRest(DataCatalogEncryptionAtRest.kms(key))
         .build());

Because encryption is fixed at construction, encryptAccount must be called before the account catalog is first used in the stack — before any Catalog.forAccount(this) call, and before any Database that uses the account catalog. Calling it after the account catalog has been materialized throws.

The account catalog's encryption is an account- and region-wide setting, managed through the singleton PutDataCatalogEncryptionSettings API. Configure it in exactly one stack. Configuring it from multiple stacks in the same account and region makes those stacks overwrite one another at deploy time, and the result is order-dependent. Unlike duplicate settings within a single stack (which CloudFormation rejects), this cross-stack conflict is not caught at synthesis time, because each stack synthesizes to its own template.

Creating a catalog

To create a new catalog resource, use the Catalog constructor. Encryption is configured through the encryptionAtRest and connectionPasswordEncryption props: 

 Catalog.Builder.create(this, "MyCatalog")
         .catalogName("my-catalog")
         .description("my catalog description")
         .build();

Encryption at rest

Configure Data Catalog encryption at rest through the encryptionAtRest option (on Catalog.encryptAccount or the Catalog constructor). It accepts a DataCatalogEncryptionAtRest describing the mode: 

 Key key;
 
 
 // SSE-KMS with a customer-managed key
 Catalog.encryptAccount(this, CatalogEncryptionOptions.builder()
         .encryptionAtRest(DataCatalogEncryptionAtRest.kms(key))
         .build());
 
 // SSE-KMS with an AWS-managed key (omit the key)
 // SSE-KMS with an AWS-managed key (omit the key)
 Catalog.Builder.create(this, "ManagedKeyCatalog")
         .catalogName("managed-key-catalog")
         .encryptionAtRest(DataCatalogEncryptionAtRest.kms())
         .build();
 
 // Disable encryption at rest
 // Disable encryption at rest
 Catalog.Builder.create(this, "PlaintextCatalog")
         .catalogName("plaintext-catalog")
         .encryptionAtRest(DataCatalogEncryptionAtRest.disabled())
         .build();

When you use SSE-KMS-WITH-SERVICE-ROLE, AWS Glue accesses the KMS key through a service role you provide. If you pass a customer-managed key, the role is automatically granted the permissions it needs to encrypt and decrypt catalog data: 

 import software.amazon.awscdk.services.iam.*;
 Key key;
 IRole role;
 
 Catalog.encryptAccount(this, CatalogEncryptionOptions.builder()
         .encryptionAtRest(DataCatalogEncryptionAtRest.kmsWithServiceRole(role, key))
         .build());

The customer-managed key, when configured, is exposed on the catalog as encryptionKey (and the connection-password key as connectionPasswordKey), so you can reference it to grant additional access. It is undefined when encryption is disabled or an AWS-managed key is used.

Connection password encryption

Independently from encryption at rest, the Data Catalog can encrypt the passwords stored in connection properties. Configure it through the connectionPasswordEncryption option: 

 Key key;
 
 Catalog.encryptAccount(this, CatalogEncryptionOptions.builder()
         .connectionPasswordEncryption(ConnectionPasswordEncryption.builder()
                 .kmsKey(key)
                 // Whether GetConnection/GetConnections return the password encrypted (default: true)
                 .returnConnectionPasswordEncrypted(true)
                 .build())
         .build());

The two encryption blocks are independent: enabling one does not require the other, and each may use a different KMS key. The customer-managed key for connection passwords is exposed as connectionPasswordKey.

Importing a catalog

You can import an existing catalog by ARN or by id. An imported catalog is a pure identity handle — it emits no resources and does not manage the catalog's encryption: 

 ICatalog byId = Catalog.fromCatalogId(this, "ById", "my-catalog-id");
 ICatalog byArn = Catalog.fromCatalogArn(this, "ByArn", "arn:aws:glue:us-east-1:123456789012:catalog/my-catalog-id");

To manage the Data Catalog encryption of a catalog you did not create in this stack, add a CfnDataCatalogEncryptionSettings resource targeting its id directly. Do this from exactly one stack: like the account catalog, a catalog has a single encryption configuration, so two settings resources targeting the same id race to overwrite one another at deploy time. Within a single stack this is caught by CloudFormation template validation (E3019, duplicate primary identifiers); across stacks it is not, since each stack synthesizes to its own template. 

 import software.amazon.awscdk.services.glue.CfnDataCatalogEncryptionSettings;
 
 
 CfnDataCatalogEncryptionSettings.Builder.create(this, "Encryption")
         .catalogId("my-catalog-id")
         .dataCatalogEncryptionSettings(DataCatalogEncryptionSettingsProperty.builder()
                 .encryptionAtRest(EncryptionAtRestProperty.builder().catalogEncryptionMode("SSE-KMS").build())
                 .build())
         .build();

Database

A Database is a logical grouping of Tables in the Glue Catalog. 

 Database.Builder.create(this, "MyDatabase")
         .databaseName("my_database")
         .description("my_database_description")
         .build();

Because a database is a container for tables and their metadata, it is retained by default when removed from the stack, to avoid accidental data loss. Set removalPolicy to RemovalPolicy.DESTROY to have it deleted instead: 

 import software.amazon.awscdk.RemovalPolicy;
 
 
 Database.Builder.create(this, "MyDatabase")
         .databaseName("my_database")
         .removalPolicy(RemovalPolicy.DESTROY)
         .build();

Table

A Glue table describes a table of data in S3: its structure (column names and types), location of data (S3 objects with a common prefix in a S3 bucket), and format for the files (Json, Avro, Parquet, etc.): 

 Database myDatabase;
 
 S3Table.Builder.create(this, "MyTable")
         .database(myDatabase)
         .columns(List.of(Column.builder()
                 .name("col1")
                 .type(Schema.STRING)
                 .build(), Column.builder()
                 .name("col2")
                 .type(Schema.array(Schema.STRING))
                 .comment("col2 is an array of strings")
                 .build()))
         .dataFormat(DataFormat.JSON)
         .build();

By default, a S3 bucket will be created to store the table's data but you can bring your own with S3TableStorage.fromBucket and set an s3Prefix: 

 Bucket myBucket;
 Database myDatabase;
 
 S3Table.Builder.create(this, "MyTable")
         .storage(S3TableStorage.fromBucket(myBucket))
         .s3Prefix("my-table/")
         // ...
         .database(myDatabase)
         .columns(List.of(Column.builder()
                 .name("col1")
                 .type(Schema.STRING)
                 .build()))
         .dataFormat(DataFormat.JSON)
         .build();

Glue tables can be configured to contain user-defined properties, to describe the physical storage of table data, through the storageParameters property: 

 Database myDatabase;
 
 S3Table.Builder.create(this, "MyTable")
         .storageParameters(List.of(StorageParameter.skipHeaderLineCount(1), StorageParameter.compressionType(CompressionType.GZIP), StorageParameter.custom("separatorChar", ",")))
         // ...
         .database(myDatabase)
         .columns(List.of(Column.builder()
                 .name("col1")
                 .type(Schema.STRING)
                 .build()))
         .dataFormat(DataFormat.JSON)
         .build();

Glue tables can also be configured to contain user-defined table properties through the parameters property: 

 Database myDatabase;
 
 S3Table.Builder.create(this, "MyTable")
         .parameters(Map.of(
                 "key1", "val1",
                 "key2", "val2"))
         .database(myDatabase)
         .columns(List.of(Column.builder()
                 .name("col1")
                 .type(Schema.STRING)
                 .build()))
         .dataFormat(DataFormat.JSON)
         .build();

Partition Keys

To improve query performance, a table can specify partitionKeys on which data is stored and queried separately. For example, you might partition a table by year and month to optimize queries based on a time window: 

 Database myDatabase;
 
 S3Table.Builder.create(this, "MyTable")
         .database(myDatabase)
         .columns(List.of(Column.builder()
                 .name("col1")
                 .type(Schema.STRING)
                 .build()))
         .partitionKeys(List.of(Column.builder()
                 .name("year")
                 .type(Schema.SMALL_INT)
                 .build(), Column.builder()
                 .name("month")
                 .type(Schema.SMALL_INT)
                 .build()))
         .dataFormat(DataFormat.JSON)
         .build();

Partition Indexes

Another way to improve query performance is to specify partition indexes. If no partition indexes are present on the table, AWS Glue loads all partitions of the table and filters the loaded partitions using the query expression. The query takes more time to run as the number of partitions increase. With an index, the query will try to fetch a subset of the partitions instead of loading all partitions of the table.

The keys of a partition index must be a subset of the partition keys of the table. You can have a maximum of 3 partition indexes per table. To specify a partition index, you can use the partitionIndexes property: 

 Database myDatabase;
 
 S3Table.Builder.create(this, "MyTable")
         .database(myDatabase)
         .columns(List.of(Column.builder()
                 .name("col1")
                 .type(Schema.STRING)
                 .build()))
         .partitionKeys(List.of(Column.builder()
                 .name("year")
                 .type(Schema.SMALL_INT)
                 .build(), Column.builder()
                 .name("month")
                 .type(Schema.SMALL_INT)
                 .build()))
         .partitionIndexes(List.of(PartitionIndex.builder()
                 .indexName("my-index") // optional
                 .keyNames(List.of("year"))
                 .build())) // supply up to 3 indexes
         .dataFormat(DataFormat.JSON)
         .build();

Alternatively, you can call the addPartitionIndex() function on a table: 

 S3Table myTable;
 
 myTable.addPartitionIndex(PartitionIndex.builder()
         .indexName("my-index")
         .keyNames(List.of("year"))
         .build());

Partition Filtering

If you have a table with a large number of partitions that grows over time, consider using AWS Glue partition indexing and filtering. 

 Database myDatabase;
 
 S3Table.Builder.create(this, "MyTable")
         .database(myDatabase)
         .columns(List.of(Column.builder()
                 .name("col1")
                 .type(Schema.STRING)
                 .build()))
         .partitionKeys(List.of(Column.builder()
                 .name("year")
                 .type(Schema.SMALL_INT)
                 .build(), Column.builder()
                 .name("month")
                 .type(Schema.SMALL_INT)
                 .build()))
         .dataFormat(DataFormat.JSON)
         .enablePartitionFiltering(true)
         .build();

Partition Projection

Partition projection allows Athena to automatically add new partitions as new data arrives, without requiring ALTER TABLE ADD PARTITION statements. This improves query performance and reduces management overhead by eliminating the need to manually manage partition metadata.

For more information, see the AWS documentation on partition projection.

INTEGER Projection

For partition keys with sequential numeric values: 

 Database myDatabase;
 
 S3Table.Builder.create(this, "MyTable")
         .database(myDatabase)
         .columns(List.of(Column.builder()
                 .name("data")
                 .type(Schema.STRING)
                 .build()))
         .partitionKeys(List.of(Column.builder()
                 .name("year")
                 .type(Schema.INTEGER)
                 .build()))
         .dataFormat(DataFormat.JSON)
         .partitionProjection(Map.of(
                 "year", PartitionProjectionConfiguration.integer(IntegerPartitionProjectionConfigurationProps.builder()
                         .min(2020)
                         .max(2023)
                         .interval(1) // optional, defaults to 1
                         .digits(4)
                         .build())))
         .build();

DATE Projection

For partition keys with date or timestamp values. Supports both fixed dates and relative dates using NOW: 

 Database myDatabase;
 
 S3Table.Builder.create(this, "MyTable")
         .database(myDatabase)
         .columns(List.of(Column.builder()
                 .name("data")
                 .type(Schema.STRING)
                 .build()))
         .partitionKeys(List.of(Column.builder()
                 .name("date")
                 .type(Schema.STRING)
                 .build()))
         .dataFormat(DataFormat.JSON)
         .partitionProjection(Map.of(
                 "date", PartitionProjectionConfiguration.date(DatePartitionProjectionConfigurationProps.builder()
                         .min("2020-01-01")
                         .max("2023-12-31")
                         .format("yyyy-MM-dd")
                         .interval(1) // optional, defaults to 1
                         .intervalUnit(DateIntervalUnit.DAYS)
                         .build())))
         .build();

You can also use relative dates with NOW: 

 Database myDatabase;
 
 S3Table.Builder.create(this, "MyTable")
         .database(myDatabase)
         .columns(List.of(Column.builder()
                 .name("data")
                 .type(Schema.STRING)
                 .build()))
         .partitionKeys(List.of(Column.builder()
                 .name("date")
                 .type(Schema.STRING)
                 .build()))
         .dataFormat(DataFormat.JSON)
         .partitionProjection(Map.of(
                 "date", PartitionProjectionConfiguration.date(DatePartitionProjectionConfigurationProps.builder()
                         .min("NOW-3YEARS")
                         .max("NOW")
                         .format("yyyy-MM-dd")
                         .build())))
         .build();

ENUM Projection

For partition keys with a known set of values: 

 Database myDatabase;
 
 S3Table.Builder.create(this, "MyTable")
         .database(myDatabase)
         .columns(List.of(Column.builder()
                 .name("data")
                 .type(Schema.STRING)
                 .build()))
         .partitionKeys(List.of(Column.builder()
                 .name("region")
                 .type(Schema.STRING)
                 .build()))
         .dataFormat(DataFormat.JSON)
         .partitionProjection(Map.of(
                 "region", PartitionProjectionConfiguration.enum(EnumPartitionProjectionConfigurationProps.builder()
                         .values(List.of("us-east-1", "us-west-2", "eu-west-1"))
                         .build())))
         .build();

INJECTED Projection

For custom partition values injected at query time: 

 Database myDatabase;
 
 S3Table.Builder.create(this, "MyTable")
         .database(myDatabase)
         .columns(List.of(Column.builder()
                 .name("data")
                 .type(Schema.STRING)
                 .build()))
         .partitionKeys(List.of(Column.builder()
                 .name("custom")
                 .type(Schema.STRING)
                 .build()))
         .dataFormat(DataFormat.JSON)
         .partitionProjection(Map.of(
                 "custom", PartitionProjectionConfiguration.injected()))
         .build();

Multiple Partition Projections

You can configure partition projection for multiple partition keys: 

 Database myDatabase;
 
 S3Table.Builder.create(this, "MyTable")
         .database(myDatabase)
         .columns(List.of(Column.builder()
                 .name("data")
                 .type(Schema.STRING)
                 .build()))
         .partitionKeys(List.of(Column.builder()
                 .name("year")
                 .type(Schema.INTEGER)
                 .build(), Column.builder()
                 .name("month")
                 .type(Schema.INTEGER)
                 .build(), Column.builder()
                 .name("region")
                 .type(Schema.STRING)
                 .build()))
         .dataFormat(DataFormat.JSON)
         .partitionProjection(Map.of(
                 "year", PartitionProjectionConfiguration.integer(IntegerPartitionProjectionConfigurationProps.builder()
                         .min(2020)
                         .max(2023)
                         .build()),
                 "month", PartitionProjectionConfiguration.integer(IntegerPartitionProjectionConfigurationProps.builder()
                         .min(1)
                         .max(12)
                         .digits(2)
                         .build()),
                 "region", PartitionProjectionConfiguration.enum(EnumPartitionProjectionConfigurationProps.builder()
                         .values(List.of("us-east-1", "us-west-2"))
                         .build())))
         .build();

Glue Connections

Glue connections allow external data connections to third party databases and data warehouses. However, these connections can also be assigned to Glue Tables, allowing you to query external data sources using the Glue Data Catalog.

Whereas S3Table will point to (and if needed, create) a bucket to store the tables' data, ExternalTable will point to an existing table in a data source. For example, to create a table in Glue that points to a table in Redshift: 

 Connection myConnection;
 Database myDatabase;
 
 ExternalTable.Builder.create(this, "MyTable")
         .connection(myConnection)
         .externalDataLocation("default_db_public_example") // A table in Redshift
         // ...
         .database(myDatabase)
         .columns(List.of(Column.builder()
                 .name("col1")
                 .type(Schema.STRING)
                 .build()))
         .dataFormat(DataFormat.JSON)
         .build();

Data Quality Ruleset

A DataQualityRuleset defines a set of data quality rules — authored in Glue's Data Quality Definition Language (DQDL) — that are evaluated against a table in the Data Catalog. 

 DataQualityRuleset.Builder.create(this, "MyRuleset")
         .rulesetName("my_ruleset")
         .dqdl(Dqdl.fromString("Rules = [ RowCount > 100, IsComplete \"order_id\" ]"))
         .targetTable(new DataQualityTargetTable("my_database", "my_table"))
         .build();

Build the DQDL document with Dqdl.fromString(...). Glue parses and validates the DQDL when the ruleset is deployed; see the DQDL reference for the full rule syntax.

Encryption

When the table creates its own S3 bucket (S3TableStorage.managedBucket, the default), that bucket enforces SSL: a bucket policy denies any request made over plain HTTP. If you bring your own bucket with S3TableStorage.fromBucket, enabling enforceSSL on it is your responsibility.

Server-side encryption applies only to a bucket the table manages. Choose it with storage: glue.S3TableStorage.managedBucket(...):

  • S3Managed - (default) Server side encryption (SSE-S3) with an Amazon S3-managed key.

 Database myDatabase;
 
 S3Table.Builder.create(this, "MyTable")
         .storage(S3TableStorage.managedBucket(S3TableEncryption.s3Managed()))
         // ...
         .database(myDatabase)
         .columns(List.of(Column.builder()
                 .name("col1")
                 .type(Schema.STRING)
                 .build()))
         .dataFormat(DataFormat.JSON)
         .build();

  • Kms - Server-side encryption (SSE-KMS) with an AWS KMS Key managed by the account owner.

 Database myDatabase;
 
 // KMS key is created automatically
 // KMS key is created automatically
 S3Table.Builder.create(this, "MyTable")
         .storage(S3TableStorage.managedBucket(S3TableEncryption.kms()))
         // ...
         .database(myDatabase)
         .columns(List.of(Column.builder()
                 .name("col1")
                 .type(Schema.STRING)
                 .build()))
         .dataFormat(DataFormat.JSON)
         .build();
 
 // with an explicit KMS key
 // with an explicit KMS key
 S3Table.Builder.create(this, "MyTable")
         .storage(S3TableStorage.managedBucket(S3TableEncryption.kms(new Key(this, "MyKey"))))
         // ...
         .database(myDatabase)
         .columns(List.of(Column.builder()
                 .name("col1")
                 .type(Schema.STRING)
                 .build()))
         .dataFormat(DataFormat.JSON)
         .build();

  • KmsManaged - Server-side encryption (SSE-KMS), like Kms, except with an AWS KMS Key managed by the AWS Key Management Service.

 Database myDatabase;
 
 S3Table.Builder.create(this, "MyTable")
         .storage(S3TableStorage.managedBucket(S3TableEncryption.kmsManaged()))
         // ...
         .database(myDatabase)
         .columns(List.of(Column.builder()
                 .name("col1")
                 .type(Schema.STRING)
                 .build()))
         .dataFormat(DataFormat.JSON)
         .build();

Client-side encryption (CSE-KMS) is independent of the bucket's server-side encryption and works with either a managed or an existing bucket. Configure it with clientSideEncryption: 

 Database myDatabase;
 
 // KMS key is created automatically
 // KMS key is created automatically
 S3Table.Builder.create(this, "MyTable")
         .clientSideEncryption(TableClientSideEncryption.kms())
         // ...
         .database(myDatabase)
         .columns(List.of(Column.builder()
                 .name("col1")
                 .type(Schema.STRING)
                 .build()))
         .dataFormat(DataFormat.JSON)
         .build();
 
 // with an explicit KMS key
 // with an explicit KMS key
 S3Table.Builder.create(this, "MyTable")
         .clientSideEncryption(TableClientSideEncryption.kms(new Key(this, "MyKey")))
         // ...
         .database(myDatabase)
         .columns(List.of(Column.builder()
                 .name("col1")
                 .type(Schema.STRING)
                 .build()))
         .dataFormat(DataFormat.JSON)
         .build();

To store the table's data in an existing bucket, use glue.S3TableStorage.fromBucket(bucket). CDK does not manage that bucket's server-side encryption, so an encryption choice can never be paired with a provided bucket — but client-side encryption still applies.

Marking table data as encrypted

Both S3Table and ExternalTable set the has_encrypted_data table parameter, which Athena reads when querying client-side (CSE-KMS) encrypted datasets. It defaults to true. Set hasEncryptedData to false when the underlying data is not encrypted: 

 Database myDatabase;
 
 S3Table.Builder.create(this, "MyTable")
         .hasEncryptedData(false)
         .database(myDatabase)
         .columns(List.of(Column.builder()
                 .name("col1")
                 .type(Schema.STRING)
                 .build()))
         .dataFormat(DataFormat.JSON)
         .build();

Do not set has_encrypted_data through the free-form parameters map as well - a value there that conflicts with hasEncryptedData is rejected at synthesis time.

Types

A table's schema is a collection of columns, each of which have a name and a type. Types are recursive structures, consisting of primitive and complex types: 

 Database myDatabase;
 
 S3Table.Builder.create(this, "MyTable")
         .columns(List.of(Column.builder()
                 .name("primitive_column")
                 .type(Schema.STRING)
                 .build(), Column.builder()
                 .name("array_column")
                 .type(Schema.array(Schema.INTEGER))
                 .comment("array<integer>")
                 .build(), Column.builder()
                 .name("map_column")
                 .type(Schema.map(Schema.STRING, Schema.TIMESTAMP))
                 .comment("map<string,timestamp>")
                 .build(), Column.builder()
                 .name("struct_column")
                 .type(Schema.struct(List.of(Column.builder()
                         .name("nested_column")
                         .type(Schema.DATE)
                         .comment("nested comment")
                         .build())))
                 .comment("struct<nested_column:date COMMENT 'nested comment'>")
                 .build()))
         // ...
         .database(myDatabase)
         .dataFormat(DataFormat.JSON)
         .build();

For a type the Schema factories don't model, use glue.Schema.custom('...'), which takes the raw Glue input string.

Public FAQ

What are we launching today?

We’re launching new features to an AWS CDK Glue L2 Construct to provide best-practice defaults and convenience methods to create Glue Jobs, Connections, Triggers, Workflows, and the underlying permissions and configuration.

Why should I use this Construct?

Developers should use this Construct to reduce the amount of boilerplate code and complexity each individual has to navigate, and make it easier to create best-practice Glue resources.

What’s not in scope?

Glue Crawlers and other resources that are now managed by the AWS LakeFormation team are not in scope for this effort. Developers should use existing methods to create these resources, and the new Glue L2 construct assumes they already exist as inputs. While best practice is for application and infrastructure code to be as close as possible for teams using fully-implemented DevOps mechanisms, in practice these ETL scripts are likely managed by a data science team who know Python or Scala and don’t necessarily own or manage their own infrastructure deployments. We want to meet developers where they are, and not assume that all of the code resides in the same repository, Developers can automate this themselves via the CDK, however, if they do own both.

Validating Glue version and feature use per AWS region at synth time is also not in scope. AWS’ intention is for all features to eventually be propagated to all Global regions, so the complexity involved in creating and updating region- specific configuration to match shifting feature sets does not out-weigh the likelihood that a developer will use this construct to deploy resources to a region without a particular new feature to a region that doesn’t yet support it without researching or manually attempting to use that feature before developing it via IaC. The developer will, of course, still get feedback from the underlying Glue APIs as CloudFormation deploys the resources similar to the current CDK L1 Glue experience.