AWS Glue Construct Library

Version 2.273.0-alpha.0

Separate package

@aws-cdk/aws-glue-alpha is not part of aws-cdk-lib — install it separately:

npm install @aws-cdk/aws-glue-alpha
pip install aws-cdk.aws-glue-alpha

Deprecated

This API may emit warnings. Backward compatibility is not guaranteed.

All constructs moved to aws-cdk-lib/aws-glue.

This module is part of the AWS Cloud Development Kit project.

README

AWS Glue is a serverless data integration service that makes it easier to discover, prepare, move, and integrate data from multiple sources for analytics, machine learning (ML), and application development.

The Glue L2 construct has convenience methods working backwards from common use cases and sets required parameters to defaults that align with recommended best practices for each job type. It also provides customers with a balance between flexibility via optional parameter overrides, and opinionated interfaces that discouraging anti-patterns, resulting in reduced time to develop and deploy new resources.

References

Glue Launch Announcement
Glue Documentation
Glue L1 (CloudFormation) Constructs
Prior version of the @aws-cdk/aws-glue-alpha module

Create a Glue Job

A Job encapsulates a script that connects to data sources, processes them, and then writes output to a data target. There are four types of Glue Jobs: Spark (ETL and Streaming), Python Shell, and Flex Jobs. Most of the required parameters for these jobs are common across all types, but there are a few differences depending on the languages supported and features provided by each type. For all job types, the L2 defaults to AWS best practice recommendations, such as:

Use of Secrets Manager for Connection JDBC strings
Glue job autoscaling
Default parameter values for Glue job creation

This iteration of the L2 construct introduces breaking changes to the existing glue-alpha-module, but these changes streamline the developer experience, introduce new constants for defaults, and replacing synth-time validations with interface contracts for enforcement of the parameter combinations that Glue supports. As an opinionated construct, the Glue L2 construct does not allow developers to create resources that use non-current versions of Glue or deprecated language dependencies (e.g. deprecated versions of Python). As always, L1s allow you to specify a wider range of parameters if you need or want to use alternative configurations.

Optional and required parameters for each job are enforced via interface rather than validation; see Glue's public documentation for more granular details.

Spark Jobs

ETL Jobs

ETL jobs support pySpark and Scala languages, for which there are separate but similar constructors. ETL jobs default to the G1 worker type, but you can override this default with other supported worker type values (G1, G2, G4 and G8). ETL jobs defaults to Glue version 4.0, which you can override to 3.0. The following ETL features are enabled by default: —enable-metrics, —enable-continuous-cloudwatch-log. The Spark UI (—enable-spark-ui) is off by default; enable it by setting the sparkUI prop. You can find more details about version, worker type and other features in Glue's public documentation.

Note on continuous logging and encryption: Because continuous logging is enabled by default, job driver and executor stdout/stderr are streamed to CloudWatch. Unless you attach a SecurityConfiguration with cloudWatchEncryption, these logs are written to the account-shared, default Glue log group (/aws-glue/jobs/logs-v2/), which is not encrypted with a customer-managed key. Since job logs can contain sensitive runtime data (SQL statements, row values, error stack traces), attach a SecurityConfiguration with cloudWatchEncryption for regulated workloads. The construct emits a synthesis-time warning when continuous logging is on and no SecurityConfiguration is attached.

Reference the pyspark-etl-jobs.test.ts and scalaspark-etl-jobs.test.ts unit tests for examples of required-only and optional job parameters when creating these types of jobs.

For the sake of brevity, examples are shown using the pySpark job variety.

Example with only required parameters:

import * as cdk from 'aws-cdk-lib';
import * as iam from 'aws-cdk-lib/aws-iam';
declare const stack: cdk.Stack;
declare const role: iam.IRole;
declare const script: glue.Code;
new glue.PySparkEtlJob(stack, 'PySparkETLJob', {
  role,
  script,
  jobName: 'PySparkETLJob',
});

python

import aws_cdk as cdk
import aws_cdk.aws_iam as iam
# stack: cdk.Stack
# role: iam.IRole
# script: glue.Code

glue.PySparkEtlJob(stack, "PySparkETLJob",
    role=role,
    script=script,
    job_name="PySparkETLJob"
)

java

import software.amazon.awscdk.*;
import software.amazon.awscdk.services.iam.*;
Stack stack;
IRole role;
Code script;

PySparkEtlJob.Builder.create(stack, "PySparkETLJob")
        .role(role)
        .script(script)
        .jobName("PySparkETLJob")
        .build();

csharp

using Amazon.CDK;
using Amazon.CDK.AWS.IAM;
Stack stack;
IRole role;
Code script;

new PySparkEtlJob(stack, "PySparkETLJob", new PySparkEtlJobProps {
    Role = role,
    Script = script,
    JobName = "PySparkETLJob"
});

import cdk "github.com/aws/aws-cdk-go/awscdk"
import iam "github.com/aws/aws-cdk-go/awscdk"
var stack Stack
var role IRole
var script Code

glue.NewPySparkEtlJob(stack, jsii.String("PySparkETLJob"), &PySparkEtlJobProps{
	Role: Role,
	Script: Script,
	JobName: jsii.String("PySparkETLJob"),
})

Example with optional override parameters:

import * as cdk from 'aws-cdk-lib';
import * as iam from 'aws-cdk-lib/aws-iam';
declare const stack: cdk.Stack;
declare const role: iam.IRole;
declare const script: glue.Code;
new glue.PySparkEtlJob(stack, 'PySparkETLJob', {
  jobName: 'PySparkETLJobCustomName',
  description: 'This is a description',
  role,
  script,
  glueVersion: glue.GlueVersion.V5_1,
  continuousLogging: { enabled: false },
  workerConfiguration: {
    workerType: glue.WorkerType.G_2X,
    numberOfWorkers: 2,
  },
  maxConcurrentRuns: 100,
  timeout: cdk.Duration.hours(2),
  connections: [glue.Connection.fromConnectionName(stack, 'Connection', 'connectionName')],
  securityConfiguration: glue.SecurityConfiguration.fromSecurityConfigurationName(stack, 'SecurityConfig', 'securityConfigName'),
  tags: {
    FirstTagName: 'FirstTagValue',
    SecondTagName: 'SecondTagValue',
    XTagName: 'XTagValue',
  },
  maxRetries: 2,
});

python

import aws_cdk as cdk
import aws_cdk.aws_iam as iam
# stack: cdk.Stack
# role: iam.IRole
# script: glue.Code

glue.PySparkEtlJob(stack, "PySparkETLJob",
    job_name="PySparkETLJobCustomName",
    description="This is a description",
    role=role,
    script=script,
    glue_version=glue.GlueVersion.V5_1,
    continuous_logging=glue.ContinuousLoggingProps(enabled=False),
    worker_configuration=glue.WorkerConfiguration(
        worker_type=glue.WorkerType.G_2X,
        number_of_workers=2
    ),
    max_concurrent_runs=100,
    timeout=cdk.Duration.hours(2),
    connections=[glue.Connection.from_connection_name(stack, "Connection", "connectionName")],
    security_configuration=glue.SecurityConfiguration.from_security_configuration_name(stack, "SecurityConfig", "securityConfigName"),
    tags={
        "FirstTagName": "FirstTagValue",
        "SecondTagName": "SecondTagValue",
        "XTagName": "XTagValue"
    },
    max_retries=2
)

java

import software.amazon.awscdk.*;
import software.amazon.awscdk.services.iam.*;
Stack stack;
IRole role;
Code script;

PySparkEtlJob.Builder.create(stack, "PySparkETLJob")
        .jobName("PySparkETLJobCustomName")
        .description("This is a description")
        .role(role)
        .script(script)
        .glueVersion(GlueVersion.V5_1)
        .continuousLogging(ContinuousLoggingProps.builder().enabled(false).build())
        .workerConfiguration(WorkerConfiguration.builder()
                .workerType(WorkerType.G_2X)
                .numberOfWorkers(2)
                .build())
        .maxConcurrentRuns(100)
        .timeout(Duration.hours(2))
        .connections(List.of(Connection.fromConnectionName(stack, "Connection", "connectionName")))
        .securityConfiguration(SecurityConfiguration.fromSecurityConfigurationName(stack, "SecurityConfig", "securityConfigName"))
        .tags(Map.of(
                "FirstTagName", "FirstTagValue",
                "SecondTagName", "SecondTagValue",
                "XTagName", "XTagValue"))
        .maxRetries(2)
        .build();

csharp

using Amazon.CDK;
using Amazon.CDK.AWS.IAM;
Stack stack;
IRole role;
Code script;

new PySparkEtlJob(stack, "PySparkETLJob", new PySparkEtlJobProps {
    JobName = "PySparkETLJobCustomName",
    Description = "This is a description",
    Role = role,
    Script = script,
    GlueVersion = GlueVersion.V5_1,
    ContinuousLogging = new ContinuousLoggingProps { Enabled = false },
    WorkerConfiguration = new WorkerConfiguration {
        WorkerType = WorkerType.G_2X,
        NumberOfWorkers = 2
    },
    MaxConcurrentRuns = 100,
    Timeout = Duration.Hours(2),
    Connections = new [] { Connection.FromConnectionName(stack, "Connection", "connectionName") },
    SecurityConfiguration = SecurityConfiguration.FromSecurityConfigurationName(stack, "SecurityConfig", "securityConfigName"),
    Tags = new Dictionary<string, string> {
        { "FirstTagName", "FirstTagValue" },
        { "SecondTagName", "SecondTagValue" },
        { "XTagName", "XTagValue" }
    },
    MaxRetries = 2
});

import cdk "github.com/aws/aws-cdk-go/awscdk"
import iam "github.com/aws/aws-cdk-go/awscdk"
var stack Stack
var role IRole
var script Code

glue.NewPySparkEtlJob(stack, jsii.String("PySparkETLJob"), &PySparkEtlJobProps{
	JobName: jsii.String("PySparkETLJobCustomName"),
	Description: jsii.String("This is a description"),
	Role: Role,
	Script: Script,
	GlueVersion: glue.GlueVersion_V5_1,
	ContinuousLogging: &ContinuousLoggingProps{
		Enabled: jsii.Boolean(false),
	},
	WorkerConfiguration: &WorkerConfiguration{
		WorkerType: glue.WorkerType_G_2X,
		NumberOfWorkers: jsii.Number(2),
	},
	MaxConcurrentRuns: jsii.Number(100),
	Timeout: cdk.Duration_Hours(jsii.Number(2)),
	Connections: []IConnection{
		glue.Connection_FromConnectionName(stack, jsii.String("Connection"), jsii.String("connectionName")),
	},
	SecurityConfiguration: glue.SecurityConfiguration_FromSecurityConfigurationName(stack, jsii.String("SecurityConfig"), jsii.String("securityConfigName")),
	Tags: map[string]*string{
		"FirstTagName": jsii.String("FirstTagValue"),
		"SecondTagName": jsii.String("SecondTagValue"),
		"XTagName": jsii.String("XTagValue"),
	},
	MaxRetries: jsii.Number(2),
})

Streaming Jobs

Streaming jobs are similar to ETL jobs, except that they perform ETL on data streams using the Apache Spark Structured Streaming framework. Some Spark job features are not available to Streaming ETL jobs. They support Scala and pySpark languages. PySpark streaming jobs run on Python 3. It defaults to the G1 worker type and Glue 4.0, both of which you can override. The following best practice features are enabled by default: —enable-metrics, —enable-continuous-cloudwatch-log. The Spark UI (—enable-spark-ui) is off by default; enable it by setting the sparkUI prop.

Reference the pyspark-streaming-jobs.test.ts and scalaspark-streaming-jobs.test.ts unit tests for examples of required-only and optional job parameters when creating these types of jobs.

Example with only required parameters:

import * as cdk from 'aws-cdk-lib';
import * as iam from 'aws-cdk-lib/aws-iam';
declare const stack: cdk.Stack;
declare const role: iam.IRole;
declare const script: glue.Code;
new glue.PySparkStreamingJob(stack, 'ImportedJob', { role, script });

python

import aws_cdk as cdk
import aws_cdk.aws_iam as iam
# stack: cdk.Stack
# role: iam.IRole
# script: glue.Code

glue.PySparkStreamingJob(stack, "ImportedJob", role=role, script=script)

java

import software.amazon.awscdk.*;
import software.amazon.awscdk.services.iam.*;
Stack stack;
IRole role;
Code script;

PySparkStreamingJob.Builder.create(stack, "ImportedJob").role(role).script(script).build();

csharp

using Amazon.CDK;
using Amazon.CDK.AWS.IAM;
Stack stack;
IRole role;
Code script;

new PySparkStreamingJob(stack, "ImportedJob", new PySparkStreamingJobProps { Role = role, Script = script });

import cdk "github.com/aws/aws-cdk-go/awscdk"
import iam "github.com/aws/aws-cdk-go/awscdk"
var stack Stack
var role IRole
var script Code

glue.NewPySparkStreamingJob(stack, jsii.String("ImportedJob"), &PySparkStreamingJobProps{
	Role: Role,
	Script: Script,
})

Example with optional override parameters:

import * as cdk from 'aws-cdk-lib';
import * as iam from 'aws-cdk-lib/aws-iam';
declare const stack: cdk.Stack;
declare const role: iam.IRole;
declare const script: glue.Code;
new glue.PySparkStreamingJob(stack, 'PySparkStreamingJob', {
  jobName: 'PySparkStreamingJobCustomName',
  description: 'This is a description',
  role,
  script,
  glueVersion: glue.GlueVersion.V5_1,
  continuousLogging: { enabled: false },
  workerConfiguration: {
    workerType: glue.WorkerType.G_2X,
    numberOfWorkers: 2,
  },
  maxConcurrentRuns: 100,
  timeout: cdk.Duration.hours(2),
  connections: [glue.Connection.fromConnectionName(stack, 'Connection', 'connectionName')],
  securityConfiguration: glue.SecurityConfiguration.fromSecurityConfigurationName(stack, 'SecurityConfig', 'securityConfigName'),
  tags: {
    FirstTagName: 'FirstTagValue',
    SecondTagName: 'SecondTagValue',
    XTagName: 'XTagValue',
  },
  maxRetries: 2,
});

python

import aws_cdk as cdk
import aws_cdk.aws_iam as iam
# stack: cdk.Stack
# role: iam.IRole
# script: glue.Code

glue.PySparkStreamingJob(stack, "PySparkStreamingJob",
    job_name="PySparkStreamingJobCustomName",
    description="This is a description",
    role=role,
    script=script,
    glue_version=glue.GlueVersion.V5_1,
    continuous_logging=glue.ContinuousLoggingProps(enabled=False),
    worker_configuration=glue.WorkerConfiguration(
        worker_type=glue.WorkerType.G_2X,
        number_of_workers=2
    ),
    max_concurrent_runs=100,
    timeout=cdk.Duration.hours(2),
    connections=[glue.Connection.from_connection_name(stack, "Connection", "connectionName")],
    security_configuration=glue.SecurityConfiguration.from_security_configuration_name(stack, "SecurityConfig", "securityConfigName"),
    tags={
        "FirstTagName": "FirstTagValue",
        "SecondTagName": "SecondTagValue",
        "XTagName": "XTagValue"
    },
    max_retries=2
)

java

import software.amazon.awscdk.*;
import software.amazon.awscdk.services.iam.*;
Stack stack;
IRole role;
Code script;

PySparkStreamingJob.Builder.create(stack, "PySparkStreamingJob")
        .jobName("PySparkStreamingJobCustomName")
        .description("This is a description")
        .role(role)
        .script(script)
        .glueVersion(GlueVersion.V5_1)
        .continuousLogging(ContinuousLoggingProps.builder().enabled(false).build())
        .workerConfiguration(WorkerConfiguration.builder()
                .workerType(WorkerType.G_2X)
                .numberOfWorkers(2)
                .build())
        .maxConcurrentRuns(100)
        .timeout(Duration.hours(2))
        .connections(List.of(Connection.fromConnectionName(stack, "Connection", "connectionName")))
        .securityConfiguration(SecurityConfiguration.fromSecurityConfigurationName(stack, "SecurityConfig", "securityConfigName"))
        .tags(Map.of(
                "FirstTagName", "FirstTagValue",
                "SecondTagName", "SecondTagValue",
                "XTagName", "XTagValue"))
        .maxRetries(2)
        .build();

csharp

using Amazon.CDK;
using Amazon.CDK.AWS.IAM;
Stack stack;
IRole role;
Code script;

new PySparkStreamingJob(stack, "PySparkStreamingJob", new PySparkStreamingJobProps {
    JobName = "PySparkStreamingJobCustomName",
    Description = "This is a description",
    Role = role,
    Script = script,
    GlueVersion = GlueVersion.V5_1,
    ContinuousLogging = new ContinuousLoggingProps { Enabled = false },
    WorkerConfiguration = new WorkerConfiguration {
        WorkerType = WorkerType.G_2X,
        NumberOfWorkers = 2
    },
    MaxConcurrentRuns = 100,
    Timeout = Duration.Hours(2),
    Connections = new [] { Connection.FromConnectionName(stack, "Connection", "connectionName") },
    SecurityConfiguration = SecurityConfiguration.FromSecurityConfigurationName(stack, "SecurityConfig", "securityConfigName"),
    Tags = new Dictionary<string, string> {
        { "FirstTagName", "FirstTagValue" },
        { "SecondTagName", "SecondTagValue" },
        { "XTagName", "XTagValue" }
    },
    MaxRetries = 2
});

import cdk "github.com/aws/aws-cdk-go/awscdk"
import iam "github.com/aws/aws-cdk-go/awscdk"
var stack Stack
var role IRole
var script Code

glue.NewPySparkStreamingJob(stack, jsii.String("PySparkStreamingJob"), &PySparkStreamingJobProps{
	JobName: jsii.String("PySparkStreamingJobCustomName"),
	Description: jsii.String("This is a description"),
	Role: Role,
	Script: Script,
	GlueVersion: glue.GlueVersion_V5_1,
	ContinuousLogging: &ContinuousLoggingProps{
		Enabled: jsii.Boolean(false),
	},
	WorkerConfiguration: &WorkerConfiguration{
		WorkerType: glue.WorkerType_G_2X,
		NumberOfWorkers: jsii.Number(2),
	},
	MaxConcurrentRuns: jsii.Number(100),
	Timeout: cdk.Duration_Hours(jsii.Number(2)),
	Connections: []IConnection{
		glue.Connection_FromConnectionName(stack, jsii.String("Connection"), jsii.String("connectionName")),
	},
	SecurityConfiguration: glue.SecurityConfiguration_FromSecurityConfigurationName(stack, jsii.String("SecurityConfig"), jsii.String("securityConfigName")),
	Tags: map[string]*string{
		"FirstTagName": jsii.String("FirstTagValue"),
		"SecondTagName": jsii.String("SecondTagValue"),
		"XTagName": jsii.String("XTagValue"),
	},
	MaxRetries: jsii.Number(2),
})

Flex Jobs

The flexible execution class is appropriate for non-urgent jobs such as pre-production jobs, testing, and one-time data loads. Flexible jobs default to Glue version 5.0 and worker type G_1X. The following best practice features are enabled by default: —enable-metrics, —enable-continuous-cloudwatch-log The Spark UI (—enable-spark-ui) is off by default; enable it by setting the sparkUI prop.

Reference the pyspark-flex-etl-jobs.test.ts and scalaspark-flex-etl-jobs.test.ts unit tests for examples of required-only and optional job parameters when creating these types of jobs.

Example with only required parameters:

import * as cdk from 'aws-cdk-lib';
import * as iam from 'aws-cdk-lib/aws-iam';
declare const stack: cdk.Stack;
declare const role: iam.IRole;
declare const script: glue.Code;
new glue.PySparkFlexEtlJob(stack, 'ImportedJob', { role, script });

python

import aws_cdk as cdk
import aws_cdk.aws_iam as iam
# stack: cdk.Stack
# role: iam.IRole
# script: glue.Code

glue.PySparkFlexEtlJob(stack, "ImportedJob", role=role, script=script)

java

import software.amazon.awscdk.*;
import software.amazon.awscdk.services.iam.*;
Stack stack;
IRole role;
Code script;

PySparkFlexEtlJob.Builder.create(stack, "ImportedJob").role(role).script(script).build();

csharp

using Amazon.CDK;
using Amazon.CDK.AWS.IAM;
Stack stack;
IRole role;
Code script;

new PySparkFlexEtlJob(stack, "ImportedJob", new PySparkFlexEtlJobProps { Role = role, Script = script });

import cdk "github.com/aws/aws-cdk-go/awscdk"
import iam "github.com/aws/aws-cdk-go/awscdk"
var stack Stack
var role IRole
var script Code

glue.NewPySparkFlexEtlJob(stack, jsii.String("ImportedJob"), &PySparkFlexEtlJobProps{
	Role: Role,
	Script: Script,
})

Example with optional override parameters:

import * as cdk from 'aws-cdk-lib';
import * as iam from 'aws-cdk-lib/aws-iam';
declare const stack: cdk.Stack;
declare const role: iam.IRole;
declare const script: glue.Code;
new glue.PySparkFlexEtlJob(stack, 'pySparkFlexEtlJob', {
  jobName: 'pySparkFlexEtlJob',
  description: 'This is a description',
  role,
  script,
  glueVersion: glue.GlueVersion.V5_1,
  continuousLogging: { enabled: false },
  workerConfiguration: {
    workerType: glue.WorkerType.G_2X,
    numberOfWorkers: 2,
  },
  maxConcurrentRuns: 100,
  timeout: cdk.Duration.hours(2),
  connections: [glue.Connection.fromConnectionName(stack, 'Connection', 'connectionName')],
  securityConfiguration: glue.SecurityConfiguration.fromSecurityConfigurationName(stack, 'SecurityConfig', 'securityConfigName'),
  tags: {
    FirstTagName: 'FirstTagValue',
    SecondTagName: 'SecondTagValue',
    XTagName: 'XTagValue',
  },
  maxRetries: 2,
});

python

import aws_cdk as cdk
import aws_cdk.aws_iam as iam
# stack: cdk.Stack
# role: iam.IRole
# script: glue.Code

glue.PySparkFlexEtlJob(stack, "pySparkFlexEtlJob",
    job_name="pySparkFlexEtlJob",
    description="This is a description",
    role=role,
    script=script,
    glue_version=glue.GlueVersion.V5_1,
    continuous_logging=glue.ContinuousLoggingProps(enabled=False),
    worker_configuration=glue.WorkerConfiguration(
        worker_type=glue.WorkerType.G_2X,
        number_of_workers=2
    ),
    max_concurrent_runs=100,
    timeout=cdk.Duration.hours(2),
    connections=[glue.Connection.from_connection_name(stack, "Connection", "connectionName")],
    security_configuration=glue.SecurityConfiguration.from_security_configuration_name(stack, "SecurityConfig", "securityConfigName"),
    tags={
        "FirstTagName": "FirstTagValue",
        "SecondTagName": "SecondTagValue",
        "XTagName": "XTagValue"
    },
    max_retries=2
)

java

import software.amazon.awscdk.*;
import software.amazon.awscdk.services.iam.*;
Stack stack;
IRole role;
Code script;

PySparkFlexEtlJob.Builder.create(stack, "pySparkFlexEtlJob")
        .jobName("pySparkFlexEtlJob")
        .description("This is a description")
        .role(role)
        .script(script)
        .glueVersion(GlueVersion.V5_1)
        .continuousLogging(ContinuousLoggingProps.builder().enabled(false).build())
        .workerConfiguration(WorkerConfiguration.builder()
                .workerType(WorkerType.G_2X)
                .numberOfWorkers(2)
                .build())
        .maxConcurrentRuns(100)
        .timeout(Duration.hours(2))
        .connections(List.of(Connection.fromConnectionName(stack, "Connection", "connectionName")))
        .securityConfiguration(SecurityConfiguration.fromSecurityConfigurationName(stack, "SecurityConfig", "securityConfigName"))
        .tags(Map.of(
                "FirstTagName", "FirstTagValue",
                "SecondTagName", "SecondTagValue",
                "XTagName", "XTagValue"))
        .maxRetries(2)
        .build();

csharp

using Amazon.CDK;
using Amazon.CDK.AWS.IAM;
Stack stack;
IRole role;
Code script;

new PySparkFlexEtlJob(stack, "pySparkFlexEtlJob", new PySparkFlexEtlJobProps {
    JobName = "pySparkFlexEtlJob",
    Description = "This is a description",
    Role = role,
    Script = script,
    GlueVersion = GlueVersion.V5_1,
    ContinuousLogging = new ContinuousLoggingProps { Enabled = false },
    WorkerConfiguration = new WorkerConfiguration {
        WorkerType = WorkerType.G_2X,
        NumberOfWorkers = 2
    },
    MaxConcurrentRuns = 100,
    Timeout = Duration.Hours(2),
    Connections = new [] { Connection.FromConnectionName(stack, "Connection", "connectionName") },
    SecurityConfiguration = SecurityConfiguration.FromSecurityConfigurationName(stack, "SecurityConfig", "securityConfigName"),
    Tags = new Dictionary<string, string> {
        { "FirstTagName", "FirstTagValue" },
        { "SecondTagName", "SecondTagValue" },
        { "XTagName", "XTagValue" }
    },
    MaxRetries = 2
});

import cdk "github.com/aws/aws-cdk-go/awscdk"
import iam "github.com/aws/aws-cdk-go/awscdk"
var stack Stack
var role IRole
var script Code

glue.NewPySparkFlexEtlJob(stack, jsii.String("pySparkFlexEtlJob"), &PySparkFlexEtlJobProps{
	JobName: jsii.String("pySparkFlexEtlJob"),
	Description: jsii.String("This is a description"),
	Role: Role,
	Script: Script,
	GlueVersion: glue.GlueVersion_V5_1,
	ContinuousLogging: &ContinuousLoggingProps{
		Enabled: jsii.Boolean(false),
	},
	WorkerConfiguration: &WorkerConfiguration{
		WorkerType: glue.WorkerType_G_2X,
		NumberOfWorkers: jsii.Number(2),
	},
	MaxConcurrentRuns: jsii.Number(100),
	Timeout: cdk.Duration_Hours(jsii.Number(2)),
	Connections: []IConnection{
		glue.Connection_FromConnectionName(stack, jsii.String("Connection"), jsii.String("connectionName")),
	},
	SecurityConfiguration: glue.SecurityConfiguration_FromSecurityConfigurationName(stack, jsii.String("SecurityConfig"), jsii.String("securityConfigName")),
	Tags: map[string]*string{
		"FirstTagName": jsii.String("FirstTagValue"),
		"SecondTagName": jsii.String("SecondTagValue"),
		"XTagName": jsii.String("XTagValue"),
	},
	MaxRetries: jsii.Number(2),
})

Python Shell Jobs

Python shell jobs support a Python version that depends on the AWS Glue version you use. These can be used to schedule and run tasks that don't require an Apache Spark environment. Python shell jobs default to Python 3.9 and a MaxCapacity of 0.0625. Python 3.9 supports pre-loaded analytics libraries, enabled by default (librarySet: glue.LibrarySet.ANALYTICS). Set librarySet: glue.LibrarySet.NONE when your libraries are custom or conflict with the pre-installed ones.

Reference the pyspark-shell-job.test.ts unit tests for examples of required-only and optional job parameters when creating these types of jobs.

Example with only required parameters:

import * as cdk from 'aws-cdk-lib';
import * as iam from 'aws-cdk-lib/aws-iam';
declare const stack: cdk.Stack;
declare const role: iam.IRole;
declare const script: glue.Code;
new glue.PythonShellJob(stack, 'ImportedJob', { role, script });

python

import aws_cdk as cdk
import aws_cdk.aws_iam as iam
# stack: cdk.Stack
# role: iam.IRole
# script: glue.Code

glue.PythonShellJob(stack, "ImportedJob", role=role, script=script)

java

import software.amazon.awscdk.*;
import software.amazon.awscdk.services.iam.*;
Stack stack;
IRole role;
Code script;

PythonShellJob.Builder.create(stack, "ImportedJob").role(role).script(script).build();

csharp

using Amazon.CDK;
using Amazon.CDK.AWS.IAM;
Stack stack;
IRole role;
Code script;

new PythonShellJob(stack, "ImportedJob", new PythonShellJobProps { Role = role, Script = script });

import cdk "github.com/aws/aws-cdk-go/awscdk"
import iam "github.com/aws/aws-cdk-go/awscdk"
var stack Stack
var role IRole
var script Code

glue.NewPythonShellJob(stack, jsii.String("ImportedJob"), &PythonShellJobProps{
	Role: Role,
	Script: Script,
})

Example with optional override parameters:

import * as cdk from 'aws-cdk-lib';
import * as iam from 'aws-cdk-lib/aws-iam';
declare const stack: cdk.Stack;
declare const role: iam.IRole;
declare const script: glue.Code;
declare const extraPythonFile: glue.Code;
new glue.PythonShellJob(stack, 'PythonShellJob', {
  jobName: 'PythonShellJobCustomName',
  description: 'This is a description',
  pythonVersion: glue.PythonVersion.THREE_NINE,
  maxCapacity: glue.MaxCapacity.DPU_1,
  role,
  script,
  extraPythonFiles: [extraPythonFile],
  glueVersion: glue.GlueVersion.V3_0,
  continuousLogging: { enabled: false },
  maxConcurrentRuns: 100,
  timeout: cdk.Duration.hours(2),
  connections: [glue.Connection.fromConnectionName(stack, 'Connection', 'connectionName')],
  securityConfiguration: glue.SecurityConfiguration.fromSecurityConfigurationName(stack, 'SecurityConfig', 'securityConfigName'),
  tags: {
    FirstTagName: 'FirstTagValue',
    SecondTagName: 'SecondTagValue',
    XTagName: 'XTagValue',
  },
  maxRetries: 2,
});

python

import aws_cdk as cdk
import aws_cdk.aws_iam as iam
# stack: cdk.Stack
# role: iam.IRole
# script: glue.Code
# extra_python_file: glue.Code

glue.PythonShellJob(stack, "PythonShellJob",
    job_name="PythonShellJobCustomName",
    description="This is a description",
    python_version=glue.PythonVersion.THREE_NINE,
    max_capacity=glue.MaxCapacity.DPU_1,
    role=role,
    script=script,
    extra_python_files=[extra_python_file],
    glue_version=glue.GlueVersion.V3_0,
    continuous_logging=glue.ContinuousLoggingProps(enabled=False),
    max_concurrent_runs=100,
    timeout=cdk.Duration.hours(2),
    connections=[glue.Connection.from_connection_name(stack, "Connection", "connectionName")],
    security_configuration=glue.SecurityConfiguration.from_security_configuration_name(stack, "SecurityConfig", "securityConfigName"),
    tags={
        "FirstTagName": "FirstTagValue",
        "SecondTagName": "SecondTagValue",
        "XTagName": "XTagValue"
    },
    max_retries=2
)

java

import software.amazon.awscdk.*;
import software.amazon.awscdk.services.iam.*;
Stack stack;
IRole role;
Code script;
Code extraPythonFile;

PythonShellJob.Builder.create(stack, "PythonShellJob")
        .jobName("PythonShellJobCustomName")
        .description("This is a description")
        .pythonVersion(PythonVersion.THREE_NINE)
        .maxCapacity(MaxCapacity.DPU_1)
        .role(role)
        .script(script)
        .extraPythonFiles(List.of(extraPythonFile))
        .glueVersion(GlueVersion.V3_0)
        .continuousLogging(ContinuousLoggingProps.builder().enabled(false).build())
        .maxConcurrentRuns(100)
        .timeout(Duration.hours(2))
        .connections(List.of(Connection.fromConnectionName(stack, "Connection", "connectionName")))
        .securityConfiguration(SecurityConfiguration.fromSecurityConfigurationName(stack, "SecurityConfig", "securityConfigName"))
        .tags(Map.of(
                "FirstTagName", "FirstTagValue",
                "SecondTagName", "SecondTagValue",
                "XTagName", "XTagValue"))
        .maxRetries(2)
        .build();

csharp

using Amazon.CDK;
using Amazon.CDK.AWS.IAM;
Stack stack;
IRole role;
Code script;
Code extraPythonFile;

new PythonShellJob(stack, "PythonShellJob", new PythonShellJobProps {
    JobName = "PythonShellJobCustomName",
    Description = "This is a description",
    PythonVersion = PythonVersion.THREE_NINE,
    MaxCapacity = MaxCapacity.DPU_1,
    Role = role,
    Script = script,
    ExtraPythonFiles = new [] { extraPythonFile },
    GlueVersion = GlueVersion.V3_0,
    ContinuousLogging = new ContinuousLoggingProps { Enabled = false },
    MaxConcurrentRuns = 100,
    Timeout = Duration.Hours(2),
    Connections = new [] { Connection.FromConnectionName(stack, "Connection", "connectionName") },
    SecurityConfiguration = SecurityConfiguration.FromSecurityConfigurationName(stack, "SecurityConfig", "securityConfigName"),
    Tags = new Dictionary<string, string> {
        { "FirstTagName", "FirstTagValue" },
        { "SecondTagName", "SecondTagValue" },
        { "XTagName", "XTagValue" }
    },
    MaxRetries = 2
});

import cdk "github.com/aws/aws-cdk-go/awscdk"
import iam "github.com/aws/aws-cdk-go/awscdk"
var stack Stack
var role IRole
var script Code
var extraPythonFile Code

glue.NewPythonShellJob(stack, jsii.String("PythonShellJob"), &PythonShellJobProps{
	JobName: jsii.String("PythonShellJobCustomName"),
	Description: jsii.String("This is a description"),
	PythonVersion: glue.PythonVersion_THREE_NINE,
	MaxCapacity: glue.MaxCapacity_DPU_1,
	Role: Role,
	Script: Script,
	ExtraPythonFiles: []Code{
		extraPythonFile,
	},
	GlueVersion: glue.GlueVersion_V3_0,
	ContinuousLogging: &ContinuousLoggingProps{
		Enabled: jsii.Boolean(false),
	},
	MaxConcurrentRuns: jsii.Number(100),
	Timeout: cdk.Duration_Hours(jsii.Number(2)),
	Connections: []IConnection{
		glue.Connection_FromConnectionName(stack, jsii.String("Connection"), jsii.String("connectionName")),
	},
	SecurityConfiguration: glue.SecurityConfiguration_FromSecurityConfigurationName(stack, jsii.String("SecurityConfig"), jsii.String("securityConfigName")),
	Tags: map[string]*string{
		"FirstTagName": jsii.String("FirstTagValue"),
		"SecondTagName": jsii.String("SecondTagValue"),
		"XTagName": jsii.String("XTagValue"),
	},
	MaxRetries: jsii.Number(2),
})

Ray Jobs

⚠️ DEPRECATED: AWS Glue for Ray is closed to new customers as of April 30, 2026 and is in maintenance mode. Migrate to Amazon EKS with KubeRay Operator.

The RayJob construct, Runtime.RAY_TWO_FOUR, and JobType.RAY are deprecated and will be removed in a future release.

Metrics Control

By default, Glue jobs enable CloudWatch metrics (--enable-metrics) and observability metrics (--enable-observability-metrics) for monitoring and debugging. You can disable these metrics to reduce CloudWatch costs:

import * as cdk from 'aws-cdk-lib';
import * as iam from 'aws-cdk-lib/aws-iam';
declare const stack: cdk.Stack;
declare const role: iam.IRole;
declare const script: glue.Code;

// Disable both metrics for cost optimization
new glue.PySparkEtlJob(stack, 'CostOptimizedJob', {
  role,
  script,
  enableMetrics: false,
  enableObservabilityMetrics: false,
});

// Selective control - keep observability, disable profiling
new glue.PySparkEtlJob(stack, 'SelectiveJob', {
  role,
  script,
  enableMetrics: false,
  // enableObservabilityMetrics defaults to true
});

python

import aws_cdk as cdk
import aws_cdk.aws_iam as iam
# stack: cdk.Stack
# role: iam.IRole
# script: glue.Code


# Disable both metrics for cost optimization
glue.PySparkEtlJob(stack, "CostOptimizedJob",
    role=role,
    script=script,
    enable_metrics=False,
    enable_observability_metrics=False
)

# Selective control - keep observability, disable profiling
glue.PySparkEtlJob(stack, "SelectiveJob",
    role=role,
    script=script,
    enable_metrics=False
)

java

import software.amazon.awscdk.*;
import software.amazon.awscdk.services.iam.*;
Stack stack;
IRole role;
Code script;


// Disable both metrics for cost optimization
// Disable both metrics for cost optimization
PySparkEtlJob.Builder.create(stack, "CostOptimizedJob")
        .role(role)
        .script(script)
        .enableMetrics(false)
        .enableObservabilityMetrics(false)
        .build();

// Selective control - keep observability, disable profiling
// Selective control - keep observability, disable profiling
PySparkEtlJob.Builder.create(stack, "SelectiveJob")
        .role(role)
        .script(script)
        .enableMetrics(false)
        .build();

csharp

using Amazon.CDK;
using Amazon.CDK.AWS.IAM;
Stack stack;
IRole role;
Code script;


// Disable both metrics for cost optimization
// Disable both metrics for cost optimization
new PySparkEtlJob(stack, "CostOptimizedJob", new PySparkEtlJobProps {
    Role = role,
    Script = script,
    EnableMetrics = false,
    EnableObservabilityMetrics = false
});

// Selective control - keep observability, disable profiling
// Selective control - keep observability, disable profiling
new PySparkEtlJob(stack, "SelectiveJob", new PySparkEtlJobProps {
    Role = role,
    Script = script,
    EnableMetrics = false
});

import cdk "github.com/aws/aws-cdk-go/awscdk"
import iam "github.com/aws/aws-cdk-go/awscdk"
var stack Stack
var role IRole
var script Code


// Disable both metrics for cost optimization
// Disable both metrics for cost optimization
glue.NewPySparkEtlJob(stack, jsii.String("CostOptimizedJob"), &PySparkEtlJobProps{
	Role: Role,
	Script: Script,
	EnableMetrics: jsii.Boolean(false),
	EnableObservabilityMetrics: jsii.Boolean(false),
})

// Selective control - keep observability, disable profiling
// Selective control - keep observability, disable profiling
glue.NewPySparkEtlJob(stack, jsii.String("SelectiveJob"), &PySparkEtlJobProps{
	Role: Role,
	Script: Script,
	EnableMetrics: jsii.Boolean(false),
})

This feature is available for all Spark job types (ETL, Streaming, Flex).

Job Arguments

Glue jobs are configured through a map of name-value arguments (DefaultArguments). This construct manages several of these arguments on your behalf and exposes each one through a dedicated, strongly-typed prop:

Managed argument(s)	Prop
`--enable-continuous-cloudwatch-log`, `--continuous-log-*`	`continuousLogging`
`--enable-metrics`	`enableMetrics`
`--enable-observability-metrics`	`enableObservabilityMetrics`
`--enable-spark-ui`, `--spark-event-logs-path`	`sparkUI`
`--job-language`, `--class`	job class / `className`
`--extra-jars`, `--user-jars-first`, `--extra-py-files`, `--extra-files`	`extraJars`, `extraJarsFirst`, `extraPythonFiles`, `extraFiles`
`library-set`	`librarySet` (Python Shell)

The defaultArguments prop is the escape hatch for arguments this construct does not model. Use it for any argument without a dedicated prop:

import * as cdk from 'aws-cdk-lib';
import * as iam from 'aws-cdk-lib/aws-iam';
declare const stack: cdk.Stack;
declare const role: iam.IRole;
declare const script: glue.Code;

new glue.PySparkEtlJob(stack, 'PySparkETLJob', {
  role,
  script,
  defaultArguments: {
    // an argument this construct does not manage
    '--enable-glue-datacatalog': 'true',
  },
});

python

import aws_cdk as cdk
import aws_cdk.aws_iam as iam
# stack: cdk.Stack
# role: iam.IRole
# script: glue.Code


glue.PySparkEtlJob(stack, "PySparkETLJob",
    role=role,
    script=script,
    default_arguments={
        # an argument this construct does not manage
        "--enable-glue-datacatalog": "true"
    }
)

java

import software.amazon.awscdk.*;
import software.amazon.awscdk.services.iam.*;
Stack stack;
IRole role;
Code script;


PySparkEtlJob.Builder.create(stack, "PySparkETLJob")
        .role(role)
        .script(script)
        .defaultArguments(Map.of(
                // an argument this construct does not manage
                "--enable-glue-datacatalog", "true"))
        .build();

csharp

using Amazon.CDK;
using Amazon.CDK.AWS.IAM;
Stack stack;
IRole role;
Code script;


new PySparkEtlJob(stack, "PySparkETLJob", new PySparkEtlJobProps {
    Role = role,
    Script = script,
    DefaultArguments = new Dictionary<string, string> {
        // an argument this construct does not manage
        { "--enable-glue-datacatalog", "true" }
    }
});

import cdk "github.com/aws/aws-cdk-go/awscdk"
import iam "github.com/aws/aws-cdk-go/awscdk"
var stack Stack
var role IRole
var script Code


glue.NewPySparkEtlJob(stack, jsii.String("PySparkETLJob"), &PySparkEtlJobProps{
	Role: Role,
	Script: Script,
	DefaultArguments: map[string]*string{
		// an argument this construct does not manage
		"--enable-glue-datacatalog": jsii.String("true"),
	},
})

To keep a single, unambiguous way to express each intent, setting a construct-managed argument (any argument in the table above) or a Glue-reserved argument (--debug, --mode, --JOB_NAME, --endpoint) through defaultArguments throws at synthesis time. This holds even when the feature is turned off — for example, enableMetrics: false combined with defaultArguments: { '--enable-metrics': '' } throws rather than silently re-enabling metrics. Configure managed arguments through their dedicated prop instead — for example, use continuousLogging: { enabled: false } rather than defaultArguments: { '--enable-continuous-cloudwatch-log': 'false' }.

Enable Job Run Queuing

AWS Glue job queuing monitors your account level quotas and limits. If quotas or limits are insufficient to start a Glue job run, AWS Glue will automatically queue the job and wait for limits to free up. Once limits become available, AWS Glue will retry the job run. Glue jobs will queue for limits like max concurrent job runs per account, max concurrent Data Processing Units (DPU), and resource unavailable due to IP address exhaustion in Amazon Virtual Private Cloud (Amazon VPC).

Enable job run queuing by setting the jobRunQueuingEnabled property to true.

import * as cdk from 'aws-cdk-lib';
import * as iam from 'aws-cdk-lib/aws-iam';
declare const stack: cdk.Stack;
declare const role: iam.IRole;
declare const script: glue.Code;
new glue.PySparkEtlJob(stack, 'PySparkETLJob', {
  role,
  script,
  jobName: 'PySparkETLJob',
  jobRunQueuingEnabled: true
});

python

import aws_cdk as cdk
import aws_cdk.aws_iam as iam
# stack: cdk.Stack
# role: iam.IRole
# script: glue.Code

glue.PySparkEtlJob(stack, "PySparkETLJob",
    role=role,
    script=script,
    job_name="PySparkETLJob",
    job_run_queuing_enabled=True
)

java

import software.amazon.awscdk.*;
import software.amazon.awscdk.services.iam.*;
Stack stack;
IRole role;
Code script;

PySparkEtlJob.Builder.create(stack, "PySparkETLJob")
        .role(role)
        .script(script)
        .jobName("PySparkETLJob")
        .jobRunQueuingEnabled(true)
        .build();

csharp

using Amazon.CDK;
using Amazon.CDK.AWS.IAM;
Stack stack;
IRole role;
Code script;

new PySparkEtlJob(stack, "PySparkETLJob", new PySparkEtlJobProps {
    Role = role,
    Script = script,
    JobName = "PySparkETLJob",
    JobRunQueuingEnabled = true
});

import cdk "github.com/aws/aws-cdk-go/awscdk"
import iam "github.com/aws/aws-cdk-go/awscdk"
var stack Stack
var role IRole
var script Code

glue.NewPySparkEtlJob(stack, jsii.String("PySparkETLJob"), &PySparkEtlJobProps{
	Role: Role,
	Script: Script,
	JobName: jsii.String("PySparkETLJob"),
	JobRunQueuingEnabled: jsii.Boolean(true),
})

Uploading scripts from the CDK app repository to S3

Similar to other L2 constructs, the Glue L2 automates uploading local scripts to S3. Use glue.Code.fromAsset(path) to point at a script in your local file structure; it is uploaded to the CDK-managed asset bucket. To reference a script that already exists in S3, use glue.Code.fromBucket(bucket, key), which performs no upload. A script is required for every job.

Reference the unit tests for examples of repo and S3 code target examples.

Workflow Triggers

You can use Glue workflows to create and visualize complex extract, transform, and load (ETL) activities involving multiple crawlers, jobs, and triggers. Standalone triggers are an anti-pattern, so you must create triggers from within a workflow using the L2 construct.

Within a workflow object, there are functions to create different types of triggers with actions and predicates. You add triggers to the workflow, and each trigger references the jobs or crawlers it runs as its actions.

startOnCreation applies to scheduled triggers (and, via ConditionalTriggerOptions, conditional triggers) only. It defaults to false, but you can override it if you prefer for your trigger to start on creation.

Reference the workflow-triggers.test.ts unit tests for examples of creating workflows and triggers.

import * as cdk from 'aws-cdk-lib';
import * as iam from 'aws-cdk-lib/aws-iam';
declare const stack: cdk.Stack;
declare const role: iam.IRole;
declare const script: glue.Code;

// Create a job to run from the workflow
const job = new glue.PySparkEtlJob(stack, 'Job', { role, script });

// Create a workflow and add a trigger that runs the job
const workflow = new glue.Workflow(stack, 'Workflow');
workflow.addOnDemandTrigger('OnDemandTrigger', {
  actions: [glue.Action.job(job)],
});

python

import aws_cdk as cdk
import aws_cdk.aws_iam as iam
# stack: cdk.Stack
# role: iam.IRole
# script: glue.Code


# Create a job to run from the workflow
job = glue.PySparkEtlJob(stack, "Job", role=role, script=script)

# Create a workflow and add a trigger that runs the job
workflow = glue.Workflow(stack, "Workflow")
workflow.add_on_demand_trigger("OnDemandTrigger",
    actions=[glue.Action.job(job)]
)

java

import software.amazon.awscdk.*;
import software.amazon.awscdk.services.iam.*;
Stack stack;
IRole role;
Code script;


// Create a job to run from the workflow
PySparkEtlJob job = PySparkEtlJob.Builder.create(stack, "Job").role(role).script(script).build();

// Create a workflow and add a trigger that runs the job
Workflow workflow = new Workflow(stack, "Workflow");
workflow.addOnDemandTrigger("OnDemandTrigger", OnDemandTriggerOptions.builder()
        .actions(List.of(Action.job(job)))
        .build());

csharp

using Amazon.CDK;
using Amazon.CDK.AWS.IAM;
Stack stack;
IRole role;
Code script;


// Create a job to run from the workflow
var job = new PySparkEtlJob(stack, "Job", new PySparkEtlJobProps { Role = role, Script = script });

// Create a workflow and add a trigger that runs the job
var workflow = new Workflow(stack, "Workflow");
workflow.AddOnDemandTrigger("OnDemandTrigger", new OnDemandTriggerOptions {
    Actions = new [] { Action.Job(job) }
});

import cdk "github.com/aws/aws-cdk-go/awscdk"
import iam "github.com/aws/aws-cdk-go/awscdk"
var stack Stack
var role IRole
var script Code


// Create a job to run from the workflow
job := glue.NewPySparkEtlJob(stack, jsii.String("Job"), &PySparkEtlJobProps{
	Role: Role,
	Script: Script,
})

// Create a workflow and add a trigger that runs the job
workflow := glue.NewWorkflow(stack, jsii.String("Workflow"))
workflow.AddOnDemandTrigger(jsii.String("OnDemandTrigger"), &OnDemandTriggerOptions{
	Actions: []Action{
		glue.Action_Job(job),
	},
})

1. On-Demand Triggers

On-demand triggers can start glue jobs or crawlers. This construct provides convenience functions to create on-demand crawler or job triggers. The constructor takes an optional description parameter, but abstracts the requirement of an actions list using the job or crawler objects using conditional types.

2. Scheduled Triggers

Use addScheduledTrigger with a TriggerSchedule to fire on a cron schedule. TriggerSchedule.daily() and TriggerSchedule.weekly() are convenience factories; TriggerSchedule.cron(...) lets you build any schedule from the existing event Schedule class without writing raw cron expressions. The L2 extracts the expression that Glue requires from the TriggerSchedule.

import * as cdk from 'aws-cdk-lib';
import * as iam from 'aws-cdk-lib/aws-iam';
declare const stack: cdk.Stack;
declare const role: iam.IRole;
declare const script: glue.Code;
const job = new glue.PySparkEtlJob(stack, 'Job', { role, script });
const workflow = new glue.Workflow(stack, 'Workflow');

workflow.addScheduledTrigger('WeeklyTrigger', {
  actions: [glue.Action.job(job)],
  schedule: glue.TriggerSchedule.weekly(),
});

python

import aws_cdk as cdk
import aws_cdk.aws_iam as iam
# stack: cdk.Stack
# role: iam.IRole
# script: glue.Code

job = glue.PySparkEtlJob(stack, "Job", role=role, script=script)
workflow = glue.Workflow(stack, "Workflow")

workflow.add_scheduled_trigger("WeeklyTrigger",
    actions=[glue.Action.job(job)],
    schedule=glue.TriggerSchedule.weekly()
)

java

import software.amazon.awscdk.*;
import software.amazon.awscdk.services.iam.*;
Stack stack;
IRole role;
Code script;

PySparkEtlJob job = PySparkEtlJob.Builder.create(stack, "Job").role(role).script(script).build();
Workflow workflow = new Workflow(stack, "Workflow");

workflow.addScheduledTrigger("WeeklyTrigger", ScheduledTriggerOptions.builder()
        .actions(List.of(Action.job(job)))
        .schedule(TriggerSchedule.weekly())
        .build());

csharp

using Amazon.CDK;
using Amazon.CDK.AWS.IAM;
Stack stack;
IRole role;
Code script;

var job = new PySparkEtlJob(stack, "Job", new PySparkEtlJobProps { Role = role, Script = script });
var workflow = new Workflow(stack, "Workflow");

workflow.AddScheduledTrigger("WeeklyTrigger", new ScheduledTriggerOptions {
    Actions = new [] { Action.Job(job) },
    Schedule = TriggerSchedule.Weekly()
});

import cdk "github.com/aws/aws-cdk-go/awscdk"
import iam "github.com/aws/aws-cdk-go/awscdk"
var stack Stack
var role IRole
var script Code

job := glue.NewPySparkEtlJob(stack, jsii.String("Job"), &PySparkEtlJobProps{
	Role: Role,
	Script: Script,
})
workflow := glue.NewWorkflow(stack, jsii.String("Workflow"))

workflow.AddScheduledTrigger(jsii.String("WeeklyTrigger"), &ScheduledTriggerOptions{
	Actions: []Action{
		glue.Action_Job(job),
	},
	Schedule: glue.TriggerSchedule_Weekly(),
})

3. Event Triggers

Use addEventTrigger for EventBridge event-based triggers. There are two types: batching and non-batching. For batching triggers, you must specify batchSize. For non-batching triggers, batchSize defaults to 1. For both, batchWindow defaults to 900 seconds, but you can override the window to align with your workload's requirements.

4. Conditional Triggers

Conditional triggers have a predicate and actions associated with them. The trigger actions are executed when the predicateCondition is true.

Connection Properties

A Connection allows Glue jobs, crawlers and development endpoints to access certain types of data stores.

Secrets Management Manage JDBC connection credentials in Secrets Manager and pass the secret to the connection via the secret property (see the example below), rather than embedding credentials in properties.
Networking - the CDK determines the best fit subnet for Glue connection configuration Configure VPC placement through the network property, built with ConnectionNetwork.subnet(subnet) to pin a specific subnet, or ConnectionNetwork.vpc(vpc, vpcSubnets?) to let the L2 select one via the existing EC2 Subnet Selection library. A Glue connection targets a single subnet, so the first subnet of the selection is used. The two factories are mutually exclusive, so a subnet and a VPC can never be combined.

Pin the connection to a specific subnet:

declare const securityGroup: ec2.SecurityGroup;
declare const subnet: ec2.Subnet;
new glue.Connection(this, 'MyConnection', {
  type: glue.ConnectionType.NETWORK,
  // The security groups granting AWS Glue inbound access to the data source within the VPC
  securityGroups: [securityGroup],
  // The VPC subnet which contains the data source
  network: glue.ConnectionNetwork.subnet(subnet),
});

python

# security_group: ec2.SecurityGroup
# subnet: ec2.Subnet

glue.Connection(self, "MyConnection",
    type=glue.ConnectionType.NETWORK,
    # The security groups granting AWS Glue inbound access to the data source within the VPC
    security_groups=[security_group],
    # The VPC subnet which contains the data source
    network=glue.ConnectionNetwork.subnet(subnet)
)

java

SecurityGroup securityGroup;
Subnet subnet;

Connection.Builder.create(this, "MyConnection")
        .type(ConnectionType.NETWORK)
        // The security groups granting AWS Glue inbound access to the data source within the VPC
        .securityGroups(List.of(securityGroup))
        // The VPC subnet which contains the data source
        .network(ConnectionNetwork.subnet(subnet))
        .build();

csharp

SecurityGroup securityGroup;
Subnet subnet;

new Connection(this, "MyConnection", new ConnectionProps {
    Type = ConnectionType.NETWORK,
    // The security groups granting AWS Glue inbound access to the data source within the VPC
    SecurityGroups = new [] { securityGroup },
    // The VPC subnet which contains the data source
    Network = ConnectionNetwork.Subnet(subnet)
});

var securityGroup SecurityGroup
var subnet Subnet

glue.NewConnection(this, jsii.String("MyConnection"), &ConnectionProps{
	Type: glue.ConnectionType_NETWORK(),
	// The security groups granting AWS Glue inbound access to the data source within the VPC
	SecurityGroups: []ISecurityGroup{
		securityGroup,
	},
	// The VPC subnet which contains the data source
	Network: glue.ConnectionNetwork_Subnet(subnet),
})

Or let the CDK select a subnet from a VPC:

declare const securityGroup: ec2.SecurityGroup;
declare const vpc: ec2.Vpc;
new glue.Connection(this, 'MyConnection', {
  type: glue.ConnectionType.NETWORK,
  securityGroups: [securityGroup],
  // vpcSubnets is optional - defaults to private subnets
  network: glue.ConnectionNetwork.vpc(vpc, { subnetType: ec2.SubnetType.PRIVATE_WITH_EGRESS }),
});

python

# security_group: ec2.SecurityGroup
# vpc: ec2.Vpc

glue.Connection(self, "MyConnection",
    type=glue.ConnectionType.NETWORK,
    security_groups=[security_group],
    # vpcSubnets is optional - defaults to private subnets
    network=glue.ConnectionNetwork.vpc(vpc, subnet_type=ec2.SubnetType.PRIVATE_WITH_EGRESS)
)

java

SecurityGroup securityGroup;
Vpc vpc;

Connection.Builder.create(this, "MyConnection")
        .type(ConnectionType.NETWORK)
        .securityGroups(List.of(securityGroup))
        // vpcSubnets is optional - defaults to private subnets
        .network(ConnectionNetwork.vpc(vpc, SubnetSelection.builder().subnetType(SubnetType.PRIVATE_WITH_EGRESS).build()))
        .build();

csharp

SecurityGroup securityGroup;
Vpc vpc;

new Connection(this, "MyConnection", new ConnectionProps {
    Type = ConnectionType.NETWORK,
    SecurityGroups = new [] { securityGroup },
    // vpcSubnets is optional - defaults to private subnets
    Network = ConnectionNetwork.Vpc(vpc, new SubnetSelection { SubnetType = SubnetType.PRIVATE_WITH_EGRESS })
});

var securityGroup SecurityGroup
var vpc Vpc

glue.NewConnection(this, jsii.String("MyConnection"), &ConnectionProps{
	Type: glue.ConnectionType_NETWORK(),
	SecurityGroups: []ISecurityGroup{
		securityGroup,
	},
	// vpcSubnets is optional - defaults to private subnets
	Network: glue.ConnectionNetwork_Vpc(vpc, &SubnetSelection{
		SubnetType: ec2.SubnetType_PRIVATE_WITH_EGRESS,
	}),
})

For RDS Connection by JDBC, it is recommended to manage credentials using AWS Secrets Manager. Pass the secret via the secret property: Glue reads the credentials at runtime through the connection's SECRET_ID, so the secret value never enters the template. Note that in this case, the subnet must have a route to the AWS Secrets Manager VPC endpoint or to the AWS Secrets Manager endpoint through a NAT gateway.

declare const securityGroup: ec2.SecurityGroup;
declare const subnet: ec2.Subnet;
declare const db: rds.DatabaseCluster;
new glue.Connection(this, "RdsConnection", {
  type: glue.ConnectionType.JDBC,
  securityGroups: [securityGroup],
  network: glue.ConnectionNetwork.subnet(subnet),
  secret: db.secret,
  properties: {
    JDBC_CONNECTION_URL: `jdbc:mysql://${db.clusterEndpoint.socketAddress}/databasename`,
    JDBC_ENFORCE_SSL: "false",
  },
});

python

# security_group: ec2.SecurityGroup
# subnet: ec2.Subnet
# db: rds.DatabaseCluster

glue.Connection(self, "RdsConnection",
    type=glue.ConnectionType.JDBC,
    security_groups=[security_group],
    network=glue.ConnectionNetwork.subnet(subnet),
    secret=db.secret,
    properties={
        "JDBC_CONNECTION_URL": f"jdbc:mysql://{db.clusterEndpoint.socketAddress}/databasename",
        "JDBC_ENFORCE_SSL": "false"
    }
)

java

SecurityGroup securityGroup;
Subnet subnet;
DatabaseCluster db;

Connection.Builder.create(this, "RdsConnection")
        .type(ConnectionType.JDBC)
        .securityGroups(List.of(securityGroup))
        .network(ConnectionNetwork.subnet(subnet))
        .secret(db.getSecret())
        .properties(Map.of(
                "JDBC_CONNECTION_URL", String.format("jdbc:mysql://%s/databasename", db.getClusterEndpoint().getSocketAddress()),
                "JDBC_ENFORCE_SSL", "false"))
        .build();

csharp

SecurityGroup securityGroup;
Subnet subnet;
DatabaseCluster db;

new Connection(this, "RdsConnection", new ConnectionProps {
    Type = ConnectionType.JDBC,
    SecurityGroups = new [] { securityGroup },
    Network = ConnectionNetwork.Subnet(subnet),
    Secret = db.Secret,
    Properties = new Dictionary<string, string> {
        { "JDBC_CONNECTION_URL", $"jdbc:mysql://{db.clusterEndpoint.socketAddress}/databasename" },
        { "JDBC_ENFORCE_SSL", "false" }
    }
});

var securityGroup SecurityGroup
var subnet Subnet
var db DatabaseCluster

glue.NewConnection(this, jsii.String("RdsConnection"), &ConnectionProps{
	Type: glue.ConnectionType_JDBC(),
	SecurityGroups: []ISecurityGroup{
		securityGroup,
	},
	Network: glue.ConnectionNetwork_Subnet(subnet),
	Secret: db.Secret,
	Properties: map[string]*string{
		"JDBC_CONNECTION_URL": fmt.Sprintf("jdbc:mysql://%v/databasename", db.clusterEndpoint.socketAddress),
		"JDBC_ENFORCE_SSL": jsii.String("false"),
	},
})

Prefer the secret property over placing credentials in properties. Connection properties are emitted verbatim into the CloudFormation template, so any credential placed there in plaintext is stored in plaintext in the template, cdk.out, and source control. If a property key looks like a credential (for example PASSWORD, SECRET, or TOKEN) and holds a plaintext literal, the construct emits a synthesis-time warning.

If you need to use a connection type that doesn't exist as a static member on ConnectionType, you can instantiate a ConnectionType object, e.g: new glue.ConnectionType('NEW_TYPE').

See Adding a Connection to Your Data Store and Connection Structure documentation for more information on the supported data stores and their configurations.

SecurityConfiguration

A SecurityConfiguration is a set of security properties that can be used by AWS Glue to encrypt data at rest.

Each encryption config is built with a factory that pairs the encryption mode with its key, so illegal combinations (such as an S3-managed encryption carrying a KMS key) cannot be expressed:

new glue.SecurityConfiguration(this, 'MySecurityConfiguration', {
  cloudWatchEncryption: glue.CloudWatchEncryption.kms(),
  jobBookmarksEncryption: glue.JobBookmarksEncryption.clientSideKms(),
  s3Encryption: glue.S3Encryption.kms(),
});

python

glue.SecurityConfiguration(self, "MySecurityConfiguration",
    cloud_watch_encryption=glue.CloudWatchEncryption.kms(),
    job_bookmarks_encryption=glue.JobBookmarksEncryption.client_side_kms(),
    s3_encryption=glue.S3Encryption.kms()
)

java

SecurityConfiguration.Builder.create(this, "MySecurityConfiguration")
        .cloudWatchEncryption(CloudWatchEncryption.kms())
        .jobBookmarksEncryption(JobBookmarksEncryption.clientSideKms())
        .s3Encryption(S3Encryption.kms())
        .build();

csharp

new SecurityConfiguration(this, "MySecurityConfiguration", new SecurityConfigurationProps {
    CloudWatchEncryption = CloudWatchEncryption.Kms(),
    JobBookmarksEncryption = JobBookmarksEncryption.ClientSideKms(),
    S3Encryption = S3Encryption.Kms()
});

glue.NewSecurityConfiguration(this, jsii.String("MySecurityConfiguration"), &SecurityConfigurationProps{
	CloudWatchEncryption: glue.CloudWatchEncryption_Kms(),
	JobBookmarksEncryption: glue.JobBookmarksEncryption_ClientSideKms(),
	S3Encryption: glue.S3Encryption_Kms(),
})

By default, a shared KMS key is created for use with the encryption configurations that require one. You can also supply your own key to any factory, for example, for CloudWatch encryption:

declare const key: kms.Key;
new glue.SecurityConfiguration(this, 'MySecurityConfiguration', {
  cloudWatchEncryption: glue.CloudWatchEncryption.kms(key),
});

python

# key: kms.Key

glue.SecurityConfiguration(self, "MySecurityConfiguration",
    cloud_watch_encryption=glue.CloudWatchEncryption.kms(key)
)

java

Key key;

SecurityConfiguration.Builder.create(this, "MySecurityConfiguration")
        .cloudWatchEncryption(CloudWatchEncryption.kms(key))
        .build();

csharp

Key key;

new SecurityConfiguration(this, "MySecurityConfiguration", new SecurityConfigurationProps {
    CloudWatchEncryption = CloudWatchEncryption.Kms(key)
});

var key Key

glue.NewSecurityConfiguration(this, jsii.String("MySecurityConfiguration"), &SecurityConfigurationProps{
	CloudWatchEncryption: glue.CloudWatchEncryption_Kms(key),
})

Use glue.S3Encryption.s3Managed() for S3-managed (SSE-S3) encryption, which takes no key.

See documentation for more info for Glue encrypting data written by Crawlers, Jobs, and Development Endpoints.

Catalog

The Glue Data Catalog is a persistent metadata store for your data assets. Every account has an implicit, account-wide catalog that always exists, and you can also create additional catalogs as AWS::Glue::Catalog resources (for example, to federate to another metastore).

A catalog's encryption is fixed when the catalog is created: a catalog either carries encryption settings or it does not. This keeps its configuration easy to reason about — there are no mutation methods that change encryption after the fact.

The account-wide catalog

Use Catalog.forAccount(scope) to obtain the implicit account catalog. It is not a CloudFormation resource — it always exists. Repeated calls within the same stack return the same instance:

const catalog = glue.Catalog.forAccount(this);

python

catalog = glue.Catalog.for_account(self)

java

ICatalog catalog = Catalog.forAccount(this);

csharp

var catalog = Catalog.ForAccount(this);

catalog := glue.Catalog_ForAccount(this)

To configure Data Catalog encryption for the account, use Catalog.encryptAccount(scope, options):

declare const key: kms.Key;
glue.Catalog.encryptAccount(this, {
  encryptionAtRest: glue.DataCatalogEncryptionAtRest.kms(key),
});

python

# key: kms.Key

glue.Catalog.encrypt_account(self,
    encryption_at_rest=glue.DataCatalogEncryptionAtRest.kms(key)
)

java

Key key;

Catalog.encryptAccount(this, CatalogEncryptionOptions.builder()
        .encryptionAtRest(DataCatalogEncryptionAtRest.kms(key))
        .build());

csharp

Key key;

Catalog.EncryptAccount(this, new CatalogEncryptionOptions {
    EncryptionAtRest = DataCatalogEncryptionAtRest.Kms(key)
});

var key Key

glue.Catalog_EncryptAccount(this, &CatalogEncryptionOptions{
	EncryptionAtRest: glue.DataCatalogEncryptionAtRest_Kms(key),
})

Because encryption is fixed at construction, encryptAccount must be called before the account catalog is first used in the stack — before any Catalog.forAccount(this) call, and before any Database that uses the account catalog. Calling it after the account catalog has been materialized throws.

The account catalog's encryption is an account- and region-wide setting, managed through the singleton PutDataCatalogEncryptionSettings API. Configure it in exactly one stack. Configuring it from multiple stacks in the same account and region makes those stacks overwrite one another at deploy time, and the result is order-dependent. Unlike duplicate settings within a single stack (which CloudFormation rejects), this cross-stack conflict is not caught at synthesis time, because each stack synthesizes to its own template.

Creating a catalog

To create a new catalog resource, use the Catalog constructor. Encryption is configured through the encryptionAtRest and connectionPasswordEncryption props:

new glue.Catalog(this, 'MyCatalog', {
  catalogName: 'my-catalog',
  description: 'my catalog description',
});

python

glue.Catalog(self, "MyCatalog",
    catalog_name="my-catalog",
    description="my catalog description"
)

java

Catalog.Builder.create(this, "MyCatalog")
        .catalogName("my-catalog")
        .description("my catalog description")
        .build();

csharp

new Catalog(this, "MyCatalog", new CatalogProps {
    CatalogName = "my-catalog",
    Description = "my catalog description"
});

glue.NewCatalog(this, jsii.String("MyCatalog"), &CatalogProps{
	CatalogName: jsii.String("my-catalog"),
	Description: jsii.String("my catalog description"),
})

Encryption at rest

Configure Data Catalog encryption at rest through the encryptionAtRest option (on Catalog.encryptAccount or the Catalog constructor). It accepts a DataCatalogEncryptionAtRest describing the mode:

declare const key: kms.Key;

// SSE-KMS with a customer-managed key
glue.Catalog.encryptAccount(this, {
  encryptionAtRest: glue.DataCatalogEncryptionAtRest.kms(key),
});

// SSE-KMS with an AWS-managed key (omit the key)
new glue.Catalog(this, 'ManagedKeyCatalog', {
  catalogName: 'managed-key-catalog',
  encryptionAtRest: glue.DataCatalogEncryptionAtRest.kms(),
});

// Disable encryption at rest
new glue.Catalog(this, 'PlaintextCatalog', {
  catalogName: 'plaintext-catalog',
  encryptionAtRest: glue.DataCatalogEncryptionAtRest.disabled(),
});

python

# key: kms.Key


# SSE-KMS with a customer-managed key
glue.Catalog.encrypt_account(self,
    encryption_at_rest=glue.DataCatalogEncryptionAtRest.kms(key)
)

# SSE-KMS with an AWS-managed key (omit the key)
glue.Catalog(self, "ManagedKeyCatalog",
    catalog_name="managed-key-catalog",
    encryption_at_rest=glue.DataCatalogEncryptionAtRest.kms()
)

# Disable encryption at rest
glue.Catalog(self, "PlaintextCatalog",
    catalog_name="plaintext-catalog",
    encryption_at_rest=glue.DataCatalogEncryptionAtRest.disabled()
)

java

Key key;


// SSE-KMS with a customer-managed key
Catalog.encryptAccount(this, CatalogEncryptionOptions.builder()
        .encryptionAtRest(DataCatalogEncryptionAtRest.kms(key))
        .build());

// SSE-KMS with an AWS-managed key (omit the key)
// SSE-KMS with an AWS-managed key (omit the key)
Catalog.Builder.create(this, "ManagedKeyCatalog")
        .catalogName("managed-key-catalog")
        .encryptionAtRest(DataCatalogEncryptionAtRest.kms())
        .build();

// Disable encryption at rest
// Disable encryption at rest
Catalog.Builder.create(this, "PlaintextCatalog")
        .catalogName("plaintext-catalog")
        .encryptionAtRest(DataCatalogEncryptionAtRest.disabled())
        .build();

csharp

Key key;


// SSE-KMS with a customer-managed key
Catalog.EncryptAccount(this, new CatalogEncryptionOptions {
    EncryptionAtRest = DataCatalogEncryptionAtRest.Kms(key)
});

// SSE-KMS with an AWS-managed key (omit the key)
// SSE-KMS with an AWS-managed key (omit the key)
new Catalog(this, "ManagedKeyCatalog", new CatalogProps {
    CatalogName = "managed-key-catalog",
    EncryptionAtRest = DataCatalogEncryptionAtRest.Kms()
});

// Disable encryption at rest
// Disable encryption at rest
new Catalog(this, "PlaintextCatalog", new CatalogProps {
    CatalogName = "plaintext-catalog",
    EncryptionAtRest = DataCatalogEncryptionAtRest.Disabled()
});

var key Key


// SSE-KMS with a customer-managed key
glue.Catalog_EncryptAccount(this, &CatalogEncryptionOptions{
	EncryptionAtRest: glue.DataCatalogEncryptionAtRest_Kms(key),
})

// SSE-KMS with an AWS-managed key (omit the key)
// SSE-KMS with an AWS-managed key (omit the key)
glue.NewCatalog(this, jsii.String("ManagedKeyCatalog"), &CatalogProps{
	CatalogName: jsii.String("managed-key-catalog"),
	EncryptionAtRest: glue.DataCatalogEncryptionAtRest_*Kms(),
})

// Disable encryption at rest
// Disable encryption at rest
glue.NewCatalog(this, jsii.String("PlaintextCatalog"), &CatalogProps{
	CatalogName: jsii.String("plaintext-catalog"),
	EncryptionAtRest: glue.DataCatalogEncryptionAtRest_Disabled(),
})

When you use SSE-KMS-WITH-SERVICE-ROLE, AWS Glue accesses the KMS key through a service role you provide. If you pass a customer-managed key, the role is automatically granted the permissions it needs to encrypt and decrypt catalog data:

import * as iam from 'aws-cdk-lib/aws-iam';
declare const key: kms.Key;
declare const role: iam.IRole;
glue.Catalog.encryptAccount(this, {
  encryptionAtRest: glue.DataCatalogEncryptionAtRest.kmsWithServiceRole(role, key),
});

python

import aws_cdk.aws_iam as iam
# key: kms.Key
# role: iam.IRole

glue.Catalog.encrypt_account(self,
    encryption_at_rest=glue.DataCatalogEncryptionAtRest.kms_with_service_role(role, key)
)

java

import software.amazon.awscdk.services.iam.*;
Key key;
IRole role;

Catalog.encryptAccount(this, CatalogEncryptionOptions.builder()
        .encryptionAtRest(DataCatalogEncryptionAtRest.kmsWithServiceRole(role, key))
        .build());

csharp

using Amazon.CDK.AWS.IAM;
Key key;
IRole role;

Catalog.EncryptAccount(this, new CatalogEncryptionOptions {
    EncryptionAtRest = DataCatalogEncryptionAtRest.KmsWithServiceRole(role, key)
});

import iam "github.com/aws/aws-cdk-go/awscdk"
var key Key
var role IRole

glue.Catalog_EncryptAccount(this, &CatalogEncryptionOptions{
	EncryptionAtRest: glue.DataCatalogEncryptionAtRest_KmsWithServiceRole(role, key),
})

The customer-managed key, when configured, is exposed on the catalog as encryptionKey (and the connection-password key as connectionPasswordKey), so you can reference it to grant additional access. It is undefined when encryption is disabled or an AWS-managed key is used.

Connection password encryption

Independently from encryption at rest, the Data Catalog can encrypt the passwords stored in connection properties. Configure it through the connectionPasswordEncryption option:

declare const key: kms.Key;
glue.Catalog.encryptAccount(this, {
  connectionPasswordEncryption: {
    kmsKey: key,
    // Whether GetConnection/GetConnections return the password encrypted (default: true)
    returnConnectionPasswordEncrypted: true,
  },
});

python

# key: kms.Key

glue.Catalog.encrypt_account(self,
    connection_password_encryption=glue.ConnectionPasswordEncryption(
        kms_key=key,
        # Whether GetConnection/GetConnections return the password encrypted (default: true)
        return_connection_password_encrypted=True
    )
)

java

Key key;

Catalog.encryptAccount(this, CatalogEncryptionOptions.builder()
        .connectionPasswordEncryption(ConnectionPasswordEncryption.builder()
                .kmsKey(key)
                // Whether GetConnection/GetConnections return the password encrypted (default: true)
                .returnConnectionPasswordEncrypted(true)
                .build())
        .build());

csharp

Key key;

Catalog.EncryptAccount(this, new CatalogEncryptionOptions {
    ConnectionPasswordEncryption = new ConnectionPasswordEncryption {
        KmsKey = key,
        // Whether GetConnection/GetConnections return the password encrypted (default: true)
        ReturnConnectionPasswordEncrypted = true
    }
});

var key Key

glue.Catalog_EncryptAccount(this, &CatalogEncryptionOptions{
	ConnectionPasswordEncryption: &ConnectionPasswordEncryption{
		KmsKey: key,
		// Whether GetConnection/GetConnections return the password encrypted (default: true)
		ReturnConnectionPasswordEncrypted: jsii.Boolean(true),
	},
})

The two encryption blocks are independent: enabling one does not require the other, and each may use a different KMS key. The customer-managed key for connection passwords is exposed as connectionPasswordKey.

Importing a catalog

You can import an existing catalog by ARN or by id. An imported catalog is a pure identity handle — it emits no resources and does not manage the catalog's encryption:

const byId = glue.Catalog.fromCatalogId(this, 'ById', 'my-catalog-id');
const byArn = glue.Catalog.fromCatalogArn(this, 'ByArn', 'arn:aws:glue:us-east-1:123456789012:catalog/my-catalog-id');

python

by_id = glue.Catalog.from_catalog_id(self, "ById", "my-catalog-id")
by_arn = glue.Catalog.from_catalog_arn(self, "ByArn", "arn:aws:glue:us-east-1:123456789012:catalog/my-catalog-id")

java

ICatalog byId = Catalog.fromCatalogId(this, "ById", "my-catalog-id");
ICatalog byArn = Catalog.fromCatalogArn(this, "ByArn", "arn:aws:glue:us-east-1:123456789012:catalog/my-catalog-id");

csharp

var byId = Catalog.FromCatalogId(this, "ById", "my-catalog-id");
var byArn = Catalog.FromCatalogArn(this, "ByArn", "arn:aws:glue:us-east-1:123456789012:catalog/my-catalog-id");

byId := glue.Catalog_FromCatalogId(this, jsii.String("ById"), jsii.String("my-catalog-id"))
byArn := glue.Catalog_FromCatalogArn(this, jsii.String("ByArn"), jsii.String("arn:aws:glue:us-east-1:123456789012:catalog/my-catalog-id"))

To manage the Data Catalog encryption of a catalog you did not create in this stack, add a CfnDataCatalogEncryptionSettings resource targeting its id directly. Do this from exactly one stack: like the account catalog, a catalog has a single encryption configuration, so two settings resources targeting the same id race to overwrite one another at deploy time. Within a single stack this is caught by CloudFormation template validation (E3019, duplicate primary identifiers); across stacks it is not, since each stack synthesizes to its own template.

import { CfnDataCatalogEncryptionSettings } from 'aws-cdk-lib/aws-glue';

new CfnDataCatalogEncryptionSettings(this, 'Encryption', {
  catalogId: 'my-catalog-id',
  dataCatalogEncryptionSettings: {
    encryptionAtRest: { catalogEncryptionMode: 'SSE-KMS' },
  },
});

python

from aws_cdk.aws_glue import CfnDataCatalogEncryptionSettings


CfnDataCatalogEncryptionSettings(self, "Encryption",
    catalog_id="my-catalog-id",
    data_catalog_encryption_settings=CfnDataCatalogEncryptionSettings.DataCatalogEncryptionSettingsProperty(
        encryption_at_rest=CfnDataCatalogEncryptionSettings.EncryptionAtRestProperty(catalog_encryption_mode="SSE-KMS")
    )
)

java

import software.amazon.awscdk.services.glue.CfnDataCatalogEncryptionSettings;


CfnDataCatalogEncryptionSettings.Builder.create(this, "Encryption")
        .catalogId("my-catalog-id")
        .dataCatalogEncryptionSettings(DataCatalogEncryptionSettingsProperty.builder()
                .encryptionAtRest(EncryptionAtRestProperty.builder().catalogEncryptionMode("SSE-KMS").build())
                .build())
        .build();

csharp

using Amazon.CDK.AWS.Glue;


new CfnDataCatalogEncryptionSettings(this, "Encryption", new CfnDataCatalogEncryptionSettingsProps {
    CatalogId = "my-catalog-id",
    DataCatalogEncryptionSettings = new DataCatalogEncryptionSettingsProperty {
        EncryptionAtRest = new EncryptionAtRestProperty { CatalogEncryptionMode = "SSE-KMS" }
    }
});

import "github.com/aws/aws-cdk-go/awscdk"


awscdk.NewCfnDataCatalogEncryptionSettings(this, jsii.String("Encryption"), &CfnDataCatalogEncryptionSettingsProps{
	CatalogId: jsii.String("my-catalog-id"),
	DataCatalogEncryptionSettings: &DataCatalogEncryptionSettingsProperty{
		EncryptionAtRest: &EncryptionAtRestProperty{
			CatalogEncryptionMode: jsii.String("SSE-KMS"),
		},
	},
})

Database

A Database is a logical grouping of Tables in the Glue Catalog.

new glue.Database(this, 'MyDatabase', {
  databaseName: 'my_database',
  description: 'my_database_description',
});

python

glue.Database(self, "MyDatabase",
    database_name="my_database",
    description="my_database_description"
)

java

Database.Builder.create(this, "MyDatabase")
        .databaseName("my_database")
        .description("my_database_description")
        .build();

csharp

new Database(this, "MyDatabase", new DatabaseProps {
    DatabaseName = "my_database",
    Description = "my_database_description"
});

glue.NewDatabase(this, jsii.String("MyDatabase"), &DatabaseProps{
	DatabaseName: jsii.String("my_database"),
	Description: jsii.String("my_database_description"),
})

Because a database is a container for tables and their metadata, it is retained by default when removed from the stack, to avoid accidental data loss. Set removalPolicy to RemovalPolicy.DESTROY to have it deleted instead:

import { RemovalPolicy } from 'aws-cdk-lib';

new glue.Database(this, 'MyDatabase', {
  databaseName: 'my_database',
  removalPolicy: RemovalPolicy.DESTROY,
});

python

from aws_cdk import RemovalPolicy


glue.Database(self, "MyDatabase",
    database_name="my_database",
    removal_policy=RemovalPolicy.DESTROY
)

java

import software.amazon.awscdk.RemovalPolicy;


Database.Builder.create(this, "MyDatabase")
        .databaseName("my_database")
        .removalPolicy(RemovalPolicy.DESTROY)
        .build();

csharp

using Amazon.CDK;


new Database(this, "MyDatabase", new DatabaseProps {
    DatabaseName = "my_database",
    RemovalPolicy = RemovalPolicy.DESTROY
});

import "github.com/aws/aws-cdk-go/awscdk"


glue.NewDatabase(this, jsii.String("MyDatabase"), &DatabaseProps{
	DatabaseName: jsii.String("my_database"),
	RemovalPolicy: awscdk.RemovalPolicy_DESTROY,
})

Table

A Glue table describes a table of data in S3: its structure (column names and types), location of data (S3 objects with a common prefix in a S3 bucket), and format for the files (Json, Avro, Parquet, etc.):

declare const myDatabase: glue.Database;
new glue.S3Table(this, 'MyTable', {
  database: myDatabase,
  columns: [{
    name: 'col1',
    type: glue.Schema.STRING,
  }, {
    name: 'col2',
    type: glue.Schema.array(glue.Schema.STRING),
    comment: 'col2 is an array of strings' // comment is optional
  }],
  dataFormat: glue.DataFormat.JSON,
});

python

# my_database: glue.Database

glue.S3Table(self, "MyTable",
    database=my_database,
    columns=[glue.Column(
        name="col1",
        type=glue.Schema.STRING
    ), glue.Column(
        name="col2",
        type=glue.Schema.array(glue.Schema.STRING),
        comment="col2 is an array of strings"
    )],
    data_format=glue.DataFormat.JSON
)

java

Database myDatabase;

S3Table.Builder.create(this, "MyTable")
        .database(myDatabase)
        .columns(List.of(Column.builder()
                .name("col1")
                .type(Schema.STRING)
                .build(), Column.builder()
                .name("col2")
                .type(Schema.array(Schema.STRING))
                .comment("col2 is an array of strings")
                .build()))
        .dataFormat(DataFormat.JSON)
        .build();

csharp

Database myDatabase;

new S3Table(this, "MyTable", new S3TableProps {
    Database = myDatabase,
    Columns = new [] { new Column {
        Name = "col1",
        Type = Schema.STRING
    }, new Column {
        Name = "col2",
        Type = Schema.Array(Schema.STRING),
        Comment = "col2 is an array of strings"
    } },
    DataFormat = DataFormat.JSON
});

var myDatabase Database

glue.NewS3Table(this, jsii.String("MyTable"), &S3TableProps{
	Database: myDatabase,
	Columns: []Column{
		&Column{
			Name: jsii.String("col1"),
			Type: glue.Schema_STRING(),
		},
		&Column{
			Name: jsii.String("col2"),
			Type: glue.Schema_Array(glue.Schema_STRING()),
			Comment: jsii.String("col2 is an array of strings"),
		},
	},
	DataFormat: glue.DataFormat_JSON(),
})

By default, a S3 bucket will be created to store the table's data but you can bring your own with S3TableStorage.fromBucket and set an s3Prefix:

declare const myBucket: s3.Bucket;
declare const myDatabase: glue.Database;
new glue.S3Table(this, 'MyTable', {
  storage: glue.S3TableStorage.fromBucket(myBucket),
  s3Prefix: 'my-table/',
  // ...
  database: myDatabase,
  columns: [{
    name: 'col1',
    type: glue.Schema.STRING,
  }],
  dataFormat: glue.DataFormat.JSON,
});

python

# my_bucket: s3.Bucket
# my_database: glue.Database

glue.S3Table(self, "MyTable",
    storage=glue.S3TableStorage.from_bucket(my_bucket),
    s3_prefix="my-table/",
    # ...
    database=my_database,
    columns=[glue.Column(
        name="col1",
        type=glue.Schema.STRING
    )],
    data_format=glue.DataFormat.JSON
)

java

Bucket myBucket;
Database myDatabase;

S3Table.Builder.create(this, "MyTable")
        .storage(S3TableStorage.fromBucket(myBucket))
        .s3Prefix("my-table/")
        // ...
        .database(myDatabase)
        .columns(List.of(Column.builder()
                .name("col1")
                .type(Schema.STRING)
                .build()))
        .dataFormat(DataFormat.JSON)
        .build();

csharp

Bucket myBucket;
Database myDatabase;

new S3Table(this, "MyTable", new S3TableProps {
    Storage = S3TableStorage.FromBucket(myBucket),
    S3Prefix = "my-table/",
    // ...
    Database = myDatabase,
    Columns = new [] { new Column {
        Name = "col1",
        Type = Schema.STRING
    } },
    DataFormat = DataFormat.JSON
});

var myBucket Bucket
var myDatabase Database

glue.NewS3Table(this, jsii.String("MyTable"), &S3TableProps{
	Storage: glue.S3TableStorage_FromBucket(myBucket),
	S3Prefix: jsii.String("my-table/"),
	// ...
	Database: myDatabase,
	Columns: []Column{
		&Column{
			Name: jsii.String("col1"),
			Type: glue.Schema_STRING(),
		},
	},
	DataFormat: glue.DataFormat_JSON(),
})

Glue tables can be configured to contain user-defined properties, to describe the physical storage of table data, through the storageParameters property:

declare const myDatabase: glue.Database;
new glue.S3Table(this, 'MyTable', {
  storageParameters: [
    glue.StorageParameter.skipHeaderLineCount(1),
    glue.StorageParameter.compressionType(glue.CompressionType.GZIP),
    glue.StorageParameter.custom('separatorChar', ',')
  ],
  // ...
  database: myDatabase,
  columns: [{
    name: 'col1',
    type: glue.Schema.STRING,
  }],
  dataFormat: glue.DataFormat.JSON,
});

python

# my_database: glue.Database

glue.S3Table(self, "MyTable",
    storage_parameters=[
        glue.StorageParameter.skip_header_line_count(1),
        glue.StorageParameter.compression_type(glue.CompressionType.GZIP),
        glue.StorageParameter.custom("separatorChar", ",")
    ],
    # ...
    database=my_database,
    columns=[glue.Column(
        name="col1",
        type=glue.Schema.STRING
    )],
    data_format=glue.DataFormat.JSON
)

java

Database myDatabase;

S3Table.Builder.create(this, "MyTable")
        .storageParameters(List.of(StorageParameter.skipHeaderLineCount(1), StorageParameter.compressionType(CompressionType.GZIP), StorageParameter.custom("separatorChar", ",")))
        // ...
        .database(myDatabase)
        .columns(List.of(Column.builder()
                .name("col1")
                .type(Schema.STRING)
                .build()))
        .dataFormat(DataFormat.JSON)
        .build();

csharp

Database myDatabase;

new S3Table(this, "MyTable", new S3TableProps {
    StorageParameters = new [] { StorageParameter.SkipHeaderLineCount(1), StorageParameter.CompressionType(CompressionType.GZIP), StorageParameter.Custom("separatorChar", ",") },
    // ...
    Database = myDatabase,
    Columns = new [] { new Column {
        Name = "col1",
        Type = Schema.STRING
    } },
    DataFormat = DataFormat.JSON
});

var myDatabase Database

glue.NewS3Table(this, jsii.String("MyTable"), &S3TableProps{
	StorageParameters: []StorageParameter{
		glue.StorageParameter_SkipHeaderLineCount(jsii.Number(1)),
		glue.StorageParameter_CompressionType(glue.CompressionType_GZIP),
		glue.StorageParameter_Custom(jsii.String("separatorChar"), jsii.String(",")),
	},
	// ...
	Database: myDatabase,
	Columns: []Column{
		&Column{
			Name: jsii.String("col1"),
			Type: glue.Schema_STRING(),
		},
	},
	DataFormat: glue.DataFormat_JSON(),
})

Glue tables can also be configured to contain user-defined table properties through the parameters property:

declare const myDatabase: glue.Database;
new glue.S3Table(this, 'MyTable', {
  parameters: {
    key1: 'val1',
    key2: 'val2',
  },
  database: myDatabase,
  columns: [{
    name: 'col1',
    type: glue.Schema.STRING,
  }],
  dataFormat: glue.DataFormat.JSON,
});

python

# my_database: glue.Database

glue.S3Table(self, "MyTable",
    parameters={
        "key1": "val1",
        "key2": "val2"
    },
    database=my_database,
    columns=[glue.Column(
        name="col1",
        type=glue.Schema.STRING
    )],
    data_format=glue.DataFormat.JSON
)

java

Database myDatabase;

S3Table.Builder.create(this, "MyTable")
        .parameters(Map.of(
                "key1", "val1",
                "key2", "val2"))
        .database(myDatabase)
        .columns(List.of(Column.builder()
                .name("col1")
                .type(Schema.STRING)
                .build()))
        .dataFormat(DataFormat.JSON)
        .build();

csharp

Database myDatabase;

new S3Table(this, "MyTable", new S3TableProps {
    Parameters = new Dictionary<string, string> {
        { "key1", "val1" },
        { "key2", "val2" }
    },
    Database = myDatabase,
    Columns = new [] { new Column {
        Name = "col1",
        Type = Schema.STRING
    } },
    DataFormat = DataFormat.JSON
});

var myDatabase Database

glue.NewS3Table(this, jsii.String("MyTable"), &S3TableProps{
	Parameters: map[string]*string{
		"key1": jsii.String("val1"),
		"key2": jsii.String("val2"),
	},
	Database: myDatabase,
	Columns: []Column{
		&Column{
			Name: jsii.String("col1"),
			Type: glue.Schema_STRING(),
		},
	},
	DataFormat: glue.DataFormat_JSON(),
})

Partition Keys

To improve query performance, a table can specify partitionKeys on which data is stored and queried separately. For example, you might partition a table by year and month to optimize queries based on a time window:

declare const myDatabase: glue.Database;
new glue.S3Table(this, 'MyTable', {
  database: myDatabase,
  columns: [{
    name: 'col1',
    type: glue.Schema.STRING,
  }],
  partitionKeys: [{
    name: 'year',
    type: glue.Schema.SMALL_INT,
  }, {
    name: 'month',
    type: glue.Schema.SMALL_INT,
  }],
  dataFormat: glue.DataFormat.JSON,
});

python

# my_database: glue.Database

glue.S3Table(self, "MyTable",
    database=my_database,
    columns=[glue.Column(
        name="col1",
        type=glue.Schema.STRING
    )],
    partition_keys=[glue.Column(
        name="year",
        type=glue.Schema.SMALL_INT
    ), glue.Column(
        name="month",
        type=glue.Schema.SMALL_INT
    )],
    data_format=glue.DataFormat.JSON
)

java

Database myDatabase;

S3Table.Builder.create(this, "MyTable")
        .database(myDatabase)
        .columns(List.of(Column.builder()
                .name("col1")
                .type(Schema.STRING)
                .build()))
        .partitionKeys(List.of(Column.builder()
                .name("year")
                .type(Schema.SMALL_INT)
                .build(), Column.builder()
                .name("month")
                .type(Schema.SMALL_INT)
                .build()))
        .dataFormat(DataFormat.JSON)
        .build();

csharp

Database myDatabase;

new S3Table(this, "MyTable", new S3TableProps {
    Database = myDatabase,
    Columns = new [] { new Column {
        Name = "col1",
        Type = Schema.STRING
    } },
    PartitionKeys = new [] { new Column {
        Name = "year",
        Type = Schema.SMALL_INT
    }, new Column {
        Name = "month",
        Type = Schema.SMALL_INT
    } },
    DataFormat = DataFormat.JSON
});

var myDatabase Database

glue.NewS3Table(this, jsii.String("MyTable"), &S3TableProps{
	Database: myDatabase,
	Columns: []Column{
		&Column{
			Name: jsii.String("col1"),
			Type: glue.Schema_STRING(),
		},
	},
	PartitionKeys: []Column{
		&Column{
			Name: jsii.String("year"),
			Type: glue.Schema_SMALL_INT(),
		},
		&Column{
			Name: jsii.String("month"),
			Type: glue.Schema_SMALL_INT(),
		},
	},
	DataFormat: glue.DataFormat_JSON(),
})

Partition Indexes

Another way to improve query performance is to specify partition indexes. If no partition indexes are present on the table, AWS Glue loads all partitions of the table and filters the loaded partitions using the query expression. The query takes more time to run as the number of partitions increase. With an index, the query will try to fetch a subset of the partitions instead of loading all partitions of the table.

The keys of a partition index must be a subset of the partition keys of the table. You can have a maximum of 3 partition indexes per table. To specify a partition index, you can use the partitionIndexes property:

declare const myDatabase: glue.Database;
new glue.S3Table(this, 'MyTable', {
  database: myDatabase,
  columns: [{
    name: 'col1',
    type: glue.Schema.STRING,
  }],
  partitionKeys: [{
    name: 'year',
    type: glue.Schema.SMALL_INT,
  }, {
    name: 'month',
    type: glue.Schema.SMALL_INT,
  }],
  partitionIndexes: [{
    indexName: 'my-index', // optional
    keyNames: ['year'],
  }], // supply up to 3 indexes
  dataFormat: glue.DataFormat.JSON,
});

python

# my_database: glue.Database

glue.S3Table(self, "MyTable",
    database=my_database,
    columns=[glue.Column(
        name="col1",
        type=glue.Schema.STRING
    )],
    partition_keys=[glue.Column(
        name="year",
        type=glue.Schema.SMALL_INT
    ), glue.Column(
        name="month",
        type=glue.Schema.SMALL_INT
    )],
    partition_indexes=[glue.PartitionIndex(
        index_name="my-index",  # optional
        key_names=["year"]
    )],  # supply up to 3 indexes
    data_format=glue.DataFormat.JSON
)

java

Database myDatabase;

S3Table.Builder.create(this, "MyTable")
        .database(myDatabase)
        .columns(List.of(Column.builder()
                .name("col1")
                .type(Schema.STRING)
                .build()))
        .partitionKeys(List.of(Column.builder()
                .name("year")
                .type(Schema.SMALL_INT)
                .build(), Column.builder()
                .name("month")
                .type(Schema.SMALL_INT)
                .build()))
        .partitionIndexes(List.of(PartitionIndex.builder()
                .indexName("my-index") // optional
                .keyNames(List.of("year"))
                .build())) // supply up to 3 indexes
        .dataFormat(DataFormat.JSON)
        .build();

csharp

Database myDatabase;

new S3Table(this, "MyTable", new S3TableProps {
    Database = myDatabase,
    Columns = new [] { new Column {
        Name = "col1",
        Type = Schema.STRING
    } },
    PartitionKeys = new [] { new Column {
        Name = "year",
        Type = Schema.SMALL_INT
    }, new Column {
        Name = "month",
        Type = Schema.SMALL_INT
    } },
    PartitionIndexes = new [] { new PartitionIndex {
        IndexName = "my-index",  // optional
        KeyNames = new [] { "year" }
    } },  // supply up to 3 indexes
    DataFormat = DataFormat.JSON
});

var myDatabase Database

glue.NewS3Table(this, jsii.String("MyTable"), &S3TableProps{
	Database: myDatabase,
	Columns: []Column{
		&Column{
			Name: jsii.String("col1"),
			Type: glue.Schema_STRING(),
		},
	},
	PartitionKeys: []Column{
		&Column{
			Name: jsii.String("year"),
			Type: glue.Schema_SMALL_INT(),
		},
		&Column{
			Name: jsii.String("month"),
			Type: glue.Schema_SMALL_INT(),
		},
	},
	PartitionIndexes: []PartitionIndex{
		&PartitionIndex{
			IndexName: jsii.String("my-index"),
			 // optional
			KeyNames: []*string{
				jsii.String("year"),
			},
		},
	},
	 // supply up to 3 indexes
	DataFormat: glue.DataFormat_JSON(),
})

Alternatively, you can call the addPartitionIndex() function on a table:

declare const myTable: glue.S3Table;
myTable.addPartitionIndex({
  indexName: 'my-index',
  keyNames: ['year'],
});

python

# my_table: glue.S3Table

my_table.add_partition_index(
    index_name="my-index",
    key_names=["year"]
)

java

S3Table myTable;

myTable.addPartitionIndex(PartitionIndex.builder()
        .indexName("my-index")
        .keyNames(List.of("year"))
        .build());

csharp

S3Table myTable;

myTable.AddPartitionIndex(new PartitionIndex {
    IndexName = "my-index",
    KeyNames = new [] { "year" }
});

var myTable S3Table

myTable.AddPartitionIndex(&PartitionIndex{
	IndexName: jsii.String("my-index"),
	KeyNames: []*string{
		jsii.String("year"),
	},
})

Partition Filtering

If you have a table with a large number of partitions that grows over time, consider using AWS Glue partition indexing and filtering.

declare const myDatabase: glue.Database;
new glue.S3Table(this, 'MyTable', {
  database: myDatabase,
  columns: [{
      name: 'col1',
      type: glue.Schema.STRING,
  }],
  partitionKeys: [{
      name: 'year',
      type: glue.Schema.SMALL_INT,
  }, {
      name: 'month',
      type: glue.Schema.SMALL_INT,
  }],
  dataFormat: glue.DataFormat.JSON,
  enablePartitionFiltering: true,
});

python

# my_database: glue.Database

glue.S3Table(self, "MyTable",
    database=my_database,
    columns=[glue.Column(
        name="col1",
        type=glue.Schema.STRING
    )],
    partition_keys=[glue.Column(
        name="year",
        type=glue.Schema.SMALL_INT
    ), glue.Column(
        name="month",
        type=glue.Schema.SMALL_INT
    )],
    data_format=glue.DataFormat.JSON,
    enable_partition_filtering=True
)

java

Database myDatabase;

S3Table.Builder.create(this, "MyTable")
        .database(myDatabase)
        .columns(List.of(Column.builder()
                .name("col1")
                .type(Schema.STRING)
                .build()))
        .partitionKeys(List.of(Column.builder()
                .name("year")
                .type(Schema.SMALL_INT)
                .build(), Column.builder()
                .name("month")
                .type(Schema.SMALL_INT)
                .build()))
        .dataFormat(DataFormat.JSON)
        .enablePartitionFiltering(true)
        .build();

csharp

Database myDatabase;

new S3Table(this, "MyTable", new S3TableProps {
    Database = myDatabase,
    Columns = new [] { new Column {
        Name = "col1",
        Type = Schema.STRING
    } },
    PartitionKeys = new [] { new Column {
        Name = "year",
        Type = Schema.SMALL_INT
    }, new Column {
        Name = "month",
        Type = Schema.SMALL_INT
    } },
    DataFormat = DataFormat.JSON,
    EnablePartitionFiltering = true
});

var myDatabase Database

glue.NewS3Table(this, jsii.String("MyTable"), &S3TableProps{
	Database: myDatabase,
	Columns: []Column{
		&Column{
			Name: jsii.String("col1"),
			Type: glue.Schema_STRING(),
		},
	},
	PartitionKeys: []Column{
		&Column{
			Name: jsii.String("year"),
			Type: glue.Schema_SMALL_INT(),
		},
		&Column{
			Name: jsii.String("month"),
			Type: glue.Schema_SMALL_INT(),
		},
	},
	DataFormat: glue.DataFormat_JSON(),
	EnablePartitionFiltering: jsii.Boolean(true),
})

Partition Projection

Partition projection allows Athena to automatically add new partitions as new data arrives, without requiring ALTER TABLE ADD PARTITION statements. This improves query performance and reduces management overhead by eliminating the need to manually manage partition metadata.

For more information, see the AWS documentation on partition projection.

INTEGER Projection

For partition keys with sequential numeric values:

declare const myDatabase: glue.Database;
new glue.S3Table(this, 'MyTable', {
  database: myDatabase,
  columns: [{
    name: 'data',
    type: glue.Schema.STRING,
  }],
  partitionKeys: [{
    name: 'year',
    type: glue.Schema.INTEGER,
  }],
  dataFormat: glue.DataFormat.JSON,
  partitionProjection: {
    year: glue.PartitionProjectionConfiguration.integer({
      min: 2020,
      max: 2023,
      interval: 1,  // optional, defaults to 1
      digits: 4,    // optional, pads with leading zeros
    }),
  },
});

python

# my_database: glue.Database

glue.S3Table(self, "MyTable",
    database=my_database,
    columns=[glue.Column(
        name="data",
        type=glue.Schema.STRING
    )],
    partition_keys=[glue.Column(
        name="year",
        type=glue.Schema.INTEGER
    )],
    data_format=glue.DataFormat.JSON,
    partition_projection={
        "year": glue.PartitionProjectionConfiguration.integer(
            min=2020,
            max=2023,
            interval=1,  # optional, defaults to 1
            digits=4
        )
    }
)

java

Database myDatabase;

S3Table.Builder.create(this, "MyTable")
        .database(myDatabase)
        .columns(List.of(Column.builder()
                .name("data")
                .type(Schema.STRING)
                .build()))
        .partitionKeys(List.of(Column.builder()
                .name("year")
                .type(Schema.INTEGER)
                .build()))
        .dataFormat(DataFormat.JSON)
        .partitionProjection(Map.of(
                "year", PartitionProjectionConfiguration.integer(IntegerPartitionProjectionConfigurationProps.builder()
                        .min(2020)
                        .max(2023)
                        .interval(1) // optional, defaults to 1
                        .digits(4)
                        .build())))
        .build();

csharp

Database myDatabase;

new S3Table(this, "MyTable", new S3TableProps {
    Database = myDatabase,
    Columns = new [] { new Column {
        Name = "data",
        Type = Schema.STRING
    } },
    PartitionKeys = new [] { new Column {
        Name = "year",
        Type = Schema.INTEGER
    } },
    DataFormat = DataFormat.JSON,
    PartitionProjection = new Dictionary<string, PartitionProjectionConfiguration> {
        { "year", PartitionProjectionConfiguration.Integer(new IntegerPartitionProjectionConfigurationProps {
            Min = 2020,
            Max = 2023,
            Interval = 1,  // optional, defaults to 1
            Digits = 4
        }) }
    }
});

var myDatabase Database

glue.NewS3Table(this, jsii.String("MyTable"), &S3TableProps{
	Database: myDatabase,
	Columns: []Column{
		&Column{
			Name: jsii.String("data"),
			Type: glue.Schema_STRING(),
		},
	},
	PartitionKeys: []Column{
		&Column{
			Name: jsii.String("year"),
			Type: glue.Schema_INTEGER(),
		},
	},
	DataFormat: glue.DataFormat_JSON(),
	PartitionProjection: map[string]PartitionProjectionConfiguration{
		"year": glue.PartitionProjectionConfiguration_integer(&IntegerPartitionProjectionConfigurationProps{
			"min": jsii.Number(2020),
			"max": jsii.Number(2023),
			"interval": jsii.Number(1),
			 // optional, defaults to 1
			"digits": jsii.Number(4),
		}),
	},
})

DATE Projection

For partition keys with date or timestamp values. Supports both fixed dates and relative dates using NOW:

declare const myDatabase: glue.Database;
new glue.S3Table(this, 'MyTable', {
  database: myDatabase,
  columns: [{
    name: 'data',
    type: glue.Schema.STRING,
  }],
  partitionKeys: [{
    name: 'date',
    type: glue.Schema.STRING,
  }],
  dataFormat: glue.DataFormat.JSON,
  partitionProjection: {
    date: glue.PartitionProjectionConfiguration.date({
      min: '2020-01-01',
      max: '2023-12-31',
      format: 'yyyy-MM-dd',
      // `step` bundles interval + unit (supply both or neither). Optional at day
      // precision or coarser; required when the format is sub-day (e.g. hours).
      step: { interval: 1, intervalUnit: glue.DateIntervalUnit.DAYS },
    }),
  },
});

python

# my_database: glue.Database

glue.S3Table(self, "MyTable",
    database=my_database,
    columns=[glue.Column(
        name="data",
        type=glue.Schema.STRING
    )],
    partition_keys=[glue.Column(
        name="date",
        type=glue.Schema.STRING
    )],
    data_format=glue.DataFormat.JSON,
    partition_projection={
        "date": glue.PartitionProjectionConfiguration.date(
            min="2020-01-01",
            max="2023-12-31",
            format="yyyy-MM-dd",
            # `step` bundles interval + unit (supply both or neither). Optional at day
            # precision or coarser; required when the format is sub-day (e.g. hours).
            step=glue.DateProjectionStep(interval=1, interval_unit=glue.DateIntervalUnit.DAYS)
        )
    }
)

java

Database myDatabase;

S3Table.Builder.create(this, "MyTable")
        .database(myDatabase)
        .columns(List.of(Column.builder()
                .name("data")
                .type(Schema.STRING)
                .build()))
        .partitionKeys(List.of(Column.builder()
                .name("date")
                .type(Schema.STRING)
                .build()))
        .dataFormat(DataFormat.JSON)
        .partitionProjection(Map.of(
                "date", PartitionProjectionConfiguration.date(DatePartitionProjectionConfigurationProps.builder()
                        .min("2020-01-01")
                        .max("2023-12-31")
                        .format("yyyy-MM-dd")
                        // `step` bundles interval + unit (supply both or neither). Optional at day
                        // precision or coarser; required when the format is sub-day (e.g. hours).
                        .step(DateProjectionStep.builder().interval(1).intervalUnit(DateIntervalUnit.DAYS).build())
                        .build())))
        .build();

csharp

Database myDatabase;

new S3Table(this, "MyTable", new S3TableProps {
    Database = myDatabase,
    Columns = new [] { new Column {
        Name = "data",
        Type = Schema.STRING
    } },
    PartitionKeys = new [] { new Column {
        Name = "date",
        Type = Schema.STRING
    } },
    DataFormat = DataFormat.JSON,
    PartitionProjection = new Dictionary<string, PartitionProjectionConfiguration> {
        { "date", PartitionProjectionConfiguration.Date(new DatePartitionProjectionConfigurationProps {
            Min = "2020-01-01",
            Max = "2023-12-31",
            Format = "yyyy-MM-dd",
            // `step` bundles interval + unit (supply both or neither). Optional at day
            // precision or coarser; required when the format is sub-day (e.g. hours).
            Step = new DateProjectionStep { Interval = 1, IntervalUnit = DateIntervalUnit.DAYS }
        }) }
    }
});

var myDatabase Database

glue.NewS3Table(this, jsii.String("MyTable"), &S3TableProps{
	Database: myDatabase,
	Columns: []Column{
		&Column{
			Name: jsii.String("data"),
			Type: glue.Schema_STRING(),
		},
	},
	PartitionKeys: []Column{
		&Column{
			Name: jsii.String("date"),
			Type: glue.Schema_STRING(),
		},
	},
	DataFormat: glue.DataFormat_JSON(),
	PartitionProjection: map[string]PartitionProjectionConfiguration{
		"date": glue.PartitionProjectionConfiguration_date(&DatePartitionProjectionConfigurationProps{
			"min": jsii.String("2020-01-01"),
			"max": jsii.String("2023-12-31"),
			"format": jsii.String("yyyy-MM-dd"),
			// `step` bundles interval + unit (supply both or neither). Optional at day
			// precision or coarser; required when the format is sub-day (e.g. hours).
			"step": &DateProjectionStep{
				"interval": jsii.Number(1),
				"intervalUnit": glue.DateIntervalUnit_DAYS,
			},
		}),
	},
})

You can also use relative dates with NOW:

declare const myDatabase: glue.Database;
new glue.S3Table(this, 'MyTable', {
  database: myDatabase,
  columns: [{
    name: 'data',
    type: glue.Schema.STRING,
  }],
  partitionKeys: [{
    name: 'date',
    type: glue.Schema.STRING,
  }],
  dataFormat: glue.DataFormat.JSON,
  partitionProjection: {
    date: glue.PartitionProjectionConfiguration.date({
      min: 'NOW-3YEARS',
      max: 'NOW',
      format: 'yyyy-MM-dd',
    }),
  },
});

python

# my_database: glue.Database

glue.S3Table(self, "MyTable",
    database=my_database,
    columns=[glue.Column(
        name="data",
        type=glue.Schema.STRING
    )],
    partition_keys=[glue.Column(
        name="date",
        type=glue.Schema.STRING
    )],
    data_format=glue.DataFormat.JSON,
    partition_projection={
        "date": glue.PartitionProjectionConfiguration.date(
            min="NOW-3YEARS",
            max="NOW",
            format="yyyy-MM-dd"
        )
    }
)

java

Database myDatabase;

S3Table.Builder.create(this, "MyTable")
        .database(myDatabase)
        .columns(List.of(Column.builder()
                .name("data")
                .type(Schema.STRING)
                .build()))
        .partitionKeys(List.of(Column.builder()
                .name("date")
                .type(Schema.STRING)
                .build()))
        .dataFormat(DataFormat.JSON)
        .partitionProjection(Map.of(
                "date", PartitionProjectionConfiguration.date(DatePartitionProjectionConfigurationProps.builder()
                        .min("NOW-3YEARS")
                        .max("NOW")
                        .format("yyyy-MM-dd")
                        .build())))
        .build();

csharp

Database myDatabase;

new S3Table(this, "MyTable", new S3TableProps {
    Database = myDatabase,
    Columns = new [] { new Column {
        Name = "data",
        Type = Schema.STRING
    } },
    PartitionKeys = new [] { new Column {
        Name = "date",
        Type = Schema.STRING
    } },
    DataFormat = DataFormat.JSON,
    PartitionProjection = new Dictionary<string, PartitionProjectionConfiguration> {
        { "date", PartitionProjectionConfiguration.Date(new DatePartitionProjectionConfigurationProps {
            Min = "NOW-3YEARS",
            Max = "NOW",
            Format = "yyyy-MM-dd"
        }) }
    }
});

var myDatabase Database

glue.NewS3Table(this, jsii.String("MyTable"), &S3TableProps{
	Database: myDatabase,
	Columns: []Column{
		&Column{
			Name: jsii.String("data"),
			Type: glue.Schema_STRING(),
		},
	},
	PartitionKeys: []Column{
		&Column{
			Name: jsii.String("date"),
			Type: glue.Schema_STRING(),
		},
	},
	DataFormat: glue.DataFormat_JSON(),
	PartitionProjection: map[string]PartitionProjectionConfiguration{
		"date": glue.PartitionProjectionConfiguration_date(&DatePartitionProjectionConfigurationProps{
			"min": jsii.String("NOW-3YEARS"),
			"max": jsii.String("NOW"),
			"format": jsii.String("yyyy-MM-dd"),
		}),
	},
})

ENUM Projection

For partition keys with a known set of values:

declare const myDatabase: glue.Database;
new glue.S3Table(this, 'MyTable', {
  database: myDatabase,
  columns: [{
    name: 'data',
    type: glue.Schema.STRING,
  }],
  partitionKeys: [{
    name: 'region',
    type: glue.Schema.STRING,
  }],
  dataFormat: glue.DataFormat.JSON,
  partitionProjection: {
    region: glue.PartitionProjectionConfiguration.enum({
      values: ['us-east-1', 'us-west-2', 'eu-west-1'],
    }),
  },
});

python

# my_database: glue.Database

glue.S3Table(self, "MyTable",
    database=my_database,
    columns=[glue.Column(
        name="data",
        type=glue.Schema.STRING
    )],
    partition_keys=[glue.Column(
        name="region",
        type=glue.Schema.STRING
    )],
    data_format=glue.DataFormat.JSON,
    partition_projection={
        "region": glue.PartitionProjectionConfiguration.enum(
            values=["us-east-1", "us-west-2", "eu-west-1"]
        )
    }
)

java

Database myDatabase;

S3Table.Builder.create(this, "MyTable")
        .database(myDatabase)
        .columns(List.of(Column.builder()
                .name("data")
                .type(Schema.STRING)
                .build()))
        .partitionKeys(List.of(Column.builder()
                .name("region")
                .type(Schema.STRING)
                .build()))
        .dataFormat(DataFormat.JSON)
        .partitionProjection(Map.of(
                "region", PartitionProjectionConfiguration.enum(EnumPartitionProjectionConfigurationProps.builder()
                        .values(List.of("us-east-1", "us-west-2", "eu-west-1"))
                        .build())))
        .build();

csharp

Database myDatabase;

new S3Table(this, "MyTable", new S3TableProps {
    Database = myDatabase,
    Columns = new [] { new Column {
        Name = "data",
        Type = Schema.STRING
    } },
    PartitionKeys = new [] { new Column {
        Name = "region",
        Type = Schema.STRING
    } },
    DataFormat = DataFormat.JSON,
    PartitionProjection = new Dictionary<string, PartitionProjectionConfiguration> {
        { "region", PartitionProjectionConfiguration.Enum(new EnumPartitionProjectionConfigurationProps {
            Values = new [] { "us-east-1", "us-west-2", "eu-west-1" }
        }) }
    }
});

var myDatabase Database

glue.NewS3Table(this, jsii.String("MyTable"), &S3TableProps{
	Database: myDatabase,
	Columns: []Column{
		&Column{
			Name: jsii.String("data"),
			Type: glue.Schema_STRING(),
		},
	},
	PartitionKeys: []Column{
		&Column{
			Name: jsii.String("region"),
			Type: glue.Schema_STRING(),
		},
	},
	DataFormat: glue.DataFormat_JSON(),
	PartitionProjection: map[string]PartitionProjectionConfiguration{
		"region": glue.PartitionProjectionConfiguration_enum(&EnumPartitionProjectionConfigurationProps{
			"values": []*string{
				jsii.String("us-east-1"),
				jsii.String("us-west-2"),
				jsii.String("eu-west-1"),
			},
		}),
	},
})

INJECTED Projection

For custom partition values injected at query time:

declare const myDatabase: glue.Database;
new glue.S3Table(this, 'MyTable', {
  database: myDatabase,
  columns: [{
    name: 'data',
    type: glue.Schema.STRING,
  }],
  partitionKeys: [{
    name: 'custom',
    type: glue.Schema.STRING,
  }],
  dataFormat: glue.DataFormat.JSON,
  partitionProjection: {
    custom: glue.PartitionProjectionConfiguration.injected(),
  },
});

python

# my_database: glue.Database

glue.S3Table(self, "MyTable",
    database=my_database,
    columns=[glue.Column(
        name="data",
        type=glue.Schema.STRING
    )],
    partition_keys=[glue.Column(
        name="custom",
        type=glue.Schema.STRING
    )],
    data_format=glue.DataFormat.JSON,
    partition_projection={
        "custom": glue.PartitionProjectionConfiguration.injected()
    }
)

java

Database myDatabase;

S3Table.Builder.create(this, "MyTable")
        .database(myDatabase)
        .columns(List.of(Column.builder()
                .name("data")
                .type(Schema.STRING)
                .build()))
        .partitionKeys(List.of(Column.builder()
                .name("custom")
                .type(Schema.STRING)
                .build()))
        .dataFormat(DataFormat.JSON)
        .partitionProjection(Map.of(
                "custom", PartitionProjectionConfiguration.injected()))
        .build();

csharp

Database myDatabase;

new S3Table(this, "MyTable", new S3TableProps {
    Database = myDatabase,
    Columns = new [] { new Column {
        Name = "data",
        Type = Schema.STRING
    } },
    PartitionKeys = new [] { new Column {
        Name = "custom",
        Type = Schema.STRING
    } },
    DataFormat = DataFormat.JSON,
    PartitionProjection = new Dictionary<string, PartitionProjectionConfiguration> {
        { "custom", PartitionProjectionConfiguration.Injected() }
    }
});

var myDatabase Database

glue.NewS3Table(this, jsii.String("MyTable"), &S3TableProps{
	Database: myDatabase,
	Columns: []Column{
		&Column{
			Name: jsii.String("data"),
			Type: glue.Schema_STRING(),
		},
	},
	PartitionKeys: []Column{
		&Column{
			Name: jsii.String("custom"),
			Type: glue.Schema_STRING(),
		},
	},
	DataFormat: glue.DataFormat_JSON(),
	PartitionProjection: map[string]PartitionProjectionConfiguration{
		"custom": glue.PartitionProjectionConfiguration_injected(),
	},
})

Multiple Partition Projections

You can configure partition projection for multiple partition keys:

declare const myDatabase: glue.Database;
new glue.S3Table(this, 'MyTable', {
  database: myDatabase,
  columns: [{
    name: 'data',
    type: glue.Schema.STRING,
  }],
  partitionKeys: [
    {
      name: 'year',
      type: glue.Schema.INTEGER,
    },
    {
      name: 'month',
      type: glue.Schema.INTEGER,
    },
    {
      name: 'region',
      type: glue.Schema.STRING,
    },
  ],
  dataFormat: glue.DataFormat.JSON,
  partitionProjection: {
    year: glue.PartitionProjectionConfiguration.integer({
      min: 2020,
      max: 2023,
    }),
    month: glue.PartitionProjectionConfiguration.integer({
      min: 1,
      max: 12,
      digits: 2,
    }),
    region: glue.PartitionProjectionConfiguration.enum({
      values: ['us-east-1', 'us-west-2'],
    }),
  },
});

python

# my_database: glue.Database

glue.S3Table(self, "MyTable",
    database=my_database,
    columns=[glue.Column(
        name="data",
        type=glue.Schema.STRING
    )],
    partition_keys=[glue.Column(
        name="year",
        type=glue.Schema.INTEGER
    ), glue.Column(
        name="month",
        type=glue.Schema.INTEGER
    ), glue.Column(
        name="region",
        type=glue.Schema.STRING
    )
    ],
    data_format=glue.DataFormat.JSON,
    partition_projection={
        "year": glue.PartitionProjectionConfiguration.integer(
            min=2020,
            max=2023
        ),
        "month": glue.PartitionProjectionConfiguration.integer(
            min=1,
            max=12,
            digits=2
        ),
        "region": glue.PartitionProjectionConfiguration.enum(
            values=["us-east-1", "us-west-2"]
        )
    }
)

java

Database myDatabase;

S3Table.Builder.create(this, "MyTable")
        .database(myDatabase)
        .columns(List.of(Column.builder()
                .name("data")
                .type(Schema.STRING)
                .build()))
        .partitionKeys(List.of(Column.builder()
                .name("year")
                .type(Schema.INTEGER)
                .build(), Column.builder()
                .name("month")
                .type(Schema.INTEGER)
                .build(), Column.builder()
                .name("region")
                .type(Schema.STRING)
                .build()))
        .dataFormat(DataFormat.JSON)
        .partitionProjection(Map.of(
                "year", PartitionProjectionConfiguration.integer(IntegerPartitionProjectionConfigurationProps.builder()
                        .min(2020)
                        .max(2023)
                        .build()),
                "month", PartitionProjectionConfiguration.integer(IntegerPartitionProjectionConfigurationProps.builder()
                        .min(1)
                        .max(12)
                        .digits(2)
                        .build()),
                "region", PartitionProjectionConfiguration.enum(EnumPartitionProjectionConfigurationProps.builder()
                        .values(List.of("us-east-1", "us-west-2"))
                        .build())))
        .build();

csharp

Database myDatabase;

new S3Table(this, "MyTable", new S3TableProps {
    Database = myDatabase,
    Columns = new [] { new Column {
        Name = "data",
        Type = Schema.STRING
    } },
    PartitionKeys = new [] { new Column {
        Name = "year",
        Type = Schema.INTEGER
    }, new Column {
        Name = "month",
        Type = Schema.INTEGER
    }, new Column {
        Name = "region",
        Type = Schema.STRING
    } },
    DataFormat = DataFormat.JSON,
    PartitionProjection = new Dictionary<string, PartitionProjectionConfiguration> {
        { "year", PartitionProjectionConfiguration.Integer(new IntegerPartitionProjectionConfigurationProps {
            Min = 2020,
            Max = 2023
        }) },
        { "month", PartitionProjectionConfiguration.Integer(new IntegerPartitionProjectionConfigurationProps {
            Min = 1,
            Max = 12,
            Digits = 2
        }) },
        { "region", PartitionProjectionConfiguration.Enum(new EnumPartitionProjectionConfigurationProps {
            Values = new [] { "us-east-1", "us-west-2" }
        }) }
    }
});

var myDatabase Database

glue.NewS3Table(this, jsii.String("MyTable"), &S3TableProps{
	Database: myDatabase,
	Columns: []Column{
		&Column{
			Name: jsii.String("data"),
			Type: glue.Schema_STRING(),
		},
	},
	PartitionKeys: []Column{
		&Column{
			Name: jsii.String("year"),
			Type: glue.Schema_INTEGER(),
		},
		&Column{
			Name: jsii.String("month"),
			Type: glue.Schema_INTEGER(),
		},
		&Column{
			Name: jsii.String("region"),
			Type: glue.Schema_STRING(),
		},
	},
	DataFormat: glue.DataFormat_JSON(),
	PartitionProjection: map[string]PartitionProjectionConfiguration{
		"year": glue.PartitionProjectionConfiguration_integer(&IntegerPartitionProjectionConfigurationProps{
			"min": jsii.Number(2020),
			"max": jsii.Number(2023),
		}),
		"month": glue.PartitionProjectionConfiguration_integer(&IntegerPartitionProjectionConfigurationProps{
			"min": jsii.Number(1),
			"max": jsii.Number(12),
			"digits": jsii.Number(2),
		}),
		"region": glue.PartitionProjectionConfiguration_enum(&EnumPartitionProjectionConfigurationProps{
			"values": []*string{
				jsii.String("us-east-1"),
				jsii.String("us-west-2"),
			},
		}),
	},
})

Glue Connections

Glue connections allow external data connections to third party databases and data warehouses. However, these connections can also be assigned to Glue Tables, allowing you to query external data sources using the Glue Data Catalog.

Whereas S3Table will point to (and if needed, create) a bucket to store the tables' data, ExternalTable will point to an existing table in a data source. For example, to create a table in Glue that points to a table in Redshift:

declare const myConnection: glue.Connection;
declare const myDatabase: glue.Database;
new glue.ExternalTable(this, 'MyTable', {
  connection: myConnection,
  externalDataLocation: 'default_db_public_example', // A table in Redshift
  // ...
  database: myDatabase,
  columns: [{
    name: 'col1',
    type: glue.Schema.STRING,
  }],
  dataFormat: glue.DataFormat.JSON,
});

python

# my_connection: glue.Connection
# my_database: glue.Database

glue.ExternalTable(self, "MyTable",
    connection=my_connection,
    external_data_location="default_db_public_example",  # A table in Redshift
    # ...
    database=my_database,
    columns=[glue.Column(
        name="col1",
        type=glue.Schema.STRING
    )],
    data_format=glue.DataFormat.JSON
)

java

Connection myConnection;
Database myDatabase;

ExternalTable.Builder.create(this, "MyTable")
        .connection(myConnection)
        .externalDataLocation("default_db_public_example") // A table in Redshift
        // ...
        .database(myDatabase)
        .columns(List.of(Column.builder()
                .name("col1")
                .type(Schema.STRING)
                .build()))
        .dataFormat(DataFormat.JSON)
        .build();

csharp

Connection myConnection;
Database myDatabase;

new ExternalTable(this, "MyTable", new ExternalTableProps {
    Connection = myConnection,
    ExternalDataLocation = "default_db_public_example",  // A table in Redshift
    // ...
    Database = myDatabase,
    Columns = new [] { new Column {
        Name = "col1",
        Type = Schema.STRING
    } },
    DataFormat = DataFormat.JSON
});

var myConnection Connection
var myDatabase Database

glue.NewExternalTable(this, jsii.String("MyTable"), &ExternalTableProps{
	Connection: myConnection,
	ExternalDataLocation: jsii.String("default_db_public_example"),
	 // A table in Redshift
	// ...
	Database: myDatabase,
	Columns: []Column{
		&Column{
			Name: jsii.String("col1"),
			Type: glue.Schema_STRING(),
		},
	},
	DataFormat: glue.DataFormat_JSON(),
})

Data Quality Ruleset

A DataQualityRuleset defines a set of data quality rules — authored in Glue's Data Quality Definition Language (DQDL) — that are evaluated against a table in the Data Catalog.

declare const database: glue.IDatabase;
new glue.DataQualityRuleset(this, 'MyRuleset', {
  rulesetName: 'my_ruleset',
  dqdl: glue.Dqdl.fromString('Rules = [ RowCount > 100, IsComplete "order_id" ]'),
  targetTable: glue.DataQualityTargetTable.fromTableName(database, 'my_table'),
});

python

# database: glue.IDatabase

glue.DataQualityRuleset(self, "MyRuleset",
    ruleset_name="my_ruleset",
    dqdl=glue.Dqdl.from_string("Rules = [ RowCount > 100, IsComplete \"order_id\" ]"),
    target_table=glue.DataQualityTargetTable.from_table_name(database, "my_table")
)

java

IDatabase database;

DataQualityRuleset.Builder.create(this, "MyRuleset")
        .rulesetName("my_ruleset")
        .dqdl(Dqdl.fromString("Rules = [ RowCount > 100, IsComplete \"order_id\" ]"))
        .targetTable(DataQualityTargetTable.fromTableName(database, "my_table"))
        .build();

csharp

IDatabase database;

new DataQualityRuleset(this, "MyRuleset", new DataQualityRulesetProps {
    RulesetName = "my_ruleset",
    Dqdl = Dqdl.FromString("Rules = [ RowCount > 100, IsComplete \"order_id\" ]"),
    TargetTable = DataQualityTargetTable.FromTableName(database, "my_table")
});

var database IDatabase

glue.NewDataQualityRuleset(this, jsii.String("MyRuleset"), &DataQualityRulesetProps{
	RulesetName: jsii.String("my_ruleset"),
	Dqdl: glue.Dqdl_FromString(jsii.String("Rules = [ RowCount > 100, IsComplete \"order_id\" ]")),
	TargetTable: glue.DataQualityTargetTable_FromTableName(database, jsii.String("my_table")),
})

Build the DQDL document with Dqdl.fromString(...). Glue parses and validates the DQDL when the ruleset is deployed; see the DQDL reference for the full rule syntax.

Encryption

When the table creates its own S3 bucket (S3TableStorage.managedBucket, the default), that bucket enforces SSL: a bucket policy denies any request made over plain HTTP. If you bring your own bucket with S3TableStorage.fromBucket, enabling enforceSSL on it is your responsibility.

Server-side encryption applies only to a bucket the table manages. Choose it with storage: glue.S3TableStorage.managedBucket(...):

S3Managed - (default) Server side encryption (SSE-S3) with an Amazon S3-managed key.

declare const myDatabase: glue.Database;
new glue.S3Table(this, 'MyTable', {
  storage: glue.S3TableStorage.managedBucket(glue.S3TableEncryption.s3Managed()),
  // ...
  database: myDatabase,
  columns: [{
    name: 'col1',
    type: glue.Schema.STRING,
  }],
  dataFormat: glue.DataFormat.JSON,
});

python

# my_database: glue.Database

glue.S3Table(self, "MyTable",
    storage=glue.S3TableStorage.managed_bucket(glue.S3TableEncryption.s3_managed()),
    # ...
    database=my_database,
    columns=[glue.Column(
        name="col1",
        type=glue.Schema.STRING
    )],
    data_format=glue.DataFormat.JSON
)

java

Database myDatabase;

S3Table.Builder.create(this, "MyTable")
        .storage(S3TableStorage.managedBucket(S3TableEncryption.s3Managed()))
        // ...
        .database(myDatabase)
        .columns(List.of(Column.builder()
                .name("col1")
                .type(Schema.STRING)
                .build()))
        .dataFormat(DataFormat.JSON)
        .build();

csharp

Database myDatabase;

new S3Table(this, "MyTable", new S3TableProps {
    Storage = S3TableStorage.ManagedBucket(S3TableEncryption.S3Managed()),
    // ...
    Database = myDatabase,
    Columns = new [] { new Column {
        Name = "col1",
        Type = Schema.STRING
    } },
    DataFormat = DataFormat.JSON
});

var myDatabase Database

glue.NewS3Table(this, jsii.String("MyTable"), &S3TableProps{
	Storage: glue.S3TableStorage_ManagedBucket(glue.S3TableEncryption_S3Managed()),
	// ...
	Database: myDatabase,
	Columns: []Column{
		&Column{
			Name: jsii.String("col1"),
			Type: glue.Schema_STRING(),
		},
	},
	DataFormat: glue.DataFormat_JSON(),
})

Kms - Server-side encryption (SSE-KMS) with an AWS KMS Key managed by the account owner.

declare const myDatabase: glue.Database;
// KMS key is created automatically
new glue.S3Table(this, 'MyTable', {
  storage: glue.S3TableStorage.managedBucket(glue.S3TableEncryption.kms()),
  // ...
  database: myDatabase,
  columns: [{
    name: 'col1',
    type: glue.Schema.STRING,
  }],
  dataFormat: glue.DataFormat.JSON,
});

// with an explicit KMS key
new glue.S3Table(this, 'MyTable', {
  storage: glue.S3TableStorage.managedBucket(glue.S3TableEncryption.kms(new kms.Key(this, 'MyKey'))),
  // ...
  database: myDatabase,
  columns: [{
    name: 'col1',
    type: glue.Schema.STRING,
  }],
  dataFormat: glue.DataFormat.JSON,
});

python

# my_database: glue.Database

# KMS key is created automatically
glue.S3Table(self, "MyTable",
    storage=glue.S3TableStorage.managed_bucket(glue.S3TableEncryption.kms()),
    # ...
    database=my_database,
    columns=[glue.Column(
        name="col1",
        type=glue.Schema.STRING
    )],
    data_format=glue.DataFormat.JSON
)

# with an explicit KMS key
glue.S3Table(self, "MyTable",
    storage=glue.S3TableStorage.managed_bucket(glue.S3TableEncryption.kms(kms.Key(self, "MyKey"))),
    # ...
    database=my_database,
    columns=[glue.Column(
        name="col1",
        type=glue.Schema.STRING
    )],
    data_format=glue.DataFormat.JSON
)

java

Database myDatabase;

// KMS key is created automatically
// KMS key is created automatically
S3Table.Builder.create(this, "MyTable")
        .storage(S3TableStorage.managedBucket(S3TableEncryption.kms()))
        // ...
        .database(myDatabase)
        .columns(List.of(Column.builder()
                .name("col1")
                .type(Schema.STRING)
                .build()))
        .dataFormat(DataFormat.JSON)
        .build();

// with an explicit KMS key
// with an explicit KMS key
S3Table.Builder.create(this, "MyTable")
        .storage(S3TableStorage.managedBucket(S3TableEncryption.kms(new Key(this, "MyKey"))))
        // ...
        .database(myDatabase)
        .columns(List.of(Column.builder()
                .name("col1")
                .type(Schema.STRING)
                .build()))
        .dataFormat(DataFormat.JSON)
        .build();

csharp

Database myDatabase;

// KMS key is created automatically
// KMS key is created automatically
new S3Table(this, "MyTable", new S3TableProps {
    Storage = S3TableStorage.ManagedBucket(S3TableEncryption.Kms()),
    // ...
    Database = myDatabase,
    Columns = new [] { new Column {
        Name = "col1",
        Type = Schema.STRING
    } },
    DataFormat = DataFormat.JSON
});

// with an explicit KMS key
// with an explicit KMS key
new S3Table(this, "MyTable", new S3TableProps {
    Storage = S3TableStorage.ManagedBucket(S3TableEncryption.Kms(new Key(this, "MyKey"))),
    // ...
    Database = myDatabase,
    Columns = new [] { new Column {
        Name = "col1",
        Type = Schema.STRING
    } },
    DataFormat = DataFormat.JSON
});

var myDatabase Database

// KMS key is created automatically
// KMS key is created automatically
glue.NewS3Table(this, jsii.String("MyTable"), &S3TableProps{
	Storage: glue.S3TableStorage_ManagedBucket(glue.S3TableEncryption_Kms()),
	// ...
	Database: myDatabase,
	Columns: []Column{
		&Column{
			Name: jsii.String("col1"),
			Type: glue.Schema_STRING(),
		},
	},
	DataFormat: glue.DataFormat_JSON(),
})

// with an explicit KMS key
// with an explicit KMS key
glue.NewS3Table(this, jsii.String("MyTable"), &S3TableProps{
	Storage: glue.S3TableStorage_*ManagedBucket(glue.S3TableEncryption_*Kms(kms.NewKey(this, jsii.String("MyKey")))),
	// ...
	Database: myDatabase,
	Columns: []Column{
		&Column{
			Name: jsii.String("col1"),
			Type: glue.Schema_STRING(),
		},
	},
	DataFormat: glue.DataFormat_JSON(),
})

KmsManaged - Server-side encryption (SSE-KMS), like Kms, except with an AWS KMS Key managed by the AWS Key Management Service.

declare const myDatabase: glue.Database;
new glue.S3Table(this, 'MyTable', {
  storage: glue.S3TableStorage.managedBucket(glue.S3TableEncryption.kmsManaged()),
  // ...
  database: myDatabase,
  columns: [{
    name: 'col1',
    type: glue.Schema.STRING,
  }],
  dataFormat: glue.DataFormat.JSON,
});

python

# my_database: glue.Database

glue.S3Table(self, "MyTable",
    storage=glue.S3TableStorage.managed_bucket(glue.S3TableEncryption.kms_managed()),
    # ...
    database=my_database,
    columns=[glue.Column(
        name="col1",
        type=glue.Schema.STRING
    )],
    data_format=glue.DataFormat.JSON
)

java

Database myDatabase;

S3Table.Builder.create(this, "MyTable")
        .storage(S3TableStorage.managedBucket(S3TableEncryption.kmsManaged()))
        // ...
        .database(myDatabase)
        .columns(List.of(Column.builder()
                .name("col1")
                .type(Schema.STRING)
                .build()))
        .dataFormat(DataFormat.JSON)
        .build();

csharp

Database myDatabase;

new S3Table(this, "MyTable", new S3TableProps {
    Storage = S3TableStorage.ManagedBucket(S3TableEncryption.KmsManaged()),
    // ...
    Database = myDatabase,
    Columns = new [] { new Column {
        Name = "col1",
        Type = Schema.STRING
    } },
    DataFormat = DataFormat.JSON
});

var myDatabase Database

glue.NewS3Table(this, jsii.String("MyTable"), &S3TableProps{
	Storage: glue.S3TableStorage_ManagedBucket(glue.S3TableEncryption_KmsManaged()),
	// ...
	Database: myDatabase,
	Columns: []Column{
		&Column{
			Name: jsii.String("col1"),
			Type: glue.Schema_STRING(),
		},
	},
	DataFormat: glue.DataFormat_JSON(),
})

Client-side encryption (CSE-KMS) is independent of the bucket's server-side encryption and works with either a managed or an existing bucket. Configure it with clientSideEncryption:

declare const myDatabase: glue.Database;
// KMS key is created automatically
new glue.S3Table(this, 'MyTable', {
  clientSideEncryption: glue.TableClientSideEncryption.kms(),
  // ...
  database: myDatabase,
  columns: [{
    name: 'col1',
    type: glue.Schema.STRING,
  }],
  dataFormat: glue.DataFormat.JSON,
});

// with an explicit KMS key
new glue.S3Table(this, 'MyTable', {
  clientSideEncryption: glue.TableClientSideEncryption.kms(new kms.Key(this, 'MyKey')),
  // ...
  database: myDatabase,
  columns: [{
    name: 'col1',
    type: glue.Schema.STRING,
  }],
  dataFormat: glue.DataFormat.JSON,
});

python

# my_database: glue.Database

# KMS key is created automatically
glue.S3Table(self, "MyTable",
    client_side_encryption=glue.TableClientSideEncryption.kms(),
    # ...
    database=my_database,
    columns=[glue.Column(
        name="col1",
        type=glue.Schema.STRING
    )],
    data_format=glue.DataFormat.JSON
)

# with an explicit KMS key
glue.S3Table(self, "MyTable",
    client_side_encryption=glue.TableClientSideEncryption.kms(kms.Key(self, "MyKey")),
    # ...
    database=my_database,
    columns=[glue.Column(
        name="col1",
        type=glue.Schema.STRING
    )],
    data_format=glue.DataFormat.JSON
)

java

Database myDatabase;

// KMS key is created automatically
// KMS key is created automatically
S3Table.Builder.create(this, "MyTable")
        .clientSideEncryption(TableClientSideEncryption.kms())
        // ...
        .database(myDatabase)
        .columns(List.of(Column.builder()
                .name("col1")
                .type(Schema.STRING)
                .build()))
        .dataFormat(DataFormat.JSON)
        .build();

// with an explicit KMS key
// with an explicit KMS key
S3Table.Builder.create(this, "MyTable")
        .clientSideEncryption(TableClientSideEncryption.kms(new Key(this, "MyKey")))
        // ...
        .database(myDatabase)
        .columns(List.of(Column.builder()
                .name("col1")
                .type(Schema.STRING)
                .build()))
        .dataFormat(DataFormat.JSON)
        .build();

csharp

Database myDatabase;

// KMS key is created automatically
// KMS key is created automatically
new S3Table(this, "MyTable", new S3TableProps {
    ClientSideEncryption = TableClientSideEncryption.Kms(),
    // ...
    Database = myDatabase,
    Columns = new [] { new Column {
        Name = "col1",
        Type = Schema.STRING
    } },
    DataFormat = DataFormat.JSON
});

// with an explicit KMS key
// with an explicit KMS key
new S3Table(this, "MyTable", new S3TableProps {
    ClientSideEncryption = TableClientSideEncryption.Kms(new Key(this, "MyKey")),
    // ...
    Database = myDatabase,
    Columns = new [] { new Column {
        Name = "col1",
        Type = Schema.STRING
    } },
    DataFormat = DataFormat.JSON
});

var myDatabase Database

// KMS key is created automatically
// KMS key is created automatically
glue.NewS3Table(this, jsii.String("MyTable"), &S3TableProps{
	ClientSideEncryption: glue.TableClientSideEncryption_Kms(),
	// ...
	Database: myDatabase,
	Columns: []Column{
		&Column{
			Name: jsii.String("col1"),
			Type: glue.Schema_STRING(),
		},
	},
	DataFormat: glue.DataFormat_JSON(),
})

// with an explicit KMS key
// with an explicit KMS key
glue.NewS3Table(this, jsii.String("MyTable"), &S3TableProps{
	ClientSideEncryption: glue.TableClientSideEncryption_*Kms(kms.NewKey(this, jsii.String("MyKey"))),
	// ...
	Database: myDatabase,
	Columns: []Column{
		&Column{
			Name: jsii.String("col1"),
			Type: glue.Schema_STRING(),
		},
	},
	DataFormat: glue.DataFormat_JSON(),
})

To store the table's data in an existing bucket, use glue.S3TableStorage.fromBucket(bucket). CDK does not manage that bucket's server-side encryption, so an encryption choice can never be paired with a provided bucket — but client-side encryption still applies.

Marking table data as encrypted

Both S3Table and ExternalTable set the has_encrypted_data table parameter, which Athena reads when querying client-side (CSE-KMS) encrypted datasets. It defaults to true. Set hasEncryptedData to false when the underlying data is not encrypted:

declare const myDatabase: glue.Database;
new glue.S3Table(this, 'MyTable', {
  hasEncryptedData: false,
  database: myDatabase,
  columns: [{
    name: 'col1',
    type: glue.Schema.STRING,
  }],
  dataFormat: glue.DataFormat.JSON,
});

python

# my_database: glue.Database

glue.S3Table(self, "MyTable",
    has_encrypted_data=False,
    database=my_database,
    columns=[glue.Column(
        name="col1",
        type=glue.Schema.STRING
    )],
    data_format=glue.DataFormat.JSON
)

java

Database myDatabase;

S3Table.Builder.create(this, "MyTable")
        .hasEncryptedData(false)
        .database(myDatabase)
        .columns(List.of(Column.builder()
                .name("col1")
                .type(Schema.STRING)
                .build()))
        .dataFormat(DataFormat.JSON)
        .build();

csharp

Database myDatabase;

new S3Table(this, "MyTable", new S3TableProps {
    HasEncryptedData = false,
    Database = myDatabase,
    Columns = new [] { new Column {
        Name = "col1",
        Type = Schema.STRING
    } },
    DataFormat = DataFormat.JSON
});

var myDatabase Database

glue.NewS3Table(this, jsii.String("MyTable"), &S3TableProps{
	HasEncryptedData: jsii.Boolean(false),
	Database: myDatabase,
	Columns: []Column{
		&Column{
			Name: jsii.String("col1"),
			Type: glue.Schema_STRING(),
		},
	},
	DataFormat: glue.DataFormat_JSON(),
})

Do not set has_encrypted_data through the free-form parameters map as well - a value there that conflicts with hasEncryptedData is rejected at synthesis time.

Types

A table's schema is a collection of columns, each of which have a name and a type. Types are recursive structures, consisting of primitive and complex types:

declare const myDatabase: glue.Database;
new glue.S3Table(this, 'MyTable', {
  columns: [{
    name: 'primitive_column',
    type: glue.Schema.STRING,
  }, {
    name: 'array_column',
    type: glue.Schema.array(glue.Schema.INTEGER),
    comment: 'array<integer>',
  }, {
    name: 'map_column',
    type: glue.Schema.map(
      glue.Schema.STRING,
      glue.Schema.TIMESTAMP),
    comment: 'map<string,timestamp>',
  }, {
    name: 'struct_column',
    type: glue.Schema.struct([{
      name: 'nested_column',
      type: glue.Schema.DATE,
      comment: 'nested comment',
    }]),
    comment: "struct<nested_column:date COMMENT 'nested comment'>",
  }],
  // ...
  database: myDatabase,
  dataFormat: glue.DataFormat.JSON,
});

python

# my_database: glue.Database

glue.S3Table(self, "MyTable",
    columns=[glue.Column(
        name="primitive_column",
        type=glue.Schema.STRING
    ), glue.Column(
        name="array_column",
        type=glue.Schema.array(glue.Schema.INTEGER),
        comment="array<integer>"
    ), glue.Column(
        name="map_column",
        type=glue.Schema.map(glue.Schema.STRING, glue.Schema.TIMESTAMP),
        comment="map<string,timestamp>"
    ), glue.Column(
        name="struct_column",
        type=glue.Schema.struct([
            name="nested_column",
            type=glue.Schema.DATE,
            comment="nested comment"
        ]),
        comment="struct<nested_column:date COMMENT 'nested comment'>"
    )],
    # ...
    database=my_database,
    data_format=glue.DataFormat.JSON
)

java

Database myDatabase;

S3Table.Builder.create(this, "MyTable")
        .columns(List.of(Column.builder()
                .name("primitive_column")
                .type(Schema.STRING)
                .build(), Column.builder()
                .name("array_column")
                .type(Schema.array(Schema.INTEGER))
                .comment("array<integer>")
                .build(), Column.builder()
                .name("map_column")
                .type(Schema.map(Schema.STRING, Schema.TIMESTAMP))
                .comment("map<string,timestamp>")
                .build(), Column.builder()
                .name("struct_column")
                .type(Schema.struct(List.of(Column.builder()
                        .name("nested_column")
                        .type(Schema.DATE)
                        .comment("nested comment")
                        .build())))
                .comment("struct<nested_column:date COMMENT 'nested comment'>")
                .build()))
        // ...
        .database(myDatabase)
        .dataFormat(DataFormat.JSON)
        .build();

csharp

Database myDatabase;

new S3Table(this, "MyTable", new S3TableProps {
    Columns = new [] { new Column {
        Name = "primitive_column",
        Type = Schema.STRING
    }, new Column {
        Name = "array_column",
        Type = Schema.Array(Schema.INTEGER),
        Comment = "array<integer>"
    }, new Column {
        Name = "map_column",
        Type = Schema.Map(Schema.STRING, Schema.TIMESTAMP),
        Comment = "map<string,timestamp>"
    }, new Column {
        Name = "struct_column",
        Type = Schema.Struct(new [] { new Column {
            Name = "nested_column",
            Type = Schema.DATE,
            Comment = "nested comment"
        } }),
        Comment = "struct<nested_column:date COMMENT 'nested comment'>"
    } },
    // ...
    Database = myDatabase,
    DataFormat = DataFormat.JSON
});

var myDatabase Database

glue.NewS3Table(this, jsii.String("MyTable"), &S3TableProps{
	Columns: []Column{
		&Column{
			Name: jsii.String("primitive_column"),
			Type: glue.Schema_STRING(),
		},
		&Column{
			Name: jsii.String("array_column"),
			Type: glue.Schema_Array(glue.Schema_INTEGER()),
			Comment: jsii.String("array<integer>"),
		},
		&Column{
			Name: jsii.String("map_column"),
			Type: glue.Schema_Map(glue.Schema_STRING(), glue.Schema_TIMESTAMP()),
			Comment: jsii.String("map<string,timestamp>"),
		},
		&Column{
			Name: jsii.String("struct_column"),
			Type: glue.Schema_Struct([]Column{
				&Column{
					Name: jsii.String("nested_column"),
					Type: glue.Schema_DATE(),
					Comment: jsii.String("nested comment"),
				},
			}),
			Comment: jsii.String("struct<nested_column:date COMMENT 'nested comment'>"),
		},
	},
	// ...
	Database: myDatabase,
	DataFormat: glue.DataFormat_JSON(),
})

For a type the Schema factories don't model, use glue.Schema.custom('...'), which takes the raw Glue input string.

Public FAQ

What are we launching today?

We’re launching new features to an AWS CDK Glue L2 Construct to provide best-practice defaults and convenience methods to create Glue Jobs, Connections, Triggers, Workflows, and the underlying permissions and configuration.

Why should I use this Construct?

Developers should use this Construct to reduce the amount of boilerplate code and complexity each individual has to navigate, and make it easier to create best-practice Glue resources.

What’s not in scope?

Glue Crawlers and other resources that are now managed by the AWS LakeFormation team are not in scope for this effort. Developers should use existing methods to create these resources, and the new Glue L2 construct assumes they already exist as inputs. While best practice is for application and infrastructure code to be as close as possible for teams using fully-implemented DevOps mechanisms, in practice these ETL scripts are likely managed by a data science team who know Python or Scala and don’t necessarily own or manage their own infrastructure deployments. We want to meet developers where they are, and not assume that all of the code resides in the same repository, Developers can automate this themselves via the CDK, however, if they do own both.

Validating Glue version and feature use per AWS region at synth time is also not in scope. AWS’ intention is for all features to eventually be propagated to all Global regions, so the complexity involved in creating and updating region- specific configuration to match shifting feature sets does not out-weigh the likelihood that a developer will use this construct to deploy resources to a region without a particular new feature to a region that doesn’t yet support it without researching or manually attempting to use that feature before developing it via IaC. The developer will, of course, still get feedback from the underlying Glue APIs as CloudFormation deploys the resources similar to the current CDK L1 Glue experience.

Types

Name	Kind	Summary
`Catalog`	Construct (L2)	A Glue Data Catalog.
`CatalogBase`	Construct (L2)	Base class for all `ICatalog` implementations.
`Connection`	Construct (L2)	An AWS Glue connection to a data source.
`Database`	Construct (L2)	A Glue database.
`DataQualityRuleset`	Construct (L2)	A Glue Data Quality ruleset.
`ExternalTable`	Construct (L2)	A Glue table that targets an external data location (e.g. A table in a Redshift Cluster).
`Job`	Construct (L2)	A Glue Job.
`JobBase`	Construct (L2)	A base class is needed to be able to import existing Jobs into a CDK app to reference as part of a larger stack or construct.
`PySparkEtlJob`	Construct (L2)	PySpark ETL Jobs class.
`PySparkFlexEtlJob`	Construct (L2)	Flex Jobs class.
`PySparkStreamingJob`	Construct (L2)	Python Spark Streaming Jobs class.
`PythonShellJob`	Construct (L2)	Python Shell Jobs class.
`RayJob`	Construct (L2)	Ray Jobs class.
`S3Table`	Construct (L2)	A Glue table that targets a S3 dataset.
`ScalaSparkEtlJob`	Construct (L2)	Spark ETL Jobs class.
`ScalaSparkFlexEtlJob`	Construct (L2)	Scala Spark Flex ETL Jobs class.
`ScalaSparkStreamingJob`	Construct (L2)	Scala Streaming Jobs class.
`SecurityConfiguration`	Construct (L2)	A security configuration is a set of security properties that can be used by AWS Glue to encrypt data at rest.
`SparkJob`	Construct (L2)	Base class for different types of Spark Jobs.
`TableBase`	Construct (L2)	A Glue table.
`Workflow`	Construct (L2)	This module defines a construct for creating and managing AWS Glue Workflows and Triggers.
`WorkflowBase`	Construct (L2)	Base abstract class for Workflow.
`Action`	Class	An action initiated by a trigger.
`AssetCode`	Class	Job Code from a local file.
`CatalogEncryptionMode`	Enum	The encryption-at-rest mode for a Glue Data Catalog.
`CatalogEncryptionOptions`	Struct	Encryption configuration for a Glue Data Catalog.
`CatalogProps`	Struct	Construction properties for a `Catalog`.
`ClassificationString`	Class	Classification string given to tables with this data format.
`CloudWatchEncryption`	Class	CloudWatch Logs encryption configuration for a `SecurityConfiguration`.
`Code`	Class	Represents a Glue Job's Code assets (an asset can be a scripts, a jar, a python file or any other file).
`CodeConfig`	Struct	Result of binding `Code` into a `Job`.
`Column`	Struct	A column of a table.
`ColumnCountMismatchHandlingAction`	Enum	Identifies if the file contains less or more values for a row than the number of columns specified in the external table definition.
`CompressionType`	Enum	The compression type.
`Condition`	Class	A condition that determines when a conditional trigger fires.
`ConditionalTriggerOptions`	Struct	Properties for configuring a Condition (Predicate) based Glue Trigger.
`ConditionLogicalOperator`	Enum	Represents the logical operator for evaluating a single condition in the Glue Trigger API.
`ConditionOptions`	Struct	Options shared by all trigger conditions.
`ConnectionNetwork`	Class	VPC network placement for a Glue `Connection`.
`ConnectionOptions`	Struct	Base Connection Options.
`ConnectionPasswordEncryption`	Struct	Connection-password encryption configuration for a Glue Data Catalog.
`ConnectionProps`	Struct	Construction properties for `Connection`.
`ConnectionType`	Class	The type of the glue connection.
`ContinuousLoggingProps`	Struct	Properties for enabling Continuous Logging for Glue Jobs.
`CrawlerActionOptions`	Struct	Options for the execution of a crawler.
`CrawlerState`	Enum	Represents the state of a crawler for a condition in the Glue Trigger API.
`DatabaseProps`	Struct
`DataCatalogEncryptionAtRest`	Class	Encryption-at-rest configuration for a Glue Data Catalog.
`DataFormat`	Class	Defines the input/output formats and ser/de for a single DataFormat.
`DataFormatProps`	Struct	Properties of a DataFormat instance.
`DataQualityRulesetProps`	Struct	Construction properties for `DataQualityRuleset`.
`DataQualityTargetTable`	Class	The Glue table a `DataQualityRuleset` evaluates.
`DateIntervalUnit`	Enum	Date interval unit for partition projection.
`DatePartitionProjectionConfigurationProps`	Struct	Properties for DATE partition projection configuration.
`DateProjectionStep`	Struct	A required-together interval step for DATE partition projection.
`Dqdl`	Class	The Data Quality Definition Language (DQDL) document for a `DataQualityRuleset`.
`EnumPartitionProjectionConfigurationProps`	Struct	Properties for ENUM partition projection configuration.
`EventBatchingCondition`	Struct	Represents event trigger batch condition.
`EventTriggerOptions`	Struct	Properties for configuring an Event Bridge based Glue Trigger.
`ExecutionClass`	Enum	The ExecutionClass whether the job is run with a standard or flexible execution class.
`ExternalTableProps`	Struct
`GlueVersion`	Enum	AWS Glue version determines the versions of Apache Spark and Python that are available to the job.
`ICatalog`	Interface	A Glue Data Catalog, either the implicit account-wide catalog or one created as an `AWS::Glue::Catalog` resource.
`IConnection`	Interface	Interface representing a created or an imported `Connection`.
`IDatabase`	Interface
`IDataQualityRuleset`	Interface
`IJob`	Interface	Interface representing a new or an imported Glue Job.
`InputFormat`	Class	Absolute class name of the Hadoop `InputFormat` to use when reading table files.
`IntegerPartitionProjectionConfigurationProps`	Struct	Properties for INTEGER partition projection configuration.
`InvalidCharHandlingAction`	Enum	Specifies the action to perform when query results contain invalid UTF-8 character values.
`ISecurityConfiguration`	Interface	Interface representing a created or an imported `SecurityConfiguration`.
`ITable`	Interface
`IWorkflow`	Interface	The base interface for Glue Workflow.
`JobActionOptions`	Struct	Options for the execution of a job.
`JobAttributes`	Struct	A subset of Job attributes are required for importing an existing job into a CDK project.
`JobBookmarksEncryption`	Class	Job bookmarks encryption configuration for a `SecurityConfiguration`.
`JobLanguage`	Enum	Runtime language of the Glue job.
`JobProps`	Struct	JobProps will be used to create new Glue Jobs using this L2 Construct.
`JobState`	Enum	Job states emitted by Glue to CloudWatch Events.
`JobType`	Enum	The job type.
`LibrarySet`	Enum	The set of pre-installed Python libraries available to a Python shell job running Python 3.9.
`MaxCapacity`	Enum	The number of AWS Glue data processing units (DPUs) that can be allocated when this job runs.
`MetricType`	Enum	The Glue CloudWatch metric type.
`NumericOverflowHandlingAction`	Enum	Specifies the action to perform when ORC data contains an integer (for example, BIGINT or int64) that is larger than the column definition (for example, SMALLINT or int16).
`OnDemandTriggerOptions`	Struct	Properties for configuring an on-demand Glue Trigger.
`OrcColumnMappingType`	Enum	Specifies how to map columns when the table uses ORC data format.
`OutputFormat`	Class	Absolute class name of the Hadoop `OutputFormat` to use when writing table files.
`PartitionIndex`	Struct	Properties of a Partition Index.
`PartitionProjectionConfiguration`	Class	Factory class for creating partition projection configurations.
`PartitionProjectionType`	Enum	Partition projection type.
`Predicate`	Struct	Represents a trigger predicate.
`PredicateLogical`	Enum
`PySparkEtlJobProps`	Struct	Properties for creating a Python Spark ETL job.
`PySparkFlexEtlJobProps`	Struct	Properties for PySparkFlexEtlJob.
`PySparkStreamingJobProps`	Struct	Properties for creating a Python Spark ETL job.
`PythonShellJobProps`	Struct	Properties for creating a Python Shell job.
`PythonVersion`	Enum	Python version.
`RayJobProps`	Struct	Properties for creating a Ray Glue job.
`Runtime`	Enum	AWS Glue runtime determines the runtime engine of the job.
`S3Code`	Class	Glue job Code from an S3 bucket.
`S3Encryption`	Class	S3 encryption configuration for a `SecurityConfiguration`.
`S3EncryptionMode`	Enum	Encryption mode for S3.
`S3TableEncryption`	Class	Server-side encryption for the S3 bucket that a managed `S3Table` creates.
`S3TableProps`	Struct
`S3TableStorage`	Class	Where an `S3Table` stores its data.
`ScalaSparkEtlJobProps`	Struct	Properties for creating a Scala Spark ETL job.
`ScalaSparkFlexEtlJobProps`	Struct	Properties for a `ScalaSparkFlexEtlJob`.
`ScalaSparkStreamingJobProps`	Struct	Properties for creating a Scala Spark ETL job.
`ScheduledTriggerOptions`	Struct	Properties for configuring a scheduled Glue Trigger.
`Schema`	Class
`SecurityConfigurationProps`	Struct	Constructions properties of `SecurityConfiguration`.
`SerializationLibrary`	Class	Serialization library to use when serializing/deserializing (SerDe) table records.
`SparkExtraCodeProps`	Struct	Code props for different {@link Code} assets used by different types of Spark jobs.
`SparkJobProps`	Struct	Common properties for different types of Spark jobs.
`SparkUILoggingLocation`	Struct	The Spark UI logging location.
`SparkUIProps`	Struct	Properties for enabling Spark UI monitoring feature for Spark-based Glue jobs.
`StartableTriggerOptions`	Struct	Base options for triggers that can be started when they are created.
`StorageParameter`	Class	A storage parameter. The list of storage parameters available is not exhaustive and other keys may be used.
`StorageParameters`	Enum	The storage parameter keys that are currently known, this list is not exhaustive and other keys may be used.
`SurplusBytesHandlingAction`	Enum	Specifies how to handle data being loaded that exceeds the length of the data type defined for columns containing VARBYTE data.
`SurplusCharHandlingAction`	Enum	Specifies how to handle data being loaded that exceeds the length of the data type defined for columns containing VARCHAR, CHAR, or string data.
`TableAttributes`	Struct
`TableBaseProps`	Struct
`TableClientSideEncryption`	Class	Client-side encryption for an `S3Table`'s data.
`TriggerOptions`	Struct	Properties for configuring a Glue Trigger.
`TriggerSchedule`	Class	Represents a trigger schedule.
`Type`	Class	The type of a column in a table schema.
`WorkerConfiguration`	Struct	The worker configuration for a Spark job.
`WorkerType`	Enum	The type of predefined worker that is allocated when a job runs.
`WorkflowAttributes`	Struct	Properties for importing a Workflow using its attributes.
`WorkflowProps`	Struct	Properties for defining a Workflow.
`WriteParallel`	Enum	Specifies how to handle data being loaded that exceeds the length of the data type defined for columns containing VARCHAR, CHAR, or string data.

AWS Glue Construct Library ​

README ​

References ​

Create a Glue Job ​

Spark Jobs ​

ETL Jobs ​

Streaming Jobs ​

Flex Jobs ​

Python Shell Jobs ​

Ray Jobs ​

Metrics Control ​

Job Arguments ​

Enable Job Run Queuing ​

Uploading scripts from the CDK app repository to S3 ​

Workflow Triggers ​

1. On-Demand Triggers ​

2. Scheduled Triggers ​

3. Event Triggers ​

4. Conditional Triggers ​

Connection Properties ​

SecurityConfiguration ​

Catalog ​

The account-wide catalog ​

Creating a catalog ​

Encryption at rest ​

Connection password encryption ​

Importing a catalog ​

Database ​

Table ​

Partition Keys ​

Partition Indexes ​

Partition Filtering ​

Partition Projection ​

INTEGER Projection ​

DATE Projection ​

ENUM Projection ​

INJECTED Projection ​

Multiple Partition Projections ​

Glue Connections ​

Data Quality Ruleset ​

Encryption ​

Marking table data as encrypted ​

Types ​

Public FAQ ​

What are we launching today? ​

Why should I use this Construct? ​

What’s not in scope? ​

Types ​

AWS Glue Construct Library

README

References

Create a Glue Job

Spark Jobs

ETL Jobs

Streaming Jobs

Flex Jobs

Python Shell Jobs

Ray Jobs

Metrics Control

Job Arguments

Enable Job Run Queuing

Uploading scripts from the CDK app repository to S3

Workflow Triggers

1. On-Demand Triggers

2. Scheduled Triggers

3. Event Triggers

4. Conditional Triggers

Connection Properties

SecurityConfiguration

Catalog

The account-wide catalog

Creating a catalog

Encryption at rest

Connection password encryption

Importing a catalog

Database

Table

Partition Keys

Partition Indexes

Partition Filtering

Partition Projection

INTEGER Projection

DATE Projection

ENUM Projection

INJECTED Projection

Multiple Partition Projections

Glue Connections

Data Quality Ruleset

Encryption

Marking table data as encrypted

Types

Public FAQ

What are we launching today?

Why should I use this Construct?

What’s not in scope?

Types