Amazon Redshift will no longer support the use of Python UDFs after June 30, 2026. We will start enforcing it in phases. For more information on the details of Python end of life and migration options, see the blog post that was published on June 30, 2025.

System table integration with S3 Tables

Use system table integration with Amazon S3 Tables to retain your Amazon Redshift system table data beyond the 7-day in-cluster retention period. This helps you meet compliance, auditing, and observability requirements. When you enable this feature, Redshift writes system table logs to S3 Tables in Apache Iceberg format and manages partitioning, compaction, and snapshot maintenance. You can keep this data for as long as you need for analysis and auditing. Because the data is stored in open Iceberg format, you can query it with Redshift, Amazon Athena, or any other Iceberg-compatible engine. You can also use AI agent skills for natural-language queries (see System table skills for AWS Agent Toolkit on the GitHub website).

When you enable this feature through the console, the Redshift CLI, or SDK, Redshift begins writing new rows from the system tables you select to S3 Tables shortly thereafter at a fixed frequency. You choose which system tables to publish; S3 Tables enforces the retention duration that you configure.

This supports the following use cases:

This feature is available for Amazon Redshift Provisioned RA3 and RG instances and Amazon Redshift Serverless in all AWS commercial Regions where Redshift and S3 Tables are supported. In this topic, data warehouse refers to either a provisioned cluster or a Redshift Serverless workgroup.

How it works

After you enable the feature and select one or more supported system tables, Redshift writes newly completed records from those system tables to S3 Tables. Data is delivered in batches at a fixed frequency.

The system tables live in your AWS account but are service-managed: AWS owns and operates the data delivery and table management, including the following:

There is no infrastructure to build or maintain and no impact to your workloads.

You configure how long to retain the data through S3 Tables record expiration. For more information, see Configuring retention.

Because the tables are managed by AWS, you cannot modify or delete the rows that Redshift delivers. The delivered data is immutable. You only control read access to the tables. For more information, see Data governance.

Only completed activity is delivered to S3 Tables. The system tables in your data warehouse can show in-flight activity, but only records that have reached a final state are copied. For example, a completed, aborted, or canceled query is delivered, but a query that is still running is not delivered until it reaches a final state.

Enabling system table integration with S3 Tables

To enable system table integration with S3 Tables, use the existing Redshift APIs with a new log destination type of s3table. When enabled, Redshift creates an S3 table bucket named aws-redshift in your account. You manage the feature with the same operations you use for other log destinations: for provisioned clusters, enable-logging, disable-logging, and describe-logging-status; for Redshift Serverless, update-namespace and get-namespace.

Permissions

The principal that enables, modifies, or disables the feature must have the following permissions:

After the bucket is created, Redshift creates namespaces and tables within it using a service trust relationship between Redshift and S3 Tables. The enabling principal does not need permissions to create namespaces or tables.

Deployment options

You can configure delivery in one of two patterns. A given data warehouse uses one pattern at a time.

The deployment model determines how data is organized in S3 Tables. In the per-warehouse model, the S3 Tables namespace name contains a unique identifier for the data warehouse, so each warehouse's data is physically separated into its own namespace. In the consolidated model, the S3 Tables namespace name contains the AWS account number, and data from all warehouses in that account and Region is stored together in the same namespace and tables.

Both options are supported within a single AWS account and a single AWS Region. To analyze data across Regions or accounts, combine results from each Region's or account's tables at query time. For cross-account access, you can use AWS Glue Data Catalog sharing. For more information, see Granting cross-account access in the AWS Glue Developer Guide.

Using the console

To stop publishing a system table, return to the settings and remove it. Data already published is retained until you configure expiration through S3 Tables.

You can review the status of the integration from the cluster or namespace detail page. Expand View S3 table mapping to see the mapping between each system table and its corresponding S3 table name and namespace. You can also view the published data from the S3 Tables console.

Using the AWS CLI

The following parameters configure S3 Tables publishing:

Provisioned clusters

Enable delivery. --log-exports accepts all or a space-separated list of supported system tables, and --s3-table-granularity accepts cluster (per-warehouse) or account (consolidated).

aws redshift enable-logging \
    --cluster-identifier my-redshift-cluster \
    --log-destination-type s3table \
    --log-exports all \
    --s3-table-granularity account

To enable delivery for specific system tables only:

aws redshift enable-logging \
    --cluster-identifier my-redshift-cluster \
    --log-destination-type s3table \
    --log-exports sys_query_history sys_query_text sys_userlog \
    --s3-table-granularity cluster

Check delivery status.

aws redshift describe-logging-status \
    --cluster-identifier my-redshift-cluster

Disable delivery for all system tables, or for specific tables with --log-exports:

aws redshift disable-logging \
    --cluster-identifier my-redshift-cluster \
    --log-destination-type s3table \
    --log-exports sys_stream_scan_states

Redshift Serverless

In Redshift Serverless, you configure the feature per namespace with update-namespace. System table names are passed on --s3-table-names (not --log-exports).

Enable delivery.

aws redshift-serverless update-namespace \
    --namespace-name my-namespace \
    --log-destination-type s3table \
    --s3-table-action Enable \
    --s3-table-names all \
    --s3-table-granularity namespace

Disable delivery for specific system tables (or all):

aws redshift-serverless update-namespace \
    --namespace-name my-namespace \
    --log-destination-type s3table \
    --s3-table-action Disable \
    --s3-table-names sys_stream_scan_states

Check delivery status.

aws redshift-serverless get-namespace \
    --namespace-name my-namespace

To resume delivery after the S3 table bucket or a table was dropped, run the enable command again.

Registering with AWS Glue Data Catalog

Before you can query the retained data with Redshift, Athena, or other analytics services, the service-managed S3 table bucket must be integrated with AWS Glue Data Catalog. This is a one-time step per account and Region. If you have already integrated S3 Tables with AWS Glue Data Catalog, no additional action is required.

For integration instructions, see Integrating Amazon S3 Tables with AWS Glue Data Catalog in the AWS Glue Developer Guide.

Configuring encryption

By default, data delivered to S3 Tables is encrypted with Amazon S3-managed keys (SSE-S3). To use an AWS KMS customer managed key instead, specify its ARN with --s3-table-kms-key-id when you enable delivery:

aws redshift enable-logging \
    --cluster-identifier my-redshift-cluster \
    --log-destination-type s3table \
    --log-exports all \
    --s3-table-granularity account \
    --s3-table-kms-key-id arn:aws:kms:us-west-2:111122223333:key/key-id

If you use a customer managed key, its key policy must allow the Redshift system table integration service principal to generate data keys when it writes your system tables, and the S3 Tables maintenance service principal to use the key during table maintenance (such as compaction). Add the following statements to the key policy, replacing REGION, ACCOUNT, and KEY_ID with your own values.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "EnableRedshiftSystemTableKeyUsage",
      "Effect": "Allow",
      "Principal": {
        "Service": "systemtables.redshift.amazonaws.com"
      },
      "Action": [
        "kms:DescribeKey",
        "kms:GenerateDataKey",
        "kms:Decrypt"
      ],
      "Resource": "arn:aws:kms:REGION:ACCOUNT:key/KEY_ID",
      "Condition": {
        "StringEquals": {
          "aws:SourceAccount": "ACCOUNT"
        }
      }
    },
    {
      "Sid": "EnableS3TableMaintenanceKeyUsage",
      "Effect": "Allow",
      "Principal": {
        "Service": "maintenance.s3tables.amazonaws.com"
      },
      "Action": [
        "kms:GenerateDataKey",
        "kms:Decrypt"
      ],
      "Resource": "arn:aws:kms:REGION:ACCOUNT:key/KEY_ID",
      "Condition": {
        "StringLike": {
          "kms:EncryptionContext:aws:s3:arn": "arn:aws:s3tables:REGION:ACCOUNT:bucket/aws-redshift/*"
        }
      }
    }
  ]
}

If the key policy does not grant these permissions, Redshift cannot write to the encrypted S3 Tables and delivery fails.

Supported system tables

You can select any of the following SYS_* monitoring views to integrate with S3 Tables. For a description of the columns in each view, choose the view name.

Metadata columns added to each table

Each S3 table contains all of the columns from the source SYS_* view, plus the following five metadata columns that Redshift adds to identify the source of each row and when it was delivered.

Configuring retention

Redshift does not define a retention policy for the data it delivers to S3 Tables. You configure how long to keep the data by setting a record expiration policy through S3 Tables. You specify the number of days (for example, 5 days, 100 days, or 365 days), and S3 Tables automatically removes data that ages past the configured duration.

If you do not configure an expiration policy, data is kept indefinitely.

When you disable the integration or remove a system table, Redshift stops writing new data but previously delivered data remains in S3 Tables and continues to be subject to whatever expiration policy you have configured.

For more information, see Configuring record expiration for S3 table buckets in the Amazon S3 User Guide.

Data governance

You govern access to the retained system table data; Redshift does not apply read permissions on your behalf and does not mask data.

Delivery and lifecycle behavior

Checking delivery status

You can review the current configuration and the most recent delivery time per system table at any time:

Redshift Serverless

Delivery does not keep a workgroup awake or consume your compute. Before the workgroup pauses, Redshift ensures any pending batch is delivered.

Best practices

Billing

Writing system table data to S3 Tables is free. You are charged for standard S3 Tables storage and maintenance for the retained data, and for the query engine you use to read the data, according to that engine's pricing. For more information, see Amazon S3 Tables pricing.

Considerations and limitations