Overview How catalog metadata export works Enable catalog metadata export using the AWS CLI Catalog system tables reference Querying catalog system tables

Export metadata to S3 Tables (preview)

Note

Catalog metadata export is available as a preview feature. This feature is subject to change before general availability.

Overview

You can export your AWS Glue Data Catalog metadata into managed catalog system tables stored in Amazon S3 Tables. When you enable the S3 Tables export, your technical and business metadata is stored in read-only Apache Iceberg tables in the AWS managed aws-catalog table bucket, making them queryable using SQL with AWS analytics services like , Amazon Redshift, and Amazon EMR.

Within the aws-catalog table bucket, system tables reside in the following namespaces:

glue — contains tables and table_columns.
catalog_metadata — contains glossaries, glossary_terms, associated_glossary_terms, attachments, and item_attachments.

The following system tables are currently supported:

System table	Contains
`tables`	Technical and business metadata for each table in the Data Catalog, such as its name, database, description, storage location, and format.
`table_columns`	The columns of each catalog table, including each column's name, data type, description, and whether it is a partition key.
`glossaries`	The business glossaries in your catalog, including each glossary's name, description, and status.
`glossary_terms`	The terms defined in your glossaries, including each term's name, descriptions, parent glossary, and status.
`associated_glossary_terms`	The associations between glossary terms and assets, including the asset identifier and glossary term identifier.
`attachments`	Custom-form metadata attached at the asset level, including the form type and the attachment content.
`item_attachments`	Custom-form metadata attached to items within an asset, such as individual columns, including the iterable form name, item name, form type, and attachment content.

For the full column-level schema of each table, see Catalog system tables reference.

How catalog metadata export works

After you enable export, the Data Catalog performs a one-time backfill of your existing catalog metadata and then keeps the system tables up to date as you make changes. Newly written or updated metadata typically becomes visible in the system tables within about 5 minutes.

The system tables reside in the glue and catalog_metadata namespaces within the aws-catalog table bucket. A given system table appears the first time metadata of that kind exists in your catalog. For example, if you enable export and your catalog contains only tables and databases, you see only the table-related system tables. The glossary system tables appear after you create your first glossary.

Enable catalog metadata export using the AWS CLI

Catalog metadata export is an account-level setting. You enable or disable it with the AWS CLI.

Prerequisites

The IAM identity you use must have permission to call the export configuration operations glue:PutDataCatalogExportConfiguration and glue:GetDataCatalogExportConfiguration, as well as s3tables:CreateTable. For more information about S3 Tables permissions, see Creating tables in S3 table buckets in the Amazon Simple Storage Service User Guide.


{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "glue:PutDataCatalogExportConfiguration",
                "glue:GetDataCatalogExportConfiguration"
            ],
            "Resource": "*"
        }
    ]
}

All data in S3 Tables, including Data Catalog metadata, is encrypted with SSE-S3 encryption by default. You can choose to encrypt your catalog export with AWS KMS keys (SSE-KMS). If you choose to encrypt with AWS KMS keys, you must add additional permissions by taking the following steps:

Grant the Data Catalog export service principal and the S3 Tables maintenance service principal permissions to use your KMS key (key policy).
Grant the IAM principal that runs the export permissions to use the KMS key (identity policy).

Granting the Data Catalog export service principal and S3 Tables maintenance service principal permissions to your KMS key

To allow AWS Glue to export encrypted catalog metadata and to allow automatic table maintenance like compaction and unreferenced file removal on the exported tables, you must grant the following service principals access to your KMS key:

systemtables.catalog.amazonaws.com — exports encrypted catalog metadata to S3 Tables.
maintenance.s3tables.amazonaws.com — performs automatic table maintenance on the exported tables.

AWS Glue writes to the AWS managed table bucket named aws-catalog. The KMS key used for encrypting the S3 Tables export does not need to be the same as the one used to encrypt the Data Catalog at rest. AWS Glue supports only symmetric KMS keys. To grant these service principals access, you can use the following example key policy. In this policy, the maintenance.s3tables.amazonaws.com service principal is granted permission to use a specific KMS key for encrypting and decrypting tables in the aws-catalog table bucket. For more information about the S3 Tables maintenance service principal, see Permission requirements for S3 Tables SSE-KMS encryption in the Amazon Simple Storage Service User Guide.


{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowSystemTablesDescribeKey",
            "Effect": "Allow",
            "Principal": {
                "Service": "systemtables.catalog.amazonaws.com"
            },
            "Action": [
                "kms:DescribeKey"
            ],
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "aws:SourceAccount": "account-id"
                }
            }
        },
        {
            "Sid": "AllowSystemTablesEncryptDecrypt",
            "Effect": "Allow",
            "Principal": {
                "Service": "systemtables.catalog.amazonaws.com"
            },
            "Action": [
                "kms:GenerateDataKey",
                "kms:Decrypt"
            ],
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "aws:SourceAccount": "account-id"
                },
                "StringLike": {
                    "kms:EncryptionContext:aws:s3:arn": "arn:aws:s3tables:region:account-id:bucket/aws-catalog/table/*",
                    "kms:ViaService": "s3.region.amazonaws.com"
                }
            }
        },
        {
            "Sid": "AllowS3TablesMaintenanceEncryptDecrypt",
            "Effect": "Allow",
            "Principal": {
                "Service": "maintenance.s3tables.amazonaws.com"
            },
            "Action": [
                "kms:GenerateDataKey",
                "kms:Decrypt"
            ],
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "aws:SourceAccount": "account-id"
                },
                "StringLike": {
                    "kms:EncryptionContext:aws:s3:arn": "arn:aws:s3tables:region:account-id:bucket/aws-catalog/*"
                }
            }
        }
    ]
}

IAM permissions required for the principal for exporting

Grant the principals that run exports the following permissions. The following policy grants the IAM principal access to decrypt a specific AWS Glue Data Catalog, scoped with the glue_catalog_id encryption context (the value is your catalogId).


{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowDescribeKey",
            "Effect": "Allow",
            "Action": [
                "kms:DescribeKey"
            ],
            "Resource": "arn:aws:kms:region:account-id:key/key-id",
            "Condition": {
                "StringEquals": {
                    "aws:SourceAccount": "account-id"
                }
            }
        },
        {
            "Sid": "AllowExportOfGlueDataCatalogByCatalogId",
            "Effect": "Allow",
            "Action": [
                "kms:Decrypt",
                "kms:GenerateDataKey"
            ],
            "Resource": "arn:aws:kms:region:account-id:key/key-id",
            "Condition": {
                "StringEquals": {
                    "aws:SourceAccount": "account-id",
                    "kms:EncryptionContext:glue_catalog_id": "account-id"
                }
            }
        }
    ]
}

To enable catalog metadata export

Run the following command to enable export:


aws glue put-data-catalog-export-configuration \
    --export-setting ENABLED

Confirm the configuration by running the following command:


aws glue get-data-catalog-export-configuration

Example response:


{
    "ExportSetting": "ENABLED",
    "Status": "ENABLED",
    "S3TableBucketArn": "arn:aws:s3tables:us-east-1:111122223333:bucket/aws-catalog"
}

The Status field transitions from ENABLING to ENABLED as the initial backfill progresses.

To disable catalog metadata export

Run the following command:


aws glue put-data-catalog-export-configuration \
    --export-setting DISABLED

Catalog system tables reference

All system tables reside in the glue or catalog_metadata namespaces within the aws-catalog table bucket. Every table includes the following system columns:

Column	Type	Description
`ingestion_time`	timestamp	Time the record was ingested into the system table (UTC).
`schema_version`	string	Schema version identifier for the table.

tables

Contains one row per catalog table.

Column	Type	Description
`id`	string	Unique identifier for the table.
`catalog_id`	string	Catalog identifier.
`catalog_name`	string	Catalog name.
`update_time`	timestamp	Last update timestamp.
`created_time`	timestamp	Creation timestamp.
`created_by_id`	string	Identifier of the principal that created the table.
`created_by_type`	string	Type of the creating principal.
`description`	string	Table description.
`name`	string	Table name.
`namespace`	string	Namespace of the table.
`region_name`	string	AWS Region.
`tags`	map<string, string>	Resource tags.
`type`	string	Asset type.
`updated_by_id`	string	Identifier of the principal that last updated the table.
`updated_by_type`	string	Type of the updating principal.
`database_name`	string	Database containing the table.
`input_format`	string	Input format class.
`lake_formation_registration`	string	AWS Lake Formation registration status.
`output_format`	string	Output format class.
`owner`	string	Table owner.
`retention`	int	Retention period.
`serde_library`	string	Serialization/deserialization library.
`table_data_format`	string	Data format.
`table_data_location`	string	Data location URI.
`table_format`	string	Table format (for example, Iceberg, Hudi).
`table_type`	string	Table type.

table_columns

Contains one row per column in a catalog table.

Column	Type	Description
`id`	string	Composite identifier in the format `assetId#columnName`.
`asset_id`	string	Identifier of the parent table.
`column_name`	string	Column name.
`description`	string	Column description.
`type`	string	Column data type.
`is_partition_key`	boolean	Whether the column is a partition key.

attachments

Contains asset-level custom-form attachments (one row per attachment on an asset).

Column	Type	Description
`asset_id`	string	Identifier of the parent asset.
`attachment_name`	string	Attachment name.
`form_type_id`	string	Registered form type of the content.
`content_json`	string	Attachment content as a JSON-encoded string.

item_attachments

Contains attachments on items within an iterable form, such as individual columns.

Column	Type	Description
`asset_id`	string	Identifier of the parent asset.
`iterable_form_name`	string	Name of the iterable form (for example, columns).
`item_name`	string	Name of the item (for example, the column name).
`attachment_name`	string	Attachment name.
`form_type_id`	string	Registered form type of the content.
`content_json`	string	Attachment content as a JSON-encoded string.

glossaries

Contains one row per glossary.

Column	Type	Description
`id`	string	Unique glossary identifier.
`description`	string	Glossary description.
`name`	string	Glossary name.
`status`	string	Status (`ENABLED` or `DISABLED`).

glossary_terms

Contains one row per glossary term.

Column	Type	Description
`id`	string	Unique term identifier.
`glossary_id`	string	Identifier of the parent glossary.
`long_description`	string	Detailed term description.
`name`	string	Term name.
`short_description`	string	Brief term description.
`status`	string	Status (`ENABLED` or `DISABLED`).

associated_glossary_terms

Contains one row per association between a glossary term and an asset.

Column	Type	Description
`asset_id`	string	Identifier of the asset the glossary term is associated with.
`glossary_term_id`	string	Identifier of the glossary term associated with the asset. Joins to the `glossary_terms` table.

Querying catalog system tables

Before you can query catalog metadata exported to S3 Tables using AWS analytics services like or Amazon EMR, you must enable analytics integration on the AWS managed aws-catalog table bucket and configure AWS Lake Formation permissions.

Prerequisites

Catalog metadata export is enabled and the status is ENABLED.
Access to or other analytics services.
Waited 5 minutes after enabling export for data to be available.

Integration overview

For detailed information about integrating S3 Tables with AWS analytics services, including prerequisites, IAM role configuration, and step-by-step procedures, see Integrating Amazon S3 Tables with AWS analytics services in the Amazon Simple Storage Service User Guide.

After you enable S3 Tables export and set up analytics integration, you can query your Data Catalog metadata using AWS analytics services such as , Amazon Redshift, and Amazon EMR. This enables you to perform metadata audits and analysis using standard SQL.

Example queries

List all catalog tables with their database and format


SELECT id, name, database_name, table_format
FROM aws_catalog.glue.tables;

Find columns with descriptions for a specific table


SELECT column_name, type, description
FROM aws_catalog.glue.table_columns
WHERE asset_id = 'table-id';

Join glossary terms to their parent glossary


SELECT g.name AS glossary_name,
       t.name AS term_name,
       t.short_description
FROM aws_catalog.catalog_metadata.glossary_terms t
JOIN aws_catalog.catalog_metadata.glossaries g
    ON t.glossary_id = g.id
WHERE g.status = 'ENABLED';

Find tables associated with a specific glossary term


SELECT t.name, t.database_name, gt.name AS term_name
FROM aws_catalog.glue.tables t
JOIN aws_catalog.catalog_metadata.associated_glossary_terms agt
    ON agt.asset_id = t.id
JOIN aws_catalog.catalog_metadata.glossary_terms gt
    ON gt.id = agt.glossary_term_id
WHERE gt.name = 'term-name';

Extract a field from a custom-form attachment


SELECT asset_id,
       attachment_name,
       json_extract_scalar(content_json, '$.owner') AS data_owner
FROM aws_catalog.catalog_metadata.attachments
WHERE form_type_id = 'your-form-type-id';

Warning Javascript is disabled or is unavailable in your browser.

To use the Amazon Web Services Documentation, Javascript must be enabled. Please refer to your browser's Help pages for instructions.

Document Conventions

Working with AWS Glue Data Catalog views

Business context