Writing data quality results to Data Catalog tables

You can configure AWS Glue Data Quality evaluation runs to automatically write results to Apache Iceberg tables in the AWS Glue Data Catalog. After you enable results output, you can query your data quality results directly using , build dashboards with visualization tools, and maintain a centralized history of data quality outcomes across your account.

You can write the following types of data quality results to Data Catalog tables:

Prerequisites

To write data quality results to Data Catalog tables, the IAM role you use for the evaluation run must have the following permissions:

The evaluation run uses the IAM role you specify to write to the results tables. This is the same role that has access to the source data table.

Configuring results output

You configure data quality results output using the --additional-run-options parameter of the StartDataQualityRulesetEvaluationRun API or the additional_options parameter in AWS Glue ETL jobs. By default, AWS Glue Data Quality does not write results to Data Catalog tables. You must explicitly enable each result type that you want to write.

Each result type has its own configuration block with a shared CatalogTableConfig structure. If you do not provide a CatalogTableConfig, AWS Glue Data Quality derives default values automatically, including the table name and Amazon S3 path.

The CatalogTableConfig structure contains the following fields:

Example: Configure rule results and profiling results

aws glue start-data-quality-ruleset-evaluation-run \
  --data-source '{
    "GlueTable": {
      "DatabaseName": "my_database",
      "TableName": "my_table"
    }
  }' \
  --role "arn:aws:iam::123456789012:role/GlueServiceRole" \
  --ruleset-names '["my_ruleset"]' \
  --additional-run-options '{
    "DataQualityRuleResults": {
      "WriteDataQualityRuleResultsEnabled": true,
      "CatalogTableConfig": {
        "DatabaseName": "quality_results",
        "TableName": "rule_results"
      }
    },
    "ProfilingResults": {
      "WriteProfilingResultsEnabled": true,
      "CatalogTableConfig": {
        "DatabaseName": "quality_results",
        "TableName": "profiles"
      }
    }
  }'

Example: Configure row-level results

For row-level results, you can also specify the type of records to include and a maximum number of rows to write.

aws glue start-data-quality-ruleset-evaluation-run \
  --data-source '{
    "GlueTable": {
      "DatabaseName": "my_database",
      "TableName": "my_table"
    }
  }' \
  --role "arn:aws:iam::123456789012:role/GlueServiceRole" \
  --ruleset-names '["my_ruleset"]' \
  --additional-run-options '{
    "RowLevelResults": {
      "MaxRowsToWrite": 5000,
      "ResultType": "FAILED_ONLY",
      "CatalogTableConfig": {
        "DatabaseName": "quality_results",
        "TableName": "row_level_results"
      }
    }
  }'

The ResultType parameter accepts the following values:

Example – Configure in AWS Glue ETL jobs

In AWS Glue ETL jobs, you configure results output using the additional_options parameter with dot-notation keys:

result = EvaluateDataQuality.process_rows(
    frame=dynamic_frame,
    ruleset=ruleset,
    publishing_options={
        "dataQualityEvaluationContext": "my_context",
        "enableDataQualityResultsPublishing": True
    },
    additional_options={
        "observations.scope": "ALL",
        "dataQualityResultsPublishing.strategy": "BEST_EFFORT",
        "dataQualityResultsPublishing.resultsFormat.profilingResults.writeProfilingResultsEnabled": "true",
        "dataQualityResultsPublishing.resultsFormat.profilingResults.catalogTableConfig.databaseName": "my_db",
        "dataQualityResultsPublishing.resultsFormat.profilingResults.catalogTableConfig.tableName": "profiling_results",
        "dataQualityResultsPublishing.resultsFormat.profilingResults.catalogTableConfig.s3Location": "s3://amzn-s3-demo-bucket/profiling/",
        "dataQualityResultsPublishing.resultsFormat.profilingResults.catalogTableConfig.catalogId": "123456789012"
    }
)

Example – Configure observation results

You can configure observation results the same way as other result types. Observation results require anomaly detection to be enabled (ObservationScope: ALL):

aws glue start-data-quality-ruleset-evaluation-run \
  --data-source '{
    "GlueTable": {
      "DatabaseName": "my_database",
      "TableName": "my_table"
    }
  }' \
  --role "arn:aws:iam::123456789012:role/GlueServiceRole" \
  --ruleset-names '["my_ruleset"]' \
  --additional-run-options '{
    "ObservationScope": "ALL",
    "ObservationResults": {
      "WriteObservationResultsEnabled": true,
      "CatalogTableConfig": {
        "DatabaseName": "quality_results",
        "TableName": "observation_results"
      }
    }
  }'

Table schemas

AWS Glue Data Quality writes each result type to a separate Iceberg table. The rule results, profiling results (including the separate distribution results table), and observation results tables are partitioned by catalog_id, database_name, table_name, and day(stored_on) to enable efficient querying. You can filter on stored_on directly for time-based queries and Iceberg handles partition pruning automatically.

Rule results table

The rule results table stores the pass or fail outcome for each rule evaluated during a data quality run.

Profiling results table

The following table describes the columns in the profiling results table. This table stores scalar statistics gathered by analyzers and rules (such as Mean, StandardDeviation, and Completeness). AWS Glue Data Quality stores distribution statistics in a separate distribution results table.

Distribution results table

The following table describes the columns in the distribution results table. Distribution results are stored separately from scalar profiling statistics, with one row per bin or category. You can configure this table within the ProfilingResults.DistributionResults block.

Row-level results table

The following table describes the columns in the row-level results table. You can use this table to identify the specific records that failed your data quality rules.

Observation results table

The observation results table stores anomaly detection predictions for each statistic on every evaluation run. The table includes all prediction outcomes: anomalies, normal values, and skipped predictions. This lets you render continuous trend charts with prediction bands.

Querying results with

After your data quality evaluation completes, you can query the results tables directly using . The following examples demonstrate common query patterns.

Example: Find failed rules for a specific run

SELECT rule_name, rule_description, evaluation_message, evaluated_metrics
FROM quality_results.rule_results
WHERE ruleset_evaluation_run_id = 'dqr-12345678'
  AND rule_result = 'FAIL'
ORDER BY rule_name;

Example: View profiling statistics over time

SELECT stored_on, statistics_value
FROM quality_results.profiles
WHERE database_name = 'my_database'
  AND table_name = 'my_table'
  AND statistic_name = 'Mean'
  AND columns_referenced = ARRAY['salary']
ORDER BY stored_on;

Example – Identify rows that failed a specific rule

SELECT *
FROM quality_results.row_level_results
WHERE data_quality_evaluation_result = 'Failed'
  AND contains(data_quality_rules_fail, 'IsComplete "email"');

Example – View a numeric histogram

SELECT bin_index, bin_lower_bound, bin_upper_bound, bin_count
FROM quality_results.distributions
WHERE column_name = 'salary'
  AND ruleset_evaluation_run_id = 'dqrun-abc123'
ORDER BY bin_index;

Example – View a categorical value distribution

SELECT bin_label, bin_count
FROM quality_results.distributions
WHERE column_name = 'department'
  AND ruleset_evaluation_run_id = 'dqrun-abc123'
ORDER BY bin_count DESC;

Example – Track category frequency over time

SELECT started_on, bin_count
FROM quality_results.distributions
WHERE column_name = 'status' AND bin_label = 'active'
ORDER BY started_on;

Example: View anomaly detection trends with prediction bands

SELECT o.recorded_on, 
       p.statistics_value AS actual_value,
       o.expected_value, 
       o.lower_bound, 
       o.upper_bound,
       o.prediction_outcome
FROM quality_results.profiles p
JOIN quality_results.observation_results o
  ON p.statistic_id = o.statistic_id
  AND p.ruleset_evaluation_run_id = o.ruleset_evaluation_run_id
WHERE p.database_name = 'my_database'
  AND p.table_name = 'my_table'
  AND p.statistic_name = 'RowCount'
  AND p.stored_on >= DATE '2025-03-01'
ORDER BY p.stored_on;

Example – Query latest observation state after annotations

Because the observation results table uses an append-only model, exclusion annotations add new rows. Use a deduplication query to get the latest state for each observation:

SELECT statistic_id, statistic_name, prediction_outcome, 
       expected_value, lower_bound, upper_bound, training_input, stored_on
FROM (
  SELECT *, ROW_NUMBER() OVER (
    PARTITION BY statistic_id, ruleset_evaluation_run_id 
    ORDER BY stored_on DESC
  ) AS rn
  FROM quality_results.observation_results
  WHERE database_name = 'my_database'
    AND table_name = 'my_table'
)
WHERE rn = 1
ORDER BY stored_on;

Considerations

Keep the following considerations in mind when writing data quality results to Data Catalog tables: