Recovering DLT data after stack deletion

This section describes how to restore test scenarios and run history into a new DLT stack after the original stack has been deleted. It applies when the customer’s S3 buckets and DynamoDB tables were retained (the default behavior) and the customer wants to reconnect them to a fresh deployment.

Background

When a DLT stack is deleted, the following resources are retained in the customer’s AWS account due to data protection policies:

All resource names are auto-generated by CloudFormation. They are not predictable and cannot be specified as parameters when deploying a new stack.

Important constraints

Recovery process

Step 1: Identify retained resources

Before deploying a new stack, locate the retained resources from the deleted stack.

# List retained DynamoDB tables (look for DLT naming patterns)
aws dynamodb list-tables --query "TableNames[?contains(@, 'ScenariosTable') || contains(@, 'HistoryTable')]"

# List retained S3 buckets (look for DLT naming patterns)
aws s3api list-buckets --query "Buckets[?contains(Name, 'dlttestrunnerstoragedlts') || contains(Name, 'dltconsoleresourcesdltda')]"

If you noted the CloudFormation stack outputs before deletion, use those exact names. The relevant outputs were:

For resources not exposed as stack outputs (such as the history table), use list-stack-resources before deletion to get the physical resource IDs. This command lists every resource in the stack regardless of whether it has a corresponding output.

Step 2: Deploy a new DLT stack

Deploy a fresh DLT stack using the same CloudFormation template version (or newer). Use the same parameters as the original deployment (VPC settings, admin email, etc.).

aws cloudformation create-stack \
  --stack-name distributed-load-testing-new \
  --template-url https://s3.amazonaws.com/solutions-reference/distributed-load-testing-on-aws/latest/distributed-load-testing-on-aws.template \
  --parameters ParameterKey=AdminName,ParameterValue=<admin-name> \
               ParameterKey=AdminEmail,ParameterValue=<admin-email> \
  --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM CAPABILITY_AUTO_EXPAND

Wait for the stack to reach CREATE_COMPLETE status.

Step 3: Identify the new stack’s resources

Retrieve the resource names from the new stack’s outputs.

aws cloudformation describe-stacks \
  --stack-name distributed-load-testing-new \
  --query "Stacks[0].Outputs"

Note the values for:

For any resource not exposed as a stack output (such as the history table), use list-stack-resources to find it by its logical ID:

aws cloudformation list-stack-resources \
  --stack-name distributed-load-testing-new \
  --query "StackResourceSummaries[?contains(LogicalResourceId, 'HistoryTable')].{LogicalId:LogicalResourceId,PhysicalId:PhysicalResourceId}"

This command lists every resource the stack manages. You can filter by logical ID substring to find any resource.

Step 4: Migrate DynamoDB data

Export data from the retained tables and import it into the new tables. Use DynamoDB data export or scan-and-write operations.

Option A: Using DynamoDB Export/Import (recommended for large datasets)

For tables with thousands of items, use a DynamoDB export to S3 followed by an import:

# Export from retained table to S3
aws dynamodb export-table-to-point-in-time \
  --table-arn arn:aws:dynamodb:<region>:<account>:table/<old-scenarios-table> \
  --s3-bucket <temporary-export-bucket> \
  --s3-prefix dlt-migration/scenarios/ \
  --export-format DYNAMODB_JSON

# Wait for export to complete, then import into new table
aws dynamodb import-table \
  --s3-bucket-source S3Bucket=<temporary-export-bucket>,S3KeyPrefix=dlt-migration/scenarios/ \
  --table-creation-parameters '{"TableName":"<new-scenarios-table>","KeySchema":[{"AttributeName":"testId","KeyType":"HASH"}],"AttributeDefinitions":[{"AttributeName":"testId","AttributeType":"S"}],"BillingMode":"PAY_PER_REQUEST"}' \
  --input-format DYNAMODB_JSON

Option B: Scan and BatchWrite (recommended)

This approach reads directly from the retained tables and writes into the new stack’s tables without causing stack drift. It requires Python 3 and the boto3 library.

# Install boto3 if not already available
pip install boto3

#!/usr/bin/env python3
"""Migrate DynamoDB items directly from retained tables to new stack tables."""

import boto3

dynamodb = boto3.resource("dynamodb", region_name="<region>")

def migrate_table(source_table_name, target_table_name):
    source = dynamodb.Table(source_table_name)
    target = dynamodb.Table(target_table_name)

    # Scan all items from the retained table (handles pagination automatically)
    items = []
    response = source.scan()
    items.extend(response["Items"])
    while "LastEvaluatedKey" in response:
        response = source.scan(ExclusiveStartKey=response["LastEvaluatedKey"])
        items.extend(response["Items"])

    print(f"Scanned {len(items)} items from {source_table_name}")

    # Write items to the new table using batch_writer (handles batching automatically)
    with target.batch_writer() as batch:
        for item in items:
            batch.put_item(Item=item)

    print(f"Wrote {len(items)} items to {target_table_name}")

# Migrate scenarios
migrate_table("<old-scenarios-table>", "<new-scenarios-table>")

# Migrate history
migrate_table("<old-history-table>", "<new-history-table>")

For large tables (tens of thousands of items), consider using parallel scan segments to speed up the read phase.

Option C: Restore from PITR (if within 35-day window)

If the stack was deleted within the last 35 days and PITR was enabled (it is by default), you can restore the table to a point in time. However, PITR restores create a new table with a different name, so you would still need to copy data into the stack-managed table using Option B.

# Restore to a new temporary table
aws dynamodb restore-table-to-point-in-time \
  --source-table-name <old-scenarios-table> \
  --target-table-name dlt-scenarios-restored \
  --restore-date-time <timestamp-before-deletion>

# Then scan from dlt-scenarios-restored and batch-write into the new stack's table

Step 5: Migrate S3 data

Copy test scripts and result files from the retained bucket to the new stack’s bucket.

# Copy all objects from the old scenarios bucket to the new one
aws s3 sync \
  s3://<old-scenarios-bucket> \
  s3://<new-scenarios-bucket> \
  --source-region <region> \
  --region <region>

This copies:

Step 6: Verify the migration

Step 7: Clean up retained resources

After verifying that all data has been migrated successfully, delete the old retained resources to avoid ongoing storage costs.

# Delete old DynamoDB tables
aws dynamodb delete-table --table-name <old-scenarios-table>
aws dynamodb delete-table --table-name <old-history-table>

# Empty and delete old S3 buckets
aws s3 rm s3://<old-scenarios-bucket> --recursive
aws s3api delete-bucket --bucket <old-scenarios-bucket>

aws s3 rm s3://<old-logs-bucket> --recursive
aws s3api delete-bucket --bucket <old-logs-bucket>

aws s3 rm s3://<old-console-bucket> --recursive
aws s3api delete-bucket --bucket <old-console-bucket>

Only perform this step after confirming the new stack is fully operational with all historical data intact.

Avoiding stack drift

Stack drift occurs when the actual state of a resource differs from what CloudFormation expects. To avoid drift during this recovery process:

Writing data into CloudFormation-managed tables and buckets is safe because CloudFormation tracks the resource configuration (table schema, billing mode, encryption settings), not the data contents.

What cannot be recovered

The following are lost when a stack is deleted and cannot be restored through this process:

Preventive measures

To simplify future recovery scenarios: