Amazon DocumentDB migration runbook

This runbook provides a comprehensive guide for migrating a MongoDB database to Amazon DocumentDB using AWS Database Migration Service (DMS). It is designed to support database administrators, cloud engineers, and developers throughout the end-to-end migration journey—from initial discovery to post-migration validation.

Given the differences in implementation and supported features between MongoDB and Amazon DocumentDB, this runbook emphasizes a structured and systematic approach. It outlines essential pre-migration assessments, highlights compatibility considerations, and details the key tasks required to ensure a successful migration with minimal disruption.

The runbook is organized into the following topics:

By following the guidance in this runbook, teams can ensure a smooth, secure, and efficient transition to Amazon DocumentDB, while preserving application functionality and minimizing risk.

Compatibility

When migrating from MongoDB to Amazon DocumentDB, a thorough initial assessment and feature compatibility check is essential for a successful migration. This process begins with a comprehensive inventory of your MongoDB features, including aggregation pipeline operators, query patterns, indexes, and data models.

Since Amazon DocumentDB is compatible with MongoDB 3.6, 4.0, and 5.0 API's, applications using newer MongoDB-specific features may require refactoring. Critical areas to evaluate include sharding mechanisms(Amazon DocumentDB uses a different approach), transaction implementations, change streams functionality, and index types (particularly sparse and partial indexes).

Performance characteristics also differ, with Amazon DocumentDB optimized for enterprise workloads with predictable performance. Testing should involve running representative workloads against both systems to identify query patterns that might need optimization.

Monitoring execution plans to detect potential performance gaps is important during the assessment phase. This helps create a clear migration roadmap, identifying necessary application changes and establishing realistic timelines for a smooth transition.

Core feature compatibility

Comprehensive feature support

Enhanced AWS-integrated features

Modern architecture advantages

For a comprehensive overview of Amazon DocumentDB's capabilities, refer to the Supported MongoDB APIs, operations, and data types in Amazon DocumentDB and Functional differences: Amazon DocumentDB and MongoDB to maximize your database's potential.

Amazon DocumentDB does not support all the indexes offered by MongoDB. We provide a free index tool to check the compatibility. We recommend running the index tool to assess incompatibility and plan workarounds accordingly.

Amazon DocumentDB compatibility assessment tool

The MongoDB to Amazon DocumentDB Compatibility Tool is an open-source utility available on GitHub that helps evaluate MongoDB workload compatibility with Amazon DocumentDB by analyzing MongoDB logs or application source code.

Key features

Assessment methods

Log-based assessment

Source code analysis

For best results, we recommend using both assessment methods when possible to get a complete picture of compatibility challenges before migration.

Workload discovery

Migrating from MongoDB to Amazon DocumentDB requires a thorough understanding of the existing database workload. Workload discovery is the process of analyzing your database usage patterns, data structures, query performance, and operational dependencies to ensure a seamless transition with minimal disruption. This section outlines the key steps involved in workload discovery to facilitate an effective migration from MongoDB to Amazon DocumentDB.

Assessing the existing MongoDB deployment

Before migration, it is crucial to evaluate the current MongoDB environment, including:

Identifying data model differences

Although Amazon DocumentDB is MongoDB-compatible, there are differences in supported features, such as:

Query and performance analysis

Understanding query behavior helps optimize migration and post-migration performance. Key areas to analyze include:

Security and access control review

Authentication and authorization

Encryption

Ensure encryption at rest and in transit aligns with compliance requirements.

Network configuration

Plan for Virtual Private Cloud (VPC) setup and security group rules.

Operational and monitoring considerations

To maintain system reliability, workload discovery should also include:

Index migration

Migrating from MongoDB to Amazon DocumentDB involves transferring not just data but also indexes to maintain query performance and optimize database operations. This section outlines the detailed step-by-step process for migrating indexes from MongoDB to Amazon DocumentDB while ensuring compatibility and efficiency.

Using the Amazon DocumentDB index tool

Clone the index tool

git clone https://github.com/aws-samples/amazon-documentdb-tools.git
cd amazon-documentdb-tools/index-tool

pip install -r requirements.txt

Export indexes from MongoDB (if migrating from MongoDB)

python3 migrationtools/documentdb_index_tool.py --dump-indexes --dir mongodb_index_export --uri
'mongodb://localhost:27017'

Export indexes from Amazon DocumentDB (if migrating from Amazon DocumentDB)

python3 migrationtools/documentdb_index_tool.py --dump-indexes --dir docdb_index_export --uri
'mongodb://user:password@mydocdb.cluster-cdtjj00yfi95.eu-west-
2.docdb.amazonaws.com:27017/?tls=true&tlsCAFile=rds-combined-ca-
bundle.pem&replicaSet=rs0&retryWrites=false'

Import indexes

python3 migrationtools/documentdb_index_tool.py --restore-indexes --skip-incompatible --dir
mongodb_index_export --uri 'mongodb://user:password@mydocdb.cluster-cdtjj00yfi95.eu-west-
2.docdb.amazonaws.com:27017/?tls=true&tlsCAFile=rds-combined-ca-
bundle.pem&replicaSet=rs0&retryWrites=false'

Verify indexes

python3 migrationtools/documentdb_index_tool.py --show-issues --dir mongodb_index_export

User migration

Migrating users from MongoDB to Amazon DocumentDB is essential for maintaining access control, authentication, and database security. This section outlines detailed steps to successfully migrate MongoDB users while preserving their roles and permissions using the Amazon DocumentDB export user tool.

Using Amazon DocumentDB export users tool

The Export Users tool exports users and roles from MongoDB or Amazon DocumentDB to JavaScript files, which can then be used to recreate them in another cluster.

Prerequisites

# Clone the repository
git clone https://github.com/awslabs/amazon-documentdb-tools.git
cd amazon-documentdb-tools/migration/export-users

# Install required dependencies
pip install pymongo

Step 1: Export users and roles

# Export users and roles to JavaScript files
python3 docdbExportUsers.py \
--users-file mongodb-users.js \
--roles-file mongodb-roles.js \
--uri "mongodb://admin:password@source-host:27017/"

Step 2: Edit the Users File

// Example of how to update the users.js file
// Find each user creation statement and add the password
db.getSiblingDB("admin").createUser({
user: "appuser",
// Add password here
pwd: "newpassword",
roles: [
{ role: "readWrite", db: "mydb" }
]
})

Step 3: Restore Custom Roles to Amazon DocumentDB

# Import roles first
mongo --ssl \
--host target-host:27017 \
--sslCAFile rds-combined-ca-bundle.pem \
--username admin \
--password password \
mongodb-roles.js

Step 4: Restore Users to Amazon DocumentDB

# Import users after roles are created
mongo --ssl \
--host target-host:27017 \
--sslCAFile rds-combined-ca-bundle.pem \
--username admin \
--password password \
mongodb-users.js

Important notes

This method provides a secure and flexible approach to migrating users and roles from MongoDB to Amazon DocumentDB while allowing for password resets during the migration process.

Data migration

Online migration

This section provides detailed steps to perform an online migration from MongoDB to Amazon DocumentDB using AWS DMS to enable minimal downtime and continuous replication. To begin, you set up an Amazon DocumentDB cluster as the target and ensure your MongoDB instance is properly configured as the source, typically requiring replica set mode for change data capture. Next, you create a DMS replication instance and define source and target endpoints with the necessary connection details. After validating the endpoints, you configure and start a migration task that can include full data load, ongoing replication, or both.

Configure target (Amazon DocumentDB)

Create a custom parameter group

See the AWS Management Console or AWS CLI procedures in Creating Amazon DocumentDB cluster parameter groups.

Create an Amazon DocumentDB cluster

Configure source

MongoDB and Amazon DocumentDB can both serve as migration sources, depending on your scenario:

Before starting migration, configure your source to allow AWS DMS access.

Create a MongoDB user with proper permissions:

db.createUser({
user: "dmsUser",
pwd: "yourSecurePassword",
roles: [{ role: "readAnyDatabase", db: "admin" }]
})

Configure network and authentication.

When configuring network connectivity for MongoDB to DMS migration:

EC2-hosted MongoDB source

On-premises MongoDB source

MongoDB Atlas source

Amazon DocumentDB source

Configure your source Amazon DocumentDB security group to allow inbound traffic from the DMS replication instance security group.

Create DMS replication instance

Configure your DMS infrastructure using the following steps:

Create DMS endpoints

Create a source endpoint

For a MongoDB source

For an Amazon DocumentDB source

Create a target endpoint (Amazon DocumentDB)

Create replication task

Offline migration

This section outlines the process to perform an offline migration from a self-managed MongoDB instance to Amazon DocumentDB using native MongoDB tools: mongodump and mongorestore.

Prerequisites

Source MongoDB requirements

Target Amazon DocumentDB requirements

General requirements

Prepare an Amazon DocumentDB cluster

Create an Amazon DocumentDB cluster in AWS:

Perform the data dump (mongodump)

Choose one of the following options to create a dump file:

Transfer dump files to restoration environment

Choose an appropriate method based on your dump size:

Restore data to Amazon DocumentDB (mongorestore)

Before starting the restore process, create the indexes in Amazon DocumentDB. You can utilize the Amazon DocumentDB Index tool to export and import indexes.

Choose one of the following options to restore data:

Monitoring

This section provides a detailed monitoring process to track the progress, performance, and health of an ongoing migration from:

MongoDB to Amazon DocumentDB

or

Amazon DocumentDB to Amazon DocumentDB

The monitoring steps apply regardless of the migration method (AWS DMS, mongodump/mongorestore, or other tools).

AWS DMS Migration monitoring (if applicable)

Monitor the following key CloudWatch metrics:

Full load phase metrics

CDC phase metrics

Resource metrics

Error metrics

Create CloudWatch alarms for critical metrics like CDCLatencyTarget and CPUUtilization to receive notifications if migration performance degrades.

DMS logs (CloudWatch logs)

DMS task status (using AWS CLI)

aws dms describe-replication-tasks --filters Name=replication-task id,Values=<task_id> --query
"ReplicationTasks[0].Status"

Expected status flow:

creating → ready → running → stopping → stopped (or failed)

Monitor using docdb-dashboarder

The docdb-dashboarder tool provides comprehensive monitoring for Amazon DocumentDB clusters by automatically generating CloudWatch dashboards with essential performance metrics. These dashboards display critical cluster-level metrics (replica lag, operation counters), instance-level metrics (CPU, memory, connections), and storage metrics (volume usage, backup storage). For migration scenarios, the tool offers specialized dashboards that track migration progress with metrics like CDC replication lag and operation rates. The dashboards can monitor multiple clusters simultaneously and include support for NVMe-backed instances. By visualizing these metrics, teams can proactively identify performance bottlenecks, optimize resource allocation, and ensure smooth operation of their Amazon DocumentDB deployments. The tool eliminates the need for manual dashboard creation while providing consistent monitoring across all environments. For setup instructions and advanced configuration options, refer to the Amazon DocumentDB Dashboarder Tool GitHub repository.

Validation

This section provides a detailed validation process to ensure data consistency, integrity, and application compatibility after migrating from:

MongoDB to Amazon DocumentDB

or

Amazon DocumentDB to Amazon DocumentDB

The validation steps apply regardless of the migration method (AWS DMS, mongodump/mongorestore, or other tools).

Validation checklist

Verify that the number of documents in each collection matches between source and target:

MongoDB source

mongo --host <source_host> --port <port> --username <user> -- password <password> --eval
"db.<collection>.count()"

Amazon DocumentDB target

mongo --host <target_host> --port <port> --username <user> -- password <password> --eval
"db.<collection>.count()"

Schema and index validation

Ensure that:

Check collections (source vs. target)

mongo --host <source_host> --eval "show collections"
mongo --host <target_host> --ssl --eval "show collections"

check indexes (Source vs. Target)

mongo --host <source_host> --eval" db.<collection>.getIndexes()"
mongo --host <target_host> --ssl –eval" db.<collection>.getIndexes()"

Compare the list of collections to ensure there are no missing or extra collections.

Verify indexes by checking index names, key definitions, unique constraints, and TTL indexes (if any).

Check schema validation rules (if using schema validation in MongoDB)

mongo --host <source_host> --eval" db.getCollectionInfos({name: '<collection>'})
[0].options.validator"
   mongo --host <target_host> --ssl –eval" db.getCollectionInfos({name: '<collection>'})[0].options.validator"

Data sampling and field-level validation

You can randomly sample documents and compare fields between source and target.

Manual sampling

Fetch five random documents (source):

mongo --host <source_host> --eval "db.<collection>.aggregate([{ \$sample: { size: 5 } }])"

Fetch the same document IDs (target):

mongo --host <target_host> --ssl –eval "db.<collection>.find({ _id: { \$in: [<list_of_ids>] } })"

Automatic sampling

import pymongo
# Connect to source and target
source_client = pymongo.MongoClient("<source_uri>")
target_client = pymongo.MongoClient("<target_uri>", ssl=True)
source_db = source_client["<db_name>"]
target_db = target_client["<db_name>"]
# Compare 100 random documents
for doc in source_db.<collection>.aggregate([{ "$sample":
{ "size": 100 } }]):
target_doc = target_db.<collection>.find_one({ "_id":
doc["_id"] })
if target_doc != doc:
print(f"❌ Mismatch in _id: {doc['_id']}")
else:
print(f"✅ Match: {doc['_id']}")

Validation using DataDiffer tool

The DataDiffer tool provides a reliable way to compare data between source and target databases.

Prerequisites

The following prerequisites must be met before installing the DataDiffer tool:

Setup and installation

Clone the repository and navigate to the DataDiffer directory

git clone https://github.com/awslabs/amazon-documentdb-tools.git
cd amazon-documentdb-tools/migration/data-differ

Install required dependencies

pip install -r requirements.txt

Running data validation

Create a configuration file (e.g., config.json) with connection details

{
"source": {
"uri": "mongodb://username:password@source-mongodb-
host:27017/?replicaSet=rs0",
"db": "your_database",
"collection": "your_collection"
},
"target": {
"uri": "mongodb://username:password@target-docdb-
cluster.region.docdb.amazonaws.com:27017/?tls=true&tlsCAFile=global-
bundle.pem&replicaSet=rs0",
"db": "your_database",
"collection": "your_collection"
},
"options": {
"batch_size": 1000,
"threads": 4,
"sample_size": 0,
"verbose": true
}
}

Run the DataDiffer tool

python differ.py --config config.json

For large collections, use sampling to validate a subset of data

python differ.py --config config.json --sample-size 10000

To validate multiple collections, create separate configuration files or use the batch mode

python differ.py --batch-config batch_config.json

Interpreting results

The tool will output:

Best practices

The following are best practices when using the DataDiffer tool: