Backfill migration
Backfill is accomplished with Reindex-from-Snapshot (RFS) by moving existing documents from your source cluster to the Amazon OpenSearch Service domain or Amazon OpenSearch Serverless collection. RFS takes a one-time snapshot of the source cluster (the only time the source is touched), reads the raw Lucene segment files directly from the snapshot in Amazon S3, extracts documents, applies transformations, and bulk-indexes them on the target. The format of Elasticsearch and OpenSearch indices is such that each shard of each index can be parsed, extracted, and reindexed independently, which means the work can be fanned out at the shard level on Amazon EKS.
This approach improves the migration experience by:
-
Removing load from the source cluster during backfill migration after the source cluster snapshot is taken
-
Enabling "hopping" across multiple major versions without having to pass through the intermediate versions
-
Creating a migration path from post-fork versions of Elasticsearch to Amazon OpenSearch Service and Amazon OpenSearch Serverless
-
Increasing the speed of backfill migration by parallelizing work at the shard level
-
Simplifying the process of pausing and resuming a migration — RFS automatically resumes from the last checkpoint when restarted, already-migrated shards are skipped, and no data is duplicated
Useful RFS tuning settings include podReplicas (number of RFS pods running in parallel — one shard per pod), maxConnections (bulk-indexer concurrency to the target), documentsPerBulkRequest (bulk batch size), maxShardSizeBytes (default 80 GiB; larger shards must be reduced before backfill via force-merge or split), initialLeaseDuration (ISO-8601 duration each worker holds a shard lease before re-acquisition; default PT10M), allowedDocExceptionTypes (list of exception class names to count as success rather than retry), and allowLooseVersionMatching.
