Migrate metadata

Metadata migration shapes the Amazon OpenSearch Service domain or Amazon OpenSearch Serverless NextGen collection so that it can accept the incoming documents and traffic. It moves index settings, index mappings, legacy and composable index templates, component templates, and aliases from the source cluster to the target before document backfill or traffic replay becomes meaningful.

In the workflow model, metadata migration is a configured phase inside the migration workflow rather than a separate manual tool run. When a metadataMigrationConfig is present in your workflow configuration, the workflow runs two logical steps:

If approval gates are enabled, the workflow pauses between these two steps so you can inspect the evaluation result before any change is written. The metadata migration tool gathers information from the source through a snapshot or through HTTP requests, and the snapshots it uses are fully compatible with the Reindex-from-Snapshot (RFS) backfill phase. For how to create the source snapshot, see Creating a snapshot.

In workflow runs, metadata evaluation and migration reports are stored as managed output artifacts on the SnapshotMigration resource. Use workflow manage and Show Output, or use workflow show --list followed by workflow show snapshotmigration.<NAME> evaluatemetadata or workflow show snapshotmigration.<NAME> migratemetadata, to review those reports before approving the next gate.

The console metadata commands described in this section run the same logic outside the workflow. Reach for them when you want to preview or apply metadata manually during validation or troubleshooting.

What metadata migration covers

Metadata migration handles the following:

It also applies built-in transformations to reconcile version-specific behavior between the source and the target. See Built-in transformations.

Plan separate work for the following — these are not migrated automatically:

For data streams, this means metadata migration does not create the target data stream, data-stream template, or backing-index structure. If you need to backfill documents that are stored in source .ds-* backing indexes, prepare the target data stream separately and configure RFS as described in Backfilling data stream documents.

Evaluate metadata

Before migrating, preview what will be migrated. In the workflow, this is the evaluateMetadata step. To run it manually from the Migration Console pod:

console metadata evaluate

This scans the source cluster, applies filtering and modifications, and produces a list of items that will be migrated. Items not listed will not be migrated. Use this as a safety check before modifying the Amazon OpenSearch Service domain or Amazon OpenSearch Serverless NextGen collection.

Example output:

Starting Metadata Evaluation
Clusters:
   Source:
      Remote Cluster: OpenSearch 1.3.16

   Target:
      Remote Cluster: OpenSearch 2.14.0

Migration Candidates:
   Index Templates:
      simple_index_template

   Component Templates:
      simple_component_template

   Indexes:
      blog_2023, movies_2023

   Aliases:
      alias1, movies-alias

Results:
   0 issue(s) detected

Run metadata migration

The migrate step applies all evaluated items to the Amazon OpenSearch Service domain or Amazon OpenSearch Serverless NextGen collection. In the workflow, this is the migrateMetadata step. To run it manually:

console metadata migrate

If you run the workflow more than once, target indexes that already exist are reported as non-fatal so workflow reconciliation can rerun safely. For manual console metadata migrate runs, pass --allow-existing-indexes true when you want the same idempotent behavior; otherwise, an existing target index is treated as a fatal error. To re-migrate an item with changed metadata, delete it from the target first, then rerun evaluate and migrate.

Example output:

Starting Metadata Migration

Clusters:
   Source:
      Snapshot: OpenSearch 1.3.16

   Target:
      Remote Cluster: OpenSearch 2.14.0

Migrated Items:
   Index Templates:
      simple_index_template

   Component Templates:
      simple_component_template

   Indexes:
      blog_2023, movies_2023

   Aliases:
      alias1, movies-alias

Results:
   0 issue(s) detected

Filter what gets migrated

Use allowlists in your workflow configuration to scope metadata migration. During a pilot, restrict the run to a small set of indexes, then widen to the full set for the real migration:

For Elasticsearch and OpenSearch regex: entries, the pattern must match the entire name. Use expressions such as regex:logs-. for prefixes or regex:. for suffixes; regex:logs matches only the exact string logs. Solr collection and core allowlists do not use regex: patterns.

These metadata allowlists are evaluated client-side after the snapshot has been taken. To exclude indices from the snapshot itself, configure the snapshot indexAllowlist in createSnapshotConfig; that field uses the source cluster’s native snapshot index-expression syntax rather than regex: patterns.

Other useful metadataMigrationConfig settings include:

Sourceless metadata evaluation

If a selected source index has _source disabled or partially filtered, metadata evaluation fails by default because document backfill cannot read a complete _source document. To opt in to sourceless migration, set enableSourcelessMigrations in both metadataMigrationConfig and documentBackfillConfig. This tells metadata evaluation to permit those indices and tells RFS to reconstruct documents from stored fields, doc_values, Lucene point values, indexed terms, and mapping-level constant values. RFS also strips copy_to target fields from reconstructed _source and can sometimes reverse-derive a missing source field from a less-lossy copy_to target.

If reconstruction from stored fields and doc_values is not sufficient, RFS can treat _recovery_source as _source when it exists in Elasticsearch 7 or OpenSearch snapshots with soft deletes. Configure that behavior with documentBackfillConfig.useRecoverySource. The metadata migration CLI accepts --use-recovery-source and the workflow schema accepts metadataMigrationConfig.useRecoverySource for parity with document backfill configuration, but metadata evaluation does not read document bodies and does not use _recovery_source. The field is transient and may not exist for every document, so use it only after validating representative documents.

The approval gates are the right place to decide whether this is acceptable for your data model. Stored fields are the closest replacement for _source; doc_values and indexed terms can change multi-value ordering, analyzed text surface form, and object-array field association.

Verify metadata

Before proceeding to backfill, confirm the target details. Depending on your configuration, check the sharding strategy or verify that index mappings are correctly defined:

console clusters curl target /<index>/_mapping?pretty
console clusters curl target /<index>/_settings?pretty

Built-in transformations

When you migrate between major versions, some field types and index structures are deprecated, removed, or renamed. Migration Assistant applies built-in transformations automatically so that source definitions land in a shape the Amazon OpenSearch Service domain or Amazon OpenSearch Serverless NextGen collection accepts.

For field types that the built-in transformations do not cover, you can supply a custom transformer. See Transform field types.

Troubleshoot metadata migration

Access detailed logs

For workflow-managed metadata migrations on Amazon EKS, use the workflow logs and managed output artifacts instead of looking for files on the Migration Console filesystem:

workflow log resource snapshotmigration.<NAME> -- --tail=200
workflow show snapshotmigration.<NAME> evaluatemetadata
workflow show snapshotmigration.<NAME> migratemetadata

Use workflow log resource --list and workflow show --list if you need the exact snapshotmigration.<NAME> resource name.

Amazon OpenSearch Service running in compatibility mode

If you encounter an error about being unable to update an Elasticsearch 7.10.2 cluster, compatibility mode may be enabled on the Amazon OpenSearch Service domain. Disable compatibility mode to continue. See Enable compatibility mode in the Amazon OpenSearch Service Developer Guide.

For broader diagnosis, see Troubleshooting.