# Online migration for Valkey or Redis OSS
<a name="OnlineMigration"></a>

 By using Online Migration, you can migrate your data from your self-hosted open-source Valkey or Redis OSS on Amazon EC2 to Amazon ElastiCache.

This refers to migration from a self-hosted instance to the ElastiCache service. For information on upgrading from Redis OSS to Valkey on ElastiCache see [Upgrading engine versions including cross engine upgradesUpgrading engine versions](VersionManagement.HowTo.md).

**Note**  
Online migration is not supported to ElastiCache serverless caches or clusters running on the r6gd node type.

## Overview
<a name="Migration-Overview"></a>

To migrate your data from open-source Valkey or Redis OSS running on Amazon EC2 to Amazon ElastiCache requires an existing or newly created Amazon ElastiCache deployment. The deployment must have a configuration that is ready for migration. It also should be in line with the configuration that you want, including attributes such as instance type, number of shards, and number of replicas. 

Online migration is designed for data migration from self-hosted open-source Valkey or Redis OSS on Amazon EC2 to ElastiCache, and not for moving data between ElastiCache clusters.

**Important**  
We strongly recommend you read the following sections in their entirety before beginning the online migration process.

The migration begins when you call the `StartMigration` API operation or AWS CLI command. When migrating Valkey or Redis OSS cluster-mode disabled clusters, the migration process makes the primary node of the ElastiCache Valkey or Redis OSS cluster a replica of your source Valkey or Redis OSS primary. When migrating Valkey or Redis OSS cluster-mode enabled clusters, the migration process makes the primary node of each ElastiCache shard a replica of your source cluster's corresponding shard owning the same slots.

After the client-side changes are ready, call the `CompleteMigration` API operation. This API operation promotes your ElastiCache deployment to your primary Valkey or Redis OSS deployment with primary and replica nodes (as applicable). Now you can redirect your client application to start writing data to ElastiCache. Throughout the migration, you can check the status of replication by running the [valkey-cli INFO](https://valkey.io/commands/info) command on your Valkey nodes and on the ElastiCache primary nodes. 

## Migration steps
<a name="Migration-Steps"></a>

The following topics outline the process for migrating your data:
+ [Preparing your source and target for migration](Migration-Prepare.md)
+ [Testing the data migration](Migration-Test.md)
+ [Starting migration](Migration-Initiate.md)
+ [Verifying the data migration progress](Migration-Verify.md)
+ [Completing the data migration](Migration-Complete.md)

# Preparing your source and target for migration
<a name="Migration-Prepare"></a>

With these steps you can prepare to migrate your data from a self-hosted Valkey or Redis source on EC2 to ElastiCache, or from a Redis OSS cluster to an ElastiCache Valkey cluster.

This refers to migration from a self-hosted instance to the ElastiCache service. For information on upgrading from Redis OSS to Valkey on ElastiCache see [Upgrading engine versions including cross engine upgradesUpgrading engine versions](VersionManagement.HowTo.md).

You must ensure that all four of the prerequisites mentioned following are satisfied before you start the migration from ElastiCache console, API or AWS CLI.

**To prepare your source and target Valkey or Redis OSS Nodes for migration**

1. Identify the target ElastiCache deployment and make sure that you can migrate data to it. 

   An existing or newly created ElastiCache deployment should meet the following requirements for migration: 
   + It is using Valkey, or Redis OSS 5.0.6 or higher.
   + It doesn't have encryption in-transit enabled.
   + It has Multi-AZ enabled.
   + It has sufficient memory available to fit the data from your Valkey or Redis OSS cluster. To configure the right reserved memory settings, see [Managing reserved memory for Valkey and Redis OSS](redis-memory-management.md).
   + For cluster-mode disabled, you can migrate directly from Valkey or Redis OSS versions 2.8.21 onward to Valkey or Redis OSS version 5.0.6 onward if are using the CLI or Valkey or Redis OSS versions 5.0.6 onward using the CLI or console. For cluster mode enabled, you can migrate directly from any cluster-mode enabled Valkey or Redis OSS version to Redis OSS version 5.0.6 onward, if are using the CLI or Redis OSS versions 5.0.6 onward using the CLI or console.
   + Number of shards in source and target match.
   + It is not part of a global datastore.
   + It has data tiering disabled.

1. Make sure that the configurations of your open-source Valkey or Redis OSS and the ElastiCache deployment are compatible. 

   At a minimum, all the following in the target ElastiCache deployment should be compatible with your Valkey or Redis OSS configuration for replication: 
   + Your cluster should not have AUTH enabled.
   + The config `protected-mode` should be set to `no`.
   + If you have `bind` configuration in your Valkey or Redis OSS config, then it should be updated to allow requests from ElastiCache nodes.
   + The number of logical databases should be the same on the ElastiCache node and your Valkey or Redis OSS cluster. This value is set using `databases` in the Valkey or Redis OSS config.
   + Valkey or Redis OSS commands that perform data modification should not be renamed to allow replication of the data to succeed. for example `sync`, `psync`, `info`, `config`, `command`, and `cluster`.
   + To replicate the data from your Valkey or Redis OSS cluster to ElastiCache, make sure that there is sufficient CPU and memory to handle this additional load. This load comes from the RDB file created by your Valkey or Redis OSS cluster and transferred over the network to ElastiCache node.
   + All Valkey or Redis OSS instances in the source cluster should be running on the same port.

1. Make sure that your instances can connect with ElastiCache by doing the following:
   + Ensure that each instance's IP address is private.
   + Assign or create the ElastiCache deployment in the same virtual private cloud (VPC) as your Valkey or Redis OSS on your instance (recommended).
   + If the VPCs are different, set up VPC peering to allow access between the nodes. For more information on VPC peering, see [Access Patterns for Accessing an ElastiCache Cache in an Amazon VPC](elasticache-vpc-accessing.md).
   + The security group attached to your Valkey or Redis OSS instances should allow inbound traffic from ElastiCache nodes.

1. Make sure that your application can direct traffic to ElastiCache nodes after migration of data is complete. For more information, see [Access Patterns for Accessing an ElastiCache Cache in an Amazon VPC](elasticache-vpc-accessing.md). 

# Testing the data migration
<a name="Migration-Test"></a>

After all prerequisites are complete, you can validate migration setup using the AWS Management Console, ElastiCache API, or AWS CLI. The following example shows using the CLI.

Test migration by calling the `test-migration` command with the following parameters:
+ `--replication-group-id` – The ID of the replication group to which data is to be migrated.
+ `--customer-node-endpoint-list` – List of endpoints from which data should be migrated. List should have only one element.

The following is an example using the CLI.

```
aws elasticache test-migration --replication-group-id test-cluster --customer-node-endpoint-list "Address='10.0.0.241',Port=6379"
```

ElastiCache will validate migration setup without any actual data migration.

# Starting migration
<a name="Migration-Initiate"></a>

After all prerequisites are complete, you can begin data migration using the AWS Management Console, ElastiCache API, or AWS CLI. For cluster-mode enabled, if slot migration differs, a resharding will be performed before live migration. The following example shows using the CLI.

**Note**  
We recommended to use `TestMigration` API to validate migration setup. But this is strictly optional.

Start migration by calling the `start-migration` command with the following parameters:
+ `--replication-group-id` – Identifier of the target ElastiCache replication group
+ `--customer-node-endpoint-list` – A list of endpoints with either DNS or IP addresses and the port where your source Valkey or Redis OSS cluster is running. The list can only take one element for both cluster-mode disabled and cluster-mode enabled. If you have enabled chained replication, the endpoint can point to a replica instead of the primary node in your Valkey or Redis OSS cluster. 

The following is an example using the CLI.

```
aws elasticache start-migration --replication-group-id test-cluster --customer-node-endpoint-list "Address='10.0.0.241',Port=6379"
```

As you run this command, the ElastiCache primary node (in each shard) configures itself to become a replica of your Valkey or Redis OSS instance (in corresponding shard owning same slots in cluster enabled redis). The status of ElastiCache cluster changes to **migrating** and data starts migrating from your Valkey or Redis OSS instance to the ElastiCache primary node. Depending on the size of the data and load on your Valkey or Redis OSS instance, the migration can take a while to complete. You can check the progress of the migration by running the [valkey-cli INFO](https://valkey.io/commands/info) command on your Valkey instance and ElastiCache primary node.

After successful replication, all writes to your Valkey or Redis OSS instances propagate to the ElastiCache cluster. You can use ElastiCache nodes for reads. However, you can't write to the ElastiCache cluster. If an ElastiCache primary node has other replica nodes connected to it, these replica nodes continue to replicate from the ElastiCache primary node. This way, all the data from your Valkey or Redis OSS cluster gets replicated to all the nodes in ElastiCache cluster.

If an ElastiCache primary node can't become a replica of your Valkey or Redis OSS instance, it retries several times before eventually promoting itself back to primary. The status of ElastiCache cluster then changes to **available**, and a replication group event about the failure to initiate the migration is sent. To troubleshoot such a failure, check the following:
+ Look at the replication group event. Use any specific information from the event to fix the migration failure.
+ If the event doesn’t provide any specific information, make sure that you have followed the guidelines in [Preparing your source and target for migration](Migration-Prepare.md).
+ Ensure that the routing configuration for your VPC and subnets allows traffic between ElastiCache nodes and your Valkey or Redis OSS instances.
+ Ensure the security group attached to your Valkey or Redis OSS instances allows input bound traffic from ElastiCache nodes.
+ Check Valkey or Redis OSS logs for your instances for more information about failures specific to replication.

# Verifying the data migration progress
<a name="Migration-Verify"></a>

After the data migration has begun, you can do the following to track its progress:
+ Verify that Valkey or Redis OSS `master_link_status` is `up` in the `INFO` command on ElastiCache primary node(s). You can also find this information in the ElastiCache console. Select the cluster and under **CloudWatch metrics**, observe **Primary Link Health Status**. After the value reaches 1, the data is in sync. 
+ You can check that the ElastiCache replica has an **online** state by running the `INFO` command on your Valkey or Redis OSS instances. Doing this also provides information about replication lag.
+ Verify low client output buffer by using the [CLIENT LIST](https://valkey.io/commands/client-list) command on your Valkey or Redis OSS instances.

After the data migration is complete, the data is in sync with any new writes coming to the primary node(s) of your Valkey or Redis OSS cluster.

# Completing the data migration
<a name="Migration-Complete"></a>

When you are ready to cut over to the ElastiCache cluster, use the `complete-migration` CLI command with the following parameters:
+ `--replication-group-id` – The identifier for the replication group.
+ `--force` – A value that forces the migration to stop without ensuring that data is in sync.

The following is an example.

```
aws elasticache complete-migration --replication-group-id test-cluster
```

As you run this command, the ElastiCache primary node (in each shard) stops replicating from your Valkey or Redis OSS instance and promotes it to primary. This promotion typically completes within minutes. To confirm the promotion to primary, check for the event `Complete Migration successful for test-cluster`. At this point, you can direct your application to ElastiCache writes and reads. ElastiCache cluster status should change from **migrating** to **available**.

If the promotion to primary fails, the ElastiCache primary node continues to replicate from your Valkey or Redis OSS instance. The ElastiCache cluster continues to be in **migrating** status, and a replication group event message about the failure is sent. To troubleshoot this failure, look at the following:
+ Check the replication group event. Use specific information from the event to fix the failure.
+ You might get an event message about data not in sync. If so, make sure that the ElastiCache primary can replicate from your Valkey or Redis OSS instance and both are in sync. If you still want to stop the migration, you can run the preceding command with the `—force` option.
+ You might get an event message if one of the ElastiCache nodes is undergoing a replacement. You can retry the complete the migration step after the replacement is complete.

# Performing online data migration using the Console
<a name="Migration-Console"></a>

You can use the AWS Management Console to migrate your data from your cluster to your Valkey or Redis OSS cluster. 

**To perform online data migration using the console**

1. Sign in to the console and open the ElastiCache console at [https://console.aws.amazon.com/elasticache/](https://console.aws.amazon.com/elasticache/home/home).

1. Either create a new Valkey or Redis OSS cluster or choose an existing cluster. Make sure that the cluster meets the following requirements:
   + Your engine version should be Valkey 7.2 and higher, or Redis OSS 5.0.6 or higher. 
   + Your cluster should not have AUTH enabled.
   + The config `protected-mode` should be set to `no`.
   + If you have `bind` configuration in your Valkey or Redis OSS config, then it should be updated to allow requests from ElastiCache nodes.
   + The number of databases should be the same between the ElastiCache node and your Valkey or Redis OSS cluster. This value is set using `databases` in the engine config.
   + Valkey or Redis OSS commands that perform data modification should not be renamed to allow replication of the data to succeed.
   + To replicate the data from your Valkey or Redis OSS cluster to ElastiCache, make sure that there is sufficient CPU and memory to handle this additional load. This load comes from the RDB file created by your Valkey or Redis OSS cluster and transferred over the network to ElastiCache node.
   + The cluster is in **available** status.

1. With your cluster selected, choose **Migrate Data from Endpoint** for **Actions**. 

1. In the **Migrate Data from Endpoint** dialog box, enter the IP address, and the port where your Valkey or Redis OSS cluster is available.
**Important**  
The IP address must be exact. If you enter the address incorrectly, the migration fails.

1. Choose **Start Migration**.

   As the cluster begins migration, it changes to **Modifying** and then **Migrating** status.

1. Monitor the migration progress by choosing **Events** on the navigation pane.

At any point during the migration process, you can stop migration. To do so, choose your cluster and choose **Stop Data Migration** for **Actions**. The cluster then goes to **Available** status.

If the migration succeeds, the cluster goes to **Available** status and the event log shows the following:

`Migration operation succeeded for replication group ElastiCacheClusterName.`

If the migration fails, the cluster goes to **Available** status and the event log shows the following:

`Migration operation failed for replication group ElastiCacheClusterName.`