Upgrade the Apache Kafka version

Open the Amazon MSK console at https://console.aws.amazon.com/msk/.

In the navigation bar, choose the Region where you created the MSK cluster.

Choose the MSK cluster which you want to upgrade.

On the Properties tab, choose Upgrade in the Apache Kafka version section.

In the Apache Kafka version section, do the following:

1. In the Choose Apache Kafka version dropdown list, choose the version to which you want to upgrade. For example, choose 3.9.x.

2. (Optional) Choose Version compatibility to view the compatibility between your cluster's current version and the version to which you want to upgrade. Then, choose Choose to proceed or choose Cancel.

3. Choose the Update cluster configuration checkbox to automatically apply a new Kafka configuration revision that's compatible with the upgraded version. This ensures compatibility and enables new features or improvements of the upgraded version. However, skip it if you want to maintain your existing custom configurations.

4. Choose Upgrade.

Run the following command, replacing ClusterArn with the Amazon Resource Name (ARN) that you obtained when you created your cluster. If you don't have the ARN for your cluster, you can find it by listing all clusters. For more information, see List Amazon MSK clusters.

aws kafka get-compatible-kafka-versions --cluster-arn ClusterArn

The output of this command includes a list of the Apache Kafka versions to which you can upgrade the cluster. It looks like the following example.

{
    "CompatibleKafkaVersions": [
        {
            "SourceVersion": "2.2.1",
            "TargetVersions": [
                "2.3.1",
                "2.4.1",
                "2.4.1.1",
                "2.5.1"
            ]
        }
    ]
}

Run the following command, replacing ClusterArn with the Amazon Resource Name (ARN) that you obtained when you created your cluster. If you don't have the ARN for your cluster, you can find it by listing all clusters. For more information, see List Amazon MSK clusters.

Replace Current-Cluster-Version with the current version of the cluster. For TargetVersion you can specify any of the target versions from the output of the previous command.

aws kafka update-cluster-kafka-version --cluster-arn ClusterArn --current-version Current-Cluster-Version --target-kafka-version TargetVersion

The output of the previous command looks like the following JSON.

{
    
    "ClusterArn": "arn:aws:kafka:us-east-1:012345678012:cluster/exampleClusterName/abcdefab-1234-abcd-5678-cdef0123ab01-2",
    "ClusterOperationArn": "arn:aws:kafka:us-east-1:012345678012:cluster-operation/exampleClusterName/abcdefab-1234-abcd-5678-cdef0123ab01-2/0123abcd-abcd-4f7f-1234-9876543210ef"
}

To get the result of the update-cluster-kafka-version operation, run the following command, replacing ClusterOperationArn with the ARN that you obtained in the output of the update-cluster-kafka-version command.

aws kafka describe-cluster-operation --cluster-operation-arn ClusterOperationArn

The output of this describe-cluster-operation command looks like the following JSON example.

{
    "ClusterOperationInfo": {
        "ClientRequestId": "62cd41d2-1206-4ebf-85a8-dbb2ba0fe259",
        "ClusterArn": "arn:aws:kafka:us-east-1:012345678012:cluster/exampleClusterName/abcdefab-1234-abcd-5678-cdef0123ab01-2",
        "CreationTime": "2021-03-11T20:34:59.648000+00:00",
        "OperationArn": "arn:aws:kafka:us-east-1:012345678012:cluster-operation/exampleClusterName/abcdefab-1234-abcd-5678-cdef0123ab01-2/0123abcd-abcd-4f7f-1234-9876543210ef",
        "OperationState": "UPDATE_IN_PROGRESS",
        "OperationSteps": [
            {
                "StepInfo": {
                    "StepStatus": "IN_PROGRESS"
                },
                "StepName": "INITIALIZE_UPDATE"
            },
            {
                "StepInfo": {
                    "StepStatus": "PENDING"
                },
                "StepName": "UPDATE_APACHE_KAFKA_BINARIES"
            },
            {
                "StepInfo": {
                    "StepStatus": "PENDING"
                },
                "StepName": "FINALIZE_UPDATE"
            }
        ],
        "OperationType": "UPDATE_CLUSTER_KAFKA_VERSION",
        "SourceClusterInfo": {
            "KafkaVersion": "2.4.1"
        },
        "TargetClusterInfo": {
            "KafkaVersion": "2.6.1"
        }
    }
}

If OperationState has the value UPDATE_IN_PROGRESS, wait a while, then run the describe-cluster-operation command again. When the operation is complete, the value of OperationState becomes UPDATE_COMPLETE. Because the time required for Amazon MSK to complete the operation varies, you might need to check repeatedly until the operation is complete.

Invoke the GetCompatibleKafkaVersions operation to get a list of the Apache Kafka versions to which you can upgrade the cluster.

Invoke the UpdateClusterKafkaVersion operation to upgrade the cluster to one of the compatible Apache Kafka versions.