Create an Apache Kafka topic - Amazon Managed Streaming for Apache Kafka
Setting up your environment for creating topicsCreating a topic and writing data to it

Create an Apache Kafka topic

In this step, you use the previously created client machine to create a topic on the serverless cluster.

Setting up your environment for creating topics

  • Before creating a topic, make sure that you've downloaded the AWS MSK IAM JAR file to your Kafka installation's libs/ directory. If you haven't done this yet, run the following command in your Kafka's libs/ directory.

    wget https://github.com/aws/aws-msk-iam-auth/releases/download/v2.3.0/aws-msk-iam-auth-2.3.0-all.jar

    This JAR file is required for IAM authentication with your MSK Serverless cluster.

  • When running Kafka commands, you might need to make sure the classpath includes the AWS MSK IAM JAR file. To do this, do one of the following:

    • Set the CLASSPATH environment variable to include your Kafka libraries as shown in the following example.

      export CLASSPATH=<path-to-your-kafka-installation>/libs/*:<path-to-your-kafka-installation>/libs/aws-msk-iam-auth-2.3.0-all.jar

    • Run Kafka commands using the full Java command with explicit classpath, as shown in the following example.

      java -cp "<path-to-your-kafka-installation>/libs/*:<path-to-your-kafka-installation>/libs/aws-msk-iam-auth-2.3.0-all.jar" org.apache.kafka.tools.TopicCommand --bootstrap-server $BS --command-config client.properties --create --topic msk-serverless-tutorial --partitions 6

Creating a topic and writing data to it

  1. In the following export command, replace my-endpoint with the bootstrap-server string you that you saved after you created the cluster. Then, go to the kafka_2.12-2.8.1/bin directory on the client machine and run the export command.

    export BS=my-endpoint

  2. Run the following command to create a topic called msk-serverless-tutorial.

    <path-to-your-kafka-installation>/bin/kafka-topics.sh --bootstrap-server $BS --command-config client.properties --create --topic msk-serverless-tutorial --partitions 6

Next Step

Produce and consume data in MSK Serverless