Creating a Lambda event source mapping for an Amazon MSK event source - AWS Lambda

Creating a Lambda event source mapping for an Amazon MSK event source

To create an event source mapping, you can use the Lambda console, the AWS Command Line Interface (CLI), or an AWS SDK.

Note

When you create the event source mapping, Lambda creates a hyperplane ENI in the private subnet that contains your MSK cluster, allowing Lambda to establish a secure connection. This hyperplane ENI allows uses the subnet and security group configuration of your MSK cluster, not your Lambda function.

The following console steps add an Amazon MSK cluster as a trigger for your Lambda function. Under the hood, this creates an event source mapping resource.

To add an Amazon MSK trigger to your Lambda function (console)

  1. Open the Function page of the Lambda console.

  2. Choose the name of the Lambda function you want to add an Amazon MSK trigger to.

  3. Under Function overview, choose Add trigger.

  4. Under Trigger configuration, choose MSK.

  5. To specify your Kafka cluster details, do the following:

    1. For MSK cluster, select your cluster.

    2. For Topic name, enter the name of the Kafka topic to consume messages from.

    3. For Consumer group ID, enter the ID of a Kafka consumer group to join, if applicable. For more information, see Customizable consumer group ID in Lambda.

  6. For Cluster authentication, make the necessary configurations. For more information about cluster authentication, see Configuring Amazon MSK ycluster authentication methods in Lambda.

    • Toggle on Use authentication if you want Lambda to perform authentication with your MSK cluster when establishing a connection. Authentication is recommended.

    • If you use authentication, for Authentication method, choose the authentication method to use.

    • If you use authentication, for Secrets Manager key, choose the Secrets Manager key that contains the authentication credentials needed to access your cluster.

  7. Under Event poller configuration, make the necessary configurations.

  8. Under Batching, make the necessary configurations. For more information about batching, see Batching behavior.

    1. For Batch size, enter the maximum number of messages to receive in a single batch.

    2. For Batch window, enter the maximum number of seconds that Lambda spends gathering records before invoking the function.

  9. Under Filtering, make the necessary configurations. For more information about filtering, see Filtering events from Amazon MSK and self-managed Apache Kafka event sources.

    • For Filter criteria, add filter criteria definitions to determine whether or not to process an event.

  10. Under Failure handling, make the necessary configurations. For more information about failure handling, see Capturing discarded batches for Amazon MSK and self-managed Apache Kafka event sources.

    • For On-failure destination, specify the ARN of your on-failure destination.

  11. For Tags, enter the tags to associate with this event source mapping.

  12. To create the trigger, choose Add.

You can also create the event source mapping using the AWS CLI with the create-event-source-mapping command. The following example creates an event source mapping to map the Lambda function my-msk-function to the AWSKafkaTopic topic, starting from the LATEST message. This command also uses the SourceAccessConfiguration object to instruct Lambda to use SASL/SCRAM authentication when connecting to the cluster.

aws lambda create-event-source-mapping \
  --event-source-arn arn:aws:kafka:us-east-1:111122223333:cluster/my-cluster/fc2f5bdf-fd1b-45ad-85dd-15b4a5a6247e-2 \
  --topics AWSKafkaTopic \
  --starting-position LATEST \
  --function-name my-kafka-function
  --source-access-configurations '[{"Type": "SASL_SCRAM_512_AUTH","URI": "arn:aws:secretsmanager:us-east-1:111122223333:secret:my-secret"}]'

If the cluster uses mTLS authentication, include a SourceAccessConfiguration object that specifies CLIENT_CERTIFICATE_TLS_AUTH and a Secrets Manager key ARN. This is shown in the following command:

aws lambda create-event-source-mapping \
  --event-source-arn arn:aws:kafka:us-east-1:111122223333:cluster/my-cluster/fc2f5bdf-fd1b-45ad-85dd-15b4a5a6247e-2 \
  --topics AWSKafkaTopic \
  --starting-position LATEST \
  --function-name my-kafka-function
  --source-access-configurations '[{"Type": "CLIENT_CERTIFICATE_TLS_AUTH","URI": "arn:aws:secretsmanager:us-east-1:111122223333:secret:my-secret"}]'

When the cluster uses IAM authentication, you don’t need a SourceAccessConfiguration object. This is shown in the following command:

aws lambda create-event-source-mapping \
  --event-source-arn arn:aws:kafka:us-east-1:111122223333:cluster/my-cluster/fc2f5bdf-fd1b-45ad-85dd-15b4a5a6247e-2 \
  --topics AWSKafkaTopic \
  --starting-position LATEST \
  --function-name my-kafka-function