SelfManagedKafkaEventSourceProps

class aws_cdk.aws_lambda_event_sources.SelfManagedKafkaEventSourceProps(*, starting_position, batch_size=None, enabled=None, max_batching_window=None, provisioned_poller_config=None, topic, bisect_batch_on_error=None, consumer_group_id=None, filter_encryption=None, filters=None, max_record_age=None, on_failure=None, report_batch_item_failures=None, retry_attempts=None, schema_registry_config=None, secret=None, starting_position_timestamp=None, bootstrap_servers, authentication_method=None, root_ca_certificate=None, security_group=None, vpc=None, vpc_subnets=None)

Bases: KafkaEventSourceProps

Properties for a self managed Kafka cluster event source.

If your Kafka cluster is only reachable via VPC make sure to configure it.

Parameters:

starting_position (StartingPosition) – Where to begin consuming the stream.
batch_size (Union[int, float, None]) – The largest number of records that AWS Lambda will retrieve from your event source at the time of invoking your function. Your function receives an event with all the retrieved records. Valid Range: - Minimum value of 1 - Maximum value of: - 1000 for DynamoEventSource - 10000 for KinesisEventSource, ManagedKafkaEventSource and SelfManagedKafkaEventSource Default: 100
enabled (Optional[bool]) – If the stream event source mapping should be enabled. Default: true
max_batching_window (Optional[Duration]) – The maximum amount of time to gather records before invoking the function. Maximum of Duration.minutes(5). Default: - Duration.seconds(0) for Kinesis, DynamoDB, and SQS event sources, Duration.millis(500) for MSK, self-managed Kafka, and Amazon MQ.
provisioned_poller_config (Union[ProvisionedPollerConfig, Dict[str, Any], None]) – Configuration for provisioned pollers that read from the event source. When specified, allows control over the minimum and maximum number of pollers that can be provisioned to process events from the source. Default: - no provisioned pollers
topic (str) – The Kafka topic to subscribe to.
bisect_batch_on_error (Optional[bool]) –
- If the function returns an error, split the batch in two and retry. Default: false
consumer_group_id (Optional[str]) – The identifier for the Kafka consumer group to join. The consumer group ID must be unique among all your Kafka event sources. After creating a Kafka event source mapping with the consumer group ID specified, you cannot update this value. The value must have a length between 1 and 200 and full the pattern ‘[a-zA-Z0-9-/:_+=.@-]’. Default: - none
filter_encryption (Optional[IKey]) – Add Customer managed KMS key to encrypt Filter Criteria. Default: - none
filters (Optional[Sequence[Mapping[str, Any]]]) – Add filter criteria to Event Source. Default: - none
max_record_age (Optional[Duration]) – The maximum age of a record that Lambda sends to a function for processing. The default value is -1, which sets the maximum age to infinite. When the value is set to infinite, Lambda never discards old records. Record are valid until it expires in the event source. Default: -1
on_failure (Optional[IEventSourceDlq]) – Add an on Failure Destination for this Kafka event. Supported destinations: - {@link KafkaDlq } - Send failed records to a Kafka topic - SNS topics - Send failed records to an SNS topic - SQS queues - Send failed records to an SQS queue - S3 buckets - Send failed records to an S3 bucket Default: - discarded records are ignored
report_batch_item_failures (Optional[bool]) –
- Allow functions to return partially successful responses for a batch of records. Default: false
retry_attempts (Union[int, float, None]) –
- Maximum number of retry attempts. Set to -1 for infinite retries (until the record expires in the event source). Default: -1 (infinite retries)
schema_registry_config (Optional[ISchemaRegistry]) – Specific configuration settings for a Kafka schema registry. Default: - none
secret (Optional[ISecret]) – The secret with the Kafka credentials, see https://docs.aws.amazon.com/msk/latest/developerguide/msk-password.html for details This field is required if your Kafka brokers are accessed over the Internet. Default: none
starting_position_timestamp (Union[int, float, None]) – The time from which to start reading, in Unix time seconds. Default: - no timestamp
bootstrap_servers (Sequence[str]) – The list of host and port pairs that are the addresses of the Kafka brokers in a “bootstrap” Kafka cluster that a Kafka client connects to initially to bootstrap itself. They are in the format abc.xyz.com:xxxx.
authentication_method (Optional[AuthenticationMethod]) – The authentication method for your Kafka cluster. Default: AuthenticationMethod.SASL_SCRAM_512_AUTH
root_ca_certificate (Optional[ISecret]) – The secret with the root CA certificate used by your Kafka brokers for TLS encryption This field is required if your Kafka brokers use certificates signed by a private CA. Default: - none
security_group (Optional[ISecurityGroup]) – If your Kafka brokers are only reachable via VPC, provide the security group here. Default: - none, required if setting vpc
vpc (Optional[IVpc]) – If your Kafka brokers are only reachable via VPC provide the VPC here. Default: none
vpc_subnets (Union[SubnetSelection, Dict[str, Any], None]) – If your Kafka brokers are only reachable via VPC, provide the subnets selection here. Default: - none, required if setting vpc

ExampleMetadata:

infused

Example:

from aws_cdk.aws_lambda_event_sources import SelfManagedKafkaEventSource, AuthenticationMethod
from aws_cdk.aws_lambda import StartingPosition, Function
from aws_cdk.aws_secretsmanager import ISecret

# With provisioned pollers and poller group for cost optimization
# my_function: Function
# kafka_credentials: ISecret

my_function.add_event_source(SelfManagedKafkaEventSource(
    bootstrap_servers=["kafka-broker1.example.com:9092", "kafka-broker2.example.com:9092"],
    topic="events-topic",
    secret=kafka_credentials,
    starting_position=StartingPosition.LATEST,
    authentication_method=AuthenticationMethod.SASL_SCRAM_512_AUTH,
    provisioned_poller_config=ProvisionedPollerConfig(
        minimum_pollers=1,
        maximum_pollers=8,
        poller_group_name="self-managed-kafka-group"
    )
))

Attributes

authentication_method

The authentication method for your Kafka cluster.

Default:: AuthenticationMethod.SASL_SCRAM_512_AUTH

batch_size

The largest number of records that AWS Lambda will retrieve from your event source at the time of invoking your function.

Your function receives an event with all the retrieved records.

Valid Range:

Minimum value of 1
Maximum value of:
- 1000 for DynamoEventSource
- 10000 for KinesisEventSource, ManagedKafkaEventSource and SelfManagedKafkaEventSource

Default:: 100

bisect_batch_on_error

If the function returns an error, split the batch in two and retry.

Default:: false

bootstrap_servers

The list of host and port pairs that are the addresses of the Kafka brokers in a “bootstrap” Kafka cluster that a Kafka client connects to initially to bootstrap itself.

They are in the format abc.xyz.com:xxxx.

consumer_group_id

The identifier for the Kafka consumer group to join.

The consumer group ID must be unique among all your Kafka event sources. After creating a Kafka event source mapping with the consumer group ID specified, you cannot update this value. The value must have a length between 1 and 200 and full the pattern ‘[a-zA-Z0-9-/:_+=.@-]’.

Default:

none

See:

https://docs.aws.amazon.com/lambda/latest/dg/with-msk.html#services-msk-consumer-group-id

enabled

If the stream event source mapping should be enabled.

Default:: true

filter_encryption

Add Customer managed KMS key to encrypt Filter Criteria.

Default:

none

See:

https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#aws-managed-cmk

filters

Add filter criteria to Event Source.

Default:

none

See:

https://docs.aws.amazon.com/lambda/latest/dg/invocation-eventfiltering.html

max_batching_window

The maximum amount of time to gather records before invoking the function.

Maximum of Duration.minutes(5).

Default:

Duration.seconds(0) for Kinesis, DynamoDB, and SQS event sources, Duration.millis(500) for MSK, self-managed Kafka, and Amazon MQ.

See:

https://docs.aws.amazon.com/lambda/latest/dg/invocation-eventsourcemapping.html#invocation-eventsourcemapping-batching

max_record_age

The maximum age of a record that Lambda sends to a function for processing.

The default value is -1, which sets the maximum age to infinite. When the value is set to infinite, Lambda never discards old records. Record are valid until it expires in the event source.

Default:: -1

on_failure

Add an on Failure Destination for this Kafka event.

Supported destinations:

{@link KafkaDlq } - Send failed records to a Kafka topic
SNS topics - Send failed records to an SNS topic
SQS queues - Send failed records to an SQS queue
S3 buckets - Send failed records to an S3 bucket

Default:

discarded records are ignored

provisioned_poller_config

Configuration for provisioned pollers that read from the event source.

When specified, allows control over the minimum and maximum number of pollers that can be provisioned to process events from the source.

Default:

no provisioned pollers

report_batch_item_failures

Allow functions to return partially successful responses for a batch of records.

Default:: false

retry_attempts

Maximum number of retry attempts.

Set to -1 for infinite retries (until the record expires in the event source).

Default:: -1 (infinite retries)

root_ca_certificate

The secret with the root CA certificate used by your Kafka brokers for TLS encryption This field is required if your Kafka brokers use certificates signed by a private CA.

Default:

none

schema_registry_config

Specific configuration settings for a Kafka schema registry.

Default:

none

secret

//docs.aws.amazon.com/msk/latest/developerguide/msk-password.html for details This field is required if your Kafka brokers are accessed over the Internet.

Default:: none
Type:: The secret with the Kafka credentials, see https

security_group

If your Kafka brokers are only reachable via VPC, provide the security group here.

Default:

none, required if setting vpc

starting_position: Where to begin consuming the stream.

starting_position_timestamp

The time from which to start reading, in Unix time seconds.

Default:

no timestamp

topic: The Kafka topic to subscribe to.

vpc

If your Kafka brokers are only reachable via VPC provide the VPC here.

Default:: none

vpc_subnets

If your Kafka brokers are only reachable via VPC, provide the subnets selection here.

Default:

none, required if setting vpc