MetadataExtractionProcessor

class aws_cdk.aws_kinesisfirehose.MetadataExtractionProcessor(options, keys)

Bases: object

The data processor for dynamic partitioning with inline parsing.

See:

https://docs.aws.amazon.com/firehose/latest/dev/dynamic-partitioning-partitioning-keys.html

ExampleMetadata:

infused

Example:

# bucket: s3.Bucket

s3_destination = firehose.S3Bucket(bucket,
    dynamic_partitioning=firehose.DynamicPartitioningProps(enabled=True),
    processors=[
        firehose.MetadataExtractionProcessor.jq16({
            "customer_id": ".customer_id",
            "device": ".type.device",
            "year": ".event_timestamp|strftime("%Y")"
        })
    ],
    data_output_prefix="!{partitionKeyFromQuery:year}/!{partitionKeyFromQuery:device}/!{partitionKeyFromQuery:customer_id}/"
)
firehose.DeliveryStream(self, "DeliveryStream",
    destination=s3_destination
)
Parameters:

Methods

bind(scope, *, role, dynamic_partitioning_enabled=None, prefix=None)

Binds this processor to a destination of a delivery stream.

Implementers should use this method to grant processor invocation permissions to the provided stream and return the necessary configuration to register as a processor.

Parameters:

  • scope (Construct)

  • role (IRole) – The IAM role assumed by Amazon Data Firehose to write to the destination that this DataProcessor will bind to.

  • dynamic_partitioning_enabled (Optional[bool]) – Whether the dynamic partitioning is enabled. Default: false

  • prefix (Optional[str]) – S3 bucket prefix. Default: - No prefix

Return type:

DataProcessorConfig

Attributes

props

The constructor props of the DataProcessor.

Static Methods

classmethod jq16(query)

Creates the inline parsing configuration with JQ 1.6 engine.

Parameters:

query (Mapping[str, str]) – A map of partition key to jq expression.

Return type:

MetadataExtractionProcessor