S3BucketProps

class aws_cdk.aws_kinesisfirehose.S3BucketProps(*, buffering_interval=None, buffering_size=None, compression=None, data_output_prefix=None, encryption_key=None, error_output_prefix=None, logging_config=None, processor=None, role=None, s3_backup=None, file_extension=None)

Bases: CommonDestinationS3Props, CommonDestinationProps

Props for defining an S3 destination of an Amazon Data Firehose delivery stream.

Parameters:

buffering_interval (Optional[Duration]) – The length of time that Firehose buffers incoming data before delivering it to the S3 bucket. Minimum: Duration.seconds(0) Maximum: Duration.seconds(900) Default: Duration.seconds(300)
buffering_size (Optional[Size]) – The size of the buffer that Amazon Data Firehose uses for incoming data before delivering it to the S3 bucket. Minimum: Size.mebibytes(1) Maximum: Size.mebibytes(128) Default: Size.mebibytes(5)
compression (Optional[Compression]) – The type of compression that Amazon Data Firehose uses to compress the data that it delivers to the Amazon S3 bucket. The compression formats SNAPPY or ZIP cannot be specified for Amazon Redshift destinations because they are not supported by the Amazon Redshift COPY operation that reads from the S3 bucket. Default: - UNCOMPRESSED
data_output_prefix (Optional[str]) – A prefix that Amazon Data Firehose evaluates and adds to records before writing them to S3. This prefix appears immediately following the bucket name. Default: “YYYY/MM/DD/HH”
encryption_key (Optional[IKey]) – The AWS KMS key used to encrypt the data that it delivers to your Amazon S3 bucket. Default: - Data is not encrypted.
error_output_prefix (Optional[str]) – A prefix that Amazon Data Firehose evaluates and adds to failed records before writing them to S3. This prefix appears immediately following the bucket name. Default: “YYYY/MM/DD/HH”
logging_config (Optional[ILoggingConfig]) – Configuration that determines whether to log errors during data transformation or delivery failures, and specifies the CloudWatch log group for storing error logs. Default: - errors will be logged and a log group will be created for you.
processor (Optional[IDataProcessor]) – The data transformation that should be performed on the data before writing to the destination. Default: - no data transformation will occur.
role (Optional[IRole]) – The IAM role associated with this destination. Assumed by Amazon Data Firehose to invoke processors and write to destinations Default: - a role will be created with default permissions.
s3_backup (Union[DestinationS3BackupProps, Dict[str, Any], None]) – The configuration for backing up source records to S3. Default: - source records will not be backed up to S3.
file_extension (Optional[str]) – Specify a file extension. It will override the default file extension appended by Data Format Conversion or S3 compression features such as .parquet or .gz. File extension must start with a period (.) and can contain allowed characters: 0-9a-z!-_.*'(). Default: - The default file extension appended by Data Format Conversion or S3 compression features

ExampleMetadata:

infused

Example:

# bucket: s3.Bucket
# Provide a Lambda function that will transform records before delivery, with custom
# buffering and retry configuration
lambda_function = lambda_.Function(self, "Processor",
    runtime=lambda_.Runtime.NODEJS_LATEST,
    handler="index.handler",
    code=lambda_.Code.from_asset(path.join(__dirname, "process-records"))
)
lambda_processor = firehose.LambdaFunctionProcessor(lambda_function,
    buffer_interval=Duration.minutes(5),
    buffer_size=Size.mebibytes(5),
    retries=5
)
s3_destination = firehose.S3Bucket(bucket,
    processor=lambda_processor
)
firehose.DeliveryStream(self, "Delivery Stream",
    destination=s3_destination
)

Attributes

buffering_interval

The length of time that Firehose buffers incoming data before delivering it to the S3 bucket.

Minimum: Duration.seconds(0) Maximum: Duration.seconds(900)

Default:: Duration.seconds(300)

buffering_size

The size of the buffer that Amazon Data Firehose uses for incoming data before delivering it to the S3 bucket.

Minimum: Size.mebibytes(1) Maximum: Size.mebibytes(128)

Default:: Size.mebibytes(5)

compression

The type of compression that Amazon Data Firehose uses to compress the data that it delivers to the Amazon S3 bucket.

The compression formats SNAPPY or ZIP cannot be specified for Amazon Redshift destinations because they are not supported by the Amazon Redshift COPY operation that reads from the S3 bucket.

Default:

UNCOMPRESSED

data_output_prefix

A prefix that Amazon Data Firehose evaluates and adds to records before writing them to S3.

This prefix appears immediately following the bucket name.

Default:: “YYYY/MM/DD/HH”
See:: https://docs.aws.amazon.com/firehose/latest/dev/s3-prefixes.html

encryption_key

The AWS KMS key used to encrypt the data that it delivers to your Amazon S3 bucket.

Default:

Data is not encrypted.

error_output_prefix

A prefix that Amazon Data Firehose evaluates and adds to failed records before writing them to S3.

This prefix appears immediately following the bucket name.

Default:: “YYYY/MM/DD/HH”
See:: https://docs.aws.amazon.com/firehose/latest/dev/s3-prefixes.html

file_extension

Specify a file extension.

It will override the default file extension appended by Data Format Conversion or S3 compression features such as .parquet or .gz.

File extension must start with a period (.) and can contain allowed characters: 0-9a-z!-_.*'().

Default:

The default file extension appended by Data Format Conversion or S3 compression features

See:

https://docs.aws.amazon.com/firehose/latest/dev/create-destination.html#create-destination-s3

logging_config

Configuration that determines whether to log errors during data transformation or delivery failures, and specifies the CloudWatch log group for storing error logs.

Default:

errors will be logged and a log group will be created for you.

processor

The data transformation that should be performed on the data before writing to the destination.

Default:

no data transformation will occur.

role

The IAM role associated with this destination.

Assumed by Amazon Data Firehose to invoke processors and write to destinations

Default:

a role will be created with default permissions.

s3_backup

The configuration for backing up source records to S3.

Default:

source records will not be backed up to S3.