Customize ingestion for a data source

You can customize vector ingestion when connecting a data source in the AWS Management Console or by modifying the value of the vectorIngestionConfiguration field when sending a CreateDataSource request.

Select a topic to learn how to include configurations for customizing ingestion when connecting to a data source:

Use smart parsing

Managed knowledge bases use smart parsing by default. Smart parsing is a service-managed parsing strategy that automatically selects the best parsing approach for your content. You do not need to configure a parsing model or provide additional settings.

To use smart parsing, you can either omit the parsingConfiguration field from the vectorIngestionConfiguration, or explicitly specify it as follows:

{
    "parsingConfiguration": {
        "parsingStrategy": "SMART_PARSING"
    }
}

Choose a chunking strategy

You can customize how the documents in your data are chunked for storage and retrieval. To learn about options for chunking data in Amazon Bedrock Knowledge Bases, see How content chunking works for knowledge bases.

In the AWS Management Console you choose the chunking strategy when connecting to a data source. With the Amazon Bedrock API, you include a ChunkingConfiguration in the chunkingConfiguration field of the VectorIngestionConfiguration.

If you omit this configuration or specify the default chunking strategy, the service uses fixed-size chunking with 300 tokens and 20% overlap.

{
    "chunkingConfiguration": {
        "chunkingStrategy": "DEFAULT"
    }
}

Expand the section that corresponds to the chunking strategy that you want to use:

{
    "chunkingStrategy": "NONE"
}

{
    "chunkingStrategy": "FIXED_SIZE",
    "fixedSizeChunkingConfiguration": {
        "maxTokens": number,
        "overlapPercentage": number
    }
}