Amazon S3

Amazon S3 is an object storage service that stores data as objects within buckets. You can connect an Amazon S3 bucket as a data source for your managed knowledge base to ingest the objects you store there.

Supported features

Prerequisites

In Amazon S3, make sure you:

In your AWS account, make sure you:

Bucket policies for cross-account and KMS-encrypted access

If your Amazon S3 bucket is in a different AWS account from your knowledge base, or if it is encrypted with a customer managed KMS key, add a resource-based policy to grant your knowledge base service role access. The following examples show the minimum statements required.

Cross-account bucket policy

In the account that owns the Amazon S3 bucket, add the following statement to the bucket policy. Replace kb-account-id with the account ID where your knowledge base is created, kb-service-role with the name of your knowledge base service role, and bucket-name with your bucket name.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowBedrockKnowledgeBaseAccess",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::kb-account-id:role/kb-service-role"
            },
            "Action": [
                "s3:GetObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::bucket-name",
                "arn:aws:s3:::bucket-name/*"
            ]
        }
    ]
}

For more general guidance, see Configure access to Amazon S3 buckets.

KMS key policy for encrypted buckets

If your Amazon S3 bucket is encrypted with a customer managed KMS key, add the following statement to the key policy. Use the same placeholders as in the cross-account bucket policy. Allow a few minutes for key policy changes to propagate.

{
    "Sid": "AllowBedrockKnowledgeBaseKmsAccess",
    "Effect": "Allow",
    "Principal": {
        "AWS": "arn:aws:iam::kb-account-id:role/kb-service-role"
    },
    "Action": [
        "kms:Decrypt",
        "kms:DescribeKey"
    ],
    "Resource": "*"
}

The "Resource": "*" here scopes to the key the policy is attached to.

How to set up an Amazon S3 data source

Setting up an Amazon S3 data source involves the following steps:

Create the data source

Console

To connect an Amazon S3 bucket to your managed knowledge base

1. Under Data source, provide a name for your data source.

2. Select Amazon S3 from the data source dropdown.

3. Under Data source location, choose whether the Amazon S3 bucket is in This AWS account or Other AWS account.

4. Enter the Amazon S3 URI of your bucket. You can choose Browse S3 to select a bucket.

5. (Optional) Enter a Metadata files prefix — the Amazon S3 prefix where document metadata files (.metadata.json) are stored for this data source.

6. (Optional) Expand S3 prefix filter patterns to include or exclude specific paths.

7. (Optional) Expand File filter patterns to add regex patterns to include or exclude specific files.

8. (Optional) To enable document-level access control, select Control document access with ACLs and provide the Amazon S3 URI of your global ACL file. This option cannot be changed after creation. For details, see Document-level access controls.

API

To create an Amazon S3 data source, send a CreateDataSource request with an Agents for Amazon Bedrock build-time endpoint. The following AWS Command Line Interface example creates a data source for a bucket in the same account, with prefix and pattern filters. For a description of each field, see the connector parameters reference that follows.

aws bedrock-agent create-data-source \
 --name "S3-connector" \
 --knowledge-base-id "your-knowledge-base-id" \
 --data-source-configuration file://s3-managed-connector.json

The s3-managed-connector.json file contains the following:

{
    "type": "MANAGED_KNOWLEDGE_BASE_CONNECTOR",
    "managedKnowledgeBaseConnectorConfiguration": {
        "connectorParameters": {
            "type": "S3",
            "version": "1",
            "connectionConfiguration": {
                "bucketName": "my-knowledge-base-bucket",
                "bucketOwnerAccountId": "123456789012"
            },
            "filterConfiguration": {
                "inclusionPrefixes": ["documents/"],
                "inclusionPatterns": [".*\\.pdf", ".*\\.txt"],
                "exclusionPatterns": [".*\\.tmp"]
            },
            "metadataFilesPrefix": "metadata/"
        }
    }
}

To enable document-level access control, set aclEnabled to true and add an aclConfiguration with globalAccessControlListS3Uri. See Document-level access controls.

For managed knowledge bases, CreateDataSource is asynchronous: the data source status transitions from CREATING to AVAILABLE when the operation completes.

Connector parameters

The data source configuration uses the following connector parameters. To connect to Amazon S3, specify S3 as the connector type in connectorParameters. For the fields that wrap connectorParameters (such as deletionProtectionConfiguration and mediaExtractionConfiguration), see Connect a data source.

Document metadata files

You can attach metadata to each document by uploading a sidecar file alongside it. For each document, create a file named filename.extension.metadata.json in the same Amazon S3 path. The metadata file must not exceed 10 KB. For example, alongside report.pdf, upload report.pdf.metadata.json with the following content:

{
    "metadataAttributes": {
        "company": {
            "value": {
                "type": "STRING",
                "stringValue": "BioPharm Innovations"
            }
        },
        "created_date": {
            "value": {
                "type": "NUMBER",
                "numberValue": 20221205
            }
        },
        "author": {
            "value": {
                "type": "STRING",
                "stringValue": "Lisa Thompson"
            }
        }
    }
}

For information on the supported attribute data types and the filtering operators you can apply at query time, see Metadata and filtering.