Configure model invocation logging in Amazon Bedrock by using AWS CloudFormation
Vikramaditya Bhatnagar, Amazon Web Services
Summary
You can configure Amazon Bedrock to collect invocation logs, model input data, and model output data for all model invocations in your AWS account. This is a best practice
This pattern provides a sample AWS CloudFormation template that uses an infrastructure as code (IaC) approach to configure model invocation logging for Amazon Bedrock. The template configures log storage in both CloudWatch Logs and Amazon S3.
Prerequisites and limitations
Prerequisites
An active AWS account
The following permissions:
Permissions to create CloudFormation stacks
Permissions to access Amazon Bedrock
Permissions to create and access Amazon S3 buckets
Permissions to create and access CloudWatch Logs log groups
Permissions to create and access AWS Lambda functions
Permissions to create and access AWS Key Management Service (AWS KMS) keys
Limitations
This pattern logs model invocations to both CloudWatch Logs and Amazon S3. It does not support choosing only one of these two services.
Architecture
Target architecture
The CloudFormation template provisions the following resources in your target AWS account:
A CloudWatch Logs log group for storing model invocation logs
An Amazon S3 bucket for storing model invocation logs and a corresponding bucket policy
An Amazon S3 bucket for storing server-side access logs and a corresponding bucket policy
An AWS Lambda function that configures logging settings in Amazon Bedrock
An AWS KMS key and a corresponding key alias
An AWS Identity and Access Management (IAM) service role for Amazon Bedrock
The following diagram shows how invocation logs are stored after you deploy the CloudFormation stack associated with this pattern. Amazon Bedrock publishes log data when the foundation model delivers text, an image, a video, or embedding data. As shown in the diagram, the Amazon S3 buckets and the CloudWatch Logs log group are encrypted with an AWS KMS key.
The diagram shows the following workflow:
A user submits a query to a foundation model in Amazon Bedrock.
Amazon Bedrock assumes the IAM service role.
Amazon Bedrock generates log data and stores it in an CloudWatch Logs log group and in an Amazon S3 bucket.
If a user reads, uploads, or deletes any files in the Amazon S3 bucket that contains the model invocation logs, those activities are logged in another Amazon S3 bucket for server-side access logs.
Automation and scale
To scale this solution, you can deploy the CloudFormation template as a stack set to multiple AWS Regions and AWS accounts. For more information, see Managing stacks across accounts and Regions with StackSets in the CloudFormation documentation.
Tools
AWS services
Amazon Bedrock is a fully managed service that makes high-performing foundation models (FMs) from leading AI companies and Amazon available for your use through a unified API.
AWS CloudFormation helps you set up AWS resources, provision them quickly and consistently, and manage them throughout their lifecycle across AWS accounts and AWS Regions.
Amazon CloudWatch Logs helps you centralize the logs from all of your systems, applications, and AWS services so you can monitor them and archive them securely.
AWS Identity and Access Management (IAM) helps you securely manage access to your AWS resources by controlling who is authenticated and authorized to use them.
AWS Lambda is a compute service that helps you run code without needing to provision or manage servers. It runs your code only when needed and scales automatically, so you pay only for the compute time that you use.
AWS Key Management Service (AWS KMS) helps you create and control cryptographic keys to help protect your data.
Amazon Simple Storage Service (Amazon S3) is an object storage service that offers industry-leading scalability, data availability, security, and performance.
Other tools
Git
is an open source, distributed version control system.
Code repository
The code for this pattern is available in the GitHub enable-bedrock-logging-using-cloudformation
Epics
| Task | Description | Skills required |
|---|---|---|
Download the CloudFormation template. | Download the CloudFormation template | Cloud architect |
Deploy the template. | Create a stack in your target account and Region. In the Parameters section, specify values for the parameters that are defined in the template. For instructions, see Creating a stack in the CloudFormation documentation. | Cloud architect |
| Task | Description | Skills required |
|---|---|---|
Enable model access. | In Amazon Bedrock, add access to the foundation model. For instructions, see Add or remove access to Amazon Bedrock foundation models in the Amazon Bedrock documentation. | Cloud architect |
Run a sample prompt. | In Amazon Bedrock playgrounds, run a sample prompt. For instructions, see Generate responses in the console using playgrounds in the Amazon Bedrock documentation. | Cloud architect |
Review the logging configuration. |
| Cloud architect |
Review the Amazon S3 bucket. |
| Cloud architect |
Review the log group. |
| Cloud architect |
Related resources
AWS documentation
Accessing an Amazon S3 bucket (Amazon S3 documentation)
Creating and managing stacks (CloudFormation documentation)
Monitor model invocation (Amazon Bedrock documentation)
Working with log groups and log streams (CloudWatch Logs documentation)
AWS blog posts
