Add observability to your Amazon Bedrock AgentCore resources

Amazon Bedrock AgentCore provides a number of built-in metrics to monitor the performance of resources for the AgentCore runtime, memory, gateway, built-in tools, and identity resource types. This default data is available in Amazon CloudWatch. To view the full range of observability data in the CloudWatch console, or to output custom runtime metrics for agents, you need to instrument your code using the AWS Distro for Open Telemetry (ADOT) SDK.

To view the observability dashboard in CloudWatch, open the Amazon CloudWatch GenAi Observability page.

See the following sections to learn more about configuring your resources to view observability metrics in the CloudWatch console generative AI observability page and in CloudWatch Logs.

Enabling AgentCore observability

To view metrics, spans, and traces generated by the AgentCore service, you first need to complete a one-time setup to turn on Amazon CloudWatch Transaction Search. To view service-provided spans for memory resources, you also need to enable tracing when you create a memory. See Enabling observability for AgentCore runtime, memory, gateway, built-in tools, and identity resources to learn more.

The following sections describe how to perform these setup actions and to enable observability in your agent code.

Enabling CloudWatch Transaction Search

You can enable CloudWatch Transaction Search either by using the CloudWatch console, or by using an API through the AWS Command Line Interface (AWS CLI) or one of the AWS SDKs.

Use one of the following procedures to enable Transaction Search.

Enabling observability in agent code for AgentCore-hosted agents

In addition to the service-generated metrics, with AgentCore you can also gather span and trace data as well as custom metrics emitted from your agent code.

When you use agent frameworks like Strands, LangChain, or CrewAI with supported third-party instrumentation libraries, the framework itself comes with built in support for OTEL and GenAI semantic conventions, and it can also be instrumented with an auto-instrumentation package such as opentelemetry-instrument-langchain. It is also possible to send Generative AI semantic conventions telemetry and spans by defining a custom tracer. AgentCore supports use of the following instrumentation libraries in your agent framework:

To view this data in the CloudWatch console generative AI observability page and in Amazon CloudWatch, you need to add the AWS Distro for Open Telemetry (ADOT) SDK to your agent code.

To add ADOT support and enable AgentCore observability, follow the steps in the following procedure.

Enabling observability for agents hosted outside of AgentCore

To enable observability for agents hosted outside of the AgentCore runtime, first follow the steps in the previous sections to enable CloudWatch Transaction Search and add the ADOT SDK to your code.

For agents running outside of the AgentCore runtime, you also need to create an agent log-group which you include in your environment variables.

Configure your AWS environment variables, and then set your Open Telemetry environment variables as shown in the following.

AWS environment variables

AWS_ACCOUNT_ID=<account id>
AWS_DEFAULT_REGION=<default region>
AWS_REGION=<region>
AWS_ACCESS_KEY_ID=<access key id>
AWS_SECRET_ACCESS_KEY=<secret key>

OTEL environment variables

AGENT_OBSERVABILITY_ENABLED=true
OTEL_PYTHON_DISTRO=aws_distro
OTEL_PYTHON_CONFIGURATOR=aws_configurator # required for ADOT Python only
OTEL_RESOURCE_ATTRIBUTES=service.name=<agent-name>,aws.log.group.names=/aws/bedrock-agentcore/runtimes/<agent-id>,cloud.resource_id=<AgentEndpointArn:AgentEndpointName> # endpoint is optional
OTEL_EXPORTER_OTLP_LOGS_HEADERS=x-aws-log-group=/aws/bedrock-agentcore/runtimes/<agent-id>,x-aws-log-stream=runtime-logs,x-aws-metric-namespace=bedrock-agentcore
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_TRACES_EXPORTER=otlp

Replace <agent-name> with your agent's name and <agent-id> with a unique identifier for your agent.

Session ID support

To propagate session ID, you need to invoke using session identifier in the OTEL baggage:

from opentelemetry import baggage 
 
ctx = baggage.set_baggage("session.id", session_id) # Set the session.id in baggage
attach(ctx) # Attach the context to make it active token

Enabling observability for AgentCore runtime, memory, gateway, built-in tools, and identity resources

When you create an AgentCore runtime resource (agent), by default, AgentCore runtime creates a CloudWatch log group for the service-provided logs. However, for memory, gateway, and built-in tool resources, AgentCore doesn't configure log destinations for you automatically.

For memory and gateway resources, you can configure log destinations either in the console or by using an AWS SDK. If you use the console to configure a CloudWatch Logs destination, the default log group name for memory and gateway resources has the form /aws/vendedlogs/bedrock-agentcore/{resource-type}/APPLICATION_LOGS/{resource-id}, where {resource-type} is either memory or gateway.

For memory and gateway logs, you can also configure log destinations in Amazon S3 logs or Firehose stream logs using the AgentCore console. To learn more about storing logs in Amazon S3 or Firehose, see Uploading, downloading, and working with objects in Amazon S3 and Creating an Amazon Data Firehose delivery stream.

To learn more about the log data output by AgentCore for memory and gateway resources see Provided log data (memory) or Provided log data (gateway).

For built-in tool resources, the AgentCore service doesn't provide logs by default, but you can output your own logs from your code. If you supply your own log outputs, you need to manually configure log destinations to store this data.

To see what observability data AgentCore provides by default for each resource type, see Amazon Bedrock AgentCore generated observability data.

Configure log destinations using the console

To configure log destinations for memory or gateway logs in the AgentCore console, use the following procedures.

Configure tracing delivery to CloudWatch using the console

This section describes how to enable trace delivery to CloudWatch to track the flow of interactions through your application allowing you to visualize requests, identify performance bottlenecks, troubleshoot errors, and optimize performance.

Configure CloudWatch resources using an AWS SDK

import boto3

def enable_observability_for_resource(resource_arn, resource_id, account_id, region='us-east-1'):
    """
    Enable observability for a Bedrock AgentCore resource (e.g., Memory Store)
    """
    logs_client = boto3.client('logs', region_name=region)

    # Step 0: Create new log group for vended log delivery
    log_group_name = f'/aws/vendedlogs/bedrock-agentcore/{resource_id}'
    logs_client.create_log_group(logGroupName=log_group_name)
    log_group_arn = f'arn:aws:logs:{region}:{account_id}:log-group:{log_group_name}'
    
    # Step 1: Create delivery source for logs
    logs_source_response = logs_client.put_delivery_source(
        name=f"{resource_id}-logs-source",
        logType="APPLICATION_LOGS",
        resourceArn=resource_arn
    )
    
    # Step 2: Create delivery source for traces  
    traces_source_response = logs_client.put_delivery_source(
        name=f"{resource_id}-traces-source", 
        logType="TRACES",
        resourceArn=resource_arn
    )
    
    # Step 3: Create delivery destinations
    logs_destination_response = logs_client.put_delivery_destination(
        name=f"{resource_id}-logs-destination",
        deliveryDestinationType='CWL',
        deliveryDestinationConfiguration={
            'destinationResourceArn': log_group_arn,
        }
    )
    
    # Traces required
    traces_destination_response = logs_client.put_delivery_destination(
        name=f"{resource_id}-traces-destination",
        deliveryDestinationType='XRAY'
    )
    
    # Step 4: Create deliveries (connect sources to destinations)
    logs_delivery = logs_client.create_delivery(
        deliverySourceName=logs_source_response['deliverySource']['name'],
        deliveryDestinationArn=logs_destination_response['deliveryDestination']['arn']
    )
    
    # Traces required
    traces_delivery = logs_client.create_delivery(
        deliverySourceName=traces_source_response['deliverySource']['name'], 
        deliveryDestinationArn=traces_destination_response['deliveryDestination']['arn']
    )
    
    print(f"Observability enabled for {resource_id}")
    return {
        'logs_delivery_id': logs_delivery['id'],
        'traces_delivery_id': traces_delivery['id']
    }

# Usage example
resource_arn = "arn:aws:bedrock-agentcore:us-east-1:123456789012:memory/my-memory-id"
resource_id = "my-memory-id" 
account_id = "123456789012"

delivery_ids = enable_observability_for_resource(resource_arn, resource_id, account_id)

Enhanced AgentCore runtime observability with custom headers

You can invoke your agent with additional HTTP headers to provide enhanced observability options. The following example shows invocations including optional additional header requests for agents hosted in the AgentCore runtime.

def invoke_agent(agent_id, payload, session_id=None):
    client = boto3.client("bedrock-agentcore", region="us-west-2")
    response = client.invoke_agent_runtime(
        agentRuntimeArn="arn:aws:bedrock-agentcore:us-west-2:111122223333:runtime/test_agent_boto2-nIg2xk3VSR",
        runtimeSessionId="12345678-1234-5678-9abc-123456789012",
        payload='{"query": "Plan a weekend in Seattle"}',
    )

You can include the following optional headers when invoking your agent to enhance observability and tracing capabilities:

Enhanced AgentCore built-in tools observability with custom headers

You can invoke your Build-in Tools with additional HTTP headers to provide enhanced observability options. You can include the following optional headers when integrating following Build-in Tools APIs to enhance observability and tracing capabilities:

The following APIs support custom headers:

Enhanced AgentCore identity observability with custom headers

You can invoke your identity resources with additional HTTP headers to provide enhanced observability options. You can include the following optional headers when integrating following identity APIs to enhance observability and tracing capabilities:

The following APIs support custom headers:

Observability best practices

Consider the following best practices when implementing observability for agents in AgentCore:

Using other observability platforms

To integrate agents hosted in the AgentCore runtime with other observability platforms to capture and view telemetry outputs, set the following environment variable:

DISABLE_ADOT_OBSERVABILITY=true

Setting this variable to true unsets the AgentCore runtime's default ADOT environment variables, ensuring that none of the default ADOT configurations are set.