Add observability to your Amazon Bedrock AgentCore resources

Amazon Bedrock AgentCore provides a number of built-in metrics to monitor the performance of resources for the AgentCore runtime, memory, gateway, and built-in tool resource types. This default data is available in Amazon CloudWatch. To view the full range of observability data in the CloudWatch console, or to output custom runtime metrics for agents, you need to instrument your code using the AWS Distro for Open Telemetry (ADOT) SDK.

To view the observability dashboard in CloudWatch, open the Amazon CloudWatch GenAi Observability page.

See the following sections to learn more about configuring your resources to view observability metrics in the CloudWatch console generative AI observability page and in CloudWatch Logs.

Enabling AgentCore runtime observability

To view metrics, spans, and traces generated by the AgentCore service, you first need to complete a one-time setup to turn on Amazon CloudWatch Transaction Search. To view service-provided spans for memory resources, you also need to enable tracing when you create a memory.

The following sections describe how to perform these setup actions and to enable observability in your agent code.

Enabling CloudWatch Transaction Search

When you first use the AgentCore console to create a memory resource, AgentCore displays a button labeled Enable Observability. The button is also displayed in the CloudWatch generative AI observability page. Choose this button to have AgentCore automatically enable Transaction Search.

If you don't see this button while configuring your agent (for example, if you don't create a memory resource in the console), you must enable observability manually by using the CloudWatch console to enable Transaction Search as described in the following procedure.

Enabling observability in agent code for AgentCore-hosted agents

In addition to the service-generated metrics, with AgentCore you can also gather span and trace data as well as custom metrics emitted from your agent code.

When you use agent frameworks like Strands, LangChain, or CrewAI with supported third-party instrumentation libraries, the framework itself comes with built in support for OTEL and GenAI semantic conventions, and it can also be instrumented with an auto-instrumentation package such as opentelemetry-instrument-langchain. It is also possible to send Generative AI semantic conventions telemetry and spans by defining a custom tracer. AgentCore supports use of the following instrumentation libraries in your agent framework:

To view this data in the CloudWatch console generative AI observability page and in Amazon CloudWatch, you need to add the AWS Distro for Open Telemetry (ADOT) SDK to your agent code.

To add ADOT support and enable AgentCore observability, follow the steps in the following procedure.

Configure Observability for agents hosted outside of the AgentCore runtime

To enable observability for agents hosted outside of the AgentCore runtime, first follow the steps in the previous sections to enable CloudWatch Transaction Search and add the ADOT SDK to your code.

For agents running outside of the AgentCore runtime, you also need to create an agent log-group which you include in your environment variables.

Configure your AWS environment variables, and then set your Open Telemetry environment variables as shown in the following.

AWS environment variables

AWS_ACCOUNT_ID=<account id>
AWS_DEFAULT_REGION=<default region>
AWS_REGION=<region>
AWS_ACCESS_KEY_ID=<access key id>
AWS_SECRET_ACCESS_KEY=<secret key>

OTEL environment variables

AGENT_OBSERVABILITY_ENABLED=true
OTEL_PYTHON_DISTRO=aws_distro
OTEL_PYTHON_CONFIGURATOR=aws_configurator # required for ADOT Python only
OTEL_RESOURCE_ATTRIBUTES=service.name=<agent-name>,aws.log.group.names=/aws/bedrock-agentcore/runtimes/<agent-id>,cloud.resource_id=<AgentEndpointArn:AgentEndpointName> # endpoint is optional
OTEL_EXPORTER_OTLP_LOGS_HEADERS=x-aws-log-group=/aws/bedrock-agentcore/runtimes/<agent-id>,x-aws-log-stream=runtime-logs,x-aws-metric-namespace=bedrock-agentcore
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_TRACES_EXPORTER=otlp

Replace <agent-name> with your agent's name and <agent-id> with a unique identifier for your agent.

Session ID support

To propagate session ID, you need to invoke using session identifier in the OTEL baggage:

from opentelemetry import baggage 
 
ctx = baggage.set_baggage("session.id", session_id) # Set the session.id in baggage
attach(ctx) # Attach the context to make it active token

Enable observability for AgentCore memory, gateway, and built-in tool resources

When you create an AgentCore runtime resource (agent), by default, AgentCore runtime creates a CloudWatch log group for the service-provided logs. However, for memory, gateway, and built-in tool resources, AgentCore doesn't configure the required CloudWatch resources for you automatically. For memory resources, if you create these resources in the console, AgentCore prompts you to create log groups and enable tracing as part of the memory or gateway creation process. Be sure to complete these steps to enable observability for your resources. If you use the console to configure the CloudWatch resources, the default log group name for memory and gateway resources has the form /aws/bedrock-agentcore/{resource-id}.

For memory resources created using the AWS CLI or an AWS SDK, you need to manually configure the resources in CloudWatch to store logs and traces.

For gateway and built-in tool resources, the AgentCore service doesn't provide logs by default, but you can output your own logs from your code. If you supply your own log outputs, you need to manually configure log groups in CloudWatch Logs to store this data.

To see what observability data AgentCore provides by default for each resource type, see Amazon Bedrock AgentCore provided observability metrics.

import boto3

def enable_observability_for_resource(resource_arn, resource_id, account_id, region='us-east-1'):
    """
    Enable observability for a Bedrock AgentCore resource (e.g., Memory Store)
    """
    logs_client = boto3.client('logs', region_name=region)

    # Step 0: Create new log group for vended log delivery
    log_group_name = f'/aws/vendedlogs/bedrock-agentcore/{resource_id}'
    logs_client.create_log_group(logGroupName=log_group_name)
    log_group_arn = f'arn:aws:logs:{region}:{account_id}:log-group:{log_group_name}'
    
    # Step 1: Create delivery source for logs
    logs_source_response = logs_client.put_delivery_source(
        name=f"{resource_id}-logs-source",
        logType="APPLICATION_LOGS",
        resourceArn=resource_arn
    )
    
    # Step 2: Create delivery source for traces  
    traces_source_response = logs_client.put_delivery_source(
        name=f"{resource_id}-traces-source", 
        logType="TRACES",
        resourceArn=resource_arn
    )
    
    # Step 3: Create delivery destinations
    logs_destination_response = logs_client.put_delivery_destination(
        name=f"{resource_id}-logs-destination",
        deliveryDestinationType='CWL',
        deliveryDestinationConfiguration={
            'destinationResourceArn': log_group_arn,
        }
    )
    
    # Traces required for memory only
    traces_destination_response = logs_client.put_delivery_destination(
        name=f"{resource_id}-traces-destination",
        deliveryDestinationType='XRAY'
    )
    
    # Step 4: Create deliveries (connect sources to destinations)
    logs_delivery = logs_client.create_delivery(
        deliverySourceName=logs_source_response['deliverySource']['name'],
        deliveryDestinationArn=logs_destination_response['deliveryDestination']['arn']
    )
    
    # Traces required for memory only
    traces_delivery = logs_client.create_delivery(
        deliverySourceName=traces_source_response['deliverySource']['name'], 
        deliveryDestinationArn=traces_destination_response['deliveryDestination']['arn']
    )
    
    print(f"Observability enabled for {resource_id}")
    return {
        'logs_delivery_id': logs_delivery['id'],
        'traces_delivery_id': traces_delivery['id']
    }

# Usage example
resource_arn = "arn:aws:bedrock-agentcore:us-east-1:123456789012:memory/my-memory-id"
resource_id = "my-memory-id" 
account_id = "123456789012"

delivery_ids = enable_observability_for_resource(resource_arn, resource_id, account_id)

Enhanced AgentCore observability with custom headers

You can invoke your agent with additional HTTP headers to provide enhanced observability options. The following example shows invocations including optional additional header requests for agents hosted in the AgentCore runtime.

def invoke_agent(agent_id, payload, session_id=None):
    client = boto3.client("bedrock-agentcore", region="us-west-2")
    response = client.invoke_agent_runtime(
        agentRuntimeArn="arn:aws:bedrock-agentcore:us-west-2:864899855746:runtime/test_agent_boto2-nIg2xk3VSR",
        runtimeSessionId="12345678-1234-5678-9abc-123456789012",
        payload='{"query": "Plan a weekend in Seattle"}',
    )

You can include the following optional headers when invoking your agent to enhance observability and tracing capabilities:

Observability best practices

Consider the following best practices when implementing observability for agents in AgentCore: