Generative AI observability - Amazon CloudWatch

Generative AI observability

With Amazon CloudWatch, you can observe generative AI workloads, including Amazon Bedrock AgentCore agents, and gain insights into AI performance, health, and accuracy. CloudWatch provides pre-configured views into latency, usage, and errors of your AI workloads, allowing you to detect issues faster in components like models and agents. End-to-end prompt tracing helps you quickly identify issues in components such as knowledge bases, tools, and models. CloudWatch's AI monitoring capabilities are compatible with popular generative AI orchestration frameworks such as AWS Strands, LangChain, and LangGraph, offering flexibility with your choice of framework.

CloudWatch generative AI observability enables you to:

  • Assess AI application quality and accuracy at scale through automated monitoring, reducing manual review requirements by capturing model outputs, response quality metrics, and end-user interactions

  • Monitor model invocations, Agents (managed, self-hosted, and third-party), knowledge bases, guardrails, and tools

  • Progress from agent experimentation to production of innovative GenAI applications while ensuring superior quality, performance, and reliability. For more information, see What is Amazon Bedrock AgentCore?

  • Identify source of errors quickly using end-to-end prompt tracing, curated metrics, and logs

  • Troubleshoot issues across your entire GenAI application and underlying infrastructure, leveraging existing CloudWatch observability tools such as Application Signals, Alarms, Dashboards, Sensitive data protection, and Logs Insights

  • Access prompt traces while using Amazon Bedrock, and send structured traces of third-party models to CloudWatch using ADOT SDK. For information about adding observability to your Amazon Bedrock AgentCore agent or tool, see Amazon Bedrock AgentCore

CloudWatch generative AI observability provides two pre-built capabilities:

Note

You can use the Model Invocation dashboard by using any models for inference in Amazon Bedrock.

  • Model Invocations – Detailed metrics dashboard on model usage, token consumption, and a curated invocation logs table to view detailed input and output content of model inferences

  • Amazon Bedrock AgentCore agents – Performance and decision metrics for primitives of Amazon Bedrock AgentCore such as Agents, Memory, Built-in Tools, Gateways, and Identity

Key metrics available in these dashboards include:

  • Total and average invocations

  • Token usage (total, average per query, input, output)

  • Latency (average, P90, P99)

  • Error rates and throttling events

  • Cost attribution by application, user role, or specific user

Topics