Request Syntax URI Request Parameters Request Body Response Syntax Response Elements Errors See Also

Evaluate

Performs on-demand evaluation of agent traces using a specified evaluator. This synchronous API accepts traces in OpenTelemetry format and returns immediate scoring results with detailed explanations.

Request Syntax


POST /evaluations/evaluate/evaluatorId HTTP/1.1
Content-type: application/json

{
   "evaluationInput": { ... },
   "evaluationTarget": { ... }
}

URI Request Parameters

The request uses the following URI parameters.

evaluatorId

The unique identifier of the evaluator to use for scoring. Can be a built-in evaluator (e.g., Builtin.Helpfulness, Builtin.Correctness) or a custom evaluator ARN created through the control plane API.

Pattern: (Builtin.[a-zA-Z0-9_-]+|[a-zA-Z][a-zA-Z0-9-_]{0,99}-[a-zA-Z0-9]{10})

Required: Yes

Request Body

The request accepts the following data in JSON format.

evaluationInput

The input data containing agent session spans to be evaluated. Includes a list of spans in OpenTelemetry format from supported frameworks like Strands (AgentCore Runtime) or LangGraph with OpenInference instrumentation.

Type: EvaluationInput object

Note: This object is a Union. Only one member of this object can be specified or returned.

Required: Yes

evaluationTarget

The specific trace or span IDs to evaluate within the provided input. Allows targeting evaluation at different levels: individual tool calls, single request-response interactions (traces), or entire conversation sessions.

Type: EvaluationTarget object

Note: This object is a Union. Only one member of this object can be specified or returned.

Required: No

Response Syntax


HTTP/1.1 200
Content-type: application/json

{
   "evaluationResults": [ 
      { 
         "context": { ... },
         "errorCode": "string",
         "errorMessage": "string",
         "evaluatorArn": "string",
         "evaluatorId": "string",
         "evaluatorName": "string",
         "explanation": "string",
         "label": "string",
         "tokenUsage": { 
            "inputTokens": number,
            "outputTokens": number,
            "totalTokens": number
         },
         "value": number
      }
   ]
}

Response Elements

If the action is successful, the service sends back an HTTP 200 response.

The following data is returned in JSON format by the service.

evaluationResults

The detailed evaluation results containing scores, explanations, and metadata. Includes the evaluator information, numerical or categorical ratings based on the evaluator's rating scale, and token usage statistics for the evaluation process.

Type: Array of EvaluationResultContent objects

Errors

For information about the errors that are common to all actions, see Common Errors.

AccessDeniedException

The exception that occurs when you do not have sufficient permissions to perform an action. Verify that your IAM policy includes the necessary permissions for the operation you are trying to perform.

HTTP Status Code: 403

ConflictException

The exception that occurs when the request conflicts with the current state of the resource. This can happen when trying to modify a resource that is currently being modified by another request, or when trying to create a resource that already exists.

HTTP Status Code: 409

DuplicateIdException

An exception thrown when attempting to create a resource with an identifier that already exists.

HTTP Status Code: 409

InternalServerException

The exception that occurs when the service encounters an unexpected internal error. This is a temporary condition that will resolve itself with retries. We recommend implementing exponential backoff retry logic in your application.

HTTP Status Code: 500

ResourceNotFoundException

The exception that occurs when the specified resource does not exist. This can happen when using an invalid identifier or when trying to access a resource that has been deleted.

HTTP Status Code: 404

ServiceQuotaExceededException

The exception that occurs when the request would cause a service quota to be exceeded. Review your service quotas and either reduce your request rate or request a quota increase.

HTTP Status Code: 402

ThrottlingException

The exception that occurs when the request was denied due to request throttling. This happens when you exceed the allowed request rate for an operation. Reduce the frequency of requests or implement exponential backoff retry logic in your application.

HTTP Status Code: 429

UnauthorizedException

This exception is thrown when the JWT bearer token is invalid or not found for OAuth bearer token based access

HTTP Status Code: 401

ValidationException

The exception that occurs when the input fails to satisfy the constraints specified by the service. Check the error message for details about which input parameter is invalid and correct your request.

HTTP Status Code: 400