aws-cdk-lib.aws_bedrockagentcore.LlmAsAJudgeOptions

LlmAsAJudgeOptions Struct

Options for configuring an LLM-as-a-Judge custom evaluator.

Uses a foundation model to assess agent performance based on custom instructions and a rating scale.

Source

aws-bedrockagentcore/lib/evaluation/evaluator-config.ts

Type last changed

v2.255.0 · 19 May 2026

Example

// Create a custom LLM-as-a-Judge evaluator
const evaluator = new agentcore.Evaluator(this, 'MyEvaluator', {
  evaluatorName: 'my_custom_evaluator',
  level: agentcore.EvaluationLevel.SESSION,
  evaluatorConfig: agentcore.EvaluatorConfig.llmAsAJudge({
    instructions: 'Evaluate whether the agent response is helpful and accurate.',
    modelId: 'us.anthropic.claude-sonnet-4-6',
    ratingScale: agentcore.EvaluatorRatingScale.categorical([
      { label: 'Good', definition: 'The response is helpful and accurate.' },
      { label: 'Bad', definition: 'The response is not helpful or contains errors.' },
    ]),
  }),
});

// Use the custom evaluator in an online evaluation configuration
new agentcore.OnlineEvaluationConfig(this, 'MyEvaluation', {
  onlineEvaluationConfigName: 'my_evaluation',
  evaluators: [
    agentcore.EvaluatorSelector.builtin(agentcore.BuiltinEvaluator.HELPFULNESS),
    agentcore.EvaluatorSelector.custom(evaluator),
  ],
  dataSource: agentcore.DataSourceConfig.fromCloudWatchLogs({
    logGroupNames: ['/aws/bedrock-agentcore/my-agent'],
    serviceNames: ['my-agent.default'],
  }),
});

Properties

additionalModelRequestFields?optional{ [key: string]: any }additional_model_request_fields?optionaltyping.Mapping[str, Any]additionalModelRequestFields?optionalMap<String, Object>AdditionalModelRequestFields?optionalIDictionary<string, object>AdditionalModelRequestFields?optional*map[string]interface{}#

Additional model-specific request fields.

Default: No additional fields

inferenceConfig?optionalEvaluatorInferenceConfiginference_config?optionalEvaluatorInferenceConfiginferenceConfig?optionalEvaluatorInferenceConfigInferenceConfig?optionalEvaluatorInferenceConfigInferenceConfig?optionalEvaluatorInferenceConfig#

Optional inference configuration parameters that control model behavior during evaluation.

Default: The foundation model's default inference parameters are used

When not specified, the foundation model uses its own default values for maxTokens, temperature, and topP.

instructionsstringinstructionsstrinstructionsStringInstructionsstringInstructions*stringrequired#

The evaluation instructions that guide the language model in assessing agent performance.

notes

These instructions define the evaluation criteria, context, and expected behavior. Instructions must contain placeholders appropriate for the evaluation level (e.g., {context}, {available_tools} for SESSION level).

Note: Evaluators using reference-input placeholders (e.g., {expected_tool_trajectory}, {assertions}, {expected_response}) are only compatible with on-demand evaluation, not online evaluation.

modelIdstringmodel_idstrmodelIdStringModelIdstringModelId*stringrequired#

The identifier of the Amazon Bedrock model to use for evaluation.

Accepts standard model IDs (e.g., 'anthropic.claude-sonnet-4-6') and cross-region inference profile IDs with region prefixes (e.g., 'us.anthropic.claude-sonnet-4-6', 'eu.anthropic.claude-sonnet-4-6').

ratingScaleEvaluatorRatingScalerating_scaleEvaluatorRatingScaleratingScaleEvaluatorRatingScaleRatingScaleEvaluatorRatingScaleRatingScaleEvaluatorRatingScalerequired#

The rating scale that defines how the evaluator should score agent performance.