EvaluatorProps Struct
Properties for creating an Evaluator.
Example
ts
// Create a custom LLM-as-a-Judge evaluator
const evaluator = new agentcore.Evaluator(this, 'MyEvaluator', {
evaluatorName: 'my_custom_evaluator',
level: agentcore.EvaluationLevel.SESSION,
evaluatorConfig: agentcore.EvaluatorConfig.llmAsAJudge({
instructions: 'Evaluate whether the agent response is helpful and accurate.',
modelId: 'us.anthropic.claude-sonnet-4-6',
ratingScale: agentcore.EvaluatorRatingScale.categorical([
{ label: 'Good', definition: 'The response is helpful and accurate.' },
{ label: 'Bad', definition: 'The response is not helpful or contains errors.' },
]),
}),
});
// Use the custom evaluator in an online evaluation configuration
new agentcore.OnlineEvaluationConfig(this, 'MyEvaluation', {
onlineEvaluationConfigName: 'my_evaluation',
evaluators: [
agentcore.EvaluatorSelector.builtin(agentcore.BuiltinEvaluator.HELPFULNESS),
agentcore.EvaluatorSelector.custom(evaluator),
],
dataSource: agentcore.DataSourceConfig.fromCloudWatchLogs({
logGroupNames: ['/aws/bedrock-agentcore/my-agent'],
serviceNames: ['my-agent.default'],
}),
});
Properties
descriptionstringdescriptionstrdescriptionStringDescriptionstringDescription*string= No description#The description of the evaluator.
evaluatorConfigEvaluatorConfigevaluator_configEvaluatorConfigevaluatorConfigEvaluatorConfigEvaluatorConfigEvaluatorConfigEvaluatorConfigEvaluatorConfigrequired#The configuration that defines how the evaluator assesses agent performance.
Use EvaluatorConfig.llmAsAJudge() for model-based evaluation or EvaluatorConfig.codeBased() for Lambda-based evaluation.
evaluatorNamestringevaluator_namestrevaluatorNameStringEvaluatorNamestringEvaluatorName*stringrequired#The name of the evaluator.
Must be unique within your account. Valid characters are a-z, A-Z, 0-9, _ (underscore). Must start with a letter and can be up to 48 characters long.
levelEvaluationLevellevelEvaluationLevellevelEvaluationLevelLevelEvaluationLevelLevelEvaluationLevelrequired#The level at which the evaluator assesses agent performance.
Determines what granularity of data the evaluator operates on: tool call, trace (single request-response), or session (full conversation).
tags{ [key: string]: string }tagstyping.Mapping[str, str]tagsMap<String, String>TagsIDictionary<string, string>Tags*map[string]*string= No tags#Tags for the evaluator.
A list of key:value pairs of tags to apply to this Evaluator resource.