Evaluator

class aws_cdk.aws_bedrock_agentcore_alpha.Evaluator(scope, id, *, evaluator_config, evaluator_name, level, description=None, tags=None)

Bases: EvaluatorBase

(experimental) A custom evaluator for Amazon Bedrock AgentCore.

Custom evaluators enable you to define evaluation logic tailored to your specific use cases. Supports two evaluation strategies:

LLM-as-a-Judge: Uses a foundation model with custom instructions and a rating scale.
Code-based: Uses a Lambda function for custom evaluation logic.

Custom evaluators are used with OnlineEvaluationConfig via EvaluatorReference.custom().

Stability:: experimental
Resource:: AWS::BedrockAgentCore::Evaluator

Example:

# Create a custom LLM-as-a-Judge evaluator
evaluator = agentcore.Evaluator(self, "MyEvaluator",
    evaluator_name="my_custom_evaluator",
    level=agentcore.EvaluationLevel.SESSION,
    evaluator_config=agentcore.EvaluatorConfig.llm_as_aJudge(
        instructions="Evaluate whether the agent response is helpful and accurate.",
        model_id="us.anthropic.claude-sonnet-4-6",
        rating_scale=agentcore.EvaluatorRatingScale.categorical([label="Good", definition="The response is helpful and accurate.", label="Bad", definition="The response is not helpful or contains errors."
        ])
    )
)

# Use the custom evaluator in an online evaluation configuration
agentcore.OnlineEvaluationConfig(self, "MyEvaluation",
    online_evaluation_config_name="my_evaluation",
    evaluators=[
        agentcore.EvaluatorReference.builtin(agentcore.BuiltinEvaluator.HELPFULNESS),
        agentcore.EvaluatorReference.custom(evaluator)
    ],
    data_source=agentcore.DataSourceConfig.from_cloud_watch_logs(
        log_group_names=["/aws/bedrock-agentcore/my-agent"],
        service_names=["my-agent.default"]
    )
)

Parameters:

scope (Construct)
id (str)
evaluator_config (EvaluatorConfig) – (experimental) The configuration that defines how the evaluator assesses agent performance. Use EvaluatorConfig.llmAsAJudge() for model-based evaluation or EvaluatorConfig.codeBased() for Lambda-based evaluation.
evaluator_name (str) – (experimental) The name of the evaluator. Must be unique within your account. Valid characters are a-z, A-Z, 0-9, _ (underscore). Must start with a letter and can be up to 48 characters long.
level (EvaluationLevel) – (experimental) The level at which the evaluator assesses agent performance. Determines what granularity of data the evaluator operates on: tool call, trace (single request-response), or session (full conversation).
description (Optional[str]) – (experimental) The description of the evaluator. Default: - No description
tags (Optional[Mapping[str, str]]) – (experimental) Tags for the evaluator. A list of key:value pairs of tags to apply to this Evaluator resource. Default: - No tags

Stability:

experimental

Methods

apply_removal_policy(policy)

Apply the given removal policy to this resource.

The Removal Policy controls what happens to this resource when it stops being managed by CloudFormation, either because you’ve removed it from the CDK application or because you’ve made a change that requires the resource to be replaced.

The resource can be deleted (RemovalPolicy.DESTROY), or left in your AWS account for data recovery and cleanup later (RemovalPolicy.RETAIN).

Parameters:: policy (RemovalPolicy)
Return type:: None

grant(grantee, *actions)

(experimental) Grants IAM actions to the IAM Principal.

[disable-awslint:no-grants]

Parameters:

grantee (IGrantable) –
- The IAM principal to grant permissions to.
actions (str) –
- The actions to grant.

Return type:

Grant

Returns:

An IAM Grant object representing the granted permissions

Stability:

experimental

to_string()

Returns a string representation of this construct.

Return type:: str

with_(*mixins)

Applies one or more mixins to this construct.

Mixins are applied in order. The list of constructs is captured at the start of the call, so constructs added by a mixin will not be visited. Use multiple with() calls if subsequent mixins should apply to added constructs.

Parameters:: mixins (IMixin)
Return type:: IConstruct

Attributes

PROPERTY_INJECTION_ID = '@aws-cdk.aws-bedrock-agentcore-alpha.Evaluator'

created_at

(experimental) The timestamp when the evaluator was created.

Stability:: experimental
Attribute:: true

env

The environment this resource belongs to.

For resources that are created and managed in a Stack (those created by creating new class instances like new Role(), new Bucket(), etc.), this is always the same as the environment of the stack they belong to.

For referenced resources (those obtained from referencing methods like Role.fromRoleArn(), Bucket.fromBucketName(), etc.), they might be different than the stack they were imported into.

evaluator_arn

(experimental) The ARN of the evaluator.

Stability:: experimental
Attribute:: true

evaluator_id

(experimental) The unique identifier of the evaluator.

Stability:: experimental
Attribute:: true

evaluator_name

(experimental) The name of the evaluator.

Stability:: experimental
Attribute:: true

evaluator_ref

(experimental) A reference to this Evaluator resource.

Stability:: experimental

node: The tree node.

stack: The stack in which this resource is defined.

status

(experimental) The lifecycle status of the evaluator.

Stability:: experimental
Attribute:: true

updated_at

(experimental) The timestamp when the evaluator was last updated.

Stability:: experimental
Attribute:: true

Static Methods

classmethod from_evaluator_arn(scope, id, evaluator_arn)

(experimental) Import an existing Evaluator by its ARN.

Parameters:

scope (Construct) –
- The construct scope.
id (str) –
- Construct identifier.
evaluator_arn (str) –
- The evaluator ARN to import.

Return type:

IEvaluator

Returns:

An IEvaluator reference

Stability:

experimental

classmethod from_evaluator_attributes(scope, id, *, evaluator_arn, evaluator_id, evaluator_name=None)

(experimental) Import an existing Evaluator from its attributes.

Parameters:

scope (Construct) –
- The construct scope.
id (str) –
- Construct identifier.
evaluator_arn (str) – (experimental) The ARN of the evaluator.
evaluator_id (str) – (experimental) The ID of the evaluator.
evaluator_name (Optional[str]) – (experimental) The name of the evaluator. Default: - No name available

Return type:

IEvaluator

Returns:

An IEvaluator reference

Stability:

experimental

classmethod from_evaluator_id(scope, id, evaluator_id)

(experimental) Import an existing Evaluator by its ID.

Parameters:

scope (Construct) –
- The construct scope.
id (str) –
- Construct identifier.
evaluator_id (str) –
- The evaluator ID to import.

Return type:

IEvaluator

Returns:

An IEvaluator reference

Stability:

experimental

classmethod is_construct(x)

Checks if x is a construct.

Use this method instead of instanceof to properly detect Construct instances, even when the construct library is symlinked.

Explanation: in JavaScript, multiple copies of the constructs library on disk are seen as independent, completely different libraries. As a consequence, the class Construct in each copy of the constructs library is seen as a different class, and an instance of one class will not test as instanceof the other class. npm install will not create installations like this, but users may manually symlink construct libraries together or use a monorepo tool: in those cases, multiple copies of the constructs library can be accidentally installed, and instanceof will behave unpredictably. It is safest to avoid using instanceof, and using this type-testing method instead.

Parameters:: x (Any) – Any object.
Return type:: bool
Returns:: true if x is an object created from a class which extends Construct.

classmethod is_owned_resource(construct)

Returns true if the construct was created by CDK, and false otherwise.

Parameters:: construct (IConstruct)
Return type:: bool

classmethod is_resource(construct)

Check whether the given construct is a Resource.

Parameters:: construct (IConstruct)
Return type:: bool