Buat evaluasi online

# Create online evaluation configuration
agentcore add online-eval \
  --name "your_config_name" \
  --runtime "your_runtime_name" \
  --evaluator "Builtin.GoalSuccessRate" "Builtin.Helpfulness" \
  --sampling-rate 1.0 \
  --enable-on-create

Perintah ini menambahkan konfigurasi evaluasi online ke lokal Andaagentcore.json. Jalankan agentcore deploy untuk membuatnya di AWS akun Anda.

Masukkan nama untuk konfigurasi evaluasi online Anda.

Pilih evaluator untuk disertakan. Anda dapat memilih dari evaluator bawaan dan evaluator kustom apa pun yang telah Anda buat.

Tetapkan laju pengambilan sampel — persentase permintaan agen yang akan dievaluasi.

Pilih apakah akan mengaktifkan evaluasi secara otomatis setelah penerapan.

Tinjau konfigurasi dan tekan Enter untuk mengonfirmasi.

from bedrock_agentcore_starter_toolkit import Evaluation

# Initialize the evaluation client
eval_client = Evaluation()

# Replace these with your actual values
config_name = "YOUR_CONFIG_NAME"
agent_id = "YOUR_AGENT_ID"  # e.g., agent_myagent-ABC123xyz

# Create online evaluation configuration
config = eval_client.create_online_config(
    config_name=config_name,                  # Must use underscores, not hyphens
    agent_id=agent_id,                        # Agent ID (e.g., agent_myagent-ABC123xyz)
    sampling_rate=1.0,                        # Percentage to evaluate (0-100). 1.0 = evaluate 1% of interactions
    evaluator_list=["Builtin.GoalSuccessRate", "Builtin.Helpfulness"],  # List of evaluator IDs
    config_description="Online Evaluation Config",  # Optional description
    auto_create_execution_role=True,          # Automatically creates IAM role (default: True)
    enable_on_create=True                     # Enable immediately after creation (default: True)
)

print("✅ Online evaluation configuration created!")
print(f"Config ID: {config['onlineEvaluationConfigId']}")
print(f"Status: {config['status']}")

# Save the config ID for later operations
config_id = config['onlineEvaluationConfigId']
print(f"\nSaved config_id: {config_id}")

import boto3

# Your input log group that contains agent traces
LOG_GROUP_NAME = "/aws/agentcore/test-agent-traces"

# The service.name attribute from Otel
SERVICE_NAME = "strands_healthcare_single_agent.DEFAULT"

# The role created earlier with the required permissions for eval
role_arn = f"arn:aws:iam::{ACCOUNT_ID}:role/AgentCoreEvaluationRole"

client = boto3.client('bedrock-agentcore-control')

create_config_response = client.create_online_evaluation_config(
    onlineEvaluationConfigName="strands_healthcare_agent_1",
    description="Continuous evaluation of a healthcare agent",
    rule={
        "samplingConfig": {"samplingPercentage": 80.0}
    },
    dataSourceConfig={
        "cloudWatchLogs": {
            "logGroupNames": [LOG_GROUP_NAME],
            "serviceNames": [SERVICE_NAME]
        }
    },
    evaluators=[{"evaluatorId":"Builtin.Helpfulness"}],
    evaluationExecutionRoleArn=role_arn,
    enableOnCreate=True
)

aws bedrock-agentcore-control create-online-evaluation-config \
    --online-evaluation-config-name "strands_healthcare_agent_1" \
    --description "Continuous evaluation of a healthcare agent" \
    --rule '{"samplingConfig": {"samplingPercentage": 80.0}}' \
    --data-source-config '{"cloudWatchLogs": {"logGroupNames": ["/aws/agentcore/test-agent-traces"], "serviceNames": ["strands_healthcare_single_agent.DEFAULT"]}}' \
    --evaluators '[{"evaluatorId": "Builtin.Helpfulness"}]' \
    --evaluation-execution-role-arn "arn:aws:iam::{YOUR_ACCOUNT_ID}:role/AgentCoreEvaluationRole" \
    --enable-on-create