Run an A/B test with target-based routing

Use the target-based routing pattern when the change you are testing involves code changes, a framework upgrade, or an entirely different agent implementation. Target-based routing routes traffic between multiple versions of the same AgentCore Runtime (named endpoints), or between entirely different AgentCore Runtimes. The AgentCore Gateway registers each endpoint as a separate target and routes each session to one endpoint or the other based on the A/B test’s traffic weights.

Key configuration for target-based A/B tests:

This walkthrough deploys two versions of the customer support agent — one using Claude Sonnet (control) and one using Claude Opus (treatment) — creates named endpoints for each version, creates an A/B test, sends traffic, reviews results, and deploys the winner.

For a detailed comparison of A/B test patterns, see Choosing a pattern.

Step 1: Create the project

Create the project with the AgentCore CLI:

agentcore create --name ABTestTargetBased --no-agent
cd ABTestTargetBased

Step 2: Add the runtime

Add the agent runtime. You will deploy two versions of this runtime — one for control and one for treatment — then create named endpoints to alias each version.

agentcore add agent \
  --name csAgent \
  --language Python \
  --framework Strands \
  --model-provider Bedrock \
  --memory none \
  --build CodeZip

Project structure:

ABTestTargetBased/
├── agentcore/
│   ├── agentcore.json
│   ├── aws-targets.json
│   └── cdk/
└── app/
    └── csAgent/
        ├── main.py
        └── pyproject.toml

Step 3: Deploy control and treatment versions

Replace app/csAgent/main.py with the control version (using Claude Sonnet):

"""Customer support agent — control variant."""
from strands import Agent, tool
from strands.models.bedrock import BedrockModel
from bedrock_agentcore.runtime import BedrockAgentCoreApp

app = BedrockAgentCoreApp()
MODEL_ID = "global.anthropic.claude-sonnet-4-5-20250929-v1:0"
SYSTEM_PROMPT = "You are a helpful customer support assistant for Acme Store."


@tool
def lookup_order(order_id: str) -> str:
    """Look up an order by ID."""
    orders = {
        "ORD-1001": {"status": "delivered", "item": "Blue T-Shirt", "total": "$29.99"},
        "ORD-1002": {"status": "in_transit", "item": "Running Shoes", "est_delivery": "2026-04-05"},
        "ORD-1003": {"status": "delayed", "item": "Wireless Headphones", "days_late": 5},
    }
    return str(orders.get(order_id, {"error": f"Order {order_id} not found"}))


@tool
def initiate_return(order_id: str, reason: str) -> str:
    """Initiate a return for an order."""
    return f"Return initiated for {order_id}. Reason: {reason}. Return label sent to customer email."


@tool
def apply_discount(order_id: str, discount_percent: int, reason: str) -> str:
    """Apply a discount to an order."""
    return f"Applied {discount_percent}% discount to {order_id}. Reason: {reason}."


agent = Agent(
    model=BedrockModel(model_id=MODEL_ID),
    tools=[lookup_order, initiate_return, apply_discount],
    system_prompt=SYSTEM_PROMPT,
)


@app.entrypoint
def invoke(payload, context):
    result = agent(payload.get("prompt", "Hello"))
    return {"response": result.message["content"][0]["text"]}


if __name__ == "__main__":
    app.run()

Update app/csAgent/pyproject.toml dependencies:

dependencies = [
    "aws-opentelemetry-distro",
    "bedrock-agentcore >= 1.8.0",
    "boto3",
    "botocore[crt] >= 1.35.0",
    "strands-agents[otel] >= 1.13.0",
    "opentelemetry-distro",
    "opentelemetry-instrumentation",
]

Deploy the control version (this creates version 1):

agentcore deploy

Now update main.py to use a different model for the treatment variant and deploy (this creates version 2):

MODEL_ID = "global.anthropic.claude-opus-4-6-v1"

agentcore deploy

Create named endpoints for each version and deploy:

agentcore add runtime-endpoint \
  --runtime csAgent \
  --endpoint control \
  --version 1 \
  --description "Control variant — Claude Sonnet"

agentcore add runtime-endpoint \
  --runtime csAgent \
  --endpoint treatment \
  --version 2 \
  --description "Treatment variant — Claude Opus"

agentcore deploy

You now have:

Verify the runtime is working:

agentcore invoke --runtime csAgent --prompt "What is the status of order ORD-1003?"

You now have:

Step 4: Create online evaluation configurations

Each endpoint has its own log group (the log group name ends with the endpoint name), so you need one online evaluation config per variant:

agentcore add online-eval \
  --name controlEvalTb \
  --runtime csAgent \
  --endpoint control \
  --evaluator "Builtin.Helpfulness" \
  --sampling-rate 100.0 \
  --enable-on-create

agentcore add online-eval \
  --name treatmentEvalTb \
  --runtime csAgent \
  --endpoint treatment \
  --evaluator "Builtin.Helpfulness" \
  --sampling-rate 100.0 \
  --enable-on-create

agentcore deploy

After each deployment, note the online evaluation config ARN — you will need both when creating the A/B test.

For more details on evaluator options and configuration, see Create online evaluation.

Step 5: Create the gateway and targets

A target-based A/B test routes traffic through an AgentCore Gateway, so the gateway and its two targets must already be deployed before you start the test. Add a gateway and register each runtime endpoint as an http-runtime target, then deploy:

agentcore add gateway --name csGateway

agentcore add gateway-target \
  --name customer-support-control \
  --gateway csGateway \
  --type http-runtime \
  --runtime csAgent \
  --runtime-endpoint control

agentcore add gateway-target \
  --name customer-support-treatment \
  --gateway csGateway \
  --type http-runtime \
  --runtime csAgent \
  --runtime-endpoint treatment

agentcore deploy

Step 6: Create the A/B test

Start the A/B test with agentcore run ab-test. Each variant references one of the gateway targets you created and has its own online evaluation config. The command initiates the test directly on the service against the already-deployed gateway.

Step 7: Send traffic through the AgentCore Gateway

After the A/B test is running, send traffic through the AgentCore Gateway HTTP endpoint. The AgentCore Gateway assigns each request to a variant (control or treatment) based on the runtime session ID.

How variant assignment works

The AgentCore Gateway uses the X-Amzn-Bedrock-AgentCore-Runtime-Session-Id header to determine which target to route traffic to. This header is optional — if you do not provide it, the runtime generates a session ID automatically. The AgentCore Gateway then uses the session ID (whether you provided it or the runtime generated it) to assign the request to a variant based on your configured traffic weights.

Session assignment is sticky: once a session ID is assigned to a variant, all subsequent requests with that same session ID route to the same target. This ensures a consistent experience within a session while still distributing new sessions across variants according to your traffic split.

Generate traffic for testing

Save the following script as loadgen.sh, replacing <gateway-id> and <target-name> with the values from your deployment output. You can also copy the full invocation URL from agentcore view ab-test <ab-test-id>:

#!/bin/bash
export AWS_ACCESS_KEY_ID=$(aws configure get aws_access_key_id)
export AWS_SECRET_ACCESS_KEY=$(aws configure get aws_secret_access_key)
export AWS_SESSION_TOKEN=$(aws configure get aws_session_token)

GATEWAY_URL="https://<gateway-id>.gateway.bedrock-agentcore.us-west-2.amazonaws.com/<target-name>/invocations"

PROMPTS=(
  "What is the status of order ORD-1003?"
  "I want to return order ORD-1001, it doesn't fit."
  "My order ORD-1003 is late. Can I get a discount?"
  "Where is my order ORD-1002?"
  "I need help with a return for order ORD-1001. The color is wrong."
  "Can you check on order ORD-1003? I've been waiting forever."
  "I'd like to cancel order ORD-1002 if it hasn't shipped yet."
  "Order ORD-1003 is delayed again. This is unacceptable."
  "What's your return policy for order ORD-1001?"
  "My headphones order ORD-1003 still hasn't arrived. What can you do?"
)

for i in $(seq 1 30); do
  PROMPT="${PROMPTS[$(( (i - 1) % ${#PROMPTS[@]} ))]}"
  echo "=== Request $i: $PROMPT ==="
  curl -s --aws-sigv4 "aws:amz:us-west-2:bedrock-agentcore" \
    --user "$AWS_ACCESS_KEY_ID:$AWS_SECRET_ACCESS_KEY" \
    -H "x-amz-security-token: $AWS_SESSION_TOKEN" \
    -H "Content-Type: application/json" \
    -H "X-Amzn-Bedrock-AgentCore-Runtime-Session-Id: $(uuidgen)" \
    -d "{\"prompt\": \"$PROMPT\"}" \
    -X POST \
    "$GATEWAY_URL"
  echo ""
  sleep 2
done

Run the script:

bash loadgen.sh

Step 8: Get results

Poll the A/B test to monitor results as sample sizes grow. Polling does not affect statistical validity.

Interpreting results

Step 9: Confirm results and stop the A/B test

Once the A/B test reaches statistical significance, review the results and stop the experiment.

Step 10: Deploy the winner

After stopping the A/B test, route all traffic to the winning variant.

agentcore promote ab-test -i <ab-test-id>
agentcore deploy

promote stops the A/B test (if still running), updates the control endpoint to point to the treatment version (for example, updating control from version 1 to version 2), and removes the treatment endpoint. Run agentcore deploy to apply the changes.

Alternatively, you can manually deploy the winner by doing one of the following:

Next steps

After deploying the winner:

Understanding results

When you call GetABTest, the response includes a results object once the aggregation pipeline has processed enough sessions. The results contain per-evaluator metrics broken down by variant.

Results structure

{
  "results": {
    "analysisTimestamp": "2026-04-30T18:45:00Z",
    "evaluatorMetrics": [
      {
        "evaluatorArn": "arn:aws:bedrock-agentcore:us-west-2:123456789012:evaluator/Builtin.Helpfulness",
        "controlStats": {
          "variantName": "C",
          "sampleSize": 24,
          "mean": 0.72
        },
        "variantResults": [
          {
            "variantName": "T1",
            "sampleSize": 6,
            "mean": 0.85,
            "absoluteChange": 0.13,
            "percentChange": 18.1,
            "pValue": 0.032,
            "confidenceInterval": {
              "lower": 0.02,
              "upper": 0.24
            },
            "isSignificant": true
          }
        ]
      }
    ]
  }
}

Field reference

Troubleshooting

A/B test shows no results after sending traffic

Results do not appear immediately. The time it takes depends on the session timeout configured in your online evaluation config — a session is considered complete only after no new requests arrive within the timeout window. After a session ends, expect results within approximately 15 minutes.

If results still do not appear after this window: