feat(eval): ab-test target-based run (create)

[jariy17]

Stacked on #2113 (config-bundle run). Adds the second A/B-test create mode.

Command structure

agentcore eval ab-test target-based run
  -n, --name <name>              required   A/B test name
  -g, --gateway <id>             required   deployed gateway id
  --control <json>               required   {"gateway-target","online-eval"}  (inline · file:// · -)
  --treatment <json>             required   {"gateway-target","online-eval"}  (inline · file:// · -)
  --treatment-weight <1-99>                 control weight = 100 - this (default 50)
  --gateway-filter <json>                   GatewayFilter {"targetPaths":[...]}  (inline · file:// · -)
  --role-arn <arn>                          execution-role override (else auto-provisioned)
  --enable-on-create <true|false>           start immediately (default true; false = create PAUSED)

Splits gateway traffic between two gateway targets, each scored by its own online-eval. The target picks the runtime + endpoint, so those aren't passed separately.

Actual input → output

Input (exploratory account, us-west-2):

agentcore eval ab-test target-based run \
  --name orders-v2-canary \
  --gateway orders-gateway-demo-000 \
  --control '{"gateway-target":"orders-prod-target","online-eval":"orders-prod-quality"}' \
  --treatment '{"gateway-target":"orders-v2-target","online-eval":"orders-v2-quality"}' \
  --treatment-weight 20

Actual output — resolves + validates the gateway before any create, so an unreachable gateway fails fast, creating nothing:

Error: User: …:assumed-role/Admin/… is not authorized to perform: bedrock-agentcore:GetGateway

Output on success (shape from CreateABTest):

{
  "abTestId": "orders-v2-canary-abc123",
  "abTestArn": "arn:aws:bedrock-agentcore:us-west-2:…:ab-test/orders-v2-canary-abc123",
  "name": "orders-v2-canary",
  "status": "CREATING",
  "executionStatus": "RUNNING"
}

--enable-on-create false → NOT_STARTED (created PAUSED).

How it works

• Variants use variantConfiguration.target = { name } (vs config-bundle's configurationBundle); evaluation is perVariantOnlineEvaluationConfig: [{ name: "C", … }, { name: "T1", … }] — one online-eval per variant.
• Shared create seam: extracted EvalClient.createABTest (GetGateway → account, provision role, retryWhileRolePropagates on assume/AccessDenied, rollback on failure). Both config-bundle and target-based run now go through it — removes the duplicated role/retry block.
• --control/--treatment/--gateway-filter via SourceResolver (inline / file:// / -), same as config-bundle run.
• Validation is server-side (gateway READY, targets belong to it, each online-eval matches its target's endpoint) — the CLI resolves ids → the request and surfaces the service's 4xx.

Tests

Consolidated ab-test.test.tsx gains target-based coverage: hierarchy (ab-test → target-based → run), required-flag matrix, mis-shaped JSON, identical-target rejection, and flag→request mapping (C/T1 targets, per-variant evals, weights). 52 ab-test tests pass; typecheck 0; oxlint clean.

Deferred

• Golden fixture — recording needs a gateway wired to two runtime-backed targets + two online-evals (heavier than config-bundle's self-created bundle). The shared createABTest seam is already golden-covered by feat(eval): ab-test config-bundle run (create) #2113's config-bundle run, so target-based rides the same recorded path. Follow-up.
• TUI wizard (the doc's interactive target-based flow) — follow-up.

Review

Base = feat/eval-ab-test-config-bundle-run (stack). Review after #2113 merges, or view just this branch's diff.

[agentcore-devx-automation]

AgentCore Harness Review

Verdict: Looks good

Reviewed the top commit's diff against the stack base (feat/eval-ab-test-config-bundle-run): the new eval ab-test target-based run handler, the createTargetBasedABTest core method, the shared createABTest refactor, and the consolidated tests.

Findings:

• The extraction of EvalClient.createABTest preserves the pre-existing semantics (GetGateway → account, role provisioning, retryWhileRolePropagates gated on auto-provisioned role, rollback via deleteAbTestRole). createConfigBundleABTest now flows through it and produces the same request shape as before — no behavioral drift.
• createTargetBasedABTest correctly maps to variantConfiguration.target and evaluationConfig.perVariantOnlineEvaluationConfig (C/T1 keyed to variant names), with the same clientToken, gatewayFilter, and enableOnCreate defaults as the config-bundle path.
• Handler validation (required flags, strict {gateway-target, online-eval} schema, identical-target rejection, weight bounds 1–99, enable-on-create enum) is symmetric with config-bundle and reasonable. SourceResolver + parseJsonFlag usage matches the established pattern.
• Tests use the real TestCoreClient fake rather than mocks, exercise routing hierarchy, required-flag matrix, mis-shaped JSON, identical-target rejection, and flag→request mapping. No excessive mocking.
• Telemetry omission is consistent with the sibling eval/ab-test handlers, so no new instrumentation is expected here.
• Deferred golden fixture (per PR description) is reasonable given the shared createABTest seam is already golden-covered by #2113.

Nothing blocking — safe to merge once the stack base (#2113) lands.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 79.81651% with 44 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (feat/eval-ab-test-config-bundle-run@0f31b84). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
src/core/eval.tsx	59.43%	43 Missing ⚠️
...c/handlers/eval/ab-test/target-based/run/index.tsx	99.02%	1 Missing ⚠️

Additional details and impacted files

@@                          Coverage Diff                           @@
##             feat/eval-ab-test-config-bundle-run    #2135   +/-   ##
======================================================================
  Coverage                                       ?   97.10%           
======================================================================
  Files                                          ?      477           
  Lines                                          ?    29513           
  Branches                                       ?        0           
======================================================================
  Hits                                           ?    28658           
  Misses                                         ?      855           
  Partials                                       ?        0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[nborges-aws]

One clarifying question on target ID/name passing. Otherwise PR looks clean!

[nborges-aws]

Is target expecting a name or id here? src/handlers/eval/ab-test/target-based/run/index.tsx#L21 documents gateway-target as <id>; I'd expect input.control.gatewayTarget is then the target ID. But this field looks to expect target name?

[nborges-aws]

same question as above