feat(eval): read-only ab-test commands + TUI

[jariy17]

Adds the full read-only + write agentcore eval ab-test command family, plus the interactive TUI. Five commits.

Command + TUI structure

agentcore eval ab-test                      TUI menu shows get + list only
├── get     --id                            read  · JSON to stdout
├── list    --next-token --max-results      read  · JSON page to stdout
├── pause   --id                            write · executionStatus → PAUSED
├── resume  --id                            write · executionStatus → RUNNING
├── stop    --id                            write · executionStatus → STOPPED
└── delete  --id                            write · DeleteABTestCommand

How it works

A/B tests are imperative jobs (like batch-evaluation), not project resources. All commands are one-liners through the data-plane SDK client @aws-sdk/client-bedrock-agentcore:

• get/list → GetABTestCommand / ListABTestsCommand. get returns per-evaluator metrics inline (no CloudWatch).
• pause/resume/stop → UpdateABTestCommand({ abTestId, executionStatus }) — the same shape online-eval pause/resume uses.
• delete → DeleteABTestCommand.
• TUI: menu → picker → detail hub + JSON, modeled on runtime. Write commands are CLI-only and kept out of the interactive menu via .supportedTuiCommands("get","list") (matches online-eval).

Known / deferred

1. stop gateway-rule suggestion — after stopping, the doc calls for printing a suggested (never executed) update-gateway-rule to promote the winner. Left as a // TODO this PR; tracked as a follow-up.
2. --json error path — failures print plain-text Error: … to stderr, not a JSON error object. Shared-renderer issue, not ab-test-specific; follow-up.
3. Transition validity (pause-on-RUNNING, delete-on-RUNNING, etc.) is enforced server-side; the CLI forwards and surfaces the service error. No client-side state machine, matching online-eval.

Files

Area	Files
CLI handlers	handlers/eval/ab-test/{get,list,pause,resume,stop,delete}/index.tsx, index.tsx
TUI screens	handlers/eval/ab-test/{screen,list/screen,get/screen}.tsx, components/AbTestPicker.tsx, 5 routes in Root.tsx
Core	core/eval.tsx (getABTest/listABTests/setABTestExecutionStatus/deleteABTest), handlers/eval/types.tsx, testing/TestCoreClient.tsx

Tests

• Golden fixture (ab-test.fixture.test.tsx, like harness): real router → CoreClient → recorded SDK → matchGolden. get / list / not-found.
• Screen (ab-test.screen.test.tsx, like runtime): menu (read-only), picker, hub, json, retry.
• Command-flow (ab-test.write.test.tsx, like batch-evaluation): hierarchy = get/list/pause/resume/stop/delete; each transition's executionStatus; delete; required --id.

Bug bash — all green

• bun test src/handlers/eval src/components → 338+ pass / 0 fail; bun run typecheck → 0 errors; oxlint → clean.
• 4-agent /review-pr + a live non-destructive bug-bash on the exploratory account (us-west-2): all 6 commands dispatch correctly; write verbs return a clean not-found on a bogus id (no mutation); --id guard fires pre-network. One real bug found & fixed in this PR: the TUI menu listed write commands (dead-end) — now .supportedTuiCommands.

Follow-ups (separate PRs)

• stop's gateway-rule promotion suggestion
• structured --json error output (shared renderer)
• remove stale projectSchemas/ab-test.ts (contradicts the job model)

[agentcore-devx-automation]

Claude Security Review: no high-confidence findings. (run)

[codecov-commenter]

Codecov Report

❌ Patch coverage is 94.61279% with 16 lines in your changes missing coverage. Please review.
✅ Project coverage is 97.30%. Comparing base (343d213) to head (ba2e9cc).
⚠️ Report is 5 commits behind head on refactor.

Files with missing lines	Patch %	Lines
src/core/eval.tsx	38.46%	16 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##           refactor    #2102      +/-   ##
============================================
- Coverage     97.33%   97.30%   -0.04%     
============================================
  Files           417      428      +11     
  Lines         25250    25546     +296     
============================================
+ Hits          24578    24858     +280     
- Misses          672      688      +16     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[agentcore-devx-automation]

Claude Security Review: no high-confidence findings. (run)

[agentcore-devx-automation]

Claude Security Review: no high-confidence findings. (run)

[notgitika]

discussed offline and we see that this is the output from the API. i would imagine as customer would be confused seeing these 2 fields in the data table in TUI. we can try to unify it in the future maybe

[notgitika]

is the reason we are not using satisfies that we have treatmentname and not variantname? I see that in the fixture

[jariy17]

Good catch — switched to satisfies GetABTestResponse in c2e0c18. It immediately surfaced that the stub had treatmentName (the real metric field is variantName, matching the recorded fixture) and was missing Variant.variantConfiguration. Both fixed; the cast was hiding them.

[notgitika]

nice that we have these as just passthroughs :)

[agentcore-devx-automation]

Claude Security Review: no high-confidence findings. (run)

[agentcore-devx-automation]

Claude Security Review: no high-confidence findings. (run)

[agentcore-devx-automation]

Claude Security Review: no high-confidence findings. (run)