Generate eval cases from observed production traces (trace-replay-to-yaml)

[nabinpkl]

Status: deferred. Filed to capture the idea so we don't lose it; no immediate plan to land. Supersedes the originally-scoped #28 (synthetic generate-from-description CLI), which represents an earlier-generation pattern.

Why

Anthropic's Demystifying evals for AI agents (2026-01-09) advocates generating eval cases from observed production behavior rather than from synthetic descriptions. The frontier framing: when a real trace surfaces an interesting failure (a fabrication that slipped through, a tool fired on a refusal turn, a gate verdict that disagreed with intuition), one command should turn that trace into a permanent eval case so the same shape can never regress unnoticed.

The synthetic-from-description pattern (#28's original scope) saves keystrokes but not judgment. Generated cases need hand-tuning before their probes become sharp. Trace-replay is fundamentally different: the trace IS the case, the probes are derived from "what was true on this trace today should still be true tomorrow."

Scope

• New CLI: python -m agent_service.evals.case_from_trace --trace-id <hex> --suite <name> --case-id <slug> --out <path>
• Reads the named trace from otel.otel_traces:
  • inputs reconstructed from the AgentRequest captured at turn start (we already record mcae.request.user_question and the focus context as span attributes; verify completeness or add what's missing).
  • Probes auto-derived from observed invariants:
    • turn-root-span-present, turn-tagged-as-eval (always).
    • tool_called_with_args for each tool span observed.
    • turn_attribute_equals for tool_calls, claims_emitted, claims_approved from the observed turn root attrs.
    • gate_passed for each gate span with verdict=approved observed.
    • no_span_with_status(mcae.turn, error).
• Writes a YAML case dict; refuses to overwrite existing files unless --force.
• Run produces a freshly-baselined case in 1-2 commands: case_from_trace ... && just eval-baseline ....

Out of scope

• Inverse direction (synthetic generate-from-description). Closed via Add generate_cases CLI for LLM-authored YAML expansion #28.
• Iterative refinement (run → judge → regenerate). Different workflow.
• Multi-trace differential cases (gated vs ungated trajectory comparison). That's the cross-trace probe work blocked by Behavioral evaluation across switch combinations to separate intrinsic from gated behavior #29.
• Sensitive-data redaction. Production traces may contain PII or wallet addresses we don't want in committed eval files; needs a separate redaction pass before this CLI is safe on real production data. Today's traces are dev/eval-only, so deferred until production wiring earns it.

Done when

• Command takes a trace id and emits a valid EvalCase YAML.
• The emitted case loads via load_suite() and runs against the live agent.
• Re-running the case against a fresh trace of the same input passes baseline diff (proving the auto-derived probes encode stable invariants, not noise).
• README documents the workflow alongside the existing manual case authoring.

Refs

• Anthropic, Demystifying evals for AI agents (2026-01-09) for the framing.
• Add generate_cases CLI for LLM-authored YAML expansion #28 (closed as superseded) for the rejected synthetic-generation predecessor.
• Ship 2 (epic): agent eval substrate, layered with framework on top per ADR 14 #20 (epic, eval substrate).
• Behavioral evaluation across switch combinations to separate intrinsic from gated behavior #29 (related: switch-matrix behavior suites).