Ship 2 (epic): agent eval substrate, layered with framework on top per ADR 14

[nabinpkl]

Summary

Build the agent eval substrate so just eval evals/cases/<suite>.yaml runs a real suite end-to-end against the live stack: hits /agent/ask, captures trace_ids, runs probes against otel.otel_traces, persists per-probe JSON results + a summary, and prints pass/fail counts (with a regression diff against committed baselines).

Architectural decision recorded in ADR 14. Foundation comes from ADR 13 (OTel + Langfuse) plus Ship 1.5 (mcae.* namespace + enriched gate/claim/primitive spans, commits 1177c27 and 6d81e6b).

The four invariants this ship protects

1. A case is data, not code (YAML, loadable into a typed schema, stable IDs).
2. A probe is a predicate over an OTel trace (pure function, no framework imports).
3. A probe result is a structured artifact (JSON, queryable in ClickHouse, schema independent of any tool).
4. The agent under test is invoked exactly the way production invokes it (/agent/ask over HTTP, real OTel pipeline, real mcae.* and gen_ai.* spans).

The four-layer stack

Layer 4: Framework adapter (THIN, SWAPPABLE)  ~80 LOC, pydantic_evals seam
Layer 3: Runner (FRAMEWORK-AGNOSTIC)          ~50 LOC, orchestration
Layer 2: Probes (PURE FUNCTIONS, OURS)        ~150 LOC, 5-7 probe kinds
Layer 1: Schema (CANONICAL, OURS)             ~80 LOC, pydantic models

Lower layers know nothing about higher ones. Framework swap touches only Layer 4.

Probe vocabulary (initial)

• has_matching_span(span_name, attrs)
• tool_called_with_args(tool_name, arg_predicates) — reads gen_ai.tool.name + mcae.primitive.input
• claim_grounded_in(source_kind) — reads mcae.claim.source_kind, anchor for future sql_explore hybrid
• gate_passed(kind, version?) — reads mcae.gate.verdict + mcae.gate.version
• span_latency_p50_under(span_name, ms) — aggregates OTel Duration
• no_span_with_status(span_name, status)
• llm_call_used_model(model_name) — reads gen_ai.request.model

New probe kinds extend the schema's Literal and add a file in agent_service/evals/probes/.

Sub-tickets (sequential, each lands as its own commit)

• Ship 2 / Session 1: Layer 1 schema (cases, probes, results) #21 — Session 1: Layer 1 schema (cases, probes, results)
• Ship 2 / Session 2: Layer 2 probes (5-7 pure-function predicates over OTel traces) #22 — Session 2: Layer 2 probes (5-7 pure-function predicates)
• Ship 2 / Session 3: Layer 3 runner + persistence + agent invoker #23 — Session 3: Layer 3 runner + persistence + agent invoker
• Ship 2 / Session 4: Layer 4 pydantic_evals adapter (the swap-point seam) #24 — Session 4: Layer 4 pydantic_evals adapter
• Ship 2 / Session 5: Seed cases + just eval target + first baseline run #25 — Session 5: seed cases + just eval target + baseline run
• Ship 2 / Session 6: Regression baselines + docs #26 — Session 6: regression baselines + docs

Done when

• just eval evals/cases/wallet_profile_smoke.yaml returns a summary like 5/5 cases passed, 23/23 probes passed (vs baseline: +0 -0).
• evals/runs/<run_id>/ populated with per-probe JSON.
• New probe kinds can be added by extending the schema Literal + dropping one file in probes/.
• New cases are pure YAML edits.
• Layer 4 can be swapped (rewrite pydantic_evals_adapter.py) without touching Layers 1-3.

Trigger conditions to revisit ADR 14

• pydantic_evals introduces a feature we want that does not map cleanly to our schema.
• pydantic_evals stagnates or pivots in a direction incompatible with our use.
• Anthropic, OpenAI, or another credible 2026+ source publishes an agent-eval pattern materially different from this design (verify CURRENT per AGENTS.md research-dating rule).

References

• ADR 14 — decision rationale and four invariants
• ADR 13 — OTel substrate this ship consumes
• AGENTS.md — library acceptance bar + research-dating rule + no-adapter-bridging rule

[nabinpkl]

2026-05-05 update: Layer 4 (#24) closed without implementation; superseded by two narrower tickets (#27, #28) that add net-new capability without the framework runtime dependency. Full reasoning in ADR 14 addendum.

Revised sub-ticket list:

• Ship 2 / Session 1: Layer 1 schema (cases, probes, results) #21 — Layer 1 schema (cases, probes, results)
• Ship 2 / Session 2: Layer 2 probes (5-7 pure-function predicates over OTel traces) #22 — Layer 2 probes (5-7 pure-function predicates)
• Ship 2 / Session 3: Layer 3 runner + persistence + agent invoker #23 — Layer 3 runner + persistence + agent invoker
• Ship 2 / Session 4: Layer 4 pydantic_evals adapter (the swap-point seam) #24 — Layer 4 pydantic_evals adapter (dropped; SpanQuery requires in-process span capture, incompatible with our cross-process OTel → CH pipeline)
• Ship 2 / Session 5: Seed cases + just eval target + first baseline run #25 — Seed cases + just eval target + baseline run
• Ship 2 / Session 6: Regression baselines + docs #26 — Regression baselines + docs
• Add llm_judge probe kind for subjective grading #27 — llm_judge probe kind (replaces the LLM-as-judge slice of Ship 2 / Session 4: Layer 4 pydantic_evals adapter (the swap-point seam) #24)
• Add generate_cases CLI for LLM-authored YAML expansion #28 — generate_cases CLI (replaces the dataset-generation slice of Ship 2 / Session 4: Layer 4 pydantic_evals adapter (the swap-point seam) #24)

Architecture summary stays the same: cases as data, probes as predicates over OTel spans, results as JSON artifacts, agent invoked the way production invokes it. The "framework on top" plank turned out to be unreachable for our trace-store choice; the rest of the design held.

[nabinpkl]

2026-05-05 update: #25 done in 3a09609. Two suites, six cases, 34 probes, 30/34 pass on the baseline.

Three findings worth pinning:

• The agent does NOT dispatch the wallet_profile primitive on a known-observable wallet. Eval surfaced what looks like a real bug; baseline pins it for Ship 2 / Session 6: Regression baselines + docs #26 to diff against.
• Agent's primitive-call decision is non-deterministic on the unknown-wallet case across runs.
• Probe-shape limitations: tool_called_with_args doesn't read nested JSON paths; gate_passed conflates 'didn't run' with 'retracted'. Both are candidate follow-ons after Add llm_judge probe kind for subjective grading #27.

Revised remaining work:

• Ship 2 / Session 5: Seed cases + just eval target + first baseline run #25 — Seed cases + first baseline run
• Ship 2 / Session 6: Regression baselines + docs #26 — Regression baselines + diff_runs + docs
• Add llm_judge probe kind for subjective grading #27 — llm_judge probe kind
• Add generate_cases CLI for LLM-authored YAML expansion #28 — generate_cases CLI

Schema gap surfaced: multi-turn cases needed to probe mcae.repeat.detection. Filed as separate concern, not blocking the rest of the epic.

[nabinpkl]

2026-05-05 update: #26 done in 2fb6881. Baselines committed for both suites (18/18 + 16/16). Diff catches new failures, newly passing, and schema deltas; exits 2 on drift, 0 on clean.

Revised remaining work:

• Ship 2 / Session 5: Seed cases + just eval target + first baseline run #25 — Seed cases + first baseline run
• Ship 2 / Session 6: Regression baselines + docs #26 — Regression baselines + diff_runs + docs
• Add llm_judge probe kind for subjective grading #27 — llm_judge probe kind
• Add generate_cases CLI for LLM-authored YAML expansion #28 — generate_cases CLI

The substrate now pays rent: any future change to the agent that flips a probe in either direction is caught at just eval time.

[nabinpkl]

2026-05-06 update: #27 done in 1575390. llm_judge probe with outcome and trajectory modes; trajectory mode verified end-to-end as judge-of-judges (the eval-judge audited the constitution gate's verdict and confirmed it).

Revised remaining work:

• Ship 2 / Session 5: Seed cases + just eval target + first baseline run #25 — Seed cases + first baseline run
• Ship 2 / Session 6: Regression baselines + docs #26 — Regression baselines + diff_runs + docs
• Add llm_judge probe kind for subjective grading #27 — llm_judge probe kind
• Add generate_cases CLI for LLM-authored YAML expansion #28 — generate_cases CLI

Lessons from #27 worth keeping: free-tier OpenRouter availability is volatile across models (4 tried, 4 different failure modes). Plain-text + JSON parse beat tool-calling as the more durable design choice. json.JSONDecoder.raw_decode over regex was load-bearing for handling ${ref:N} literals in judge responses.

[nabinpkl]

Closing as shipped.

What landed under this epic

• Eval substrate (Layers 1-3 of ADR 14): typed EvalCase schema with discriminated-union ProbeSpec, runner that invokes the agent over the production HTTP/SSE path with runType=eval, ClickHouse-backed trace querying, persisted ProbeResult and RunMetadata artifacts, regression baselines with provenance fields (git_sha, agent_version, agent_primary_model, agent_policy_model, eval_judge_model).
• 11 probe kinds: has_matching_span, no_matching_span, tool_called_with_args, claim_grounded_in, gate_passed, span_latency_p50_under, no_span_with_status, llm_call_used_model, llm_judge, turn_attribute_equals, slowest_call_under_ms.
• Provider-error robustness: with_provider_retry with per-attempt timeouts + single retry across all 5 LLM call sites; infra_health.has_terminal_provider_failure flips probes to inconclusive=True on agent-side terminal failures; baseline diff skips inconclusive entries.
• Judge-bias prevention: env-derived forbidden-family list rejecting any judge model whose family matches AGENT_PRIMARY_MODEL or AGENT_POLICY_MODEL at YAML load time (preference leakage prevention per ICLR 2026).
• 8 committed eval suites, 12 cases: who_are_you, who_are_you_no_role, wallet_profile_smoke, wallet_profile_fabrication_allowed, community_summary_smoke, refusal_smoke, refusal_prompt_injection, refusal_obfuscated_injection.
• One real bug surfaced and fixed via the substrate: text-vs-tool divergence on prompt-injection vectors (Agent fails text-vs-tool divergence on roleplay and system-tag-spoofing prompt injection vectors #33), closed today.

What was deliberately NOT shipped

• Layer 4 framework adapter (pydantic_evals): dropped per ADR 14 2026-05-05 addendum. Rationale: pydantic_evals' SpanQuery requires in-process span capture (incompatible with our cross-process OTel → CH pipeline). The probe set covers what the framework would have given us; carrying the dependency would have been dead optionality.
• Synthetic case generation (the generate_dataset replacement): closed via Add generate_cases CLI for LLM-authored YAML expansion #28 → superseded by Generate eval cases from observed production traces (trace-replay-to-yaml) #30 (trace-replay-to-yaml, deferred). Frontier moved past synthetic generation toward production-trace-derived cases (Anthropic 2026-01-09 framing).

Follow-up tickets (post-epic)

Tracked separately because they extend the substrate's surface rather than complete its core:

• Behavioral evaluation across switch combinations to separate intrinsic from gated behavior #29 (behavior-story matrix suites)
• Generate eval cases from observed production traces (trace-replay-to-yaml) #30 (trace-replay-to-yaml — deferred)
• Multi-turn eval coverage: knowledge retention, context drift, goal hijacking #31 (multi-turn eval coverage — blocked on Ship 4)
• Eval substrate: distinguish improving drift from regressing drift in exit code #32 (substrate enhancement: improving vs regressing drift, judge-call inconclusive)
• Refusal-obfuscated-injection suite: holistic revisit (3 vectors, dual axes, multi-source flap) #34 (refusal-obfuscated-injection suite holistic revisit)
• Promote stay_in_role to StayInRoleSwitches sub-message for per-defense ablation #35 (StayInRoleSwitches sub-message scaffolding)
• Refactor system_v4 to industry-standard XML+markdown tagged structure with single-file composer #36 (XML-tagged prompt structure + composer — the foundation refactor)
• Article data-collection: vectors x models x switches matrix runner emitting markdown table #37 (article data-collection: vectors x models x switches matrix runner)

Refs:

• ADR 14 with all addendums (1 through 4)
• The session that shipped the work + the bug fix it surfaced (Agent fails text-vs-tool divergence on roleplay and system-tag-spoofing prompt injection vectors #33)