Verdict output contract is fragile: no structured-output path, unbounded reasoning, degradations warned once per run across a fan-out

[macanderson]

Problem

Three related robustness gaps in the verdict call's I/O (crates/stella-pipeline/src/verify.rs::parse_verifier_response, pipeline/verifier_stage.rs):

1. First-non-empty-line protocol vs reasoning models. A verifier that opens with "Here is my assessment:" yields no token → heuristic_fallback, silently converting every verdict from that model into the heuristic (which fails anything without green touched tests). The goal verifier already asks for JSON (stella-core/src/goal.rs); the pipeline verdict has no structured path. Consider a provider-parity-aware forced tool-call/JSON mode (invariant 8: declare the posture per provider, with witness tests), falling back to the token protocol where unsupported. Negation handling shipped separately; this issue is about the protocol itself.
2. Verdict::reasoning is the whole response, unbounded — it flows into the worker's revision prompt and the verdict cache. Bound it (the disclosure ladder already caps what crosses to the worker; the cap should be structural).
3. Degradation warnings are once-per-run AtomicBools (verifier_stage.rs::warn_verifier_fallback) — a best-of-N fan-out reports one caveat for N candidates, and nothing records WHICH candidates degraded. Consider per-candidate proof steps instead of (or besides) the one warning.

Also worth folding in: the trusted evidence summary has no length bound (oracle_trace grows per observation; the diff has a token budget, the trusted zone does not) — pipeline/evidence.rs since the extraction.

Verify

Each sub-fix carries its own witness: (1) a scripted verifier replying with a preamble line parses to a real verdict under the structured path; (2) a 100KB reply does not land 100KB in the revision prompt; (3) a two-candidate fan-out with a dead verifier records two degradation facts.

Related

#1483/#1501 (timeout vs fallback), the 46%-agreement measurement cited at verify.rs (asymmetric trust), #1284 (turn the independent checker on and measure).

[macanderson]

Triage 2026-08-06: P1 — the first-non-empty-line protocol silently converts entire model families' verdicts into the heuristic (which fails anything without green touched tests), a correctness defect on the shipped verdict path; the unbounded reasoning and once-per-run fan-out warnings compound it. Plan: as specced in the body; next step is the provider-parity-aware structured-output path (invariant 8: declared posture per provider with witness tests, token protocol as fallback), then the reasoning/evidence bounds and per-candidate degradation facts.

[macanderson]

Item 2 shipped as PR #1932. Design notes on the other two, from looking at both.

Item 2 — bounded reasoning (done)

Measured on the witness fixture, the old path carried 114,006 characters into the revision prompt and the verdict cache. Now clipped at MAX_VERDICT_REASONING_CHARS = 4,000, at the point the value is constructed rather than by a downstream consumer — which is what this issue means by "structural". Three witnesses: the clip (with the clipping stated, not silent), an ordinary verdict passing through untouched, and a multi-byte reply not panicking on a byte-slice.

I also checked the two places that touch reasoning afterwards — pipeline.rs's cache-reuse arm appends a fixed ~130-char note, and heuristic_fallback builds from fixed strings — so the two parse sites were the only unbounded ingress.

Item 3 — per-candidate degradation facts (NOT done; here is why it is bigger than it looks)

The obvious fix is to reset verifier_fallback_warned per candidate instead of per run. That is wrong, and the trap is worth recording before someone tries it:

• The flag is one AtomicBool on the shared Pipeline (pipeline.rs:882).
• Best-of-N candidates run concurrently (pipeline/tests/best_of_n/fanout_concurrency.rs).

So a reset-at-candidate-start races: two candidates degrading at overlapping times would produce one warning or two, nondeterministically, and the test would be flaky rather than wrong-but-stable.

The fact has to be keyed by candidate, not counted. That means:

1. Pipeline::verifier (verifier_stage.rs:77) takes no candidate identity today — its signature and every caller need it threaded through.
2. The record wants to be per-candidate state (or a Mutex<HashSet<CandidateId>>), not an AtomicBool.
3. pipeline.rs is a grandfathered god file at its ceiling, so the new state and its accessor land in a sibling — verifier_stage.rs itself has room.

The issue suggests proof steps as an alternative, which is cleaner for the consumer but costs a new stella_protocol::ProofStep variant and therefore a docs/wire/ regeneration (make wire-schema) — worth it if the deck should render per-candidate degradation, not worth it if the transcript line is enough.

Whoever takes it should decide that first, because it changes whether this is a stella-pipeline change or a protocol change.

Item 1 — structured output

Unchanged from the issue: it needs a provider-parity posture per invariant 8 (declare per provider, with a named witness test), and the fallback to the token protocol where unsupported. Largest of the three.

This issue stays open for 1 and 3.