fix(stella-pipeline): bound the verdict reasoning at the point it is constructed (#1787) - #1932
Merged
Conversation
…constructed
`Verdict::reasoning` was the verifier's whole reply, unbounded, and it travels:
into the worker's revision prompt and into the verdict cache. A reasoning model
that thinks out loud for 100 KB put 100 KB into the next worker turn — on every
revision round, at the full input rate, and left it in a cache entry that keeps
it.
Measured on the fixture in the witness below, the old code carried **114,006
characters** into the revision prompt.
`bounded_reasoning` clips at `MAX_VERDICT_REASONING_CHARS` (4,000 — roughly a
thousand tokens) and says it did.
## Three decisions worth naming
**Structural, not policy.** The disclosure ladder already caps what crosses to
the worker, but as policy applied downstream by one consumer. This bounds the
value where it is *constructed*, so a consumer added later cannot be written
that forgets — which is what this issue means by "the cap should be
structural".
**The head is kept, not the tail.** The verifier prompt asks for the verdict
token and its reasons first, so the front is the part a revision acts on. The
witness asserts `starts_with("FAIL")` so a later "keep the tail" refactor has
to argue with a test rather than with a comment.
**Clipped on a character boundary.** A verifier answering in a language of
multi-byte characters is model output, which is runtime data — a `&text[..N]`
byte slice would panic the pipeline there (invariant 5).
## Not re-unbounded downstream
Checked the two places that touch it afterwards: `pipeline.rs`'s cache-reuse
arm appends a fixed ~130-character note, and `heuristic_fallback` builds its
reasoning from fixed strings. Both are bounded by construction, so the two
parse sites were the only ingress.
## Witnesses
- `a_runaway_verifier_reply_is_clipped_before_it_reaches_the_worker` — a
>100 KB reply yields a bounded `reasoning` that **says** it was clipped (a
silent truncation reads as a verifier that simply stopped talking) and still
leads with its verdict token. Fails on the old code:
`reasoning survived at 114006 chars`.
- `an_ordinary_verdict_passes_through_the_bound_unchanged` — the bound is
invisible in the case that matters, so a fix that quietly edited every
verdict cannot pass.
- `clipping_a_multibyte_reply_does_not_panic` — the byte-slice trap.
## Scope
This is item 2 of the three #1787 lists, which says "each sub-fix carries its
own witness". Item 1 (a structured-output path for verifiers whose replies open
with a preamble) needs a provider-parity posture per invariant 8 and is a
larger change; item 3 (per-candidate degradation facts instead of one
once-per-run `AtomicBool`) is independent. The issue stays open for both.
`cargo test -p stella-pipeline` — 557 passed, 0 failed. Clippy `-D warnings`,
`fmt --check`, rustdoc `-D warnings`, `check-file-size`, `check-god-files` and
`check-left-behind` clean.
Refs #1787
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Sorry @macanderson, you have reached your weekly rate limit of 500000 diff characters.
Please try again later or upgrade to continue using Sourcery
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
Reviewer's GuideThis PR introduces a structural character-level cap on verifier verdict reasoning at construction time, ensuring oversized verifier replies are trimmed safely and explicitly before being stored or reused, and adds focused tests to validate clipping behavior, correctness on normal replies, and multibyte safety. Flow diagram for bounded verifier reasoning construction and usageflowchart LR
subgraph VerifierSide
A[verifier_model_response text]
B[parse_verifier_response]
C[bounded_reasoning]
D["Verdict.reasoning (bounded String)"]
end
subgraph DownstreamConsumers
E[worker_revision_prompt]
F[verdict_cache]
end
A --> B
B -->|calls| C
C -->|trim and clip to MAX_VERDICT_REASONING_CHARS| D
D --> E
D --> F
File-Level Changes
Possibly linked issues
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
This was referenced Aug 7, 2026
macanderson
added a commit
that referenced
this pull request
Aug 7, 2026
… candidate in a fan-out (#1787) (#1951) ## What & why Item 3 of #1787: verifier degradation was surfaced by a once-per-run `AtomicBool` warning, so a best-of-N fan-out whose verifier died (or answered off-protocol) reported one caveat for N candidates and recorded nothing about **which** candidates were judged by the deterministic heuristic. Each candidate now emits one structured `ProofStep::VerdictDegraded { candidate, reason }` fact — 1-based ordinal, `Oracle::run`'s convention — the first time its verdict degrades. Dedup state is candidate-local (`VerdictDegradation` rides `CandidateState`), never a shared flag reset at candidate boundaries: candidates run concurrently (`fanout_concurrency.rs`), and a run-wide reset would race nondeterministically. The once-per-run transcript warning is unchanged; the traces view renders the new step (`verdict degraded (candidate N): reason`), and the replay fold keys it as `proof:verdict_degraded:{candidate}`. Two file-size-driven structural moves, both following existing exemplars: - `ProofStep` moves from `event.rs` (at its ceiling) to a new `stella-protocol/src/proof.rs`, the exact `ladder.rs` pattern — re-exported from `event` and the crate root so `stella_protocol::ProofStep` and `event::ProofStep` never moved. `docs/wire/` regenerated. - `pipeline.rs` (also at its ceiling) stays under it by consolidating the two once-per-run `AtomicBool` notices into `VerifierNotices` in `verifier_stage.rs`, and `verifier()` now takes the crate's own `Spend` envelope instead of loose `budget`/`total` (the `task_frame.rs` idiom). `CandidateSlot` in `fanout_stage.rs` groups a candidate's ordinal with its workspace so the fan-out driver stays under `clippy::too_many_arguments` structurally rather than by `#[allow]`. Refs #1787 — item 1 (provider-parity-aware structured verdict output) remains open; item 2 shipped as #1932. ## The witness - [x] This PR includes a witness test (fails on `main`, passes here) `a_two_candidate_fanout_records_which_candidates_degraded` — the issue's own Verify: two isolated candidates with a tokenless verifier record facts for ordinals 1 and 2 beside exactly one transcript warning; every post-triage scripted reply is tokenless so the assertion holds under any completion order. `a_candidate_degrading_on_every_round_records_one_fact` pins per-candidate (not per-round) emission, with the provider's script consumption asserted so the second escalation provably happened. Both name `ProofStep::VerdictDegraded`, which does not exist on `main`. Plus a wire round-trip pinning the serialized shape (`proof.rs`) and the variant added to `wire_contract.rs`'s exhaustive list. ## The gate - [x] `cargo fmt --check` - [x] `cargo clippy --workspace --all-targets -- -D warnings` - [x] `cargo test -p stella-protocol -p stella-pipeline -p stella-tui` (all green; guards-fast, file-size, god-files, invariants, module-reachability, wire-schema all green — CI runs the full workspace) - [x] Docs updated where behavior changed (doc comments on every moved/new item) - [x] CLA signed - [x] `Refs #1787` above and as a commit trailer — deliberately not `Closes`: item 1 stays open ## Nothing left behind Item 1 of #1787 (structured verdict output with a declared per-provider posture) is untouched and tracked by the still-open #1787. ## Summary by Sourcery Record verifier verdict degradation per candidate in fan-out runs while refactoring proof and verifier wiring to keep file sizes and interfaces within project constraints. New Features: - Emit a structured ProofStep::VerdictDegraded fact per candidate when its model verdict falls back to the deterministic heuristic, keyed by 1-based candidate ordinal. - Expose verdict degradation in traces and replay signatures so consumers can see which candidates were judged by the heuristic and why. Bug Fixes: - Ensure best-of-N verifier fan-outs record which specific candidates degraded to the deterministic heuristic rather than only surfacing a single run-wide warning. Enhancements: - Split the proof-step vocabulary into a dedicated stella-protocol::proof module and re-export it to keep existing ProofStep paths stable while allowing the enum to grow independently of the event envelope. - Group once-per-run verifier caveat and fallback notices into a VerifierNotices struct and thread candidate ordinals through fan-out and candidate execution so degradation tracking is candidate-local and concurrency-safe. - Adjust the verifier interface to take a Spend envelope instead of separate budget and total parameters, and introduce a CandidateSlot helper to keep the fan-out driver below argument-count limits. Documentation: - Regenerate TypeScript wire typings and JSON schemas for proof events, including the new verdict_degraded proof step and updated documentation references. Tests: - Add pipeline tests that witness per-candidate verdict degradation in a two-candidate fan-out and ensure repeated degradations for a single candidate only emit one fact. - Extend protocol wire-contract tests and add a round-trip test pinning the serialized shape of the new VerdictDegraded proof step.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fix(stella-pipeline): bound the verdict reasoning at the point it is constructed
Verdict::reasoningwas the verifier's whole reply, unbounded, and it travels:into the worker's revision prompt and into the verdict cache. A reasoning model
that thinks out loud for 100 KB put 100 KB into the next worker turn — on every
revision round, at the full input rate, and left it in a cache entry that keeps
it.
Measured on the fixture in the witness below, the old code carried 114,006
characters into the revision prompt.
bounded_reasoningclips atMAX_VERDICT_REASONING_CHARS(4,000 — roughly athousand tokens) and says it did.
Three decisions worth naming
Structural, not policy. The disclosure ladder already caps what crosses to
the worker, but as policy applied downstream by one consumer. This bounds the
value where it is constructed, so a consumer added later cannot be written
that forgets — which is what this issue means by "the cap should be
structural".
The head is kept, not the tail. The verifier prompt asks for the verdict
token and its reasons first, so the front is the part a revision acts on. The
witness asserts
starts_with("FAIL")so a later "keep the tail" refactor hasto argue with a test rather than with a comment.
Clipped on a character boundary. A verifier answering in a language of
multi-byte characters is model output, which is runtime data — a
&text[..N]byte slice would panic the pipeline there (invariant 5).
Not re-unbounded downstream
Checked the two places that touch it afterwards:
pipeline.rs's cache-reusearm appends a fixed ~130-character note, and
heuristic_fallbackbuilds itsreasoning from fixed strings. Both are bounded by construction, so the two
parse sites were the only ingress.
Witnesses
a_runaway_verifier_reply_is_clipped_before_it_reaches_the_worker— aan_ordinary_verdict_passes_through_the_bound_unchanged— the bound isinvisible in the case that matters, so a fix that quietly edited every
verdict cannot pass.
clipping_a_multibyte_reply_does_not_panic— the byte-slice trap.Scope
This is item 2 of the three #1787 lists, which says "each sub-fix carries its
own witness". Item 1 (a structured-output path for verifiers whose replies open
with a preamble) needs a provider-parity posture per invariant 8 and is a
larger change; item 3 (per-candidate degradation facts instead of one
once-per-run
AtomicBool) is independent. The issue stays open for both.cargo test -p stella-pipeline— 557 passed, 0 failed. Clippy-D warnings,fmt --check, rustdoc-D warnings,check-file-size,check-god-filesandcheck-left-behindclean.Refs #1787
Summary by Sourcery
Bound verifier verdict reasoning at construction to prevent oversized reasoning payloads from propagating through the pipeline and cache.
Enhancements:
Tests: