fix(stella-pipeline): bound the verdict reasoning at the point it is constructed (#1787) - #1932
Merged
Conversation
…constructed
`Verdict::reasoning` was the verifier's whole reply, unbounded, and it travels:
into the worker's revision prompt and into the verdict cache. A reasoning model
that thinks out loud for 100 KB put 100 KB into the next worker turn — on every
revision round, at the full input rate, and left it in a cache entry that keeps
it.
Measured on the fixture in the witness below, the old code carried **114,006
characters** into the revision prompt.
`bounded_reasoning` clips at `MAX_VERDICT_REASONING_CHARS` (4,000 — roughly a
thousand tokens) and says it did.
## Three decisions worth naming
**Structural, not policy.** The disclosure ladder already caps what crosses to
the worker, but as policy applied downstream by one consumer. This bounds the
value where it is *constructed*, so a consumer added later cannot be written
that forgets — which is what this issue means by "the cap should be
structural".
**The head is kept, not the tail.** The verifier prompt asks for the verdict
token and its reasons first, so the front is the part a revision acts on. The
witness asserts `starts_with("FAIL")` so a later "keep the tail" refactor has
to argue with a test rather than with a comment.
**Clipped on a character boundary.** A verifier answering in a language of
multi-byte characters is model output, which is runtime data — a `&text[..N]`
byte slice would panic the pipeline there (invariant 5).
## Not re-unbounded downstream
Checked the two places that touch it afterwards: `pipeline.rs`'s cache-reuse
arm appends a fixed ~130-character note, and `heuristic_fallback` builds its
reasoning from fixed strings. Both are bounded by construction, so the two
parse sites were the only ingress.
## Witnesses
- `a_runaway_verifier_reply_is_clipped_before_it_reaches_the_worker` — a
>100 KB reply yields a bounded `reasoning` that **says** it was clipped (a
silent truncation reads as a verifier that simply stopped talking) and still
leads with its verdict token. Fails on the old code:
`reasoning survived at 114006 chars`.
- `an_ordinary_verdict_passes_through_the_bound_unchanged` — the bound is
invisible in the case that matters, so a fix that quietly edited every
verdict cannot pass.
- `clipping_a_multibyte_reply_does_not_panic` — the byte-slice trap.
## Scope
This is item 2 of the three #1787 lists, which says "each sub-fix carries its
own witness". Item 1 (a structured-output path for verifiers whose replies open
with a preamble) needs a provider-parity posture per invariant 8 and is a
larger change; item 3 (per-candidate degradation facts instead of one
once-per-run `AtomicBool`) is independent. The issue stays open for both.
`cargo test -p stella-pipeline` — 557 passed, 0 failed. Clippy `-D warnings`,
`fmt --check`, rustdoc `-D warnings`, `check-file-size`, `check-god-files` and
`check-left-behind` clean.
Refs #1787
Contributor
There was a problem hiding this comment.
Sorry @macanderson, you have reached your weekly rate limit of 500000 diff characters.
Please try again later or upgrade to continue using Sourcery
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
Reviewer's GuideThis PR introduces a structural character-level cap on verifier verdict reasoning at construction time, ensuring oversized verifier replies are trimmed safely and explicitly before being stored or reused, and adds focused tests to validate clipping behavior, correctness on normal replies, and multibyte safety. Flow diagram for bounded verifier reasoning construction and usageflowchart LR
subgraph VerifierSide
A[verifier_model_response text]
B[parse_verifier_response]
C[bounded_reasoning]
D["Verdict.reasoning (bounded String)"]
end
subgraph DownstreamConsumers
E[worker_revision_prompt]
F[verdict_cache]
end
A --> B
B -->|calls| C
C -->|trim and clip to MAX_VERDICT_REASONING_CHARS| D
D --> E
D --> F
File-Level Changes
Possibly linked issues
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
This was referenced Aug 7, 2026
macanderson
added a commit
that referenced
this pull request
Aug 7, 2026
… candidate in a fan-out (#1787) (#1951) ## What & why Item 3 of #1787: verifier degradation was surfaced by a once-per-run `AtomicBool` warning, so a best-of-N fan-out whose verifier died (or answered off-protocol) reported one caveat for N candidates and recorded nothing about **which** candidates were judged by the deterministic heuristic. Each candidate now emits one structured `ProofStep::VerdictDegraded { candidate, reason }` fact — 1-based ordinal, `Oracle::run`'s convention — the first time its verdict degrades. Dedup state is candidate-local (`VerdictDegradation` rides `CandidateState`), never a shared flag reset at candidate boundaries: candidates run concurrently (`fanout_concurrency.rs`), and a run-wide reset would race nondeterministically. The once-per-run transcript warning is unchanged; the traces view renders the new step (`verdict degraded (candidate N): reason`), and the replay fold keys it as `proof:verdict_degraded:{candidate}`. Two file-size-driven structural moves, both following existing exemplars: - `ProofStep` moves from `event.rs` (at its ceiling) to a new `stella-protocol/src/proof.rs`, the exact `ladder.rs` pattern — re-exported from `event` and the crate root so `stella_protocol::ProofStep` and `event::ProofStep` never moved. `docs/wire/` regenerated. - `pipeline.rs` (also at its ceiling) stays under it by consolidating the two once-per-run `AtomicBool` notices into `VerifierNotices` in `verifier_stage.rs`, and `verifier()` now takes the crate's own `Spend` envelope instead of loose `budget`/`total` (the `task_frame.rs` idiom). `CandidateSlot` in `fanout_stage.rs` groups a candidate's ordinal with its workspace so the fan-out driver stays under `clippy::too_many_arguments` structurally rather than by `#[allow]`. Refs #1787 — item 1 (provider-parity-aware structured verdict output) remains open; item 2 shipped as #1932. ## The witness - [x] This PR includes a witness test (fails on `main`, passes here) `a_two_candidate_fanout_records_which_candidates_degraded` — the issue's own Verify: two isolated candidates with a tokenless verifier record facts for ordinals 1 and 2 beside exactly one transcript warning; every post-triage scripted reply is tokenless so the assertion holds under any completion order. `a_candidate_degrading_on_every_round_records_one_fact` pins per-candidate (not per-round) emission, with the provider's script consumption asserted so the second escalation provably happened. Both name `ProofStep::VerdictDegraded`, which does not exist on `main`. Plus a wire round-trip pinning the serialized shape (`proof.rs`) and the variant added to `wire_contract.rs`'s exhaustive list. ## The gate - [x] `cargo fmt --check` - [x] `cargo clippy --workspace --all-targets -- -D warnings` - [x] `cargo test -p stella-protocol -p stella-pipeline -p stella-tui` (all green; guards-fast, file-size, god-files, invariants, module-reachability, wire-schema all green — CI runs the full workspace) - [x] Docs updated where behavior changed (doc comments on every moved/new item) - [x] CLA signed - [x] `Refs #1787` above and as a commit trailer — deliberately not `Closes`: item 1 stays open ## Nothing left behind Item 1 of #1787 (structured verdict output with a declared per-provider posture) is untouched and tracked by the still-open #1787. ## Summary by Sourcery Record verifier verdict degradation per candidate in fan-out runs while refactoring proof and verifier wiring to keep file sizes and interfaces within project constraints. New Features: - Emit a structured ProofStep::VerdictDegraded fact per candidate when its model verdict falls back to the deterministic heuristic, keyed by 1-based candidate ordinal. - Expose verdict degradation in traces and replay signatures so consumers can see which candidates were judged by the heuristic and why. Bug Fixes: - Ensure best-of-N verifier fan-outs record which specific candidates degraded to the deterministic heuristic rather than only surfacing a single run-wide warning. Enhancements: - Split the proof-step vocabulary into a dedicated stella-protocol::proof module and re-export it to keep existing ProofStep paths stable while allowing the enum to grow independently of the event envelope. - Group once-per-run verifier caveat and fallback notices into a VerifierNotices struct and thread candidate ordinals through fan-out and candidate execution so degradation tracking is candidate-local and concurrency-safe. - Adjust the verifier interface to take a Spend envelope instead of separate budget and total parameters, and introduce a CandidateSlot helper to keep the fan-out driver below argument-count limits. Documentation: - Regenerate TypeScript wire typings and JSON schemas for proof events, including the new verdict_degraded proof step and updated documentation references. Tests: - Add pipeline tests that witness per-candidate verdict degradation in a two-candidate fan-out and ensure repeated degradations for a single candidate only emit one fact. - Extend protocol wire-contract tests and add a round-trip test pinning the serialized shape of the new VerdictDegraded proof step.
1 task
macanderson
added a commit
that referenced
this pull request
Aug 7, 2026
…ngress (#1787) (#1982) > **Stacked on #1975.** Based on `unbreak-main-pipeline` because `main` does not > currently compile `stella-pipeline`'s test target; the diff below is the one > commit on top. GitHub retargets this to `main` when #1975 merges. ## What & why The last unbounded ingress into the verdict prompt, and the item #1787 folds in at the end of its body: > Also worth folding in: the trusted evidence summary has no length bound > (`oracle_trace` grows per observation; the diff has a token budget, the > trusted zone does not) — `pipeline/evidence.rs` since the extraction. `verifier_evidence_summary` renders `oracle_trace` in full. That trace gains an observation per verification round, and the repair gate (#1479) keeps granting rounds for as long as a measured budget affords them — so the one channel that grows without limit was also the one channel with no ceiling. Every other input to that prompt is bounded: the diff has a token budget, recall frames have `bound_recalled_frames`, and `Verdict::reasoning` got its cap in #1932. Bounded to the newest 24 observations, with the drop **stated in-band**: ``` oracle_trace=[…76 earlier observation(s) omitted → candidate:pass → candidate:fail → …] ``` Three choices worth naming, because each has a wrong-looking alternative: - **Newest kept, oldest dropped.** The recent runs are what the verdict weighs; a trace clipped from the front would hand the verifier a history that stops before the evidence. - **Stated, not silent.** A trace that silently began mid-run reads as the whole run — the verifier would draw conclusions about a first observation that was not the first. - **Clipped where the value is constructed**, not at a downstream consumer. That is the same "structural, not by convention" rule #1932 applied to `reasoning`, and it is why the stored `LadderSnapshot` and `verdict_provenance` are deliberately untouched: the bound is on the *prompt ingress*, not on the record. 24 is sized far above a normal run (a baseline plus a handful of rounds), so the bound only ever bites a pathological loop. It is a named constant next to its rationale rather than a literal. ## The witness - [x] This PR includes a witness test `a_pathological_oracle_trace_is_clipped_with_the_drop_stated` — a 100-observation trace renders exactly 24 entries behind the `…76 earlier observation(s) omitted →` marker, and still ends on the newest observation. `an_ordinary_oracle_trace_renders_unchanged` is the other half, and the one that matters for regression: a 5-observation trace is asserted **byte-identical to `render_oracle_trace`**, main's own unbounded function, which is still present and still used for provenance. So every prompt the bound does not bite is unchanged to the byte — which also means no verdict-reuse digest (#1431) moves for an ordinary run. Honest note on "fails on main": these pin a bound that does not exist on `main`, so the failure there is that `bounded_oracle_trace` is not defined — the same shape as #1932's witnesses for the `reasoning` cap, and the shape any "add a missing ceiling" change has. The behavioural claim is carried by the second test, which compares against main's function directly rather than against a copied expectation. ## The gate - `cargo test -p stella-pipeline` — 585 + 5 + 2 + 5 + 6 + 4 = **607 passed, 0 failed** - `cargo clippy -p stella-pipeline --all-targets` — clean - `cargo fmt --check -p stella-pipeline` — clean - `scripts/check-file-size.sh` — OK, none grew (`evidence.rs` was extracted from `pipeline.rs` precisely so this kind of channel can be added without touching a god file, and that still holds) Full workspace left to CI. ## Nothing left behind `Refs #1787`, deliberately not `Closes` — this is the folded-in bound only. Item 1 (a provider-parity-aware structured verdict output path, invariant 8) remains open and is being approached from a different angle in #1964; item 2 shipped as #1932 and item 3 as #1951. Refs #1787 ## Summary by Sourcery Bound the oracle trace rendered in the verifier prompt to a fixed number of recent observations and document the truncation in-band while preserving full traces in stored provenance. Enhancements: - Introduce a bounded oracle trace renderer for the verifier prompt, limiting the trusted-zone trace to the newest 24 observations and prefixing output with an omission marker when older entries are dropped. Tests: - Add tests ensuring pathological long oracle traces are clipped with an explicit omission notice and that ordinary short traces remain byte-identical to the existing unbounded rendering.
This was referenced Aug 7, 2026
macanderson
added a commit
that referenced
this pull request
Aug 7, 2026
…ngress (#1787) (#2002) ## Why this PR exists **#1787's fix is not in `main`.** PR #1982 carried it, but its base was the topic branch `unbreak-main-pipeline`, whose own PR (#1975) was **closed, not merged**. #1982 then merged into that dead branch, so the oracle-trace bound landed nowhere `main` can see, and nothing is carrying that branch forward. It also merged in a **broken** state. While the base was being reconciled with `main`, git's auto-merge of the two independently-written unbreaks concatenated both sides, leaving: - `struct PassingShell` and `fn shell_call_result` **defined twice** - `async fn a_revision_halts_at_the_step_where_the_tracked_test_flips` defined twice - a duplicate `ModelCallRole::Research` match arm (unreachable pattern) None of that compiles. `unbreak-main-pipeline` currently holds it; `main` is unaffected. This PR is the clean landing: **`main` plus `evidence.rs`, and nothing else.** ## What it does (#1787) Bounds the oracle trace at the verifier-prompt ingress. The trace grows once per verification round and the repair gate can keep granting rounds while a measured budget affords them — so unlike the diff, which rides under a token budget, this channel had **no ceiling at all**. - `MAX_ORACLE_TRACE_OBSERVATIONS = 24` — sized far above a normal run (baseline plus a handful of rounds) so the bound only bites a pathological loop. - `bounded_oracle_trace` keeps the **newest** observations and states the drop **in-band** (`…N earlier observation(s) omitted → …`), so the verifier reads "earlier observations exist" rather than a trace that silently starts mid-run. - The **stored snapshot keeps the full trace**; only the prompt ingress is clipped — the structural-bound rule from #1932. ## Witnesses - `a_pathological_oracle_trace_is_clipped_with_the_drop_stated` — a 100-observation trace renders clipped to the newest 24 with the omission counted in-band. - `an_ordinary_oracle_trace_renders_unchanged` — the bound does not touch a normal run, so this cannot ship as "always clip". Observations alternate pass/fail in the fixture so a clipped render is distinguishable from a repeated one. ## Verification - `cargo test -p stella-pipeline` — **585 pass**, 0 fail, including both witnesses above - `cargo fmt --check -p stella-pipeline` — clean - Diff vs `main` is exactly one file: `crates/stella-pipeline/src/pipeline/evidence.rs` (+74/−2) ## CI is red on `main`'s breaks, not this diff This branch is merged up to current `main`. Every failing step fails in a file this PR does not touch, and each already has a dedicated unbreak in flight: | Failing step | Where | Covered by | |---|---|---| | `check-file-size` | `scripts/file-size-baseline.txt` (parallel-merge skew) | **#2003**, **#2008** | | `cargo fmt --check` | not this crate's file | **#2005** | | clippy: unused `spend` / unused `mut` | `pipeline/scope_stage.rs:34` — a dead local `#1985` left behind | **#2000** | | rustdoc: unresolved `CompactionRewrite` | `stella-protocol` | **#2010** | The clippy one is worth naming precisely, since it is `stella-pipeline`: `main`'s `scope_stage.rs` binds `let mut spend = Spend { budget, total };` and then never uses it — the loop constructs a fresh `Spend` inline per iteration. `spend` occurs exactly once in the file. That is `main`'s dead local, untouched by this PR. No competing unbreak is included here on purpose — six are already open against `main`, and duplicating one is how `main` gets re-broken. ## Note on the dead branch `unbreak-main-pipeline` still holds the duplicate-definition breakage and the only copy of #1982's merge. It is not reachable from `main` and its PR is closed, so nothing needs to be reverted — but it should not be revived without first taking `main`'s copies of `flip_halt_arming.rs`, `management_prompt/tests.rs` and `scope_stage.rs`, which is what this PR does. Filed as #2001. Closes #1787
macanderson
added a commit
that referenced
this pull request
Aug 7, 2026
…ngress (#1787) (#2012) > Supersedes #1982, which was auto-closed when the branch it was stacked on > went away. Same single commit, now rebased directly on `main`. > > Note: `main` currently fails `cargo clippy -p stella-pipeline` with five > pre-existing warnings unrelated to this change (fixed in #2009), so this > PR's clippy step inherits them until that lands. ## What & why The last unbounded ingress into the verdict prompt, and the item #1787 folds in at the end of its body: > Also worth folding in: the trusted evidence summary has no length bound > (`oracle_trace` grows per observation; the diff has a token budget, the > trusted zone does not) — `pipeline/evidence.rs` since the extraction. `verifier_evidence_summary` renders `oracle_trace` in full. That trace gains an observation per verification round, and the repair gate (#1479) keeps granting rounds for as long as a measured budget affords them — so the one channel that grows without limit was also the one channel with no ceiling. Every other input to that prompt is bounded: the diff has a token budget, recall frames have `bound_recalled_frames`, and `Verdict::reasoning` got its cap in #1932. Bounded to the newest 24 observations, with the drop **stated in-band**: ``` oracle_trace=[…76 earlier observation(s) omitted → candidate:pass → candidate:fail → …] ``` Three choices worth naming, because each has a wrong-looking alternative: - **Newest kept, oldest dropped.** The recent runs are what the verdict weighs; a trace clipped from the front would hand the verifier a history that stops before the evidence. - **Stated, not silent.** A trace that silently began mid-run reads as the whole run — the verifier would draw conclusions about a first observation that was not the first. - **Clipped where the value is constructed**, not at a downstream consumer. That is the same "structural, not by convention" rule #1932 applied to `reasoning`, and it is why the stored `LadderSnapshot` and `verdict_provenance` are deliberately untouched: the bound is on the *prompt ingress*, not on the record. 24 is sized far above a normal run (a baseline plus a handful of rounds), so the bound only ever bites a pathological loop. It is a named constant next to its rationale rather than a literal. ## The witness - [x] This PR includes a witness test `a_pathological_oracle_trace_is_clipped_with_the_drop_stated` — a 100-observation trace renders exactly 24 entries behind the `…76 earlier observation(s) omitted →` marker, and still ends on the newest observation. `an_ordinary_oracle_trace_renders_unchanged` is the other half, and the one that matters for regression: a 5-observation trace is asserted **byte-identical to `render_oracle_trace`**, main's own unbounded function, which is still present and still used for provenance. So every prompt the bound does not bite is unchanged to the byte — which also means no verdict-reuse digest (#1431) moves for an ordinary run. Honest note on "fails on main": these pin a bound that does not exist on `main`, so the failure there is that `bounded_oracle_trace` is not defined — the same shape as #1932's witnesses for the `reasoning` cap, and the shape any "add a missing ceiling" change has. The behavioural claim is carried by the second test, which compares against main's function directly rather than against a copied expectation. ## The gate - `cargo test -p stella-pipeline` — 585 + 5 + 2 + 5 + 6 + 4 = **607 passed, 0 failed** - `cargo clippy -p stella-pipeline --all-targets` — clean - `cargo fmt --check -p stella-pipeline` — clean - `scripts/check-file-size.sh` — OK, none grew (`evidence.rs` was extracted from `pipeline.rs` precisely so this kind of channel can be added without touching a god file, and that still holds) Full workspace left to CI. ## Nothing left behind `Refs #1787`, deliberately not `Closes` — this is the folded-in bound only. Item 1 (a provider-parity-aware structured verdict output path, invariant 8) remains open and is being approached from a different angle in #1964; item 2 shipped as #1932 and item 3 as #1951. Refs #1787 ## Summary by Sourcery Bound the oracle trace rendered in verifier evidence summaries and added tests to cover the new trusted-zone length cap. New Features: - Introduce a bounded oracle trace renderer for verifier prompts that limits the trusted evidence summary to the newest observations while indicating omissions in-band. Tests: - Add witness tests ensuring long oracle traces are clipped with an explicit omission marker and that ordinary short traces remain byte-identical to the unbounded renderer.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fix(stella-pipeline): bound the verdict reasoning at the point it is constructed
Verdict::reasoningwas the verifier's whole reply, unbounded, and it travels:into the worker's revision prompt and into the verdict cache. A reasoning model
that thinks out loud for 100 KB put 100 KB into the next worker turn — on every
revision round, at the full input rate, and left it in a cache entry that keeps
it.
Measured on the fixture in the witness below, the old code carried 114,006
characters into the revision prompt.
bounded_reasoningclips atMAX_VERDICT_REASONING_CHARS(4,000 — roughly athousand tokens) and says it did.
Three decisions worth naming
Structural, not policy. The disclosure ladder already caps what crosses to
the worker, but as policy applied downstream by one consumer. This bounds the
value where it is constructed, so a consumer added later cannot be written
that forgets — which is what this issue means by "the cap should be
structural".
The head is kept, not the tail. The verifier prompt asks for the verdict
token and its reasons first, so the front is the part a revision acts on. The
witness asserts
starts_with("FAIL")so a later "keep the tail" refactor hasto argue with a test rather than with a comment.
Clipped on a character boundary. A verifier answering in a language of
multi-byte characters is model output, which is runtime data — a
&text[..N]byte slice would panic the pipeline there (invariant 5).
Not re-unbounded downstream
Checked the two places that touch it afterwards:
pipeline.rs's cache-reusearm appends a fixed ~130-character note, and
heuristic_fallbackbuilds itsreasoning from fixed strings. Both are bounded by construction, so the two
parse sites were the only ingress.
Witnesses
a_runaway_verifier_reply_is_clipped_before_it_reaches_the_worker— aan_ordinary_verdict_passes_through_the_bound_unchanged— the bound isinvisible in the case that matters, so a fix that quietly edited every
verdict cannot pass.
clipping_a_multibyte_reply_does_not_panic— the byte-slice trap.Scope
This is item 2 of the three #1787 lists, which says "each sub-fix carries its
own witness". Item 1 (a structured-output path for verifiers whose replies open
with a preamble) needs a provider-parity posture per invariant 8 and is a
larger change; item 3 (per-candidate degradation facts instead of one
once-per-run
AtomicBool) is independent. The issue stays open for both.cargo test -p stella-pipeline— 557 passed, 0 failed. Clippy-D warnings,fmt --check, rustdoc-D warnings,check-file-size,check-god-filesandcheck-left-behindclean.Refs #1787
Summary by Sourcery
Bound verifier verdict reasoning at construction to prevent oversized reasoning payloads from propagating through the pipeline and cache.
Enhancements:
Tests: