fix(stella-pipeline,stella-cli): verdict preamble parsing, pipeline calibration, and a revision halted on an already-green command (#1787, #1595, #1793) - #1964
Merged
Conversation
… solve rate, wall clock and tokens Three independent fixes, each on a path every benchmark task takes. **#1787 — a verifier that thinks before it answers parsed as no answer.** `parse_verifier_response` consulted only the first non-empty line, so a reply opening with "Here is my assessment:" carried no verdict token there and returned `None`. `None` is the signal for `heuristic_fallback`, which passes only on an observed flip or green touched tests and otherwise FAILS — so this did not degrade one verdict, it converted *every* verdict from a preambling model into a heuristic that mostly refuses. Reasoning models preamble by construction, which makes it a whole model family the verifier could not grade with. Two positions are now authoritative, tried in order: the reply's first non-empty line, then its last. Deliberately two and not a scan of the body — intermediate prose is where a verifier *discusses* failing tests, and reading it would reintroduce the misread the token set was narrowed to prevent. The head still wins outright, so every reply that leads with its verdict parses exactly as before. **#1793 — the measured early stop never applied to a revision.** `FlipHalt` (built from a Terminal-Bench regression: a correct solution, then the rest of the wall clock spent re-litigating whether the grader could see it) armed only from a configured `--test-command` baseline. An authored witness reaches the same precondition — an observed FAILING baseline — but in `witness_on_demand`, which never armed it. And `revise_turn` passed `None` unconditionally regardless. Both fixed, and the second is narrower than it first looks. A revision inherits the halt only while the oracle reads `FlipState::Failing`. A verifier can reject a candidate whose witness flipped — the test went green but the review found the change wanting — and arming there would end the revision the first time the model re-ran that passing test, before it addressed anything the verifier objected to. `Flipped` and `Unstable` are both excluded because a pass has been seen and a halt cannot tell a reproducible one from a flake; the sticky latch is refused for the same reason. **#1595 — the default `stella run` estimated worst.** Five of stella-cli's seven assembly sites seeded a `CalibrationMap` and handed it to the engine. The two that did not were `run_pipeline_one_shot` — which *is* `stella run` — and fleet workers, because `Pipeline` had no way to accept one. The cost is larger than a lost seed: with no map at all the correction is inert for the whole run, so the drift those turns measured was never applied to them either. `Pipeline` gains `with_calibration`, borrowed rather than owned because `CalibrationMap` is deliberately not `Clone` (an owned field would also cost `PipelineConfig` its `Clone`). It attaches to all four engines the pipeline builds — the worker's turns, the best-of-N children, and the witness author/repair — via one `attach` helper, because seeding three of four would reproduce this issue's exact shape one level down. Refs #1787 (item 1 narrowed: the token protocol is now preamble-tolerant; the forced structured-output posture remains, and item 3 is untouched) Closes #1793 Closes #1595 ## Witnesses - `a_verifier_that_preambles_is_read_at_its_conclusion` and `intermediate_prose_never_decides_a_verdict` — both fail on main (verified by reverting verify.rs alone and re-running). - `a_revision_inherits_the_halt_while_the_tracked_command_is_still_red` and `a_revision_on_a_green_or_flaky_command_is_never_halted`. - `the_pipeline_path_sizes_its_budget_with_the_callers_calibration` and `a_pipeline_lent_no_calibration_reports_the_identity_factor`, reading the factor off each engine step's `StepManifest` (`call_seq == 0`; management roles ride the same step and have no compaction budget). ## File-size baseline `agent.rs` +3 and `fleet_cmd.rs` +3 are the call-site wiring — the seed and the lend, which need the store, the config and the pipeline all in scope and so cannot live elsewhere. `pipeline/tests.rs` +1 is a `mod` declaration. Against that, `pipeline.rs` ratchets DOWN 11 lines: `with_turn_gate` moved to the new `pipeline/attachments.rs` beside `with_calibration`, and the repeated gate-attachment blocks collapsed into `attach`. ## Left behind Filed rather than fixed: #1948 (the verdict reasoning is bounded twice, in different units — a multi-byte reply is truncated to a third of the cap), #1949 (no end-to-end witness that the authored-witness path arms the halt; the `turn.halt_on_goal_met` parity row stays `ShippedUnwitnessed`), #1950 (no CLI-side witness that the two call sites do the lending; `calibration.drift` stays `ShippedUnwitnessed`). #1787's forced structured-output posture — a provider-parity declaration per invariant 8 — is deliberately NOT in this change and is the larger remaining half of that issue. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01R6sipVC9LGErHS5Sw93Ufb
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
Contributor
Reviewer's GuideFixes three Terminal-Bench-impacting defects: verifier verdict parsing now considers both the first and last non-empty lines; FlipHalt is correctly inherited into revisions only when the tracked command is still failing; and token-drift calibration is plumbed through the staged pipeline and fleet worker paths so all engines use measured drift for budgeting. Sequence diagram for FlipHalt inheritance into revision turnssequenceDiagram
participant Execute as run_candidate
participant Oracle as FlipOracle
participant Halt as FlipHalt
participant Verifier
participant Revise as Pipeline_revise_turn
participant Engine
Execute->>Oracle: observe_run(command, is_configured, baseline_output)
note over Execute,Oracle: Baseline observed as failing
Execute->>Halt: FlipHalt::new(command)
Execute->>Revise: CandidateState.flip_halt = Some(Arc<FlipHalt>)
Verifier->>Revise: reject_candidate()
Revise->>Oracle: oracle.state()
Revise->>Revise: for_revision(&flip_halt, FlipState::Failing)
note over Revise: Returns Some(Arc<FlipHalt>) only when state == Failing and halt not flipped
Revise->>Engine: run_engine_turn(engine, ..., halt = Some(FlipHalt))
Engine-->>Halt: observe(command_output)
Halt-->>Engine: is_flipped() == true
Engine-->>Revise: halt on goal met (fail→pass)
Verifier->>Revise: [revision can proceed on issues]
alt FlipState is Flipped/Unstable/None or halt already flipped
Revise->>Revise: for_revision(&flip_halt, state) == None
Revise->>Engine: run_engine_turn(engine, ..., halt = None)
note over Revise,Engine: Revision is never halted on already-green or flaky commands
end
File-Level Changes
Assessment against linked issues
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
macanderson
marked this pull request as ready for review
August 7, 2026 01:38
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Sorry @macanderson, you have reached your weekly rate limit of 500000 diff characters.
Please try again later or upgrade to continue using Sourcery
![vercel[bot]](https://avatars.githubusercontent.com/in/8329?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Additional Suggestion:
The research-stage engine skips self.attach(), so it inherits neither the pipeline's turn_gate (pause) nor its CalibrationMap (#1595 token-drift correction), unlike the four other engines the pipeline builds.
…build main landed its own #1793 fix (#1945) while this branch was open, arriving at the same two changes: `witness_on_demand` arms `state.flip_halt`, and revise turns receive it instead of `None`. Main's implementation is taken wholesale; this branch's duplicate is dropped. One refinement survives on top, as `flip_halt::for_revision`. Main gates the revision on `FlipHalt::unfired`, which refuses a latch that already fired — the halt the EXECUTE turn stopped on. It cannot cover the other way a revision opens on an already-green command, because that halt has never fired: `witness_on_demand` arms a FRESH `FlipHalt` after execution, and by then the witness it names has usually already flipped (it is written to pass once the work is done). A verifier can still reject such a candidate — a lint regression, a refuted verdict — and the revision would inherit an unfired halt on a command that is green before it starts, ending at the first step boundary where the model re-ran that test. So the oracle decides: only `FlipState::Failing` means the flip is still ahead of the turn. `for_revision` composes with `unfired` rather than replacing it. The research stage (#1778, landed on main since) built its engine without the pipeline's attachments, so it inherited neither the pause gate nor the calibration this branch adds — the exact "seeded N-1 of N engines" shape `attach` exists to prevent. It now goes through `attach` like the other four, which also fixes a pre-existing bug: a paused run's research sub-agents did not park. ## Unbreaking the pipeline test build `origin/main` at 43402ae does not compile its own test suite. Verified in a clean worktree at that commit, three errors, none of them from this branch: * `pipeline/tests/verification_hardening/flip_halt_arming.rs` (added by #1945) references `PassingShell` and `shell_call_result`, which exist nowhere in the crate — the fixtures were never committed. * `management_prompt/tests.rs`'s deliberately-exhaustive match does not cover `ModelCallRole::Research`, added later by #1953. Both are fixed here because this branch cannot be verified, let alone merged, on a base whose tests do not build. The two fixtures are reconstructed from what the test's own comments require, and the semantics are pinned by the test passing for the stated reason: `shell_call_result` puts the command under the `command` key `command_of` reads, and `PassingShell` emits the trailing `[exit code: 0]` marker `exit_status` parses — without either, the halt could never latch and `an_authored_witness_arms_the_revision_flip_halt` would have been green for no reason. `Research` joins the "never dispatched through the management chokepoint" arm because its calls are sub-agents run through `Engine::run_sub_agent_with_sender`, not `metered_raw_call`. `cargo test -p stella-pipeline`: 588 unit + 22 integration, 0 failures, including main's own arming witness. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…run-3mp8zk' into claude/terminal-bench-bugs-solo-run-3mp8zk # Conflicts: # crates/stella-pipeline/src/pipeline/witness_stage.rs # scripts/file-size-baseline.txt
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Owner
Author
|
Pushed 1.
|
macanderson
changed the title
…a private intra-doc link Both reproduce on origin/main at 43402ae and are unrelated to this branch's own changes; see #1972 for the full account. * `plan_stage` reached 8 arguments when #1953 added `research`, one past clippy's limit, so `make lint` fails workspace-wide. Fixed by taking the `Spend` the pipeline already uses for exactly this pair rather than `budget` and `total` loose — the established idiom here, and it removes a line at each of the three sites instead of adding an #[allow]. * `daemon/boot.rs`'s module doc intra-doc-links `SkipReason::NoResumePoint`, which is `pub(super)` and therefore absent from a non-private rustdoc build, failing `cargo doc -D warnings`. Demoted to a code span, matching how #1965 fixed the sibling case in stella-store. Refs #1972 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Aug 7, 2026
…od file The `Spend` bundle that fixed `plan_stage`'s clippy arity carried a three-line rationale comment into `pipeline.rs`, taking it to 3453 against a recorded ceiling of 3451 — the god file rule is that a grandfathered file is closed to growth, so the ratchet failed while clippy, rustdoc and the workspace suite all passed. The rationale moves to `Spend`'s own doc in `stage_budget.rs`, which is where a reader asking "why does this signature take a bundle?" already looks, and which is not a god file. `pipeline.rs` lands at 3450 — one under its ceiling and with the baseline untouched, rather than regenerated to make the gate agree.
Owner
Author
|
Status, and one correction I owe. Correction: the daemon test failure is not diskEarlier I attributed It is a load-sensitive timing flake: the test waits out a real 8-second Where the branch stands
Thanks for What this PR actually contains nowThree things, since #1793 landed independently on
|
macanderson
added a commit
that referenced
this pull request
Aug 7, 2026
…ngress (#1787) (#1982) > **Stacked on #1975.** Based on `unbreak-main-pipeline` because `main` does not > currently compile `stella-pipeline`'s test target; the diff below is the one > commit on top. GitHub retargets this to `main` when #1975 merges. ## What & why The last unbounded ingress into the verdict prompt, and the item #1787 folds in at the end of its body: > Also worth folding in: the trusted evidence summary has no length bound > (`oracle_trace` grows per observation; the diff has a token budget, the > trusted zone does not) — `pipeline/evidence.rs` since the extraction. `verifier_evidence_summary` renders `oracle_trace` in full. That trace gains an observation per verification round, and the repair gate (#1479) keeps granting rounds for as long as a measured budget affords them — so the one channel that grows without limit was also the one channel with no ceiling. Every other input to that prompt is bounded: the diff has a token budget, recall frames have `bound_recalled_frames`, and `Verdict::reasoning` got its cap in #1932. Bounded to the newest 24 observations, with the drop **stated in-band**: ``` oracle_trace=[…76 earlier observation(s) omitted → candidate:pass → candidate:fail → …] ``` Three choices worth naming, because each has a wrong-looking alternative: - **Newest kept, oldest dropped.** The recent runs are what the verdict weighs; a trace clipped from the front would hand the verifier a history that stops before the evidence. - **Stated, not silent.** A trace that silently began mid-run reads as the whole run — the verifier would draw conclusions about a first observation that was not the first. - **Clipped where the value is constructed**, not at a downstream consumer. That is the same "structural, not by convention" rule #1932 applied to `reasoning`, and it is why the stored `LadderSnapshot` and `verdict_provenance` are deliberately untouched: the bound is on the *prompt ingress*, not on the record. 24 is sized far above a normal run (a baseline plus a handful of rounds), so the bound only ever bites a pathological loop. It is a named constant next to its rationale rather than a literal. ## The witness - [x] This PR includes a witness test `a_pathological_oracle_trace_is_clipped_with_the_drop_stated` — a 100-observation trace renders exactly 24 entries behind the `…76 earlier observation(s) omitted →` marker, and still ends on the newest observation. `an_ordinary_oracle_trace_renders_unchanged` is the other half, and the one that matters for regression: a 5-observation trace is asserted **byte-identical to `render_oracle_trace`**, main's own unbounded function, which is still present and still used for provenance. So every prompt the bound does not bite is unchanged to the byte — which also means no verdict-reuse digest (#1431) moves for an ordinary run. Honest note on "fails on main": these pin a bound that does not exist on `main`, so the failure there is that `bounded_oracle_trace` is not defined — the same shape as #1932's witnesses for the `reasoning` cap, and the shape any "add a missing ceiling" change has. The behavioural claim is carried by the second test, which compares against main's function directly rather than against a copied expectation. ## The gate - `cargo test -p stella-pipeline` — 585 + 5 + 2 + 5 + 6 + 4 = **607 passed, 0 failed** - `cargo clippy -p stella-pipeline --all-targets` — clean - `cargo fmt --check -p stella-pipeline` — clean - `scripts/check-file-size.sh` — OK, none grew (`evidence.rs` was extracted from `pipeline.rs` precisely so this kind of channel can be added without touching a god file, and that still holds) Full workspace left to CI. ## Nothing left behind `Refs #1787`, deliberately not `Closes` — this is the folded-in bound only. Item 1 (a provider-parity-aware structured verdict output path, invariant 8) remains open and is being approached from a different angle in #1964; item 2 shipped as #1932 and item 3 as #1951. Refs #1787 ## Summary by Sourcery Bound the oracle trace rendered in the verifier prompt to a fixed number of recent observations and document the truncation in-band while preserving full traces in stored provenance. Enhancements: - Introduce a bounded oracle trace renderer for the verifier prompt, limiting the trusted-zone trace to the newest 24 observations and prefixing output with an omission marker when older entries are dropped. Tests: - Add tests ensuring pathological long oracle traces are clipped with an explicit omission notice and that ordinary short traces remain byte-identical to the existing unbounded rendering.
macanderson
added a commit
that referenced
this pull request
Aug 7, 2026
…a dangling boot.rs doc link (#1985) ## What & why `main`'s required `fmt + clippy + test` job has been **red since #1951** (five consecutive merges), so every open PR inherits a red gate. Four distinct errors are involved. This PR fixes the two that **no open PR covers**. ### 1. `cargo clippy -D warnings` — `plan_stage` has 8 arguments (limit 7) `crates/stella-pipeline/src/pipeline.rs`. #1953 added a `research` parameter, pushing `plan_stage` over `clippy::too_many_arguments`. `Spend<'_>` (`crates/stella-pipeline/src/pipeline/stage_budget.rs`) already groups the `budget`/`total` pair, and **seven** sibling stage methods take it; `plan_stage` was the last one carrying the two loose. Adopting it takes the count to 7. This is the right fix rather than `#[allow(clippy::too_many_arguments)]` because there is no argument that the lint is wrong *here* — the grouping type the lint is asking for already exists — and because `pipeline.rs` is a grandfathered god file closed to growth, which this change **shrinks**. ### 2. `cargo doc -D warnings` — unresolved link in `boot.rs` `crates/stella-cli/src/daemon/boot.rs`. #1939 left ``[`SkipReason::NoResumePoint`]`` in the module doc unresolvable (`no item named `SkipReason` in scope`), which fails `rustdoc::broken_intra_doc_links`. Qualified to the full path — the same remedy #1927 applied to *this same file* for *this same reason* after #1920. That recurrence is filed as #1986. ## What this PR deliberately does NOT fix The other two errors belong to the open PR **#1964**: - `flip_halt_arming.rs` references `PassingShell` / `shell_call_result`, test helpers #1945 landed the test file without. - `crates/stella-pipeline/src/management_prompt/tests.rs`'s exhaustive match omits `ModelCallRole::Research`, which #1953 added. Duplicating them here is how two parallel unbreak PRs collide, so they are left to #1964. **`main` needs both PRs.** This one alone leaves `cargo test` red; #1964 alone leaves clippy and rustdoc red (verified: #1964's own CI still fails both, and its clippy failure is the identical `plan_stage` error). Neither is sufficient on its own, and the merge order does not matter. Because of that, **this PR's own `cargo clippy` and `cargo test` steps will stay red until #1964 merges** — clippy `--all-targets` reaches the missing test helpers once the lib error is gone. That is expected, not a regression. ## The witness - [x] No witness test needed — a lint fix and a doc-link fix, neither a behavior change. Verified the artisanal way: - `RUSTDOCFLAGS="-D warnings" cargo doc -p stella-cli --no-deps` fails on `main` and **exits 0** with this change. - `cargo check -p stella-pipeline --all-targets` reports **only** the three #1964-owned errors; the `plan_stage` arg-count error is gone and both call sites (`pipeline/scope_stage.rs`, `pipeline/tests/management_accounting.rs`) compile. - `cargo fmt --check -p stella-pipeline -p stella-cli` exits 0. ## The gate - [x] `check-file-size.sh` and `check-god-files.sh` pass; `pipeline.rs` shrinks, so no baseline change is needed. - [x] No behavior change, no new flags, no docs pages affected. - [x] No new dependencies. ## Nothing left behind - **#1986** — `boot.rs`'s module doc has now broken `main`'s rustdoc twice in two days by the same mechanism (#1920 → #1927, then #1939 → this PR), because `ci.yml` does not run on a push to `main`. Filed as a handoff. - **#1974** — `CandidateState` is hand-built at two sites, which is how #1951's field addition silently broke PR #1962 at merge time. Noticed in the same investigation. Refs #1953, #1939, #1964
1 task
This was referenced Aug 7, 2026
macanderson
added a commit
that referenced
this pull request
Aug 7, 2026
…ngress (#1787) (#2012) > Supersedes #1982, which was auto-closed when the branch it was stacked on > went away. Same single commit, now rebased directly on `main`. > > Note: `main` currently fails `cargo clippy -p stella-pipeline` with five > pre-existing warnings unrelated to this change (fixed in #2009), so this > PR's clippy step inherits them until that lands. ## What & why The last unbounded ingress into the verdict prompt, and the item #1787 folds in at the end of its body: > Also worth folding in: the trusted evidence summary has no length bound > (`oracle_trace` grows per observation; the diff has a token budget, the > trusted zone does not) — `pipeline/evidence.rs` since the extraction. `verifier_evidence_summary` renders `oracle_trace` in full. That trace gains an observation per verification round, and the repair gate (#1479) keeps granting rounds for as long as a measured budget affords them — so the one channel that grows without limit was also the one channel with no ceiling. Every other input to that prompt is bounded: the diff has a token budget, recall frames have `bound_recalled_frames`, and `Verdict::reasoning` got its cap in #1932. Bounded to the newest 24 observations, with the drop **stated in-band**: ``` oracle_trace=[…76 earlier observation(s) omitted → candidate:pass → candidate:fail → …] ``` Three choices worth naming, because each has a wrong-looking alternative: - **Newest kept, oldest dropped.** The recent runs are what the verdict weighs; a trace clipped from the front would hand the verifier a history that stops before the evidence. - **Stated, not silent.** A trace that silently began mid-run reads as the whole run — the verifier would draw conclusions about a first observation that was not the first. - **Clipped where the value is constructed**, not at a downstream consumer. That is the same "structural, not by convention" rule #1932 applied to `reasoning`, and it is why the stored `LadderSnapshot` and `verdict_provenance` are deliberately untouched: the bound is on the *prompt ingress*, not on the record. 24 is sized far above a normal run (a baseline plus a handful of rounds), so the bound only ever bites a pathological loop. It is a named constant next to its rationale rather than a literal. ## The witness - [x] This PR includes a witness test `a_pathological_oracle_trace_is_clipped_with_the_drop_stated` — a 100-observation trace renders exactly 24 entries behind the `…76 earlier observation(s) omitted →` marker, and still ends on the newest observation. `an_ordinary_oracle_trace_renders_unchanged` is the other half, and the one that matters for regression: a 5-observation trace is asserted **byte-identical to `render_oracle_trace`**, main's own unbounded function, which is still present and still used for provenance. So every prompt the bound does not bite is unchanged to the byte — which also means no verdict-reuse digest (#1431) moves for an ordinary run. Honest note on "fails on main": these pin a bound that does not exist on `main`, so the failure there is that `bounded_oracle_trace` is not defined — the same shape as #1932's witnesses for the `reasoning` cap, and the shape any "add a missing ceiling" change has. The behavioural claim is carried by the second test, which compares against main's function directly rather than against a copied expectation. ## The gate - `cargo test -p stella-pipeline` — 585 + 5 + 2 + 5 + 6 + 4 = **607 passed, 0 failed** - `cargo clippy -p stella-pipeline --all-targets` — clean - `cargo fmt --check -p stella-pipeline` — clean - `scripts/check-file-size.sh` — OK, none grew (`evidence.rs` was extracted from `pipeline.rs` precisely so this kind of channel can be added without touching a god file, and that still holds) Full workspace left to CI. ## Nothing left behind `Refs #1787`, deliberately not `Closes` — this is the folded-in bound only. Item 1 (a provider-parity-aware structured verdict output path, invariant 8) remains open and is being approached from a different angle in #1964; item 2 shipped as #1932 and item 3 as #1951. Refs #1787 ## Summary by Sourcery Bound the oracle trace rendered in verifier evidence summaries and added tests to cover the new trusted-zone length cap. New Features: - Introduce a bounded oracle trace renderer for verifier prompts that limits the trusted evidence summary to the newest observations while indicating omissions in-band. Tests: - Add witness tests ensuring long oracle traces are clipped with an explicit omission marker and that ordinary short traces remain byte-identical to the unbounded renderer.
macanderson
added a commit
that referenced
this pull request
Aug 7, 2026
…es left behind (#2014) ## What & why `main` is red at `6c345532` on **three separate gates** — `cargo fmt --check`, `cargo clippy -D warnings`, and the file-size ratchet — plus workspace rustdoc. Every open PR inherits all of it. The cause is not one bad change. Four sessions fixed the *same* red base concurrently (#1964, #1970, #1971, and an earlier push to #1964's branch). The merge that closed #1964 resolved every overlap by **keeping both sides**, which is the dangerous resolution here: it produces code that still compiles, so nothing conflicted and nobody had to look at it, and the damage only shows up under `-D warnings`. ### clippy (`-D warnings`) — four merge artefacts | Site | Lint | |---|---| | `management_prompt/tests.rs` | `ModelCallRole::Research` appears **twice** in one `\|` chain → `unreachable_patterns` | | `pipeline/scope_stage.rs` | a hoisted `let mut spend` **and** a per-iteration inline `Spend` → `unused_variables` + `unused_mut` | | `tests/verification_hardening.rs` | `SHELL_TOOL`, `shell_call_result`, `PassingShell` duplicated into the `flip_halt_arming` child → three `dead_code` | In each case the duplicate is deleted and the *used* copy kept. For `scope_stage` that is the inline `Spend`, because the loop replans after a rejected scope card and a moved bundle could not be handed to the next attempt — #1971's comment beside it already says so. ### fmt `crates/stella-protocol/src/event/tests.rs` is missing the trailing newline `rustfmt` wants after `mod tag_table;`. Unrelated to the merges and failing on its own. ### rustdoc `StepUsage` links a bare `` [`CompactionRewrite`] ``, re-exported at the crate root but never in `event`'s scope — the next line already spells the field `crate::CompactionRewrite`, so the link now matches. This one was **invisible** until the `stella-cli` link above it was fixed: `cargo doc` stops at the first failing crate, so a broken link one dependency layer down masks every link beneath it. Third occurrence of that pattern here. ### file-size ratchet `driver.rs` and `pipeline/tests.rs` each sit one line over a stale ceiling. Regenerated with `make file-size-update` rather than hand-edited — which is why the diff mostly **tightens**: `pipeline.rs` drops 3451 → 3181 and `bus.rs` 2126 → 1891. Both were already true and neither was recorded. ## The witness No witness test: this is a build/lint/format repair with no behaviour change. The gate *is* the witness, and each failure was reproduced locally before and after. ## The gate Run on this tree, not inferred: - [x] `make guards-fast` — all 25 guards plus `cargo fmt --check` - [x] `cargo clippy --workspace --all-targets -- -D warnings` - [x] `RUSTDOCFLAGS="-D warnings" cargo doc --workspace --no-deps` (24 crates) - [x] `cargo test --workspace` - [x] `./scripts/check-file-size.sh` and `check-god-files` ## Nothing left behind Already filed and linked, not duplicated: - **#1986** — `ci.yml` does not run on a push to `main`, which is *why* all eight of these landed unnoticed. This PR is the fourth cleanup in a row caused by that gap; it is the fix worth prioritising. - **#1972** — the original red-main report this chain started from. - **#1645** — red PRs keep landing (`enforce_admins` off). - **#1977** — `ALL_ROLES` is a hand-maintained array that can silently under-test the role family; it is exactly what let the `Research` arm drift in the first place. Refs #1986 Refs #1972 Refs #1977 ## Summary by Sourcery Repair main by resolving merge artefacts and bringing formatting, linting, documentation, and file-size checks back to green. Bug Fixes: - Remove duplicated management prompt role arm to fix unreachable pattern lint. - Drop unused scope-stage budget variable to clear unused variable lints. - Delete duplicate shell tooling fakes from verification hardening tests to remove dead code lints. - Correct rustdoc link for compaction rewrite events so documentation builds cleanly. - Add missing trailing newline in event tests module to satisfy rustfmt. Enhancements: - Clarify documentation around daemon boot resume-point handling and location of shell doubles for flip-halt tests. Build: - Regenerate file-size baseline to reflect current driver and pipeline module sizes, re-aligning with the file-size ratchet checks. Tests: - Tighten test layout by consolidating flip-halt shell doubles into a single module referenced by the arming tests. Chores: - Minor comment and whitespace cleanups across daemon boot, pipeline scope stage, and event tests.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Three defects picked off the backlog for Terminal-Bench impact, one per metric. #1793 landed on
mainindependently while this was open (#1945), so this PR now defers to that implementation — see §2. What remains is described below.#1787 — solve rate: a verifier that thinks before it answers parsed as no answer
parse_verifier_responseconsulted only the first non-empty line. A reply opening withHere is my assessment:carried no verdict token there and returnedNone— andNoneis the signal forheuristic_fallback, which passes only on an observed flip or green touched tests and otherwise FAILS. So this did not degrade one verdict; it converted every verdict from a preambling model into a heuristic that mostly refuses. Reasoning models preamble by construction, which makes this a whole model family the verifier could not grade with.Two positions are now authoritative, tried in order: the reply's first non-empty line, then its last. Deliberately two, and not a scan of the body — intermediate prose is where a verifier discusses failing tests, and reading it would reintroduce the misread the token set was narrowed to prevent. The head still wins outright, so a reply that leads with its verdict parses exactly as before whatever its closing line says.
This is the narrow half of the issue's item 1. The forced structured-output path needs a provider-parity posture per invariant 8 and is deliberately not here; #1787 stays open for it and for item 3.
#1595 — cost: the default
stella runestimated worstFive of
stella-cli's seven assembly sites seeded aCalibrationMapand handed it to the engine. The two that did not wererun_pipeline_one_shot— which isstella run— and fleet workers, becausePipelinehad no way to accept one. The cost is larger than a lost seed: with no map at all the correction is inert for the whole run, so the drift those turns measured was never applied to them either.Pipelinegainswith_calibration, borrowed rather than owned becauseCalibrationMapis deliberately notClone(an owned field would also costPipelineConfigitsClone, which many call sites rely on). It attaches to every engine the pipeline builds through oneattachhelper — because seeding N-1 of N engines is exactly the shape of the bug being fixed, one level down.That turned out to be load-bearing within this PR: the review bot caught that the research stage (#1778, landed on
mainsince I wroteattach) built its engine outside it. Fixed, and it also fixes a pre-existing bug unrelated to calibration — a paused run's research sub-agents did not park, because that engine never received the turn gate either.#1793 — a revision halted on an already-green command
mainfixed #1793 first (#1945), arriving at the same two changes this branch had. Main's implementation is taken wholesale and the duplicate dropped.One refinement survives, as
flip_halt::for_revision, composed on top ofFlipHalt::unfiredrather than replacing it.unfiredrefuses a latch that already fired — the halt the execute turn stopped on. It cannot cover the other way a revision opens on an already-green command, because that halt has never fired:witness_on_demandarms a freshFlipHaltafter execution, and by then the witness it names has usually already flipped — the ordinary case, since a witness is written to pass once the work is done. A verifier can still reject such a candidate (a lint regression, a refuted verdict), and the revision then inherits an unfired halt on a command that is green before it starts, ending at the first step boundary where the model re-runs that test — before addressing anything the verifier objected to.So the oracle decides: only
FlipState::Failingmeans the flip is still ahead of the turn. Same rulerun_candidatealready applies when it refuses to arm on a green configured baseline.Closes #1595
Refs #1787
Refs #1793
Refs #1972
The witness
main, passes here)a_verifier_that_preambles_is_read_at_its_conclusionintermediate_prose_never_decides_a_verdictNonethe_pipeline_path_sizes_its_budget_with_the_callers_calibrationa_pipeline_lent_no_calibration_reports_the_identity_factora_revision_on_a_green_or_flaky_command_is_never_haltedFlipState::Failingguard — the halfunfiredcannot covera_revision_inherits_the_halt_while_the_tracked_command_is_still_redThe two #1787 witnesses were checked the artisanal way:
git stash push -- crates/stella-pipeline/src/verify.rs(source only, tests left in place) → both fail; restored → both pass.The calibration witnesses read the factor off each engine step's
StepManifest, filtered tocall_seq == 0. The pipeline's management roles ride the same step at 1, 2, … and carry the identity factor whatever the engines were lent — a raw management call has no transcript to compact — so counting them would make the test assert something untrue of the seam.The gate
cargo fmt --checkcargo clippy --workspace --all-targets -- -D warningscargo test --workspacepipeline/attachments.rsmodule doc;for_revision's rule stated where the semantics live;stella-parity'scalibration.driftrow rewritten)Closes #Nappears both here and as a commit trailerNothing left behind
Filed: verify: the verdict reasoning is bounded twice, in different units — a multi-byte reply is truncated to a third of the cap and carries two markers #1948, pipeline: no end-to-end witness that an authored witness arms the mid-turn flip halt (parity row turn.halt_on_goal_met stays unwitnessed) #1949, cli: no witness that run_pipeline_one_shot and the fleet worker actually lend their seeded CalibrationMap to the pipeline #1950, main is red on three axes: stella-pipeline tests don't compile, clippy arity, and a private intra-doc link #1972
main is red on three axes: stella-pipeline tests don't compile, clippy arity, and a private intra-doc link #1972 —
maincannot compilestella-pipeline's test suite, independent of this branch. Fixed here because this PR could not be verified on it; filed separately so unbreakingmainis tracked either way. Details in the comment above.verify: the verdict reasoning is bounded twice, in different units — a multi-byte reply is truncated to a third of the cap and carries two markers #1948 —
Verdict::reasoningis bounded twice in different units (chars at construction, bytes when forwarded); a multi-byte reply gets cut to a third of the budget with two truncation markers.pipeline: no end-to-end witness that an authored witness arms the mid-turn flip halt (parity row turn.halt_on_goal_met stays unwitnessed) #1949 — the arming half of FlipHalt never arms on the authored-witness path, and revise turns pass None even for configured commands #1793 has no end-to-end witness beyond
main's own;turn.halt_on_goal_metstaysShippedUnwitnessed.cli: no witness that run_pipeline_one_shot and the fleet worker actually lend their seeded CalibrationMap to the pipeline #1950 — nothing proves
run_pipeline_one_shotand the fleet worker do the lending. Delete either.with_calibration(..)and the workspace suite stays green — which is how the original gap survived.calibration.driftstaysShippedUnwitnessedwith itsmissingfield rewritten to say so.Ground-rule check
stella-core; no new dependenciesAnything reviewers should know?
Three commits here are not mine to claim as feature work — they unbreak the base.
mainat43402aefailscargo test -p stella-pipeline --no-runwith three errors (#1945's test fixtures were never committed; #1953's newModelCallRoledoesn't satisfy #1941's deliberately-exhaustive match), andcargo doc -D warningsfails on a private intra-doc link indaemon/boot.rsthat is byte-identical onmain. Reconstructing #1945's fixtures is the part worth a reviewer's eye: the semantics are pinned by its witness passing for the stated reason (thecommandkeycommand_ofreads; the[exit code: 0]markerexit_statusparses), but if that author intended something different, this is where to say so.The file-size baseline moves, and mostly down.
agent.rs+3 andfleet_cmd.rs+3 are the call-site wiring — the seed and the lend, which need the store, the config and the pipeline all in scope.pipeline/tests.rs+1 is amoddeclaration. Against those,pipeline.rsratchets down 11 lines:with_turn_gatemoved into the newpipeline/attachments.rsbesidewith_calibration, and the repeated gate-attachment blocks collapsed intoattach.Not measured on the bench. These are picked for Terminal-Bench impact and argued from the code and the issues, not from a before/after run — no provider credentials were available in the environment this was written in, so no benchmark run was performed. The cost and wall-clock claims are mechanism-level; #1770/#1289 are where a real measurement would land.