feat(stella-protocol): a signal-consumer ledger — every AgentEvent variant declares what reads it - #2720
Merged
Merged
Conversation
…riant declares what reads it "Produced and not consumed" is a repo-wide shape, and every instance so far was found by a bench run paying for it rather than by a test going red: flip.json written with nothing reading it (#1536); verify_done confirmations tallied but not feeding the halt, which cost solved_then_timeout four times on one certification panel before #2661 wired it; the flip transition still emitting nothing durable, so a shipped halt cannot be measured in the field. Each was fixed as whack-a-mole. This makes the class structural, copying content_free.rs element for element: a reviewed table, tests enforcing it from both sides, and negative controls proving the harness can fail. Every wire tag gets a row declaring a ConsumerPosture — Behavioral names the code that branches on it, Surfaced names the surfaces that select it, RecordedOnly and Unclassified each cite the issue where the gap is being decided. Unclassified sits under a down-only ratchet so a new variant cannot be filed away unread. Three design calls that differ from the issue's sketch, each forced by the code: - Posture and surfaces are orthogonal fields, not one enum. tool_result is both Behavioral and on the observatory whitelist; folding them together would have made #2707's parity test unsatisfiable on day one. - Surface names only the observatory and serve. The TUI and replay match AgentEvent exhaustively, so membership is compiler-guaranteed for every variant and a per-row claim would record nothing. - The ledger is a hand-maintained table rather than a third output of agent_event_tags!. That macro is the wire decoder's source of truth and event.rs sits 35 lines under the ratchet; both reasons are in the module doc so the next author does not re-litigate it. The tool_result row is the ledger earning its keep already: the intuitive consumer is loop detection, which is wrong — the detector reads the transcript, a different plane. The real event-plane consumer is the store's tool_calls projection. Refs #2701 Refs #2703
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Sorry @macanderson, you have reached your weekly rate limit of 500000 diff characters.
Please try again later or upgrade to continue using Sourcery
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
Contributor
Reviewer's GuideIntroduces a hand-maintained signal-consumer ledger for AgentEvent variants, enforcing via tests that every wire tag declares how it is consumed and tightening documentation/invariant structure around signal consumption. Sequence diagram for auditing the signal-consumer ledgersequenceDiagram
participant TestRunner
participant EventModule
participant audit_ledger
TestRunner->>EventModule: audit()
EventModule->>audit_ledger: audit_ledger(SIGNAL_CONSUMERS, KNOWN_TYPE_TAGS)
audit_ledger->>audit_ledger: check rows for DuplicateRow / UnknownTag
audit_ledger->>audit_ledger: check known_tags for MissingRow
audit_ledger-->>EventModule: Vec<LedgerViolation>
EventModule-->>TestRunner: Vec<LedgerViolation>
File-Level Changes
Possibly linked issues
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
![vercel[bot]](https://avatars.githubusercontent.com/in/8329?s=60&v=4){kind=small}
…TYPE_TAGS The symbol is in scope via the module's own use, so the bare intra-doc label already resolves and rustdoc's redundant-explicit-links lint rejects the spelled-out path. Caught by CI's doc-warnings step, which the local run of this change skipped.
…mer-ledger # Conflicts: # AGENTS.md
macanderson
enabled auto-merge (squash)
August 10, 2026 20:55
10 tasks
Removing the invariant number from this string left the parenthetical saying exactly what the prose ahead of it already said. Caught in review on #2720.
7 tasks
macanderson
added a commit
that referenced
this pull request
Aug 10, 2026
… instead of aborting (#2677) (#2744) ## What & why Under a sustained 429 brownout, `RetryPolicy::standard()`'s inline ladder (6 retries, ~16s total backoff) exhausted in seconds and a `Retry-After` past the fixed 120s ceiling failed `Terminal` outright — even with hundreds of seconds of task budget left. #2667 measured the loss: 7 attempts burned in 15s, killing a trial with 880s of its 900s budget remaining (8/16 cells in `turnloop-cert-panel1`). The comparator lineage retries 429s reset-aware for hours with chunked keep-alives. **Abort becomes recovery.** A rate limit the inline ladder cannot absorb now *parks* instead of dying, bounded by real wall-clock headroom: - **`crates/stella-core/src/retry.rs`** — a new `ParkSupervisor` port (the same seam shape as `Sleeper`: the retry driver stays pure decision logic; clock, budget, events, and the soft-stop latch never leak in). The pure planner `plan_rate_limit_park` decides the wait: a server `Retry-After` that fits the allowance is honored verbatim (waiting less than the server asked guarantees a re-429, so an unaffordable hint still fails fast — the fail-fast the old fixed ceiling provided survives, now measured against real headroom); hint-less parks follow a doubling 30s→5min backoff, the comparator's cap. Long waits sleep in 30s chunks with a supervisor tick before each one. Exemplar for the port-with-no-op-default shape: `tokio`'s injected-time seam / the crate's own `Sleeper`+`ports::Clock` discipline. - **`crates/stella-core/src/driver/rate_limit.rs`** (new sibling module, the `settlement.rs` pattern) — the whole attempt-ladder block moves out of god-file `driver.rs` (which **shrinks by 76 lines**), and the engine's supervisor lands beside it: allowance = task-deadline remaining minus a 30s wake reserve, capped at an absolute 6h; `TurnParked`/`TurnWoken` span (**no new `AgentEvent` variant** — the existing #1857 span fits and keeps this orthogonal to #2720's ledger) plus a per-chunk keep-alive `Text` narration for liveness; soft stop ends the park at the next chunk. - **L-M4 and invariant 6 hold**: `RetryPolicy::deterministic()` and every auxiliary call (`accounted_call.rs` — summarizer/triage/authoring) keep `NoParking`, matching the comparator's background-source posture; the park sits between model attempts, never mid-tool, and the budget's deadline bounds it from outside. Anthropic's `Retry-After` already reaches `RateLimited::retry_after_ms` via `parse_retry_after_ms`/`classify_http_status` (`crates/stella-model/src/http.rs`, 429 arm — wiremock-covered there); this PR consumes that hint at the seam where the abort actually happened. `stella-model` does not link `stella-core`, so the recovery witness lives at the driver seam with a scripted provider. Closes #2677 Closes #2667 ## The witness - [x] This PR includes witness tests (fail on `main`, pass here) Verified the artisanal way — `git checkout origin/main -- crates/stella-core/src/{retry,driver,accounted_call}.rs && rm crates/stella-core/src/driver/rate_limit.rs` with the test file kept (the driver witnesses use only pre-existing API), then `cargo test -p stella-core --lib parked_wait`: - `driver::tests::parked_wait::sustained_rate_limiting_parks_within_budget_and_recovers` — **fails on main** (`Aborted { reason: "model call failed: provider rate limited: throttled", kind: Failure }`), passes here: 9 hint-less 429s → 6 inline retries + 3 parks → completed step, `TurnParked`/`TurnWoken` on the stream, no `RetriesExhausted`. - `driver::tests::parked_wait::a_long_retry_after_hint_is_honored_by_parking_when_budget_allows` — **fails on main** (`Terminal: …asked to wait 300000ms… past this call's 120000ms server-hint ceiling`), passes here: the 300s hint parks (span deadline 300–338s incl. jitter) and the step completes. - `a_hint_past_the_remaining_deadline_still_fails_fast_without_parking` pins the kept fail-fast half (60s deadline vs 300s hint → clean abort, no park). - Pure-logic coverage: `retry::tests::sustained_rate_limiting_beyond_the_ladder_parks_and_recovers`, `an_abort_tick_ends_the_park_and_surfaces_the_error`, `the_deterministic_policy_never_parks_even_with_allowance`, and property `a_planned_park_never_exceeds_the_allowance_and_never_undercuts_the_hint` (proptest, per the crate's pure-engine-logic testing idiom). All pre-existing retry tests pass unchanged — `retry_with_backoff` runs `NoParking`, preserving pre-#2677 behavior exactly for supervisor-less callers. ## The gate - [x] `cargo fmt --check` - [x] `cargo clippy --workspace --all-targets -- -D warnings` (stella-core + all dependents checked) - [x] `cargo test -p stella-core` (1089 passed) — full workspace runs in CI - [x] Docs updated where behavior changed (module docs in `retry.rs`, `max_server_hint_ms` semantics, new module doc) - [x] CLA signed - [x] `Closes #N` appears both above and as commit trailers ## Nothing left behind - Filed: #2741 (runner-side halves of #2667: infrastructure classification of 429 aborts + fan-out token bucket), #2742 (529/overloaded classifies as `Transport`, so a sustained 529 brownout never reaches the park), #2743 (a soft stop during a park surfaces as a `Failure` abort instead of the graceful boundary soft-stop exit) ## Ground-rule check - [x] No I/O added to `stella-core` (the supervisor is a port; the only clock read is `Instant::now` in the driver, matching the existing neighborhood); no new deps - [x] No new outbound network calls - [x] No new cross-boundary types (`TurnParked`/`TurnWoken` reused; nothing added to `stella-protocol`) ## Anything reviewers should know? - `retry_with_backoff_observed` gained a required `&mut dyn ParkSupervisor` parameter (it is `pub(crate)`); the public `retry_with_backoff` signature is unchanged. - The exhausted-ladder error class is preserved: a rate limit that gives up hint-less still surfaces `RateLimited` (so `RetriesExhausted::retryable` stays truthful per #926); only a genuinely oversized-and-unaffordable hint keeps the explanatory `Terminal`, now naming hint, inline ceiling, and remaining headroom. - Deliberately **no new `AgentEvent` variant** to stay orthogonal to #2720's consumer ledger; if a typed per-chunk heartbeat is wanted later it should be added post-ledger. - Alternative rejected: a `max_parked_wait_ms` field on `RetryPolicy` — the allowance is dynamic (deadline-derived), so a policy constant would either lie or duplicate the supervisor's answer. ## Summary by Sourcery Add budget-aware parked waiting for sustained provider rate limiting and refactor the model-call retry ladder into a dedicated driver module. Bug Fixes: - Prevent sustained 429 rate limiting from prematurely aborting turns when there is remaining wall-clock budget. - Ensure long Retry-After hints beyond the inline ceiling are honored via parked waits when affordable, and fail fast only when they cannot fit within the remaining deadline. Enhancements: - Introduce a pluggable ParkSupervisor interface and pure park planning logic to bound and narrate long rate-limit waits without coupling retry logic to clocks or budgets. - Move the attempt ladder and its rate-limit handling out of driver.rs into a new driver::rate_limit module, reducing driver size and centralizing retry-related effects. - Keep auxiliary accounted calls and deterministic retry policy on a no-parking path so background and helper traffic continue to fail fast under rate limiting. Tests: - Add unit and property tests for park planning behavior to guarantee waits respect allowances and server hints. - Add integration-style driver tests covering sustained 429 recovery via parking, honoring long Retry-After hints within budget, and fail-fast behavior when hints exceed remaining headroom.
macanderson
added a commit
that referenced
this pull request
Aug 10, 2026
…he tag table, making totality a compile error (#2737) **#2720 has merged**; this now targets `main` directly. It was authored stacked on that PR, so the merge resolution here kept the generated ledger from this branch and the `MissingRow` message fix that landed on main with #2720. ## What & why #2720 shipped the signal-consumer ledger as a hand-maintained table enforced by a **test**. That was strictly weaker than the `E0004` tripwire `agent_event_tags!` already gave the tag list, and it shipped that way for exactly one reason, recorded in its module doc at the time: `event.rs` sat 35 lines under the 1500-line ratchet with no baseline entry, and the postures are worth roughly a hundred. This removes that constraint. The variant table moves to `crates/stella-protocol/src/event/tags.rs` and grows a `ConsumerPosture` per row, so `SIGNAL_CONSUMERS` is generated from the same list as `type_tag()` and `KNOWN_TYPE_TAGS`. `event.rs` drops from 1465 to 1350 lines. **No baseline entry was added and none moved** — `stella-protocol` still has zero grandfathered files, which is the state to keep it in. ## The witness Adding an `AgentEvent` variant with no declared consumer is now a **build** failure, not a test failure. Verified by adding a probe variant and running `cargo build` (not `cargo test`): ``` error[E0004]: non-exhaustive patterns: `&AgentEvent::HaltFired { .. }` not covered --> crates/stella-protocol/src/event/tags.rs:95:23 | 95 | match self { | ^^^^ pattern `&AgentEvent::HaltFired { .. }` not covered ``` That is the whole point of the change and the only evidence it landed, so it is quoted rather than asserted. (The probe used `HaltFired` — the variant #2704 will actually add. It will hit this and need one row.) ## What generation does not buy Three `LedgerViolation` kinds — `MissingRow`, `UnknownTag`, `DuplicateRow` — become unrepresentable for the real ledger. They are **deliberately kept**, and the module doc now says why: `audit_ledger` takes its ledger and tag list as parameters precisely so the negative controls can hand it broken input. Deleting the structural rules because one call site can no longer trip them would cost the harness its ability to prove itself with them — the vacuous-harness failure `content_free.rs` exists to prevent. The semantic half still runs and still matters: that every gap posture cites a real `#1234` reference, that a `Behavioral` row names somewhere to look, and that the posture agrees with `surfaces`. Those are judgements about a row's content, and no macro can hold an author to them. ## Compatibility `KNOWN_TYPE_TAGS` keeps its path through `event` via a re-export, so `crate::KNOWN_TYPE_TAGS`, `schema_export.rs`, `tests/wire_contract.rs`, and every existing intra-doc citation are untouched. The "Adding a variant?" propagation comment — the map to the four compile-enforced and two silent downstream matchers — moved with the table rather than being stranded in `event.rs` away from what it describes. ## Deleted tests None. ## The gate - [x] `cargo build -p stella-protocol` - [x] `cargo test -p stella-protocol` — 128 tests, all green - [x] `cargo test --workspace` — green - [x] `make lint` (clippy `-D warnings`, workspace, all targets) - [x] `RUSTDOCFLAGS="-D warnings" cargo doc` — green (run unpiped; a piped cargo doc can report a false green) - [x] `scripts/check-file-size.sh` — OK, nothing went over, no baseline change - [x] `cargo fmt --all` Closes #2730 Refs #2701 Refs #2702
11 tasks
macanderson
added a commit
that referenced
this pull request
Aug 10, 2026
… streams, with a first-byte deadline (#2686) (#2748) ## What & why A provider stream that hangs before its first byte (a proxy buffering the SSE body) or comes back as a 200 with an **empty stream** used to burn the whole retry budget re-issuing the identical streaming request into the same broken pipe: the per-chunk idle bound only fired after 120s per attempt, classified as retryable Transport, and every retry streamed again — a provider that would answer fine over a non-streaming request was treated as down. This is #2686's zappy-style "reactive resilience" fallback, adapted to Stella's accounting discipline. Three pieces, all in `stella-model` (the engine is untouched): - **A first-byte deadline, distinct from the idle bound.** `http::FIRST_BYTE_TIMEOUT` (90s, matching the comparator's stream watchdog) bounds only the FIRST body read; any chunk at all moves the stream onto the ordinary 120s `STREAM_IDLE_TIMEOUT`. It composes with — rather than duplicates — the engine's 816s idle-based model deadline: a hang now surfaces in 90s instead of 120s/816s. `http::next_stream_read` (the new `StreamRead` enum) keeps Idle distinct from Failed; `next_with_timeout` is reimplemented on top of it so there is one truth. - **A latched, bounded recovery** (`src/stream_recovery.rs`): Streaming → Probing → Confirmed. A fallback-eligible fault (hung before the first byte, or the stream died having delivered *no completion signal whatsoever*) arms the probe; the probe's unary success confirms the latch for the session (a buffering proxy buffers deterministically); a failed probe **reverts** to streaming, so one unproven fault never condemns the streaming path. The retried attempt re-issues the byte-identical body with `stream: false` (`zai/unary.rs`) through `http::unary_client`'s 600s read bound (#547's lesson — a unary call has no first token to reset the clock). - **Accounting rides the existing rails.** The recovery is deliberately split across two attempts rather than hidden inside one provider call: the faulted streaming attempt fails as retryable Transport, so the engine's retry ladder stays the single owner of retries and bills the discarded attempt through its existing per-attempt `UsageIncomplete` observer (`driver.rs::run_model_call`) — nothing arrived by construction of the eligibility rule, so `partial` is honestly `None`, exactly `http::partial_usage`'s "attach nothing rather than a zeroed envelope" discipline. A mid-stream death WITH salvage is deliberately ineligible: it keeps its retry-as-a-stream path and its `attach_partial` accounting, so there is no partial transcript to tombstone, ever. **Invariant 8 (parity, declared not assumed):** the matrix gains its third axis, `StreamFallbackPosture`. The shared chat-completions adapter's identities (`zai`, `openrouter`, `xai`, `deepseek`, `local`, settings-defined gateways) declare `UnaryFallback` with named witnesses; `anthropic`/`openai`/`gemini`/`vertex` declare `StreamingOnly` with the gap tracked in #2746; `bedrock` is `AlwaysUnary` (it calls Converse — there is no stream to fall back from). Enforced from both sides like the other two axes: `stella-cli`'s config test fails a seeded provider with no row, and the parity tests fail a row whose witness rotted. AGENTS.md invariant 8 and the crate README are updated in the same PR (stale "two axes" prose is a bug). **Exemplar:** the latch follows zappy's watchdog-and-fallback shape; the split-across-attempts structure follows this repo's own `retry.rs`/`UsageIncomplete` pattern rather than an internal hidden resend, so accounting and attempt-bounding stay owned by one place. **God files:** `zai.rs` was at its 1565-line ceiling, so the SSE aggregation half moved to `zai/stream.rs` (the `driver/settlement.rs` split pattern) and the unary path landed in `zai/unary.rs`; `zai.rs` is now under 1500 and its baseline entry is retired (minimal baseline edit — deliberately NOT a full `file-size-update` regen, to avoid the parallel-merge retighten skew). `zai/tests.rs` was also at its ceiling, so the three `reasoning_aware_max_tokens` tests moved to `zai/tests/zai_effort.rs` (tests moved, not deleted — same names, new file). The streaming and unary paths share one set of assembly helpers (`tool_call_input`, `promote_reasoning_as_text`, `fold_usage`, `final_finish_reason`) so they cannot drift. Closes #2686 ## The witness - [x] This PR includes a witness test (fails on `main`, passes here) `crates/stella-model/src/zai/tests/stream_fallback.rs`: - `a_stream_hung_before_its_first_byte_falls_back_to_a_non_streaming_request` — the headline #2686 witness. Wiremock cannot hold a response body open (its delays cover the whole response), so this one uses a minimal hand-rolled std-TCP server that answers a `"stream":true` request with headers and **not one body byte**, and a `"stream":false` request with a real completion. First call faults retryably at the first-byte deadline naming the switch; second call completes unary. - `an_empty_stream_falls_back_to_a_non_streaming_request` — the other broken shape (200 + EOF before any data), wiremock-based; the unary response carries a tool call, proving the fallback parses the whole dialect. - Controls: `a_healthy_stream_never_arms_the_fallback` (every request keeps `stream:true`), `a_stream_that_died_after_content_is_retried_as_a_stream_not_unary` (eligibility boundary), `a_failed_unary_probe_reverts_the_session_to_streaming` (wire sequence `stream → unary → stream` proves the reversion), plus latch state-machine units in `stream_recovery.rs`. **Flip evidence, run the artisanal way** (`git checkout origin/main -- crates/stella-model`, inject the empty-stream witness, `cargo test -p stella-model an_empty_stream_falls_back`): ``` test zai::tests::an_empty_stream_falls_back_to_a_non_streaming_request ... FAILED panicked at crates/stella-model/src/zai/tests.rs:1933: provider transport error: Z.ai stream ended before [DONE] — connection closed mid-response ``` — on main the fault carries no fallback and the retry re-streams; with the change the same suite is green (382 passed). The hung-first-byte witness additionally cannot compile on main (`with_first_byte_deadline` and the latch don't exist). ## The gate - [x] `cargo fmt --check` - [x] `cargo clippy -p stella-model -p stella-cli --all-targets -- -D warnings` (exit 0) - [x] `cargo test -p stella-model` (382 passed) and `stella-cli`'s seeded-provider parity tests (3 passed) after rebasing onto `origin/main` at #2720's ledger merge - [x] `RUSTDOCFLAGS="-D warnings" cargo doc --no-deps -p stella-model -p stella-cli` — exit 0, checked unpiped - [x] Docs updated: AGENTS.md invariant 8 (three axes), `crates/stella-model/README.md` (layout + god-file list), module docs - [x] `Closes #2686` appears both here and as a commit trailer ## Nothing left behind - #2746 — extend the fallback to the `anthropic`, `openai`, `gemini`/`vertex` dialects (written as a full handoff; the `StreamingOnly` parity rows cite it). Bedrock needs nothing. - Not fixed here by design: a proxy that heartbeats keep-alive bytes forever while buffering the real body resets the idle clock and is not caught by either deadline — same as before this PR; noted in the aggregator's first-chunk comment and left out of #2746's scope deliberately (no such middlebox has been observed). ## Summary by Sourcery Introduce a streaming→unary fallback in the shared OpenAI-compatible adapter when streams hang before the first byte or return empty, backed by a per-session latch and a first-byte timeout, while keeping retry and usage accounting unchanged. New Features: - Add a per-session streaming recovery latch that can reroute retried attempts from streaming to unary when a stream is hung or empty. - Provide a non-streaming (unary) completion path for the shared chat-completions adapter that mirrors the streaming path’s assembly semantics. - Define a first-byte timeout for streaming responses to detect hung streams separately from inter-chunk idle time. Enhancements: - Refactor the Zai adapter into separate streaming and unary modules sharing common assembly helpers to prevent divergence between delivery paths. - Extend the provider parity matrix with a StreamFallbackPosture axis, declaring each provider’s streaming recovery behavior and enforcing completeness via tests. - Add a generalized stream-read helper that distinguishes idle, end-of-stream, and transport failures for more precise handling of streaming faults. Documentation: - Update AGENTS.md and the stella-model README to describe the new StreamFallbackPosture axis and the split Zai streaming/unary modules, including stream-recovery and first-byte timeout plumbing. Tests: - Add a dedicated stream_fallback test suite for Zai covering hung-first-byte, empty-stream, healthy-stream control, mid-stream death, and failed unary probe behaviors. - Update parity and CLI config tests to require StreamFallbackPosture rows and witness tests for providers that claim a unary fallback. - Move reasoning-aware max-tokens tests into a separate zai_effort test module to stay within file-size limits. --------- Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com> Co-authored-by: Vercel <vercel[bot]@users.noreply.github.com>
macanderson
added a commit
that referenced
this pull request
Aug 10, 2026
…rrors (#2680) (#2752) ## What & why Even with #2739's usage-anchored accounting, estimation error survives — the anchor prices only the tail since the last provider report — and when the estimate undershoots, the provider rejects the request as too large. That rejection used to classify as `ProviderError::Terminal` and abort the turn. This PR converts it into a recovered step: zappy's reactive-compact rung adapted to Stella's accounting discipline. **Classification (stella-model / stella-protocol).** `ProviderError` gains a `ContextOverflow { message }` variant — never retryable as-is (a verbatim re-issue rejects identically). The shared funnel every adapter already uses (`http::classify_http_status`) detects it: HTTP 413 unconditionally, HTTP 400 by narrow per-dialect signatures (Anthropic `prompt is too long`, OpenAI `context_length_exceeded` code / `maximum context length` / `exceeds the context window`, Bedrock `Input is too long for requested model`, Gemini/Vertex `exceeds the maximum number of tokens`). Per **invariant 8** the divergence is declared, not assumed: `provider_parity` gains a third axis, `OverflowPosture`, one row per provider id — `Detected` rows name a witness test (existence-checked, like the other two axes), `BestEffort` rows declare that the wire shape is unverified and degrades safely to today's abort. `stella-cli` config tests enforce row completeness; upgrade work for the five BestEffort rows is #2750. **Recovery (stella-core, `driver/overflow_recovery.rs` — sibling module, no growth into the god files).** On an overflow failure the engine clamps the next compaction budget *below the rejected transcript's own estimate* (rung 1 sheds a quarter of the estimated weight, rung 2 half) and re-runs the step. The repair is the **existing compaction machinery** — the pure retention/dedup/aging passes against the clamped budget, then the configured overflow summarizer — no parallel path; the anchored budget from #2739 feeds the same seam and the clamp is a `min` on top of it. The transcript stays well-paired (the failed call appended nothing) and the retried call advances the step index so receipts never collide. **Latching / death-spiral guard.** Per-turn down-only latch: at most 2 rungs, the counter never resets within the turn, clamps are monotone tighter and never loosen — so error → recover → error burns at most 2 extra calls, ever. A spent ladder surfaces exactly the unrecovered shape (`RetriesExhausted` + non-retryable `Error` + breaker `record_failure`). Not checkpointed, like `length_continuations`: a resume re-permits only a bounded allowance. **What consumers see.** While a rung is armed the overflow is withheld from the terminal channels — announced only as `Error { retryable: true }` (the summarizer-failure notice shape) plus the `model.request.failed` lifecycle signal — so an observer never tears down a mid-recovery session. **No new `AgentEvent` variant**: recovery reuses `Error`/`Compaction`/`UsageIncomplete`, so #2720's consumer ledger needs no new row. A recoverable overflow also does **not** feed the #2734 breaker — an oversized request is the engine's accounting miss, not provider ill-health; the exhausted-abort path keeps today's breaker behavior exactly. **Accounting.** Every rejected attempt was a paid dispatch and still bills through the ordinary per-attempt `UsageIncomplete { reason: ProviderError }` observer (verified by test). No timings ride `ToolOutput`. **Wire.** `stella-serve`'s `ProviderErrorWire` mirrors the variant both directions; `docs/wire/` regenerated — additive-only (`kind: "context_overflow"`). Exemplar for the shape: the ladder/latch mirrors `driver/usage_anchor.rs` + `length_continuations` (per-turn bounded allowance), and classification follows `http.rs`'s established status-ladder idiom. Closes #2680 Refs #2671 ## The witness - [x] This PR includes a witness test (fails on `main`, passes here) `stella-core` (`driver/tests/context_overflow.rs`): - `a_context_overflow_is_recovered_by_forced_compaction_and_a_reissued_call` — scripted provider rejects once with overflow; on main the turn aborts, here it completes in exactly 2 calls, with the terminal events withheld. - `repeated_overflow_is_latched_and_aborts_after_the_ladder_is_spent` — always-rejecting provider gets exactly 1 + MAX_RECOVERY_RUNGS = 3 paid attempts, then aborts terminally (the bound test). - `every_rejected_overflow_attempt_is_billed_through_usage_incomplete` — one envelope per rejected attempt. `stella-model` (http.rs + `zai/tests/error_classify.rs`): per-dialect classification witnesses (anthropic/openai/gemini/bedrock/413), the precision guard `an_unrelated_400_never_classifies_as_context_overflow`, and an end-to-end wiremock funnel test on the shared chat-completions adapter. **Flip evidence, artisanal:** the true on-main run fails by compile (the variant genuinely doesn't exist there), so the behavioral flip was shown by ablation of the exact interceptions main lacks — with `run_model_call`'s overflow interception disabled, all 3 core tests FAIL (`0 passed; 3 failed`); restored, `3 passed`. With `classify_http_status`'s overflow arm disabled, the 6 classification/funnel witnesses FAIL; restored, all pass. ## The gate - [x] `cargo fmt --check` - [x] `cargo clippy --workspace --all-targets -- -D warnings` - [x] `cargo test -p stella-core -p stella-model -p stella-serve -p stella-cli` (one failure: `daemon::…grace_period`, the known full-suite-load flake #2393 — passes alone) - [x] Docs updated where behavior changed (module docs, parity matrix, wire schema regenerated) - [x] CLA signed - [x] `Closes #2680` appears both above and as a commit trailer ## Nothing left behind - [x] Filed: #2750 (verify + upgrade the five BestEffort overflow rows to Detected), #2751 (the summarizer-request-itself-overflows rung — zappy's head-truncation, the one branch recovery still can't repair). The full-suite daemon flake hit during the gate is already tracked as #2393. ## Ground-rule check - [x] No I/O added to `stella-core` (the latch is pure data on `TurnState`; recovery reuses existing async seams); no new deps - [x] No new outbound network calls - [x] New cross-boundary shape: `ProviderErrorWire::ContextOverflow` rides the existing serde round-trip surface; `docs/wire` diff is additive-only ## Anything reviewers should know? - The clamp deliberately persists for the rest of the turn even after a committed call: dropping it once the anchor re-bases would let a compaction budget configured above the provider's real window oscillate grow → overflow → recover; keeping it costs bounded over-compaction and buys monotone convergence. - The recovered path skips `outcomes.record_failure` on purpose (an oversized request isn't provider ill-health); the exhausted path records it exactly as before, so #2734's breaker behavior is unchanged for unrecovered failures. ## Summary by Sourcery Introduce reactive recovery for provider context-window overflow errors, allowing the engine to compact the transcript under a tightened budget and re-issue the model call instead of aborting the turn. New Features: - Add a ProviderError::ContextOverflow variant and wire representation so overflow rejections are classified distinctly from terminal errors and can trigger recovery. - Implement per-turn overflow recovery latch and budget clamp that drive forced compaction and a bounded number of re-issued calls when context overflow is detected. Enhancements: - Extend HTTP status classification and provider parity matrix with per-provider context-overflow detection signatures, plus tests to guard precision and coverage. - Ensure overflow recovery integrates with existing lifecycle events, usage accounting, breakers, and checkpoint behavior without changing terminal semantics when recovery is exhausted. Documentation: - Update wire schemas and parity-matrix documentation to describe the new context-overflow error kind and overflow posture axis. Tests: - Add stella-core tests exercising overflow recovery behavior, ladder bounds, and UsageIncomplete billing envelopes, plus stella-model tests for per-provider overflow classification and end-to-end funneling through the shared adapter.
2 tasks
macanderson
added a commit
that referenced
this pull request
Aug 10, 2026
…build error (#2755) ## What & why Invariant #10 says the signal-consumer ledger's totality is enforced by tests — "adding a variant without declaring what consumes it is a red test." That was true when #2720 landed and stopped being true two commits later: #2737 generated the ledger rows from the tag table, so the check is now an `E0004` at `cargo build`. Caught by re-reading main after both merges landed. Per CLAUDE.md a stale doc is a bug, and this one is mine — I wrote the prose in #2720 and changed the mechanism in #2737 without chasing the claim. The correction is not cosmetic: it **understates** the guarantee, and it misdirects the next person adding an `AgentEvent` variant about where the failure will surface. It also tightens the closing "what it does not prove" paragraph, which is the part most worth keeping exact — totality is now compiler-enforced, while issue citation and posture coherence remain test-enforced. Saying "tests" for both blurs the line that paragraph exists to draw. ## The witness - [x] No witness needed — documentation only, correcting a claim about existing behavior. The behavior it now describes is witnessed in #2737, where a probe variant produces `error[E0004]: non-exhaustive patterns`. ## Deleted tests None. ## The gate - [x] `scripts/check-invariants.sh` — OK, 10 invariants, one normative home, 28 citations resolve Refs #2701 Refs #2730 ## Summary by Sourcery Documentation: - Clarify that adding an AgentEvent variant without a consumer now triggers a compile-time E0004 error rather than a failing test and tighten the distinction between compiler- and test-enforced guarantees in the invariant description.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
"Produced and not consumed" is a repo-wide shape (#2701), and every instance so far was found by a bench run paying for it rather than by a test going red:
flip.jsonwritten with nothing reading it (#1536);verify_doneconfirmations tallied but not feeding the halt, which costsolved_then_timeoutfour times on one certification panel before #2661 wired it; the flip transition still emitting nothing durable, so a shipped halt cannot be measured in the field. Each was fixed as whack-a-mole.This makes the class structural.
crates/stella-protocol/src/event/consumers.rsholds one row perAgentEventwire tag, each declaring aConsumerPosture:Behavioral { site }— something branches on it;sitenames the code.Surfaced— rendering is the realized value; requires a non-emptysurfaces.RecordedOnly { issue }— persisted and nothing else; a declared gap.Unclassified { issue }— nobody has audited it yet, under a down-only ratchet.Exemplar named, per CLAUDE.md:
crates/stella-store/src/content_free.rs, copied element for element — reviewed table,audit_*returning violations rather than panicking, tests enforcing totality from both sides, and negative controls proving the harness can fail.DRAIN_FORMATS'sNotYetBuilt { issue }is the direct ancestor of the two gap postures.Three design calls that differ from the issue's sketch
Each was forced by reading the code, and each is recorded in the module doc so the next author does not re-litigate it:
surfacesare orthogonal fields, not one enum.tool_resultis bothBehavioraland on the observatory whitelist. The issue'sSurfaced { surfaces }variant would have made observatory: pin the 8-type journal whitelist to the signal-consumer ledger with a parity test (worker: haiku) #2707's observatory↔ledger parity test unsatisfiable on day one.Surfacenames only the observatory and serve. The TUI (model::Model::apply,textline::event_line,deck::trace_of) and replay (replay::event_signature) matchAgentEventexhaustively, so membership is compiler-guaranteed for every variant. A field whose value is identical on every row records nothing; this enum names only surfaces that choose, which is what makes a surface claim falsifiable.agent_event_tags!. Generating it from the macro would make totality compile-enforced rather than test-enforced — strictly stronger, and deliberately not done: that macro is the wire decoder's source of truth (a tag missing fromKNOWN_TYPE_TAGSsilently demotes every event carrying it toUnknown), andevent.rssits 35 lines under the 1500-line ratchet with no baseline entry while the postures add roughly forty. Splitting the tag macro out is worthwhile but should not be smuggled into this PR.The ledger earned its keep before merge
The
tool_resultrow's obvious consumer is loop detection. That is wrong: the detector reads the transcript (loop_evidence::recent_call_recordsoverCompletionMessages), a different plane that happens to carry the same facts. The real event-plane consumer isstella-store/src/tool_calls.rs::project_event. Both of my first-draftsitestrings named symbols that do not exist (turn_call_records,JournalBuilder); I caught them by opening the files. That is exactly the staleness the module doc declares as not machine-checked — stated plainly rather than implied.PR #2710 (open,
Closes #2700) also appends an invariant numbered #9 to AGENTS.md. Two PRs writing one number is how a citation silently resolves to the wrong invariant —scripts/check-invariants.shvalidates citations against1..count, so a duplicate #9 passes the guard while meaning the wrong thing.This PR defers to the older one:
cargo test) #10, with a9. *(reserved …)*placeholder line naming docs(agents): codify the tool-first single-purpose invariant (#9) #2710 so the gap is explained rather than mysterious.check-invariantsis green (it counts matching lines and does not require contiguity).consumers.rsrefers to the invariant by name only, so renumbering is a one-line AGENTS.md edit with zero code churn.The witness
the_ledger_is_total_over_known_type_tags. Verified fail-on-old the artisanal way — deleting theproofrow and loweringMAX_UNCLASSIFIEDmakes it red with an actionable message:The ratchet demonstrated itself on its first run, failing at
36 rows are Unclassified but MAX_UNCLASSIFIED is 35before the constant was set.Nine negative controls, including
the_control_fixture_itself_passes— without it every other control could be firing on a fixture that was broken to begin with, which is the vacuous-harness failurecontent_free.rsexists to prevent.Deleted tests
None.
The gate
make guards-fast— includingcheck-invariants: OK — 9 invariants … 26 citation(s) resolveandcheck-file-size: OKmake lint(clippy-D warnings, workspace, all targets)cargo fmt --allcargo test -p stella-protocol— 14 new tests greenmake gatefull workspace test tier — running; will reportNo baseline widened, no
#[allow]added, no new dependencies.Follow-on
#2703 populates the remaining 36
Unclassifiedrows and drivesMAX_UNCLASSIFIEDto zero. #2707 turnstags_surfaced_by(Surface::Observatory)into a two-sided parity test against the observatory's own whitelist.Refs #2701
Refs #2703
Summary by Sourcery
Introduce a signal-consumer ledger for AgentEvent variants and codify the invariant that every emitted signal declares its consumers.
New Features:
Enhancements:
cargo test) #10 to describe the signal-consumer ledger and its enforcement strategy.Tests: