feat(stella-core): mid-turn model fallback on retries-exhausted — re-resolve through the router, repair the transcript, continue the turn - #2769
Merged
Conversation
When a model call exhausts its retries against the active provider, the engine now re-resolves the role through the new FallbackResolver port — backed by the router whose circuit breaker the failing calls already fed (#2673/#2734) — and continues the turn on the replacement instead of aborting: transcript repaired through close_open_tool_calls, terminal events withheld exactly as overflow recovery withholds its own (#2680), the swap announced as ProviderFallback plus a retryable Error notice, and at most one swap per engine (the set-once override IS the latch, so two sick providers can never ping-pong). Every burned attempt still bills through the per-attempt UsageIncomplete observer. The bare CLI loops attach a router-backed SessionFallback beside the session router at both run_turn engine sites; the budget helpers move to agent/budget.rs to keep agent.rs under its god-file ceiling; the parity matrix gains the provider.midturn_fallback row (pipeline execute-stage wiring is #2765). Closes #2679 Refs #2733, #2734, #2765
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Sorry @macanderson, you have reached your weekly rate limit of 500000 diff characters.
Please try again later or upgrade to continue using Sourcery
Contributor
Reviewer's GuideImplements mid-turn provider fallback when a model retry ladder exhausts by introducing a FallbackResolver port, a per-engine provider override latch, and router-based re-resolution, along with CLI wiring, transcript repair, and parity/docs updates; also refactors CLI budget helpers into a separate module. Sequence diagram for mid-turn provider fallback on exhausted retriessequenceDiagram
participant Engine
participant Provider as PrimaryProvider
participant ProviderOutcomes
participant FallbackResolver
participant Router
participant NewProvider
participant Events as EventSender
Engine->>Provider: drive_attempt_ladder(model_call)
Provider-->>Engine: error (retry loop exhausts)
Engine->>ProviderOutcomes: record_failure(active_provider.id)
Engine-->>Engine: ModelCallFailure::Exhausted{message,attempt_reasons,retryable}
Engine->>Events: MODEL_REQUEST_FAILED
Engine->>Engine: attempt_provider_fallback(message,state,events)
alt fallback_resolver_attached_and_latch_free
Engine->>FallbackResolver: resolve_fallback(failed_provider_id)
FallbackResolver->>Router: resolve(Role::Worker)
Router-->>FallbackResolver: decision (healthy provider)
FallbackResolver-->>Engine: ResolvedFallback{provider,reason}
Engine-->>Engine: provider_override.set(resolved.provider)
Engine->>Engine: close_open_tool_calls(FALLBACK_TOOL_RESULT,events)
Engine->>Events: ProviderFallback{from,to,reason}
Engine->>Events: Error{message: ...,retryable: true}
Engine-->>Engine: state.step += 1
Engine->>NewProvider: retried_model_call via active_provider()
else no_fallback_or_resolution_back_to_failed_provider
Engine->>Events: AgentEvent::RetriesExhausted{attempts,reasons,retryable}
Engine->>Events: AgentEvent::Error{message,retryable}
Engine-->>Engine: StepOutcome::Aborted{Failure}
end
File-Level Changes
Assessment against linked issues
Possibly linked issues
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
macanderson
enabled auto-merge (squash)
August 10, 2026 23:40
macanderson
disabled auto-merge
August 10, 2026 23:42
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
The engine held one provider for the whole turn, so a retry ladder exhausting against a wedged provider ended the turn
Aborted { Failure }even when a healthy fallback was configured and resolvable (driver.rs's abort path) — every completed step's work stranded. This PR is the last ticket of the Phase 2 reliability chain: on retries-exhausted the engine now re-resolves through the router and continues the turn on the replacement.Trigger classes. Everything that surfaces as the new
ModelCallFailure::Exhausted— transport, 5xx, auth, and rate limiting that outlived #2744's parked recovery.ContextOverflowkeeps its own rung (#2752); #2748's stream fallback still runs below, inside the attempt. The park composes rather than races: parking happens inside the ladder, the fallback only after the ladder gives up, and a soft stop typed during a park still wins at the next step boundary.Router interplay (the #2734 seam, used as designed). The fallback is a re-resolution via the new
stella_core::ports::FallbackResolverport, never a hardcoded list.drive_attempt_ladderfeedsrecord_failurebefore settlement asks for a fallback, soRouter::resolvealready routes around the sick provider; a resolution landing back on the failed provider is a refusal and the turn aborts exactly as before. The bare CLI loops attach a router-backedSessionFallback(agent/engine.rs) at bothrun_turnengine sites — thesession_routerdoc's declared destiny for #2679.Transcript repair, deterministic. The failed call appended nothing, so the engine's own path is already well-paired; caller-supplied history with an orphaned
tool_useis closed through the sameclose_open_tool_callsrepair the cancel/soft-stop exits use (stub named for the swap, mirrored onto the event stream). Model-signed thinking blocks — the other thing a naive switch replays into a 400 — are structurally absent:CompletionMessagecarries no reasoning blocks, so there is nothing to strip; the module doc records that argument.Latch/bound.
Engine::provider_overrideis a set-once cell and the set IS the latch: at most one swap per engine, ever — two sick providers cannot ping-pong. The override persists for the engine's remaining turns; the breaker's cooldown/half-open cycle is what routes fresh engines back to a recovered primary.What consumers see. The ladder now withholds the terminal
RetriesExhausted/Errorpair (mirroring #2752's overflow arm) and settlement emits it only when no fallback fires — byte-identical to the old shape, pinned by a control test. A latched swap emits the existingAgentEvent::ProviderFallback(+ a retryableErrornotice), so no new AgentEvent variant and no new consumer-ledger row (invariant #10 satisfied by reuse); the variant's doc and the generateddocs/wiredescriptions are updated (description-only diff, no shape change).Accounting. Every burned attempt still bills through the per-attempt
UsageIncompleteobserver — witnessed. No timings rideToolOutput.God files. New logic is in sibling modules (
driver/model_fallback.rs, theoverflow_recovery.rs/settlement.rspattern; tests indriver/tests/model_fallback.rs).agent.rssat exactly at its ceiling, so the self-contained budget helpers moved to a newagent/budget.rs(move, not rewrite; re-exported so every caller path is unchanged) — agent.rs lands 40 lines under its inherited size.Parity (invariant #8's cross-surface sibling).
stella-paritygains theprovider.midturn_fallbackrow claimingwith_fallback_resolver(the entry-point sweep enforces this); CLI posture isShippedUnwitnessedwith the gap cited (#2733 for the attachment witness, #2765 for pipeline wiring), APINotApplicablefor the same reason asprovider.breaker_feedback.UNWITNESSED_BASELINE4 → 5 — the declared-debt direction the ratchet exists to make visible, not an expedient.Exemplar. The module shape, withheld-events discipline, latch bound, and test suite deliberately mirror
driver/overflow_recovery.rs+driver/tests/context_overflow.rs(#2752), this repo's canonical recovery-rung shape.Closes #2679
Refs #2733, #2734, #2744, #2748, #2752, #2765
The witness
main, passes here)crates/stella-core/src/driver/tests/model_fallback.rs:exhausted_retries_swap_to_the_resolved_fallback_and_the_turn_completes— the headline IMPROVEMENT: mid-turn model fallback on retries-exhausted — repair orphaned tool_use, strip reasoning blocks, continue instead of aborting #2679 witness: terminally failing primary, healthy resolved fallback; on main the turn aborts, here it completes with the terminal channels silent and the swap announced.a_sick_fallback_is_never_swapped_again— the bound: one ladder per provider, the resolver asked exactly once, second exhaustion terminal (no ping-pong).the_transcript_handed_to_the_fallback_is_well_paired— pairing asserted on the transcript the replacement provider actually received, orphan stubbed with the swap's wording.every_attempt_before_the_swap_is_billed_through_usage_incomplete— 1 attempt + 2 retries = 3 envelopes, then the rescue.without_a_resolver_the_terminal_surfacing_is_unchanged(the relocated emission is byte-identical),a_resolution_back_onto_the_failed_provider_is_refused.Flip evidence, artisanal (the ablation form, as in #2752 — the true on-main run cannot compile because the port doesn't exist there): with the one interception main lacks disabled (
if false && self.attempt_provider_fallback(...)insettle_model_call_failure), the suite runs2 passed; 4 failed— all four behavior witnesses FAIL (turn aborts, exactly main's behavior) while both abort-path controls stay green; restored,6 passed; 0 failed.The gate
cargo fmt --check(viamake guards-fast, exit 0 — all toolchain-free guards incl. file-size, god-files, left-behind, typed-errors, module-reachability)cargo clippy -p stella-core -p stella-cli --all-targets -- -D warnings— exit 0cargo test -p stella-core -p stella-cli -p stella-parity— 1130 + 1693 + 9 (+ integration targets), 0 failed;cargo test -p stella-pipeline -p stella-serve -p stella-protocol— 0 failedmake wire-schema— regenerated, description-only diff (read above)RUSTDOCFLAGS="-D warnings" cargo doc --no-deps -p stella-core -p stella-cli -p stella-parity -p stella-protocol --document-private-items --keep-going— exit code 0, checked unpipedsession_routerdoc un-staled)Closes #2679appears both above and as a commit trailerNothing left behind
FallbackResolverport, written as a handoff. The CLI-attachment witness gap was already tracked in breaker feedback (#2673) follow-up: bare engine sites still feed no breaker, the summarizer call records nothing, and the run_turn attachment lacks a CLI witness #2733 (cited by the parity row rather than duplicated).Ground-rule check
stella-core(the resolver is a port; the CLI impl owns the adapter build); no new depsProviderFallbackreused;docs/wirediff is description-onlyAnything reviewers should know?
drive_attempt_ladderintosettle_model_call_failure(a settlement decision now that exhaustion has a recovery). The control test pins the observable shape; the one ordering change is that themodel.request.failedbus signal now precedes the terminal pair — cross-channel (hook bus vs event stream), so no consumer can order them anyway.ModelCallFailure::Fatalis renamed/retyped toExhausted { message, attempt_reasons, retryable }— it ispub(crate), no external surface.subagent.rs's engine assembly.Summary by Sourcery
Introduce mid-turn provider fallback in the engine so exhausted retry ladders can re-resolve through the router and continue the turn on a replacement provider, while preserving existing terminal behavior when no fallback is available.
New Features:
Bug Fixes:
Enhancements:
Documentation:
Tests:
Chores: