Skip to content

fix: pace retryable partial-start failures in code-mode composites - #4150

Merged
aheritier merged 2 commits into
mainfrom
fix/4067-partial-start-backoff
Sep 3, 2026
Merged

aheritier merged 2 commits into
mainfrom
fix/4067-partial-start-backoff

Conversation

@aheritier

Copy link
Copy Markdown
Collaborator

🤖 Automated implementer agentthis PR was posted by the implementer bot from Docker Agentic Platform, not by a human developer

Fixes #4067.

PR #4062 exempted PartialStartError from the backoff gate so a composite toolset's healthy subset keeps listing while degraded, but this also cleared the gate on every retryable partial failure (e.g. a RAG toolset inside a code_mode composite hitting 429s), so the failed subset burst-retried on every turn just like #4060.

startLocked now calls setStartBackoff instead of resetStartBackoff for a PartialStartError: it arms the gate when the aggregated cause is retryable (errors.As walks the whole errors.Join tree, so any single retryable inner cause is enough) and still resets it otherwise, preserving today's fail-fast behaviour for non-retryable partial failures (e.g. auth deferrals). s.started stays latched either way, so the healthy subset is unaffected.

Testing:

  • New unit tests in pkg/tools/startable_backoff_test.go: retryable-cause arms the gate; mixed auth+retryable cause still arms the gate (ANY-cause semantics) while staying non-auth-classified.
  • New end-to-end test in pkg/tools/codemode/codemode_test.go through the real codemode.Wrap() + StartableToolSet stack: gated retry doesn't re-invoke either inner, healthy tools stay listed, degraded inner recovers once the window elapses.
  • task build / go test ./... / golangci-lint run / go run ./lint . all pass (the only test failure, pkg/rag/treesitter, is a pre-existing sandbox limitation — missing gcc/CGO — reproduced identically on main).

Docs updated: docs/tools/mcp/index.md and docs/features/code-mode/index.md no longer claim this pacing is unapplied in code mode.

PR #4062 exempted PartialStartError from the backoff gate so a
composite toolset's healthy subset keeps listing while degraded, but
this also cleared the gate on every retryable partial failure (e.g. a
RAG toolset inside a code_mode composite hitting 429s), so the failed
subset burst-retried on every turn just like #4060.

startLocked now calls setStartBackoff instead of resetStartBackoff for
a PartialStartError: it arms the gate when the aggregated cause is
retryable (errors.As walks the whole errors.Join tree, so any single
retryable inner cause is enough) and still resets it otherwise,
preserving today's fail-fast behaviour for non-retryable partial
failures. s.started stays latched either way, so the healthy subset
is unaffected.

Fixes #4067
@aheritier
aheritier requested a review from a team as a code owner September 3, 2026 16:33

@aheritier aheritier left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated implementer agentthis comment was posted by the implementer bot from Docker Agentic Platform, not by a human developer

Review-of-record (self-review of my own PR; delegated an independent review to a reviewer sub-agent beforehand, which returned approve with no findings — see summary below). Self-approval is blocked for the PR author, so posting as COMMENTED with concrete evidence for maintainers to verify.

Core fixpkg/tools/startable.go:585 (startLocked's PartialStartError branch): replaced s.resetStartBackoff() with s.setStartBackoff(err). setStartBackoff (pkg/tools/startable.go:463-481) internally calls startBackoffRetryable(err) (pkg/tools/startable_backoff.go:51-59), which does errors.Is/errors.As — both walk the whole errors.Join tree that PartialStartError.Unwrap() (pkg/tools/startable.go:142) exposes. So:

  • A single retryable inner cause (429/503/etc via *modelerrors.StatusError, or lifecycle.ErrCrashLooping) anywhere in the joined batch arms the gate — even mixed with a non-retryable/auth cause (ANY-cause semantics, deliberately different from AuthOnly's ALL-cause semantics at pkg/tools/startable.go:113-122).
  • A batch with no retryable cause falls through to resetStartBackoff() inside setStartBackoff (pkg/tools/startable.go:467-470), preserving today's fail-fast behaviour for non-retryable/auth-deferral partial failures.
  • s.started = true (pkg/tools/startable.go:582) is latched unconditionally before the gate call, so the healthy subset keeps listing regardless of gate state.

Tests:

  • pkg/tools/startable_backoff_test.go: renamed the old test to TestStartableToolSet_NonRetryablePartialStartClearsBackoffGate (narrowed doc to the non-retryable contract) and added TestStartableToolSet_RetryablePartialStartArmsBackoffGate + TestStartableToolSet_MixedAuthPartialStartArmsBackoffGate (pins ANY-cause semantics: mixed auth+retryable arms the gate while staying IsAuthorizationRequired() == false).
  • pkg/tools/codemode/codemode_test.go: TestCodeModeTool_RateLimitedInnerPacesRetry drives the real codemode.Wrap() + tools.NewStartable() stack end-to-end — confirms the gated turn doesn't re-invoke either inner's Start, the healthy tool declarations stay listed throughout, and the degraded inner recovers once the window elapses.

Validation: go build ./..., go test ./pkg/tools/... ./pkg/agent/... ./pkg/runtime/... (and full go test ./... — only the pre-existing, unrelated pkg/rag/treesitter CGO/gcc gap fails, reproduced identically on main), golangci-lint run --allow-parallel-runners (0 issues), go run ./lint . (no offenses), go mod tidy --diff (clean).

Docs: docs/tools/mcp/index.md and docs/features/code-mode/index.md updated to drop the "not yet applied in code mode" caveat, with verified cross-link anchors.

../mcp/index.md from docs/features/code-mode/ resolves to the
nonexistent docs/features/mcp/index.md; the mcp docs actually live at
docs/tools/mcp/index.md.
@aheritier aheritier added area/docs Documentation changes area/mcp MCP protocol, MCP tool servers, integration area/tools For features/issues/fixes related to the usage of built-in and MCP tools kind/fix PR fixes a bug (maps to fix:). Use on PRs only. labels Sep 3, 2026
@aheritier
aheritier merged commit 1650e99 into main Sep 3, 2026
19 checks passed
@aheritier
aheritier deleted the fix/4067-partial-start-backoff branch September 3, 2026 17:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/docs Documentation changes area/mcp MCP protocol, MCP tool servers, integration area/tools For features/issues/fixes related to the usage of built-in and MCP tools kind/fix PR fixes a bug (maps to fix:). Use on PRs only.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Rate-limit backoff not applied to code-mode composite's partial-start subset retry (follow-up to #4060)

2 participants