Skip to content

fix(selfhost): AI-provider per-attempt logging amplifies one retry loop into up to 6 Sentry errors #5046

Description

@JSONbored

Summary

logSelfHostAiProviderFailed (src/selfhost/ai.ts) fires console.error (Sentry-visible) on every subscription-CLI attempt failure, not just the final one — contradicting the documented "per-attempt=warn, exhausted=error" policy runWorkersOpinion's own logging already follows (src/services/ai-review.ts, comment citing #26).

Evidence

GITTENSORY-K (selfhost_ai_provider_failed: claude_stalled_no_output) has 2077+ occurrences and is still climbing. Pulled raw events: multiple events share the same jobId/trace_id with attempt: 0, attempt: 1, etc. — confirming these are individual retries within one review call, each independently escalated to Sentry.

runWorkersOpinion retries up to 3× per model × 2 models (6 attempts max) per reviewer call. Its own per-attempt log (ai_review_provider_attempt_failed) is deliberately console.warn (not Sentry-visible) with an explicit comment: "Per-attempt logs are warn (noisy retries, skipped by the central Sentry forwarder); the exhausted summary is error (#26)." But the LOWER-LEVEL logSelfHostAiProviderFailed, called from inside createClaudeCodeAi/createCodexAi's own catch blocks on every throw, ignores that policy and escalates every attempt regardless — the two layers disagree, and the lower one wins (Sentry sees it either way).

Two other callers (src/review/planner.ts, src/services/ai-slop.ts) have their own retry loops (2× and 3× respectively) with zero logging of their own — they rely entirely on this same low-level log for any visibility, so a blind fix that always downgrades to warn would leave them completely blind to persistent failures.

Fix

Thread a finalAttempt?: boolean field through AiRunOptions (mirroring the existing attempt/jobId correlation fields). logSelfHostAiProviderFailed logs at warn only when finalAttempt === false; unset (single-shot callers) or true stays at error, matching current behavior. runWorkersOpinion, runPlannerModel, and runWorkersSlopOpinion compute it as attempt === maxAttempts - 1 && modelIndex === models.length - 1 and pass it through.

Deliverables

  • finalAttempt threaded through AiRunOptions and logSelfHostAiProviderFailed.
  • All three retry-loop callers updated to compute and pass it.
  • Regression tests: a non-final attempt logs warn (not Sentry-visible), the final attempt still logs error, a single-shot caller with no finalAttempt set still logs error.

Metadata

Metadata

Assignees

No one assigned

    Labels

    maintainer-onlyOwner-only work — yields no Gittensor points.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions