fix(queue): prefer exact github rate-limit admission buckets - #2417
Conversation
|
Warning 🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨 ⏸️ Gittensory review result - manual review recommendedReview updated: 2026-07-01 20:49:44 UTC
⏸️ Suggested Action - Manual Review Review summary Nits — 5 non-blocking
Review context
Contributor next steps
Signal definitions
🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed 💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →. Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers.
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #2417 +/- ##
==========================================
- Coverage 95.83% 95.83% -0.01%
==========================================
Files 224 224
Lines 25006 25001 -5
Branches 9096 9092 -4
==========================================
- Hits 23964 23959 -5
Misses 428 428
Partials 614 614
🚀 New features to boost your workflow:
|
A newer unkeyed/null legacy rate-limit observation could override an older-but-healthy exact installation-keyed observation purely on recency, deferring an entire installation's webhook queue even though that installation's own budget was fine. This happened repeatedly on self-host: ~147 pending github-webhook jobs piled up with last_error 'github rate-limit webhook admission' while the unkeyed fallback bucket (a different, unrelated consumer) cycled through exhaustion and reset faster than the installation-scoped exact observation was refreshed. fallbackObservationCanOverrideExact now only lets a newer fallback win when doing so is MORE permissive than the exact reading (clearing a stale exhaustion) — never when it would introduce a delay the exact observation alone would not have. A fallback still governs when no exact observation exists at all. matchesGitHubRateLimitAdmissionTarget had the same failure class in the reactive (post-failure) defer path: a confirmed rate-limit error on a job with no admission key (legacy/unknown actor work) unconditionally parked every OTHER pending job regardless of its own key. It now only parks other null-keyed candidates, matching the precedent already used for keyed blocked targets.
…austion too Review feedback on the initial fix correctly identified that the new rule was asymmetric without justification: a null/unkeyed observation is not proven to report on the SAME budget as an exact installation key, so its signal is equally untrustworthy in BOTH directions, not just when it would introduce a new restriction. Simplify to: once an exact observation exists for an admission key, it alone governs (its own reset_at already bounds how long an exhaustion can block admission); the fallback only ever applies when no exact observation exists at all.
89d09c1 to
4456074
Compare
Summary
github-webhookqueue even though that installation's own REST budget was fine.github-webhookjobs stuck withlast_error = "github rate-limit webhook admission", whilegithub_rate_limit_observationsshowed an exhausted unkeyed row racing ahead of a healthyinstallation:<id>row.fallbackObservationCanOverrideExactnow only lets a newer fallback win when doing so is more permissive than the exact reading (clearing a stale exhaustion) — never when it would introduce a delay the exact observation alone would not have. A fallback still governs admission when no exact observation exists at all.matchesGitHubRateLimitAdmissionTargethad the same failure class in the reactive (post-failure) defer path: a confirmed rate-limit error on a job with no admission key unconditionally parked every other pending job regardless of its own key. It now only parks other null-keyed candidates, matching the precedent already used for keyed blocked targets.Test plan
npx vitest run test/unit/selfhost-queue-common.test.ts test/unit/selfhost-pg-queue.test.ts test/unit/selfhost-sqlite-queue.test.ts— all passing, including new regressions for: healthy exact + newer exhausted fallback (no delay), no exact + exhausted fallback (delay), exact-alone exhausted (delay), a newer fallback still clearing a stale exact exhaustion (invariant), background-kind parity, and thematchesGitHubRateLimitAdmissionTargetnull-blocked-target scoping fixnpm run test:coverage— full suite green, 100% patch coverage on changedsrc/**lines/branchesnpm run typecheck— cleangit diff --check— cleannpm run test:ci+npm audit --audit-level=moderate— clean