fix(kiro): distinguish confirmed credit exhaustion from ambiguous 402s#2664
Open
SemonCat wants to merge 3 commits into
Open
fix(kiro): distinguish confirmed credit exhaustion from ambiguous 402s#2664SemonCat wants to merge 3 commits into
SemonCat wants to merge 3 commits into
Conversation
A Kiro 402 is currently always treated the same way: a flat 2-minute cooldown
(open-sse/config/errorConfig.js ERROR_RULES). That's fine for an ambiguous 402
(bad payment method, suspended account) but wrong for a confirmed monthly
credit-limit exhaustion, where the account will keep 402ing every 2 minutes
until the real reset — wasting requests and cycling through combo/account
fallback for no reason.
A real CodeWhisperer GenerateAssistantResponse 402 for this case has a
recognizable, verified shape (AWS ServiceQuotaExceededException):
{ "message": "You have reached the limit.",
"cause": { "$metadata": { "httpStatusCode": 402 },
"name": "ServiceQuotaExceededException",
"reason": "MONTHLY_REQUEST_COUNT" } }
KiroExecutor.parseError() now matches this shape (and the flattened variant
some surfaces use) to classify the 402 as confirmed vs. ambiguous. Ambiguous
402s are untouched and keep the existing 2-minute cooldown.
The 402 body itself never carries a reset time, so on a confirmed match this
makes a best-effort follow-up call to Kiro's own quota API (GetUsageLimits),
which does surface a trustworthy resetAt (services/usage/kiro.js already
parses `nextDateReset`/`resetDate`). If that lookup is unreachable, times out,
or shows nothing depleted, it falls back to a bounded default instead of
guessing a date.
That resetsAtMs already flows through the existing pipeline (parseUpstreamError
-> createErrorResult -> markAccountUnavailable), which previously capped any
resetsAtMs-derived cooldown at a shared 30-minute ceiling meant for short-window
rate limits (e.g. Codex). Applying that ceiling to a month-scale Kiro reset would
re-probe a known-exhausted account every 30 minutes. markAccountUnavailable now
looks up a per-provider cap (RESET_COOLDOWN_CAP_MS), with Kiro set to 24h — so a
near-term reset is honored as-is, and a far-future reset is capped to a daily
probe, recovering within a day of an early provider-side change (manual top-up,
plan change) without hammering the account meanwhile. Other providers'
resetsAtMs handling (e.g. Codex) is unchanged.
Tests: confirmed exhaustion + resetsAtMs derivation, quota-lookup-failure and
nothing-depleted fallback, ambiguous 402 preserving the old cooldown, the daily
probe cap vs. a near-term reset, the existing 30-min cap for other providers,
lock expiry, and multi-account fallback around a locked Kiro account.
Limitations: the lock is still per (account, model) like every other error type
here, so an account rotating through several Kiro model aliases can still probe
each alias independently within the cap window; making credit exhaustion an
account-wide lock is a natural follow-up but is out of scope for this change.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Keep retry timing, status, and error tied to the earliest model lock owner, and clear stale error codes after successful recovery.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Today every Kiro 402 gets the same flat 2-minute cooldown (
open-sse/config/errorConfig.jsERROR_RULES,status: 402). That's a reasonable default for an ambiguous 402 (bad payment method, suspended account), but wrong for a confirmed monthly credit-limit exhaustion: the account keeps getting re-tried and re-402ing every 2 minutes until the real reset, burning requests and cycling needlessly through account/combo fallback — sometimes for weeks.This PR adds a narrow, backward-compatible path: only a recognized exhaustion signal gets special handling; everything else keeps today's behavior exactly as-is.
Is a Kiro 402 distinguishable from "really out of credits"? Yes — for one well-known shape
A CodeWhisperer
GenerateAssistantResponse402 for monthly-quota exhaustion has a documented, reproducible shape (AWSServiceQuotaExceededException), confirmed via public reports (e.g. aws/aws-toolkit-vscode#8577):{ "code": "QModelResponse", "name": "Error", "message": "You have reached the limit.", "cause": { "$metadata": { "httpStatusCode": 402 }, "name": "ServiceQuotaExceededException", "reason": "MONTHLY_REQUEST_COUNT" } }KiroExecutor.parseError()(open-sse/executors/kiro.js) now matches this shape — and a flattened variant some surfaces use (name/reasonat the top level instead of nested undercause) — to classify a 402 as confirmed vs. ambiguous. Anything that doesn't match (unrecognized body, non-JSON, a differentreason) is left ambiguous and keeps the exact current 2-minute cooldown.I also checked the related credits/quota reports in this repo (#1328, #1854, #1901, #2374) and the recent Kiro PRs #2580/#2618/#2639 — none of them touch 402 classification or the reactive cooldown path; #1854 (open) adds a complementary proactive pre-flight quota check that skips known-depleted Kiro accounts before dispatch. This PR is the reactive fallback for when that pre-flight either isn't present or is stale — the two aren't in conflict.
Where does the reset time come from?
The 402 body itself never carries a reset date (confirmed against the real-world shape above). The only trustworthy source is Kiro's own quota endpoint,
GetUsageLimits, which 9router already parses (open-sse/services/usage/kiro.js,nextDateReset/resetDate). So on a confirmed match,parseErrormakes a best-effort follow-up call to that endpoint (8s timeout) and takes the earliestresetAtamong fully-depleted quota buckets.If that lookup is unreachable, times out, or nothing comes back depleted, it does not guess a date — it falls back to a fixed default (see below) rather than fabricating provider semantics.
Why the existing
resetsAtMscap needed a per-provider overrideresetsAtMsalready flows through the pipeline (parseUpstreamError→createErrorResult→markAccountUnavailable), and was previously capped at a single sharedMAX_RATE_LIMIT_COOLDOWN_MS(30 min) — sized for short-window resets like Codex'sresets_at(hours). Applying that cap to a month-scale Kiro reset would mean re-probing a known-exhausted account every 30 minutes.markAccountUnavailable(src/sse/services/auth.js) now looks up a per-provider cap viaRESET_COOLDOWN_CAP_MS(open-sse/config/errorConfig.js), with Kiro set to 24h:Other providers (Codex, etc.) are untouched — they still resolve to
MAX_RATE_LIMIT_COOLDOWN_MS.Behavior
Tests
New:
tests/unit/kiro-credit-exhaustion.test.js,tests/unit/kiro-credit-cooldown.test.jsresetsAtMsderived from the quota APIresetsAtMs(old cooldown preserved)name/reason) also recognizedresetsAtMscap (Codex) unchangedFull suite: ran on a clean
mastercheckout before and after this change — identical set of unrelated failures both times (environment-dependent: live-network tests, CWD-relativefs.readFileSyncinsecurity-audit.test.js, SQLite concurrency timing, golden snapshot drift). No new failures introduced.npx eslintclean on all changed files.Limitations
(account, model), like every other error type in this file — an account rotating through several Kiro model aliases can independently probe each alias within the cap window. Making confirmed credit exhaustion an account-wide lock is a natural follow-up, but changing that shared locking mechanism felt out of scope for a focused fix — happy to do it here if maintainers would rather have it bundled.🤖 Generated with Claude Code
Account-lock aggregation hardening
This branch now also keeps retry timing, HTTP status, and error text paired to the same locked account when every account is unavailable. A transient 502/503 account with the earliest 30-second retry can no longer inherit MONTHLY_REQUEST_COUNT from a different 402 account. Genuine all-account quota exhaustion remains 402.
The follow-up also clears stale errorCode after a successful cooldown recovery and makes model-specific plus account-wide locks compose by waiting for the later active expiry on each account before selecting the earliest account across the pool.
Regression coverage adds the mixed 3x402/120s plus 2x502/503/30s case, genuine all-402 preservation, cooldown recovery, stale error cleanup, and overlapping model/account lock expiry. Focused verification: 4 files, 24 tests; ESLint and production build pass. Independent Claude Code review verdict: SHIP.