feat(api-rs,slackbotv2): keep working when a provider runs out of credits

[snario]

The problem, in plain terms

If the provider behind your default harness runs out of credits — a ChatGPT/Codex usage cap, an exhausted OpenAI or Anthropic balance — the whole assistant goes down. Every message, in every thread, fails with:

Execution failed: You've hit your usage limit. Visit chatgpt.com/codex/settings/usage to purchase more credits or try again at Aug 11th, 2026 12:18 AM.

…and it stays that way until a human notices, tops up credits, or manually switches threads to another harness. For deployments that run on subscription credits, this is the most common way the bot dies — and it always happens at the worst time, because credits run out precisely when the bot is being used heavily. This exact outage hit a live deployment this week: Codex credits ran out on a Friday afternoon with the reset three days away.

The frustrating part: most deployments already have a second, fully-credentialed harness sitting idle (Claude Code next to Codex, or vice versa). Today Centaur won't touch it unless a user knows to type --claude in every affected thread.

What this PR does

One new deployment value:

slackbotv2:
  quotaFallbackHarness: claudecode   # any harness wire value; empty = feature off

With that set, running out of credits stops being an outage:

1. A turn fails with a quota error, once.
2. slackbotv2 immediately restarts that thread on the fallback harness — exactly as if the user had typed --claude — re-feeds the thread transcript, and replays the failed message.
3. The user gets their answer from the fallback harness, seconds later, in the same thread. From their side it's a brief hiccup, not a dead bot.

The switch is sticky per thread (mid-conversation threads don't bounce back and forth during the outage), while new threads still start on the default harness — so when credits reset, the deployment drifts back to normal on its own, with zero operator action.

Sensible edges: a fallback that itself fails is terminal (a per-message guard prevents ping-pong loops), an explicit --codex-style flag from the user always beats the automatic switch, and plain rate limits are deliberately not treated as quota exhaustion (they're transient; switching harnesses over them would cause churn).

How it works

• api-rs — terminal_failure_class() gains a "quota" class covering the Codex CLI's usage-limit prose, the raw OpenAI insufficient_quota / usage_not_included codes (as surfaced by both codex and nanocodex), and Anthropic's credit-balance message. The class is stamped onto the session.execution_failed event as a new additive failure_class field, so consumers react to a machine-readable taxonomy instead of regex-matching human error prose. Same values as the existing finished-execution metric label.
• slackbotv2 — captures the terminal failure off the rendered event stream; on a quota class it pins the thread's sticky harness override (model/provider tombstoned, like any harness switch), forgets the failed message's executed-state, and re-forwards it once through the normal path — reusing the existing harness-conflict restart and transcript-preamble machinery rather than inventing a parallel one.
• Plus a slackbotv2_quota_harness_fallbacks_total{outcome} counter, chart plumbing for the new value/env, and a configuration-reference docs row.

Testing

• cargo test -p centaur-session-runtime terminal_failure_class — includes a new test pinning the quota classification (and that plain rate limits stay out of it)
• services/slackbotv2: bun run check:types clean; bun test test — 233 tests, 0 fail, including new SSE failure_class propagation tests
• The quota classification was validated against a real usage-limit failure on a live deployment (the error text above is verbatim from it)

Deployment note

With claudeCodeAuthMode: access_token, api-rs validates at startup that the anthropic-claude broker credential exists in iron-control (the auth fragment references it) — seed that credential before pointing quotaFallbackHarness at claudecode, or api-rs crashloops on the 422. Verified on a live deployment.

🤖 Generated with Claude Code