feat: implement circuit breaker, retry with backoff, and graceful degradation for Stellar calls

[trustosaretin]

Closes #6

Type of Change

• New feature (non-breaking change that adds functionality)
• Bug fix
• Breaking change
• Refactoring (no functional or behavioral changes)
• Performance improvement
• Documentation update

---

Summary

All Stellar network calls (Horizon + Soroban RPC) had no retry logic, no circuit breaker, and no timeouts. During network issues, requests would block indefinitely, exhausting the connection pool and starving the event loop. This PR adds retry with exponential backoff, a circuit breaker, configurable timeouts, a Redis-backed retry queue for graceful degradation, and expanded health endpoints with dependency checks.

Motivation / Context

Fixes #6

During Stellar network outages, every reward claim, credential mint, and account funding call hangs indefinitely. Fastify's connection pool fills up, new requests queue behind blocked ones, and the entire API becomes unresponsive. Brief 502/503 hiccups are treated as permanent failures.

Detailed Changes

src/stellar/resilience.ts (new)

• Retry policy via cockatiel: 3 attempts, exponential backoff, only retries transient errors (ECONNREFUSED, ETIMEDOUT, 502, 503, etc.)
• Manual circuit breaker: opens after 5 consecutive transient failures, half-open after 30s
• Configurable read timeout (10s) and write timeout (30s)

src/stellar/client.ts

• getAccount: wrapped with circuit breaker + retry + 10s read timeout
• submitTransaction: wrapped with circuit breaker + retry + 30s write timeout
• callContract: wrapped with circuit breaker + retry + 30s write timeout
• Added getHorizonServer() for health checks

src/services/retry-queue.ts (new)

• Redis-backed FIFO queue using LPUSH/RPOP
• Max 10 retries per job, 30s retry interval
• Background processor starts with server, stops on graceful shutdown

src/modules/rewards/reward.service.ts

• claimReward catches circuit breaker errors specifically
• When circuit breaker is open, queues the claim via retry queue
• Returns { queued: true, txHash: null } instead of hard error

src/modules/rewards/reward.types.ts

• RewardClaimResult.txHash is now string | null
• Added queued: boolean field

src/server.ts

• /health: runs PostgreSQL, Redis, and Stellar Horizon checks; returns 200/503
• /health/live: liveness probe (always 200)
• /health/ready: readiness probe (200 only if all deps up)
• Starts/stops retry processor with server lifecycle

Tests

• tests/unit/services/resilience.test.ts: 9 tests covering retry, circuit breaker, timeout
• tests/unit/services/retry-queue.test.ts: 7 tests covering queue operations and processor

Testing

• Unit tests: 27 total — all pass
• Typecheck: 0 errors
• Lint: 0 errors (warnings only)

Breaking Changes

No. The API response for reward claims now includes queued and nullable txHash, but this is additive — existing clients that don't check these fields are unaffected.

Checklist

• All existing tests pass locally
• New tests added for new logic
• No new compiler warnings or linting errors introduced
• Typecheck passes
• Lint passes

Closes #6

[DeFiVC]

Thanks for tackling this — it's a solid implementation of circuit breaker, retry, and graceful degradation. The core architecture is sound. However, there are a few issues I'd like addressed before merging:

---

CI Failure

The Lint & Typecheck step passes ✅ but the Test job fails at npm run db:generate (drizzle-kit). This appears to be a pre-existing CI config issue unrelated to your changes, but the PR needs green CI before merge. Please investigate — likely the drizzle.config.ts is missing or has an issue in the CI environment.

---

Code Issues

1. Duplicated reward processing logic (server.ts:40-100 vs reward.service.ts)

processRetryJob in server.ts reimplements the entire reward claim flow (DB lookups, proof creation, invokeContract, DB updates). If reward.service.ts changes, this copy will silently drift. Consider extracting a shared method like executeRewardClaim(submission, user, score) in reward.service.ts and calling it from both paths.

2. Manual circuit breaker + cockatiel installed but unused for it

You installed cockatiel (which provides circuitBreaker) but implemented a manual circuit breaker in resilience.ts. The issue specifically suggested using cockatiel's circuit breaker. Using the library version would:

• Reduce code you maintain
• Be consistent with the retry policy (which does use cockatiel)
• Remove the mutable module-level state (circuitState, failureCount, lastFailureTime) which leaks between tests

3. Silent job loss on max retries (retry-queue.ts:41-46)

When requeueReward hits MAX_RETRIES (10), the job is dropped with only a log message. The reward is permanently lost with no dead-letter queue, no DB status update, and no notification. At minimum, update the quiz_submissions record to flag it as failed so it doesn't silently disappear.

4. Module-level mutable state in resilience.ts

circuitState, failureCount, and lastFailureTime are module-level let variables. This means:

• They're shared across all callers (which is correct for a singleton circuit breaker)
• But they leak between tests — resilience.test.ts calls getCircuitState() but never resets the state, so test order matters

If keeping the manual implementation, add a resetCircuitBreaker() export for tests.

5. Fragile transient error detection (resilience.ts:8-20)

isTransientError uses string matching (msg.includes("502"), msg.includes("network")). This could false-positive on unrelated error messages. Consider checking err.name (e.g., FetchError, HttpError) or HTTP status codes from the Stellar SDK's error types.

6. Missing: Fastify hookTimeout (Issue #6 requirement E)

The issue requested configuring hookTimeout on the Fastify instance to prevent route handlers from blocking indefinitely. This wasn't implemented. Not blocking — just noting it's an open item from the issue spec.

---

Minor Nits

• server.ts imports invokeContract and createQuizProof which are only used by processRetryJob. If the duplication is resolved (point 1), these imports can be removed from server.ts.
• The retry-queue.test.ts doesn't test the processor's behavior when processFn returns false (requeue path).

---

What's Good

• Retry with exponential backoff via cockatiel is well-implemented
• Circuit breaker threshold (5 failures) and half-open (30s) are reasonable defaults
• Read/write timeout differentiation (10s/30s) is sensible
• Health endpoints with dependency checks are clean and useful
• Queue processor with graceful shutdown is solid
• Test coverage for retry, circuit breaker, and queue is thorough

Please address points 1-5 and get CI green, then I'll approve.

[trustosaretin]

Thanks for the thorough review — all 5 points addressed:

1. Duplicated reward processing logic ✅

Extracted processRewardClaim() as a standalone exported function in reward.service.ts. Both the direct claim path and processRetryJob in server.ts now call this shared method. Removed the duplicated imports (invokeContract, createQuizProof, schema) from server.ts.

2. Manual circuit breaker + cockatiel ✅

I initially tried using cockatiel's circuitBreaker() but it has an internal bug in v4 (Cannot read properties of undefined (reading 'success') in CircuitBreakerPolicy.execute). The manual implementation is kept with resetCircuitBreaker() exported for test isolation. If the cockatiel issue is resolved in a future version, it's a straightforward swap.

3. Silent job loss on max retries ✅

Added reward_failed boolean column to quiz_submissions (schema + migration 0002). When requeueReward hits MAX_RETRIES, it now calls db.update(quizSubmissions).set({ rewardFailed: true }) so the submission is flagged as failed rather than silently dropped.

4. Module-level mutable state ✅

Added resetCircuitBreaker() export. All resilience tests call it in beforeEach to prevent state leaking between tests.

5. Fragile transient error detection ✅

Improved isTransientError():

• Checks err.name for FetchError / HttpError (Stellar SDK error types)
• Uses regex \b(502|503|504)\b instead of msg.includes("502") to avoid false positives
• Removed overly broad matches like "network" and "timeout"

Bonus: hookTimeout

hookTimeout is not in Fastify 5's type definitions (FastifyHttpOptions), so adding it would cause a type error. This is a Fastify 5 limitation — the option may need to be set differently or wait for Fastify to expose it in types.

CI

Added drizzle.config.ts so npm run db:generate works in CI. All unit tests pass (28/28), typecheck clean, lint clean.

[DeFiVC]

Great work addressing the feedback! All previous items are resolved — the extracted processRewardClaim(), resetCircuitBreaker(), reward_failed column, improved error detection, and additional tests are all solid improvements.

One remaining issue: CI test failure

The Lint & Typecheck passes ✅, but npm run test fails with:

Error: process.exit unexpectedly called with "1"
❯ loadConfig src/config/index.ts:43:13
❯ src/server.ts:6:1

Root cause: Your PR's changes to server.ts (new imports for db, redis, stellarClient, processRewardClaim, etc.) mean that any test file that transitively imports server.ts will trigger loadConfig(), which calls process.exit(1) when CI env vars are missing.

Fix: Either:

1. Mock ../config/index.js in test files that transitively import server.ts, or
2. Restructure server.ts so the start() function doesn't execute at module load time (currently start() is called at the bottom of the file — moving it to a conditional or using a separate entrypoint would prevent side effects during tests)

The cleanest approach is option 2 — the export { buildApp } line is there for testing, but start() at the bottom runs unconditionally when the module is loaded. Wrapping it in:

if (process.env.NODE_ENV !== "test") {
  start();
}

or using a separate index.ts entrypoint would fix this.

Once CI is green, I'll approve. The code itself looks good.

[trustosaretin]

Fixed — wrapped start() in NODE_ENV !== 'test' guard so test imports don't trigger loadConfig() and process.exit(1). CI should be green now.

[DeFiVC]

The NODE_ENV guard on start() is the right idea, but the CI still fails because the root cause is earlier in the import chain:

Error: process.exit unexpectedly called with "1"
❯ loadConfig src/config/index.ts:43:13
❯ src/config/index.ts:48:23    ← export const config = loadConfig()
❯ src/server.ts:6:1            ← import { config } from "./config/index.js"

config/index.ts line 48 calls loadConfig() at module level — it runs the moment any file imports config, before start() is ever reached. The NODE_ENV guard doesn't help because the import on server.ts:6 triggers loadConfig() unconditionally.

Fix options (pick one):

1. Lazy config — change config/index.ts to export a getter instead of calling loadConfig() at the top level:

   let _config: Env | null = null;
   export function getConfig(): Env {
     if (!_config) _config = loadConfig();
     return _config;
   }

   Then update server.ts to use getConfig() inside buildApp() instead of importing config at the top.

2. Move config import into buildApp() — keep loadConfig() at module level but don't import it in server.ts at the top. Import it inside buildApp() where it's actually needed.

3. Mock config/index.ts in all test files — this is the most fragile approach and doesn't scale.

Option 1 is cleanest — it defers validation until the config is actually used, which is what you want anyway (tests can import the module without triggering process.exit).

[trustosaretin]

Fixed — two changes:

1. Lazy config (config/index.ts): Wrapped config export in a Proxy so loadConfig() only runs on first property access, not at module import time. This means test files that transitively import config won't trigger process.exit(1) — the validation only happens when code actually reads a config value. All existing config.X call sites work unchanged.

2. CI env vars (.github/workflows/ci.yml): Added the missing Stellar env vars (STELLAR_HORIZON_URL, STELLAR_PLATFORM_SECRET, contract IDs) and corrected SOROBAN_RPC_URL → STELLAR_SOROBAN_RPC_URL to match the env schema.

The e2e test failures that remain are pre-existing (auth check ordering + missing DB services locally) — they'll pass in CI with the PostgreSQL and Redis service containers.

[DeFiVC]

All checks pass — LGTM! 🎉

Review summary:

• ✅ CI green (Lint & Typecheck + Test — 36 tests passing)
• ✅ Lazy config with test mode fallback — clean solution
• ✅ CI workflow has all required Stellar env vars
• ✅ Circuit breaker, retry, timeout, graceful degradation all implemented
• ✅ Deduplicated reward claim logic via shared processRewardClaim()
• ✅ reward_failed column for permanent failures
• ✅ Health endpoints with dependency checks
• ✅ Tests cover retry, circuit breaker, queue, and requeue paths

Nice work on this one — thorough implementation and good iteration on the feedback. Approving.