[Expert] Implement circuit breaker, retry with backoff, and graceful degradation for Stellar network calls

[DeFiVC]

Description

The API makes direct synchronous calls to Stellar Horizon and Soroban RPC with no retry logic, no circuit breaker, and no graceful degradation. During Stellar network issues, every request will block until timeout (60s), exhausting the connection pool and starving the Node.js event loop.

Problem Analysis

Current Stellar call paths

All external Stellar calls go through src/stellar/client.ts and src/stellar/transactions.ts:

Method	File:Line	External Call	Timeout
getAccount	client.ts:33-44	Horizon API	None
submitTransaction	client.ts:46-67	Horizon API	None
callContract	client.ts:70-86	Soroban RPC	None
invokeContract	transactions.ts:15-48	Soroban RPC (simulate + submit)	60s tx timeout
fundAccount	accounts.ts:22-41	Horizon API	None

What happens during Stellar outage

1. Every reward claim, credential mint, and account funding call hangs
2. Fastify's connection pool fills up with pending promises
3. New requests queue behind the blocked ones
4. After ~30s, clients timeout and retry, making it worse
5. After ~60s, the transaction time bounds expire, but the HTTP request may still be pending
6. The entire API becomes unresponsive

No retry for transient failures

Horizon returns HTTP 502/503 during brief outages. Currently these are treated as permanent failures.

No circuit breaker

There is no mechanism to short-circuit calls when Stellar is known to be down.

Required Implementation

A. Retry with Exponential Backoff

Install cockatiel (or implement manually) for retry policies:

// New file: src/stellar/resilience.ts
import { retry, circuitBreaker, timeout } from "cockatiel";

// Retry policy: 3 attempts, exponential backoff starting at 500ms
export const stellarRetry = retry(
  (details) => details.attemptNumber * 500,
  { maxAttempts: 3 }
);

// Circuit breaker: open after 5 consecutive failures, half-open after 30s
export const stellarCircuitBreaker = circuitBreaker(
  (err: Error) => {
    // Only break on network/transient errors, not business errors
    return err.message.includes("ECONNREFUSED") ||
           err.message.includes("ETIMEDOUT") ||
           err.message.includes("502") ||
           err.message.includes("503");
  },
  {
    halfOpenAfter: 30_000,
    breaker: {
      threshold: 5,
      duration: 60_000,
    },
  }
);

// Timeout: 10s for read operations, 30s for write operations
export const readTimeout = timeout(10_000);
export const writeTimeout = timeout(30_000);

B. Wrap Stellar Client Methods

// Updated src/stellar/client.ts
async submitTransaction(
  txEnvelope: StellarSdk.Transaction | StellarSdk.FeeBumpTransaction
): Promise<StellarSdk.Horizon.SubmitTransactionResponse> {
  return stellarCircuitBreaker.execute(() =>
    stellarRetry.execute(() =>
      writeTimeout.execute(async () => {
        try {
          const result = await this.horizon.submitTransaction(txEnvelope);
          if (result.status === "error") {
            throw new StellarError(result);
          }
          return result;
        } catch (err) {
          logger.error({ err }, "Stellar submitTransaction failed");
          throw err;
        }
      })
    )
  );
}

C. Graceful Degradation

For operations where Stellar availability is non-critical, degrade gracefully:

// In reward.service.ts
async claimReward(userId: string, submissionId: string) {
  // ... validation ...

  let txHash: string | null = null;
  let onChainSuccess = false;

  try {
    const result = await stellarClient.invokeContract(...);
    txHash = result.hash;
    onChainSuccess = true;
  } catch (err) {
    if (err instanceof CircuitBreakerError) {
      logger.warn({ submissionId }, "Stellar circuit breaker open — queuing reward");
      // Queue for later processing instead of failing
      await this.queueRewardForLater(submissionId, userId, score);
      return { success: true, queued: true, txHash: null };
    }
    throw err;
  }

  // ... DB updates ...
}

D. Background Retry Queue

For rewards that fail due to Stellar unavailability, implement a background queue:

// New file: src/services/retry-queue.ts
// Use Redis list as a simple FIFO queue
// Process with a background worker that retries every 30s
// Max retries: 10, then alert and mark as failed

E. Request Timeout on Fastify

// In server.ts
const app = Fastify({
  // ...
  hookTimeout: 15_000,     // 15s max for route handlers
  bodyLimit: 1048576,       // 1MB body limit
});

// Per-route timeout for Stellar-dependent endpoints
app.addHook("onSend", async (request, reply) => {
  if (request.url.includes("/rewards/claim") || request.url.includes("/credentials/mint")) {
    reply.header("X-Timeout", "30");
  }
});

F. Health Check with Dependency Verification

// Updated health endpoint
app.get("/health", async (request, reply) => {
  const checks = await Promise.allSettled([
    db.execute(sql`SELECT 1`),           // PostgreSQL
    redis.ping(),                         // Redis
    stellarClient.horizon.root(),        // Stellar Horizon
  ]);

  const status = checks.every(c => c.status === "fulfilled") ? "healthy" : "degraded";

  return reply.status(status === "healthy" ? 200 : 503).send({
    status,
    timestamp: new Date().toISOString(),
    uptime: process.uptime(),
    checks: {
      database: checks[0].status === "fulfilled" ? "ok" : "error",
      redis: checks[1].status === "fulfilled" ? "ok" : "error",
      stellar: checks[2].status === "fulfilled" ? "ok" : "error",
    },
  });
});

// Separate liveness probe (always 200 if process is running)
app.get("/health/live", async () => ({ status: "ok" }));

// Readiness probe (200 only if all deps are up)
app.get("/health/ready", async (request, reply) => {
  // ... same as /health but only returns 200 when fully healthy
});

Dependencies to Add

npm install cockatiel

Testing Requirements

• Mock Stellar Horizon to return 502/503 and verify retry attempts
• Mock Stellar to be completely unavailable and verify circuit breaker opens
• Verify that after circuit breaker opens, requests fail fast (< 100ms)
• Verify half-open state allows one probe request through
• Test the background retry queue processes items when Stellar recovers
• Load test: 100 concurrent reward claims with Stellar returning 503 — verify no crash

References

• Cockatiel: https://cockatiel.sijmber.nl/
• Circuit breaker pattern: https://martinfowler.com/bliki/CircuitBreaker.html
• Stellar Horizon error codes: https://developers.stellar.org/docs/data/horizon/api-reference/errors/

[trustosaretin]

I'd like to work on this issue.

Every Stellar network call in src/stellar/client.ts and src/stellar/transactions.ts — getAccount, submitTransaction, callContract, invokeContract, fundAccount — makes a raw HTTP request with no retry, no circuit breaker, and no timeout. During a Stellar outage or even a brief 502/503 hiccup, these calls block indefinitely, exhausting Fastify's connection pool, starving the Node.js event loop, and cascading into a full API-wide hang.

My approach
I will introduce cockatiel to layer retry with exponential backoff, a circuit breaker, and configurable timeouts around all Stellar client methods. A lightweight Redis-backed retry queue will handle reward claims that fail due to Stellar unavailability, and the health endpoint will be expanded to report dependency status with proper readiness/liveness probes.

Implementation plan

1. Install cockatiel and create the resilience module
   • Add cockatiel as a dependency
   • Create src/stellar/resilience.ts exporting stellarRetry (3 attempts, exponential backoff starting at 500ms), stellarCircuitBreaker (opens after 5 consecutive transient failures, half-open after 30s), readTimeout (10s), and writeTimeout (30s)
   • The circuit breaker filter should only trigger on ECONNREFUSED, ETIMEDOUT, HTTP 502, and 503 — not on business logic errors
2. Wrap src/stellar/client.ts methods with resilience policies
   • getAccount (line 33): wrap with stellarCircuitBreaker.execute(() => stellarRetry.execute(() => readTimeout.execute(...)))
   • submitTransaction (line 46): wrap with circuit breaker + retry + writeTimeout (30s)
   • callContract (line 70): wrap with circuit breaker + retry + writeTimeout
   • Add structured logging (logger.error) on each final failure after retries are exhausted
3. Wrap src/stellar/transactions.ts invokeContract (line 15-48)
   • Wrap the simulate + submit flow in the circuit breaker and write timeout
   • The 60s transaction time bounds remain; the writeTimeout (30s) ensures we fail fast before the on-chain time bounds expire, preventing orphaned pending transactions
4. Add graceful degradation to src/modules/rewards/reward.service.ts
   • In claimReward, catch CircuitBreakerError specifically
   • When the circuit breaker is open, queue the reward claim via the Redis-backed retry queue instead of returning an error
   • Return { success: true, queued: true, txHash: null } so the user knows the claim is pending
   • Create src/services/retry-queue.ts using Redis LPUSH/BRPOP as a FIFO queue with max 10 retries and a 30s retry interval
5. Expand health endpoint with dependency checks and liveness/readiness probes
   • /health: run Promise.allSettled over PostgreSQL SELECT 1, redis.ping(), and stellarClient.horizon.root() — return 200 if all healthy, 503 if degraded
   • /health/live: always 200 if process is running (liveness probe)
   • /health/ready: 200 only if all dependency checks pass (readiness probe)
   • Expose individual dependency status in the response body
6. Write tests for retry, circuit breaker, and degradation
   • Mock Stellar Horizon to return 502/503, verify retry fires 3 times with increasing delays
   • Mock Stellar to be completely unavailable, verify circuit breaker opens and subsequent requests fail fast (< 100ms)
   • Verify half-open state allows one probe request through
   • Test that claimReward queues the claim when the circuit breaker is open and the user receives a queued: true response
   • Test the retry queue processes items when Stellar recovers and marks them complete

Impact

• Prevents API-wide cascading failures during Stellar network outages — connection pool exhaustion and event loop starvation are eliminated by short-circuiting when Stellar is down.
• Improves user experience for transient failures: brief 502/503 hiccups are automatically retried without user intervention, and reward claims degrade gracefully into a queued state rather than returning hard errors.
• Aligns the API with production resilience best practices: circuit breaker, retry, and timeout are standard patterns for any external dependency in a financial system.
• The expanded health endpoints enable Kubernetes or load balancer probes to route traffic away from degraded instances, supporting zero-downtime deployments and horizontal scaling.

I'm ready to start immediately and can have a PR up within a reasonable timeline. Happy to adjust the plan if you have preferences on any of the details.

[grantfox-oss]

🦊 GrantFox — @trustosaretin has been assigned to this issue as part of the Official Campaign campaign!

Next steps:

1. Open a Pull Request referencing this issue (e.g., Closes #6)
2. Your PR will be reviewed by the ChainLearnOfficial maintainers

Good luck! Track your progress on GrantFox.

[grantfox-oss]

🎉 This issue has been marked as completed on GrantFox as part of the Official Campaign campaign!

@trustosaretin's PR #18 was approved and merged by @DeFiVC.

🏆 @trustosaretin: You earned 35 FoxPoints for this contribution! Your current tier: Explorer (83 total points). Track your full progress on GrantFox.

👏 Great work, @trustosaretin! Keep contributing to ChainLearnOfficial.