feat: indexer feature flag startup check, readiness latency, route-safe error mapping

docs/architecture/route-error-mapping.md

@@ -0,0 +1,41 @@
+# Route-safe unknown error mapping
+
+The global error middleware (`src/middlewares/error.middleware.ts`) routes
+known errors (Zod validation, JWT, Prisma, `ApiError`, payload-too-large,
+malformed JSON) through dedicated branches that already produce the right
+status code, error code and message.
+
+For everything else — an unhandled `throw`, a third-party library exception,
+a programming bug — the fallback path delegates to
+`mapUnknownRouteError()` in `src/utils/route-error.utils.ts`.
+
+## Why a shared helper
+
+- One place to evolve the unknown-error envelope shape.
+- Guaranteed inclusion of the route context id (`req.requestId`) so an
+  operator can correlate a 500 response with server-side logs.
+- Production responses never leak `error.message`, `stack`, or the raw
+  error object. Development responses include them for local debugging.
+
+## Envelope shape
+
+Production:
+
+```json
+{
+   "success": false,
+   "code": "INTERNAL_ERROR",
+   "message": "Internal server error",
+   "requestId": "8c4d…"
+}
+```
+
+Development additionally includes `stack` and `error` for fast iteration.
+
+## What the helper does NOT do
+
+It is the **fallback only**. Validation errors, auth errors, Prisma errors
+and explicit `ApiError` instances continue to be mapped by their dedicated
+branches in the global middleware. Do not reach for this helper to wrap a
+known-shape error — throw an `ApiError` (or one of the helpers in
+`error.middleware.ts`) instead.

docs/health-endpoints.md

@@ -0,0 +1,47 @@
+# Health and Readiness Endpoints
+
+The health module exposes three endpoints, each with a different contract.
+
+## `GET /api/v1/health` — liveness
+
+A minimal "the process is up" check. Always `200` while the event loop is
+healthy. Safe for load balancers and uptime monitors that should not fan out
+to dependencies.
+
+## `GET /api/v1/health/ready` — readiness
+
+Probes critical dependencies (database, cache config). Returns `200` when
+every probe passes and `503` otherwise.
+
+### Response shape
+
+```json
+{
+   "ready": true,
+   "timestamp": "2026-04-28T16:00:00.000Z",
+   "latencyMs": 7,
+   "checks": [
+      { "name": "database", "status": "ok", "latencyMs": 6 },
+      { "name": "cache", "status": "ok" }
+   ]
+}
+```
+
+### Fields
+
+| Field       | Type              | Notes                                                                                                                                          |
+| ----------- | ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
+| `ready`     | boolean           | `true` only when every check is `ok`. Maps directly to the HTTP status (`200` vs `503`).                                                       |
+| `timestamp` | string (ISO-8601) | When the response was built.                                                                                                                   |
+| `latencyMs` | number            | **Total** wall-clock duration of the readiness probe — sum of every check plus orchestration overhead. Useful for dashboards and SLO tracking. |
+| `checks`    | array             | Per-dependency results, each with its own `name`, `status`, optional `latencyMs`, and optional `error`.                                        |
+
+The payload is intentionally public-safe: no internal hostnames,
+connection strings, or stack traces are included even when a check fails.
+
+## `GET /api/v1/health/detailed` — diagnostics
+
+Full system snapshot including memory, uptime, system info, database
+response time, chain-sync lag and per-service health flags. Intended for
+operators rather than load balancers — cheaper liveness/readiness paths
+should be preferred for automated probing.

docs/indexer/FEATURE_FLAGS.md

@@ -0,0 +1,45 @@
+# Indexer Feature Flags
+
+The indexer subsystem is gated by a small set of boolean env vars plus the
+threshold values they reference. Every flag is validated at boot by
+`runIndexerFeatureFlagsStartupCheck()` in
+`src/utils/indexer-flags-startup-check.utils.ts`. Misconfigurations are
+collected into a single error so an operator can fix every issue in one
+pass — the server refuses to start until all listed issues are resolved.
+
+## Flags
+
+| Env var                                   | Type            | Default          | Purpose                                                                                       |
+| ----------------------------------------- | --------------- | ---------------- | --------------------------------------------------------------------------------------------- |
+| `ENABLE_INDEXER_DEDUPE`                   | boolean         | `true`           | Drop duplicate chain events by `(transactionHash, eventIndex)` before processing.             |
+| `ENABLE_INDEXER_DLQ`                      | boolean         | `true`           | Move events that fail after retries to the dead-letter queue. Requires dedupe to be enabled.  |
+| `ENABLE_INDEXER_CURSOR_STALENESS_WARNING` | boolean         | `true`           | Emit a structured `warn` log when the indexer cursor falls behind by more than the threshold. |
+| `INDEXER_JITTER_FACTOR`                   | number `[0, 1]` | `0.1`            | Random jitter applied to indexer poll intervals.                                              |
+| `INDEXER_CURSOR_STALE_AGE_WARNING_MS`     | integer ms      | `300000` (5 min) | How old the cursor must be before the staleness warning fires.                                |
+
+## Cross-field invariants enforced at startup
+
+- `INDEXER_JITTER_FACTOR` must be a finite number in `[0, 1]`.
+- `INDEXER_CURSOR_STALE_AGE_WARNING_MS` must be a positive integer.
+- If `ENABLE_INDEXER_CURSOR_STALENESS_WARNING=true`, the threshold must be
+  at least `1000ms`. Either disable the flag or raise the threshold.
+- `ENABLE_INDEXER_DLQ=true` requires `ENABLE_INDEXER_DEDUPE=true`. The DLQ
+  uses the dedupe key to identify repeated failures; running DLQ without
+  dedupe causes duplicate DLQ entries.
+
+## Failure mode
+
+If any invariant fails, the server logs a structured `error` entry with
+the full list of issues and exits with code `1`:
+
+```
+Refusing to start: indexer feature flags are misconfigured
+  issues:
+    - INDEXER_JITTER_FACTOR must be a number between 0 and 1 (got 5)
+    - ENABLE_INDEXER_DLQ is on but ENABLE_INDEXER_DEDUPE is off — …
+```
+
+## Validating locally
+
+Set the env vars you want to verify and run `pnpm dev`. The startup log
+emits one info line confirming the validated values.

src/config.ts

@@ -58,6 +58,13 @@ export const envSchema = z.object({
    BACKGROUND_JOB_LOCK_TTL_MS: z.coerce.number().int().positive().default(300000),
    CREATOR_LIST_SLOW_QUERY_THRESHOLD_MS: z.coerce.number().int().positive().default(500),
    INDEXER_CURSOR_STALE_AGE_WARNING_MS: z.coerce.number().int().positive().default(300000),
+
+   // Indexer feature flags — toggle individual indexer behaviors. The startup
+   // check in src/utils/indexer-flags-startup-check.utils.ts enforces the
+   // cross-field invariants between these flags and their threshold values.
+   ENABLE_INDEXER_DEDUPE: z.coerce.boolean().default(true),
+   ENABLE_INDEXER_DLQ: z.coerce.boolean().default(true),
+   ENABLE_INDEXER_CURSOR_STALENESS_WARNING: z.coerce.boolean().default(true),
 });
 
 export const envConfig = envSchema.parse(process.env);

src/middlewares/error.middleware.ts

@@ -6,6 +6,7 @@ import chalk from 'chalk';
 import { z } from 'zod';
 import { ErrorCode, ErrorCodeType } from '../constants/error.constants';
 import { logger } from '../utils/logger.utils';
+import { mapUnknownRouteError } from '../utils/route-error.utils';
 
 export class ApiError extends Error {
    statusCode: number;
@@ -165,22 +166,14 @@ export const errorHandler: ErrorRequestHandler = (
       `${method === 'GET' ? chalkColor.getReq(method) : chalkColor.postReq(method)} ${protocol}://${hostname}:${envConfig.PORT || 3000}${originalUrl}`
    );
 
-   // Default error response
-   const statusCode = err.statusCode || err.status || 500;
-   const message =
-      envConfig.MODE === 'production'
-         ? 'Internal server error'
-         : err.message || 'Something went wrong!';
-
-   res.status(statusCode).json({
-      success: false,
-      code: err.errorCode || ErrorCode.INTERNAL_ERROR,
-      message,
-      ...(envConfig.MODE === 'development' && {
-         stack: err.stack,
-         error: err,
-      }),
+   // Default fallback for unknown errors — delegated to a shared helper so
+   // route-safe envelopes stay consistent and include the request id for
+   // correlation. Known-error branches above handle their own mappings.
+   const { statusCode, body } = mapUnknownRouteError(err, {
+      requestId: req.requestId,
+      includeDebug: envConfig.MODE === 'development',
    });
+   res.status(statusCode).json(body);
 };
 
 // Helper functions for common errors

src/modules/health/health.controllers.test.ts

@@ -0,0 +1,74 @@
+import { Request, Response } from 'express';
+
+jest.mock('../../config', () => ({
+   envConfig: { MODE: 'test', PORT: 3000 },
+   appConfig: { allowedOrigins: [] },
+}));
+
+jest.mock('../../utils/prisma.utils', () => ({
+   prisma: {
+      $queryRaw: jest.fn(),
+   },
+}));
+
+import { readinessCheck } from './health.controllers';
+import { prisma } from '../../utils/prisma.utils';
+
+const queryRawMock = prisma.$queryRaw as unknown as jest.Mock;
+
+function buildRes() {
+   const res = {
+      statusCode: 0,
+      body: undefined as unknown,
+      status(code: number) {
+         this.statusCode = code;
+         return this;
+      },
+      json(payload: unknown) {
+         this.body = payload;
+         return this;
+      },
+   };
+   return res as unknown as Response & { statusCode: number; body: any };
+}
+
+describe('readinessCheck()', () => {
+   beforeEach(() => {
+      queryRawMock.mockReset();
+   });
+
+   it('includes a top-level latencyMs field in the response metadata', async () => {
+      queryRawMock.mockResolvedValue([{ '?column?': 1 }]);
+      const res = buildRes();
+
+      await readinessCheck({} as Request, res);
+
+      expect(res.statusCode).toBe(200);
+      expect(res.body.ready).toBe(true);
+      expect(typeof res.body.latencyMs).toBe('number');
+      expect(res.body.latencyMs).toBeGreaterThanOrEqual(0);
+   });
+
+   it('still reports latencyMs when a check fails (returns 503)', async () => {
+      queryRawMock.mockRejectedValue(new Error('connection refused'));
+      const res = buildRes();
+
+      await readinessCheck({} as Request, res);
+
+      expect(res.statusCode).toBe(503);
+      expect(res.body.ready).toBe(false);
+      expect(typeof res.body.latencyMs).toBe('number');
+   });
+
+   it('keeps the existing per-check latencyMs alongside the new top-level field', async () => {
+      queryRawMock.mockResolvedValue([{ '?column?': 1 }]);
+      const res = buildRes();
+
+      await readinessCheck({} as Request, res);
+
+      const dbCheck = res.body.checks.find((c: any) => c.name === 'database');
+      expect(dbCheck.status).toBe('ok');
+      expect(typeof dbCheck.latencyMs).toBe('number');
+      expect(typeof res.body.latencyMs).toBe('number');
+   });
+});

src/modules/health/health.controllers.ts

@@ -158,6 +158,7 @@ export const simpleHealthCheck = (_: Request, res: Response): void => {
 
 export const readinessCheck = async (_: Request, res: Response): Promise<void> => {
   const checks: ReadinessCheck[] = [];
+  const overallStart = Date.now();
 
   // DB check
   const dbStart = Date.now();
@@ -189,6 +190,7 @@ export const readinessCheck = async (_: Request, res: Response): Promise<void> =
   res.status(ready ? 200 : 503).json({
     ready,
     timestamp: new Date().toISOString(),
+    latencyMs: Date.now() - overallStart,
     checks,
   });
 };

src/server.ts

@@ -5,10 +5,30 @@ import { envConfig } from './config';
 import { logger } from './utils/logger.utils';
 import { prisma } from './utils/prisma.utils';
 import { verifyMigrationChecksums } from './utils/migration-checksum.utils';
+import {
+   IndexerFlagsConfigError,
+   runIndexerFeatureFlagsStartupCheck,
+} from './utils/indexer-flags-startup-check.utils';
 
 
 async function startServer() {
    try {
+      // Validate indexer feature flags before any code paths read them. We
+      // fail fast here so operators see every misconfiguration at once
+      // instead of cryptic runtime errors later in the boot sequence.
+      try {
+         runIndexerFeatureFlagsStartupCheck();
+      } catch (err) {
+         if (err instanceof IndexerFlagsConfigError) {
+            logger.error(
+               { issues: err.issues },
+               'Refusing to start: indexer feature flags are misconfigured'
+            );
+            process.exit(1);
+         }
+         throw err;
+      }
+
       await prisma.$connect();
       logger.info('Connected to database');