feat(observability): add Prometheus metrics, audit logging, and OpenTelemetry tracing

[brightpixel-dev]

Closes #9

What changed

• src/metrics/index.ts — prom-client registry with 6 metrics: HTTP request counter, HTTP latency histogram, Stellar tx duration histogram (per-method, success/error), quiz submission counter (passed/failed), reward claims counter (success/queued), credentials minted counter; default Node.js process metrics collected
• src/metrics/fastify-hook.ts — Fastify onRequest/onResponse hooks that record count and latency for every route, labelled by method, route pattern, and status code
• src/server.ts — registers metrics hook, exposes GET /metrics (Prometheus text format), imports initTracing() before Fastify setup, calls shutdownTracing() on clean exit
• src/audit/index.ts — typed auditLog(event, fields) helper that emits a pino log entry with audit: true; events: quiz.submitted, reward.claimed, reward.queued, credential.minted, auth.login, auth.login_failed
• src/tracing.ts — OTel NodeSDK with OTLP/HTTP exporter (configurable via OTEL_EXPORTER_OTLP_ENDPOINT), Fastify, pg, and ioredis auto-instrumentations; disabled when OTEL_SDK_DISABLED=true
• reward.service.ts — Stellar tx duration measured around invokeContract, rewardClaimsTotal incremented on success/queued, auditLog called on both paths
• credential.service.ts — same Stellar tx timing pattern for mint_credential, credentialsMintedTotal incremented, auditLog on success
• quiz.service.ts — quizSubmissionsTotal incremented with passed/failed label, auditLog on submission

Why

Closes issue #9 — without metrics there is no visibility into request latency, Stellar contract call health, or financial event throughput. Audit logs provide a tamper-evident record of every on-chain operation for compliance. OTel traces allow distributed request correlation across Fastify → Postgres → Redis → Stellar.

How to test

# Start the server
npm run dev

# Scrape metrics endpoint
curl http://localhost:3000/metrics

# Look for these metric families:
#   http_requests_total
#   http_request_duration_seconds
#   stellar_tx_duration_seconds
#   quiz_submissions_total
#   reward_claims_total
#   credentials_minted_total
#   process_cpu_user_seconds_total   (default Node metrics)

# For tracing: point OTEL_EXPORTER_OTLP_ENDPOINT at a local Jaeger/Tempo
# For audit logs: grep for "audit":true in pino output after a quiz submit

[DeFiVC]

Solid observability implementation that covers metrics, audit logging, and tracing — but must resolve merge conflicts before merging.

Blocking Issues

1. ❌ Merge Conflicts — mergeStateStatus: "DIRTY". The PR is based on stale main. src/server.ts has new health check endpoints (/health, /health/live, /health/ready) and a stellarClient import that the PR's diff doesn't account for. Rebase needed:

   git fetch origin
   git rebase origin/main
   # resolve conflicts in src/server.ts
   git rebase --continue

2. ⚠️ CI Not Run — statusCheckRollup is empty (likely due to conflict state). Must verify npm run typecheck, npm run lint, and npm test pass after rebase.

Code Issues

3. Missing system gauges — Issue #9 requested db_active_connections and redis_connected gauges for infrastructure health. Not implemented.

4. src/audit/index.ts — Missing ip and userAgent fields in AuditFields for security event tracing (issue #9 specified these for failed auth attempts).

What's Good

• Clean module separation: metrics/, audit/, tracing/
• Correct metric types: counters for totals, histograms for latency with sensible buckets
• Audit logging covers all financial events (reward claims, credential mints, quiz submissions)
• Tracing disabled by default via OTEL_SDK_DISABLED — safe for dev
• initTracing() before Fastify setup captures full request lifecycle
• shutdownTracing() on clean exit prevents span data loss

Please rebase on current main, resolve conflicts, and verify CI passes.

[brightpixel-dev]

Hi @DeFiVC — all review items have been addressed and the branch has been force-pushed. Here's a summary of what was done:

1. Merge conflict resolved
Rebased on current main. The only conflict was in quiz.service.ts imports — the AI-quiz PR (#20) had added generateQuizFromAI. Kept both that import and the observability imports.

2. Missing infra gauges added

• db_active_connections — prom-client collect() callback samples pool.totalCount - pool.idleCount on each /metrics scrape (zero overhead at idle)
• redis_connected — 0/1 gauge updated via ioredis connect/close/error events
• Exported pool from config/database.ts and call setupInfraMetrics(pool, redis) in buildApp() before route registration

3. ip and userAgent added to AuditFields
Both fields are now in the interface so auth routes can pass request.ip and request.headers['user-agent'] when calling auditLog("auth.login_failed", { ip, userAgent, ... }).

4. Typecheck passes
npx tsc --noEmit exits clean after rebase and all changes.

Ready for re-review.

[DeFiVC]

All blocking issues from the previous review have been addressed. The PR now delivers a complete observability stack that fully satisfies issue #9.

What's Good

• Infrastructure gauges: db_active_connections (samples pool.totalCount - pool.idleCount) and redis_connected (tracks connect/close/error events) — both use prom-client's collect() callback for lazy evaluation
• Audit fields complete: ip and userAgent now present in AuditFields for security event tracing
• Clean rebase: Conflicts resolved, upstream AI-quiz PR changes incorporated
• CI green: Lint & Typecheck and Test both pass

Summary

• 6 Prometheus metrics (HTTP, Stellar tx, quiz, reward, credential) + 2 infrastructure gauges
• Structured audit logging for all financial events
• OpenTelemetry tracing with Fastify, pg, and ioredis auto-instrumentation
• /metrics endpoint for Prometheus scraping

Approving — the implementation is solid, well-structured, and addresses all requirements from issue #9.