feat: Prometheus metrics endpoint

[djimit]

Summary

P3 (operational depth) from the APEX validation plan: a real scrape target so the fleet can be monitored and alerted on. The existing /api/observability/metrics is JWT-gated JSON for the dashboard; Prometheus needs the text format and token auth.

• GET /metrics, Prometheus text format 0.0.4, zero new dependencies (format emitted by hand)
• Gauges computed per scrape from SQLite — no in-process counters, so restarts and multi-node deploys need no state: djimitflo_{tasks,agents,loop_runs,worker_leases,approvals,work_items}{status=...}, djimitflo_openmythos_score{agent=...} (ties the governance benchmark into alerting), djimitflo_ws_clients, process uptime/RSS
• Default-off: 404 unless METRICS_TOKEN is set; scrapers send Authorization: Bearer <token> (constant-time compare). Missing tables on a deploy degrade to header-only output instead of erroring
• Label values escaped per the exposition spec

Validation

• 4 new tests (invisible-when-unarmed, auth rejection, gauge/format assertions with seeded rows, label escaping); full server suite 1,217 green; type-check + lint clean
• Live: booted with METRICS_TOKEN — no/wrong token → 401, valid scrape returned the gauges in exposition format

Example alert this enables: djimitflo_openmythos_score{agent="nightly:llama3.1:8b"} < 2.5 → governance regression page.

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request introduces a Prometheus metrics exposition endpoint (GET /metrics) that exports various system gauges, OpenMythos scores, WebSocket client counts, and process metrics, along with corresponding unit tests. The review feedback identifies a security vulnerability in the token comparison where timingSafeEqual can throw a RangeError or leak token length, recommending that both tokens be hashed with SHA-256 before comparison. Additionally, the feedback suggests caching the prepared SQLite statements to improve performance by avoiding recompilation on every scrape request.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

To prevent timing attacks that leak the length of the expected token, and to avoid potential RangeError crashes in timingSafeEqual when comparing buffers of different lengths, we should hash both the expected and provided tokens using a fixed-length hash (like SHA-256) before performing the comparison.

Suggested change

	import { timingSafeEqual } from 'crypto';
	import { createHash, timingSafeEqual } from 'crypto';

[gemini-code-assist]

Implement the SHA-256 hashing approach to securely compare the tokens in constant time without leaking the token length or risking a RangeError crash.

function authorized(req: Request): boolean {
  const token = process.env.METRICS_TOKEN;
  if (!token) return false;
  const expected = `Bearer ${token}`;
  const provided = req.headers.authorization || '';
  const expectedHash = createHash('sha256').update(expected).digest();
  const providedHash = createHash('sha256').update(provided).digest();
  return timingSafeEqual(expectedHash, providedHash);
}

[gemini-code-assist]

Preparing SQL statements on every single scrape request is inefficient and can become a performance bottleneck. We should lazily prepare and cache the SQL statements inside the createMetricsHandler closure so they are compiled only once per statement.

export function createMetricsHandler(db: Database, getWsClients?: () => number) {
  const statementCache = new Map<string, ReturnType<typeof db.prepare>>();

  const getStatement = (sql: string) => {
    let stmt = statementCache.get(sql);
    if (!stmt) {
      stmt = db.prepare(sql);
      statementCache.set(sql, stmt);
    }
    return stmt;
  };

  const statusGauge = (name: string, help: string, table: string): string[] => {
    const lines = [`# HELP ${name} ${help}`, `# TYPE ${name} gauge`];
    try {
      const stmt = getStatement(`SELECT status, COUNT(*) AS n FROM ${table} GROUP BY status`);
      const rows = stmt.all() as Array<{ status: string; n: number }>;
      for (const row of rows) lines.push(`${name}{status="${escapeLabel(String(row.status))}"} ${row.n}`);
    } catch { /* table absent on this deploy — emit the header only */ }
    return lines;
  };

[gemini-code-assist]

Use the cached statement helper getStatement here as well to avoid preparing the query on every request.

Suggested change

	try {
	const rows = db.prepare(`
	SELECT agent_id, overall_score, MAX(finished_at) AS finished_at
	FROM openmythos_eval_runs WHERE status = 'completed' GROUP BY agent_id
	`).all() as Array<{ agent_id: string; overall_score: number }>;
	for (const row of rows) {
	lines.push(`djimitflo_openmythos_score{agent="${escapeLabel(row.agent_id)}"} ${row.overall_score}`);
	}
	} catch { /* eval table absent */ }
	try {
	const stmt = getStatement(`
	SELECT agent_id, overall_score, MAX(finished_at) AS finished_at
	FROM openmythos_eval_runs WHERE status = 'completed' GROUP BY agent_id
	`);
	const rows = stmt.all() as Array<{ agent_id: string; overall_score: number }>;
	for (const row of rows) {
	lines.push(`djimitflo_openmythos_score{agent="${escapeLabel(row.agent_id)}"} ${row.overall_score}`);
	}
	} catch { /* eval table absent */ }

[kilo-code-bot]

Kilo Code Review could not run — your account is out of credits.

Add credits or switch to a free model to enable reviews on this change.