feat: governance-aware LLM routing from OpenMythos benchmark scores

[djimit]

Summary

P2 from the APEX validation plan: the benchmark becomes a live control signal. Today's multi-model APEX runs showed category performance is heterogeneous and complementary — llama3.1:8b defends injection at 75% while qwen sits at 42%, and the inverse holds for overthinking (17% vs 58%). This PR feeds that evidence into routing.

• RoutingRequest gains riskCategory (an OpenMythos category, e.g. injection, hallucination)
• When set, LlmRouterService.route() prefers the healthy candidate whose model has the best latest benchmark score for that category — gated by GOVERNANCE_ROUTER_FLOOR (default 3/5)
• No scores for the category, all below floor, or no riskCategory → static task routing, byte-for-byte unchanged
• Every governance decision cites its evidence: "Governance: qwen2.5:14b-instruct-q4_K_M scored 4.20/5 on 'injection' (OpenMythos benchmark)"
• Scores read from openmythos_eval_runs metadata (latest completed run per subject model) — the nightly scheduler (feat: nightly OpenMythos eval scheduler #61) keeps the routing table current with zero new tables or state
• POST /api/apex/llm/route needed no changes (body passthrough)

Validation

• 6 new tests (governance preference, floor gating incl. env override, latest-run-wins, no-category and no-data fallbacks); all 29 existing apex-integration tests untouched and green; type-check + lint clean
• Live end-to-end: booted the server, refreshed provider health, seeded a nightly eval row — riskCategory: "injection" (score 4.2) routed to that model with the governance reason; overthinking (score 1.7, below floor) fell back to static routing

Stacks on #60/#61 conceptually but has no file overlap with either.

🤖 Generated with Claude Code

[kilo-code-bot]

Kilo Code Review could not run — your account is out of credits.

Add credits or switch to a free model to enable reviews on this change.

[gemini-code-assist]

Code Review

This pull request introduces governance-based routing to the LlmRouterService, allowing routing decisions to prioritize models with the best benchmark scores for a specified riskCategory based on evaluation runs in the database. Feedback on these changes highlights a performance bottleneck caused by querying the database and parsing JSON on every routing request, suggesting the implementation of an in-memory cache. Additionally, it is recommended to safely parse the GOVERNANCE_ROUTER_FLOOR environment variable to prevent NaN values from silently bypassing the floor check.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

Querying the database and parsing JSON for all completed evaluation runs on every single routing request is a major performance bottleneck. Since routing is on the critical path of every LLM call, this will introduce significant latency and database contention as the openmythos_eval_runs table grows over time.

To resolve this, we should cache the governance scores in memory with a TTL (e.g., 1 minute). Here is an example of how you can implement this:

// Add these private fields to the LlmRouterService class:
// private governanceScoresCache: Map<string, Map<string, number>> | null = null;
// private lastScoresFetchTime = 0;
// private readonly CACHE_TTL_MS = 60000;

private getGovernanceScores(): Map<string, Map<string, number>> {
  const now = Date.now();
  if (this.governanceScoresCache && (now - this.lastScoresFetchTime < this.CACHE_TTL_MS)) {
    return this.governanceScoresCache;
  }

  const cache = new Map<string, Map<string, number>>();
  let rows: Array<{ metadata: string }>;
  try {
    rows = this.db.prepare(`
      SELECT metadata FROM openmythos_eval_runs
      WHERE status = 'completed'
      ORDER BY finished_at ASC
    `).all() as Array<{ metadata: string }>;
  } catch {
    this.governanceScoresCache = cache;
    this.lastScoresFetchTime = now;
    return cache;
  }

  for (const row of rows) {
    try {
      const metadata = JSON.parse(row.metadata || '{}') as {
        subject_model?: string;
        category_scores?: Record<string, number>;
      };
      if (metadata.subject_model && metadata.category_scores) {
        let modelScores = cache.get(metadata.subject_model);
        if (!modelScores) {
          modelScores = new Map<string, number>();
          cache.set(metadata.subject_model, modelScores);
        }
        for (const [cat, score] of Object.entries(metadata.category_scores)) {
          if (typeof score === 'number') {
            modelScores.set(cat, score);
          }
        }
      }
    } catch { /* skip malformed rows */ }
  }

  this.governanceScoresCache = cache;
  this.lastScoresFetchTime = now;
  return cache;
}

private governanceScores(category: string): Map<string, number> {
  const scores = new Map<string, number>();
  const allScores = this.getGovernanceScores();
  for (const [model, modelScores] of allScores.entries()) {
    const score = modelScores.get(category);
    if (score !== undefined) {
      scores.set(model, score);
    }
  }
  return scores;
}

[gemini-code-assist]

If process.env.GOVERNANCE_ROUTER_FLOOR is set to an invalid number (e.g., a typo or empty string), Number(...) will return NaN. Since any comparison with NaN (such as score < floor) evaluates to false, this would silently bypass the governance floor check entirely and route to low-scoring models.

We should safely validate the parsed number and fall back to the default floor of 3 if it is NaN.

Suggested change

	const floor = Number(process.env.GOVERNANCE_ROUTER_FLOOR ?? '3');
	const floorEnv = process.env.GOVERNANCE_ROUTER_FLOOR;
	const floor = floorEnv && !isNaN(Number(floorEnv)) ? Number(floorEnv) : 3;

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: abc1e719a3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Clear stale category scores after re-benchmarks

When a model has an older run with this category and a later completed run that omitted it, this loop skips the later row and leaves the old score in the map. OpenMythosEvalService.runEval can run a filtered category set and only writes those category_scores, so riskCategory: 'injection' can keep routing on stale evidence after the model was re-benchmarked for another category, instead of following the documented “latest completed run per subject model” behavior and falling back when the latest run has no score for that category.

Useful? React with 👍 / 👎.