feat: surface OpenMythos governance scorecard in dashboard and MCP#60
feat: surface OpenMythos governance scorecard in dashboard and MCP#60djimit wants to merge 2 commits into
Conversation
The OpenMythos benchmark (39 pre-registered APEX rounds) had zero product surface: no dashboard page, no MCP tools, and getGovernanceTrend had no consumer. This adds the read path end to end: - server: OpenMythosEvalService.listRuns + getLeaderboard, exposed as GET /api/openmythos/runs and /leaderboard (read:evidence) - mcp-server: djimitflo_openmythos_leaderboard + djimitflo_openmythos_score tools (read-only over openmythos_eval_runs) - dashboard: /governance page — per-agent leaderboard with category chips and trend, plus recent eval runs with oracle provenance Verified live: booted server on a temp DB, seeded a run, both endpoints return correct data through auth + RBAC. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Kilo Code Review could not run — your account is out of credits. Add credits or switch to a free model to enable reviews on this change. |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces the OpenMythos Governance Benchmark feature, adding a new Governance Scorecard page to the dashboard, corresponding API endpoints, backend database schema updates, and new MCP tools. Feedback focuses on optimizing database performance by replacing N+1 queries and correlated subqueries with window functions in both the backend service and MCP tools, as well as adding optional chaining to prevent potential runtime crashes in the UI when rendering run IDs.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| getLeaderboard(): AgentScore[] { | ||
| const agents = this.db.prepare(` | ||
| SELECT DISTINCT agent_id FROM openmythos_eval_runs WHERE status = 'completed' | ||
| `).all() as Array<{ agent_id: string }>; | ||
|
|
||
| // ponytail: one getAgentScore query per agent — fine at fleet sizes, batch when agents > ~1k | ||
| return agents | ||
| .map((a) => this.getAgentScore(a.agent_id)) | ||
| .filter((s): s is AgentScore => s !== null) | ||
| .sort((a, b) => b.overallScore - a.overallScore); | ||
| } |
There was a problem hiding this comment.
The current implementation of getLeaderboard() suffers from the getAgentScore() (one to get the latest run and another to get the last 5 runs for trend calculation).\n\nAt scale, this will result in significant performance degradation. Since SQLite 3.25+ supports window functions, we can fetch the latest and second-latest completed runs for all agents in a single, highly efficient query using ROW_NUMBER() OVER (PARTITION BY agent_id ORDER BY finished_at DESC). This reduces the database roundtrips from
getLeaderboard(): AgentScore[] {\n const rows = this.db.prepare(\n \"WITH ranked_runs AS (\" +\n \" SELECT \" +\n \" id, agent_id, overall_score, completed_cases, finished_at, metadata,\" +\n \" ROW_NUMBER() OVER (PARTITION BY agent_id ORDER BY finished_at DESC) as rn\" +\n \" FROM openmythos_eval_runs\" +\n \" WHERE status = 'completed'\" +\n \")\" +\n \"SELECT id, agent_id, overall_score, completed_cases, finished_at, metadata, rn\" +\n \"FROM ranked_runs\" +\n \"WHERE rn <= 2\" +\n \"ORDER BY agent_id, rn\"\n ).all() as Array<{\n id: string;\n agent_id: string;\n overall_score: number;\n completed_cases: number;\n finished_at: string;\n metadata: string;\n rn: number;\n }>;\n\n const agentGroups = new Map<string, typeof rows>();\n for (const row of rows) {\n if (!agentGroups.has(row.agent_id)) {\n agentGroups.set(row.agent_id, []);\n }\n agentGroups.get(row.agent_id)!.push(row);\n }\n\n const leaderboard: AgentScore[] = [];\n for (const [agentId, runs] of agentGroups.entries()) {\n const latest = runs[0];\n const prev = runs[1];\n\n let trend: 'improving' | 'stable' | 'declining' = 'stable';\n if (prev) {\n const diff = latest.overall_score - prev.overall_score;\n if (diff > 0.1) trend = 'improving';\n else if (diff < -0.1) trend = 'declining';\n }\n\n let metadata: { category_scores?: Record<string, number> } = {};\n try {\n metadata = JSON.parse(latest.metadata || '{}');\n } catch {}\n\n leaderboard.push({\n agentId,\n overallScore: latest.overall_score,\n categoryScores: metadata.category_scores || {},\n totalCases: latest.completed_cases,\n lastEvalAt: latest.finished_at,\n trend,\n });\n }\n\n return leaderboard.sort((a, b) => b.overallScore - a.overallScore);\n }| const rows = db.prepare(` | ||
| SELECT r.id, r.agent_id, r.status, r.total_cases, r.completed_cases, r.overall_score, r.started_at, r.finished_at, r.metadata | ||
| FROM openmythos_eval_runs r | ||
| WHERE r.status = 'completed' AND r.finished_at = ( | ||
| SELECT MAX(r2.finished_at) FROM openmythos_eval_runs r2 | ||
| WHERE r2.agent_id = r.agent_id AND r2.status = 'completed' | ||
| ) | ||
| ORDER BY r.overall_score DESC | ||
| `).all() as RunRow[]; |
There was a problem hiding this comment.
The current query uses a correlated subquery (r.finished_at = (SELECT MAX(r2.finished_at) ...)) to find the latest completed run per agent. This results in an openmythos_eval_runs for every single row in the table.\n\nWe can optimize this to a single scan using window functions (ROW_NUMBER() OVER (PARTITION BY agent_id ORDER BY finished_at DESC)), which is standard, highly efficient, and performs significantly better as the history grows.
const rows = db.prepare(\n \"WITH ranked_runs AS (\" +\n \" SELECT id, agent_id, status, total_cases, completed_cases, overall_score, started_at, finished_at, metadata,\" +\n \" ROW_NUMBER() OVER (PARTITION BY agent_id ORDER BY finished_at DESC) as rn\" +\n \" FROM openmythos_eval_runs\" +\n \" WHERE status = 'completed'\" +\n \")\" +\n \"SELECT id, agent_id, status, total_cases, completed_cases, overall_score, started_at, finished_at, metadata\" +\n \"FROM ranked_runs\" +\n \"WHERE rn = 1\" +\n \"ORDER BY overall_score DESC\"\n ).all() as RunRow[];| {runs.map((run) => ( | ||
| <div key={run.id} className="flex items-center gap-3 flex-wrap border-l-2 border-border pl-3 py-1"> | ||
| <FlaskConical className="w-3.5 h-3.5 text-foreground-tertiary" /> | ||
| <span className="text-xs font-mono text-foreground-tertiary">{run.id.slice(0, 8)}</span> |
There was a problem hiding this comment.
If run.id is ever undefined or null due to an unexpected API response or malformed data, calling .slice(0, 8) directly will throw a TypeError and crash the entire page. Using optional chaining with a fallback prevents runtime crashes and ensures a more robust UI.
| <span className="text-xs font-mono text-foreground-tertiary">{run.id.slice(0, 8)}</span> | |
| <span className=\"text-xs font-mono text-foreground-tertiary\">{run.id?.slice(0, 8) ?? 'n/a'}</span> |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9062eed5a8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| // ponytail: one getAgentScore query per agent — fine at fleet sizes, batch when agents > ~1k | ||
| return agents | ||
| .map((a) => this.getAgentScore(a.agent_id)) |
There was a problem hiding this comment.
When any completed run has malformed metadata, this new aggregate path calls getAgentScore(), whose JSON.parse(run.metadata || '{}') is unguarded; one bad agent row will throw and make /api/openmythos/leaderboard fail for the whole dashboard even though other agents have valid scores and listRuns() explicitly tolerates malformed metadata. Please isolate malformed rows or share the safe metadata parsing used by the new run list/MCP code.
Useful? React with 👍 / 👎.
…ax query - getAgentScore: guard JSON.parse so one malformed run row cannot sink the whole leaderboard (regression test added) - mcp leaderboard: replace correlated subquery with SQLite grouped-max - dashboard: null-safe run.id slice Kept the per-agent query in getLeaderboard as-is; it is ponytail-marked with its ceiling (batch when agents > ~1k). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 participant
Summary
P1 from the APEX validation plan: the OpenMythos Governance Benchmark — the platform's most differentiated capability, with 39 pre-registered APEX research rounds behind it — had zero product surface: no dashboard page, no MCP tools, and
getGovernanceTrendhad no consumer. This PR ships the read path end to end.Server
OpenMythosEvalService.listRuns(limit)— recent eval runs with subject model + oracle/judge provenance from run metadataOpenMythosEvalService.getLeaderboard()— latest completed score per agent, best firstGET /api/openmythos/runsandGET /api/openmythos/leaderboard(read:evidence)MCP server
djimitflo_openmythos_leaderboard— latest score per agent with category breakdowndjimitflo_openmythos_score— one agent's latest score + 10-run trendopenmythos_eval_runs; evals are started via REST, not MCPDashboard
/governance, ShieldCheck nav item): per-agent leaderboard with 0–5 score, trend arrow, category chips colored by score, and a recent-runs list showing subject model, status, and oracle provenanceValidation
🤖 Generated with Claude Code