feat: nightly OpenMythos eval scheduler#61
Conversation
Scheduled governance evals so the leaderboard fills with production data instead of staying empty. Default-off; armed via OPENMYTHOS_NIGHTLY_ENABLED with a comma-separated subject-model list. One run per model per UTC day under agent id nightly:<model>, deduped against openmythos_eval_runs so restarts don't double-run. When OPENMYTHOS_ORACLE_ANCHORS_PATH is set only the oracle-anchored subset runs (deterministic, minutes per model); models run sequentially to avoid saturating the Ollama host. Verified live: booted with the scheduler armed against workstation Ollama; it ran 78 oracle cases unattended and scored 2.692 — identical to both manual APEX validation runs of the same subset. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Kilo Code Review could not run — your account is out of credits. Add credits or switch to a free model to enable reviews on this change. |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces OpenMythosNightlyService to run scheduled governance evaluations and populate the leaderboard, along with its corresponding unit tests and integration into the server startup. The review feedback highlights two critical issues: first, shouldRun uses SQLite's date('now') instead of the passed now parameter, which makes tests non-deterministic and mixes timezones; second, tick lacks a concurrency guard to prevent overlapping runs if a tick takes longer than an hour, and fails to catch errors during setup, which could lead to unhandled promise rejections.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| shouldRun(model: string, now: Date = new Date()): boolean { | ||
| if (now.getHours() < this.targetHour()) return false; | ||
| const row = this.db.prepare(` | ||
| SELECT COUNT(*) AS n FROM openmythos_eval_runs | ||
| WHERE agent_id = ? AND status IN ('running', 'completed') | ||
| AND substr(started_at, 1, 10) = date('now') | ||
| `).get(`nightly:${model}`) as { n: number }; | ||
| return row.n === 0; | ||
| } |
There was a problem hiding this comment.
The shouldRun method currently uses SQLite's date('now') to check if a run has already been recorded today. This introduces two issues:
- It ignores the
nowparameter passed to the function, making the query non-deterministic and hard to test (the test will always query against the actual current system date). - It mixes the server's local timezone (used in
now.getHours()) with UTC (used in SQLite'sdate('now')), which can lead to unexpected behavior around timezone boundaries.
To make this deterministic and fully respect the now parameter, we should format now as a UTC date string (YYYY-MM-DD) and pass it as a parameter to the SQL query.
shouldRun(model: string, now: Date = new Date()): boolean {
if (now.getHours() < this.targetHour()) return false;
const dateStr = now.toISOString().slice(0, 10);
const row = this.db.prepare(
"SELECT COUNT(*) AS n FROM openmythos_eval_runs WHERE agent_id = ? AND status IN ('running', 'completed') AND substr(started_at, 1, 10) = ?"
).get("nightly:" + model, dateStr) as { n: number };
return row.n === 0;
}| async tick(now: Date = new Date()): Promise<string[]> { | ||
| const ran: string[] = []; | ||
| const caseIds = this.anchorCaseIds(); | ||
| for (const model of this.models()) { | ||
| if (!this.shouldRun(model, now)) continue; | ||
| try { | ||
| const result = await this.evalService.runEval(`nightly:${model}`, undefined, model, caseIds); | ||
| console.log(`OpenMythos nightly: ${model} → ${result.overallScore.toFixed(3)} (${result.completedCases}/${result.totalCases} cases)`); | ||
| ran.push(model); | ||
| } catch (error) { | ||
| console.error(`OpenMythos nightly: ${model} failed —`, error instanceof Error ? error.message : error); | ||
| } | ||
| } | ||
| return ran; | ||
| } |
There was a problem hiding this comment.
If a scheduler tick takes longer than an hour (e.g., if multiple models are run sequentially or Ollama is slow/hanging), the next hourly interval will fire and start a concurrent tick. Although shouldRun checks for 'running' status, models that haven't started yet in the first tick could be picked up by the second tick, leading to concurrent model execution and saturating the Ollama host.
Additionally, if anchorCaseIds() throws an error (e.g., due to a missing or malformed anchors file), the promise returned by tick() will reject. Since the interval timer does not catch this rejection, it will result in an unhandled promise rejection, which can crash the server.
We should introduce a concurrency guard (private isRunning = false; at the class level) and wrap the entire tick() execution in a try-catch-finally block to handle setup errors and ensure the guard is always reset.
Please add private isRunning = false; to the class properties, and update tick() as follows:
async tick(now: Date = new Date()): Promise<string[]> {
if (this.isRunning) {
console.warn("OpenMythos nightly: previous tick is still running, skipping this tick");
return [];
}
this.isRunning = true;
const ran: string[] = [];
try {
const caseIds = this.anchorCaseIds();
for (const model of this.models()) {
if (!this.shouldRun(model, now)) continue;
try {
const result = await this.evalService.runEval("nightly:" + model, undefined, model, caseIds);
console.log("OpenMythos nightly: " + model + " → " + result.overallScore.toFixed(3) + " (" + result.completedCases + "/" + result.totalCases + " cases)");
ran.push(model);
} catch (error) {
console.error("OpenMythos nightly: " + model + " failed —", error instanceof Error ? error.message : error);
}
}
} catch (error) {
console.error("OpenMythos nightly scheduler tick failed:", error instanceof Error ? error.message : error);
} finally {
this.isRunning = false;
}
return ran;
}![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6e8fbb6937
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| console.warn('OpenMythos nightly: enabled but OPENMYTHOS_NIGHTLY_MODELS is empty — not arming'); | ||
| return false; | ||
| } | ||
| this.timer = setInterval(() => void this.tick(), HOUR_MS); |
There was a problem hiding this comment.
When a nightly pass runs longer than an hour, this interval can enter tick() again while the prior pass is still awaiting runEval. The per-model SQL check only marks the model currently being evaluated as running, so the second pass can start a later due model concurrently (for example, model-a still running while model-b starts), defeating the sequential model execution guarantee and potentially doubling the Ollama worker-pool load. Please add a service-level in-flight guard or schedule the next pass only after the previous one settles.
Useful? React with 👍 / 👎.
| const row = this.db.prepare(` | ||
| SELECT COUNT(*) AS n FROM openmythos_eval_runs | ||
| WHERE agent_id = ? AND status IN ('running', 'completed') | ||
| AND substr(started_at, 1, 10) = date('now') |
There was a problem hiding this comment.
When the server timezone is not UTC, this mixes the local-hour gate (now.getHours()) with SQLite's UTC date('now'). After UTC midnight but before the next local target hour, the UTC date has advanced with no row while the local hour may still be past yesterday's target, so the hourly tick starts the next day's run immediately instead of at OPENMYTHOS_NIGHTLY_HOUR (for example, afternoon/evening on US timezones). Use the same clock/timezone for both the due-hour check and the dedupe day key.
Useful? React with 👍 / 👎.
1 participant
Summary
Completes the P1 governance-scorecard arc (#60 ships the read path; this ships the data supply): a scheduler that runs OpenMythos governance evals on a nightly cadence so the leaderboard fills with real production data.
OPENMYTHOS_NIGHTLY_ENABLED=true+OPENMYTHOS_NIGHTLY_MODELS=<comma list>; hour configurable viaOPENMYTHOS_NIGHTLY_HOUR(default 3, server-local).nightly:<model>, deduped againstopenmythos_eval_runsso restarts don't double-run; boot does a catch-up tick.OPENMYTHOS_ORACLE_ANCHORS_PATHset, only the oracle-anchored subset runs — deterministic scoring, ~2–5 min per model. Models run sequentially to avoid saturating the Ollama host; one model's failure doesn't stop the rest.Validation
Pairs with #60; no conflicts (touches only new files + a 5-line startup hook in index.ts).
🤖 Generated with Claude Code