Skip to content

calibration: persist counterfactual-replay results to audit_events so prompt drift is a trend, not a per-PR comment #8764

Description

@JSONbored

Context

Parent: #8757. The calibration-advisory CI check posts counterfactual-replay (judge-prompt base-vs-head) results as PR comments only — nothing durable, so there is no prompt-quality trend over time and nothing a future dashboard can aggregate. persistThresholdBacktestRuns (src/services/threshold-backtest-run.ts) is the established pattern: structured result -> recordAuditEvent with a dedicated event type.

Requirements

  • Persist each replay run's structured comparison (not rendered Markdown) under a new sibling event type, keyed by repo+PR, from the CI script's existing D1-write path (mirror how the corpus-export script already reads/writes).
  • Additive; the PR comment stays.
  • Fail-open: persistence failure must never fail the advisory CI job (memory of the advisory-jobs-fail-open rule).

Deliverables

  • The persistence write + event type.
  • Tests for the persisted shape + the fail-open path.

Test Coverage Requirements

99%+ patch coverage, branch-counted on the persistence logic (CI-script thin-IO exemption applies per repo precedent).

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions