Skip to content

Extract a shared, deployment-agnostic calibration/signal-tracking module for ORB + AMS #7982

Description

@JSONbored

Implements #7980.

Problem

Every piece of existing self-correction infrastructure (pr_outcome ground truth, reversal detection, precision circuit-breakers in auto-tune.ts, contributor_gate_history, predicted_gate_calls) is wired specifically into ORB's PR-review pipeline and its own DB tables/env shape. AMS makes its own class of deterministic, potentially-wrong calls (policy/eligibility checks — e.g. the label/CONTRIBUTING.md heuristics documented in packages/loopover-miner/docs/ams-contribution-signal-inventory.md) with no equivalent infrastructure at all, and no shared home for it. Building the new pieces (sub-issues 3-6 of #7980) directly into ORB's src/review/src/queue would repeat this mistake for a second subsystem.

Area

Architecture — new home for shared logic, most likely packages/loopover-engine (already the pattern for cross-consumer pure logic ported for reuse, per auto-tune.ts's own "self-contained native port... the host wires those at cutover" convention) or a new package if loopover-engine's existing scope doesn't fit.

Proposal

  • Define a minimal, storage-agnostic interface for: (a) recording a "rule fired" event (rule id, outcome, target), (b) recording a "human override" event, (c) computing a repeat-count / precision-over-time report from those events. auto-tune.ts's FlagStore-injection pattern is the precedent that proves this can be done without dragging in DB/env specifics.
  • Two thin adapters: one for ORB (wired to review_audit/audit_events, both cloud D1 and self-hosted Postgres/SQLite), one for AMS (wired to whatever AMS's own event/signal store is — audit that as part of this issue).
  • Explicitly NOT a rewrite of the existing ORB pr_outcome/reversal system — this wraps/reuses it where it already exists (mirrors auto-tune.ts's own "pure logic ported, host wires the storage" shape) and gives AMS the same shape of primitive.

Deliverables

  • The shared package/module with a documented interface + tests, no behavior change yet for either consumer (sub-issues 3-6 of Epic: false-positive detection & self-correction for deterministic ORB/AMS gates #7980 are what actually use it).
  • An ORB adapter and an AMS adapter, each with their own tests.
  • A short design doc noting what AMS-specific event AMS should start recording to plug in (exact event shape depends on auditing AMS's current signal-recording, which is part of this issue's own scope).

Resources

  • src/review/auto-tune.ts (the FlagStore-injection precedent to mirror)
  • src/review/outcomes-wire.ts, src/review/parity.ts (existing ORB-side pieces to wrap, not replace)
  • src/ams/, packages/loopover-miner/ (AMS side to audit)
  • packages/loopover-engine/ (likely new home)

Boundaries

No new user-facing behavior in this issue — foundation only. Sub-issues 3-6 of #7980 depend on this landing first (or can stub the interface and swap in the real module once ready — maintainer's call at implementation time).

maintainer-only — cross-cutting architecture, not a build task for contributors.

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions