feat(orb): gate determinism — deterministic test-evidence facts, decision confidence, and replayable time (#8833, #8834, #9028) (#9256)

* fix(review): demote whole-PR test-absence blockers the path classifier contradicts (#8833)

Whether a PR carries test-path evidence is a deterministic fact — signals/test-evidence.ts's
isTestPath owns it, and slop.ts's missing_test_evidence finding already decides its severity.
The reviewer prompt nonetheless fed the same fact as free text and let the model re-decide
blocker-vs-nit, so a model could close a PR on a claim the engine had already answered the
other way.

Extends the existing parse-time fact-authority pattern (demoteCiClaimBlockers #8845,
demoteEvidenceAbsenceBlockers #8961) to this criterion: a whole-PR "no tests were added"
blocker is demoted to a nit when the classifier confirms the PR does change test paths.
Fires only in the arm where the claim is provably a fact error — a genuine test-absence
blocker on a genuinely test-free PR is untouched — and demotes rather than drops, so the
observation still reaches the human.

Coverage-DEPTH claims that narrow to a specific target ("no tests for the nullish branch")
are deliberately excluded: the classifier cannot check those, so they remain the model's
judgment to make and keep blocking.

* feat(orb): capture the decision-time wall clock so staleness rules are replayable (#9028)

gate.requireFreshRebaseWindow compared the base branch's tip against an inline Date.now()
read inside maybeForceFreshRebase. Time is a decision INPUT, and nothing recorded which
instant the comparison used — so re-deriving such a decision later could silently reach the
opposite answer purely because the wall clock had moved, and report it as a match.

The decision pass now takes ONE Date.now() reading, records it into
decision_replay_inputs.replay_json as `clock`, and passes it to every clock-dependent rule
instead of each calling the clock itself. Both staleness rules move to a pure, clock-injected
module: isWithinFreshRebaseWindow takes the instant explicitly, and isBaseStaleByAheadBy is
stated as the commit-count comparison it is — provably instant-independent, not merely
assumed so.

replayDecision gains a stage-0 `clock` check: replaying at the recorded instant (the CLI
default) is bit-exact, while naming a different instant reports a `clock` divergence rather
than silently certifying a re-derivation that never reproduced the original evaluation.
Records written before this change carry no instant, so the stage is skipped rather than
guessed. The CLI exposes it as `--at <epoch ms>`.

Also corrects the decision_replay_inputs migration reference in two doc comments (0181 is
alert_dedup_claims; the table is created in 0182).

* feat(orb): record a per-decision confidence signal from inter-run agreement (#8834)

Verbalized confidence alone is poorly calibrated — #8845 already had to stop reading an
absent confidence as certainty. Sampling-based consistency is the better-behaved signal, and
the risk-control literature this epic builds on finds two samples capture most of the benefit.

Scores inter-run agreement across the reviewer stances the engine ALREADY produces (#8229's
reviewerVotes) and folds it into the verbalized confidence, at zero additional AI spend. The
combined score multiplies the two so it is monotonically below either input — a judgment is
only as trustworthy as both how sure the judge said it was and how reproducibly the judges
reached it, which is the property an abstention threshold depends on.

A lone run is recorded as UNCORROBORATED at a 0.5 agreement floor rather than fabricated
unanimity, so a single-reviewer or budget-degraded review records a strictly lower confidence
than a genuinely corroborated one. Zero samples refuses to invent a score at all.

The signal rides to DecisionRecord.aiAgreement (schema v5) so every decision joins the
risk-control calibration set with its reproducibility attached. Deliberately ADDITIVE: it
does not re-route the gate. Disagreement already routes to a hold today — differing stances
ARE the ai_review_split finding, which blocks or holds via the existing confidence floor — so
a second parallel route would double-count the same evidence instead of measuring it.

* test(orb): update decision-record schema assertions for the v5 agreement field (#8834)

The DECISION_RECORD_SCHEMA_VERSION bump to v5 moves the pinned version assertions in the
backfill bundle and queue decision-record suites. The #9124 record test additionally carries
the agreement on its cached finding and asserts it threads through to the record, mirroring
exactly how that test already proves modelIds/promptDigest pass through rather than being
re-derived.

---------

Co-authored-by: JSONbored <airdroptopian@gmail.com>

Author: JSONbored

scripts/replay-decision.ts

@@ -4,6 +4,12 @@
 //
 //   node --experimental-strip-types scripts/replay-decision.ts <bundle.json>
 //   ... | node --experimental-strip-types scripts/replay-decision.ts -
+//   node --experimental-strip-types scripts/replay-decision.ts <bundle.json> --at <epoch ms>
+//
+// #9028: `--at` names the instant to replay AT. Omit it to replay at the instant the decision itself recorded
+// (`replayInput.clock.nowMs`) — the bit-exact case. Passing a DIFFERENT instant exits 1 with a `clock`
+// divergence rather than reporting a match: time is a decision INPUT, and a clock-dependent rule
+// (`gate.requireFreshRebaseWindow`) can flip purely because the wall clock moved.
 //
 // The bundle is one JSON object: { record: {...decision_records row}, replayInput: {...replay_json} }.
 // EXTRACT (operator, against the instance DB):
@@ -24,8 +30,13 @@
 import { readFileSync } from "node:fs";
 import { replayDecision, type DecisionReplayInput, type ReplayableRecord } from "../src/review/decision-replay";
 
-/** Parse + normalize a bundle (snake_case SQL rows accepted) and replay it. Exported for tests. */
-export function runReplayBundle(raw: string): { outcome: ReturnType<typeof replayDecision> | null; error?: string } {
+/** Parse + normalize a bundle (snake_case SQL rows accepted) and replay it. Exported for tests.
+ *
+ *  #9028: `atMs` names the instant to replay AT. Omitted (the default) replays at the instant the decision
+ *  recorded, which is the bit-exact case. Supplying a DIFFERENT instant is reported as a `clock` divergence,
+ *  never silently accepted — a clock-dependent rule can legitimately flip its answer as the wall clock moves,
+ *  so "it still matches at a different instant" is not a re-derivation of the original decision. */
+export function runReplayBundle(raw: string, atMs?: number): { outcome: ReturnType<typeof replayDecision> | null; error?: string } {
   let bundle: { record?: Record<string, unknown>; replayInput?: unknown };
   try {
     bundle = JSON.parse(raw) as never;
@@ -47,18 +58,25 @@ export function runReplayBundle(raw: string): { outcome: ReturnType<typeof repla
   if (!record || !replayInput || !Array.isArray(replayInput.findings) || typeof replayInput.evaluated !== "object") {
     return { outcome: null, error: "bundle must carry {record: {id, reason_code|reasonCode, action}, replayInput: {findings, policy, evaluated}}" };
   }
-  return { outcome: replayDecision(record, replayInput) };
+  return { outcome: replayDecision(record, replayInput, atMs === undefined ? {} : { nowMs: atMs }) };
 }
 
 const invokedDirectly = process.argv[1]?.endsWith("replay-decision.ts") === true;
 if (invokedDirectly) {
-  const source = process.argv[2];
+  const argv = process.argv.slice(2);
+  const atIndex = argv.indexOf("--at");
+  const atRaw = atIndex === -1 ? undefined : argv[atIndex + 1];
+  if (atIndex !== -1 && (atRaw === undefined || !Number.isFinite(Number(atRaw)))) {
+    console.error("replay-decision: --at requires a Unix-epoch-milliseconds value");
+    process.exit(2);
+  }
+  const source = argv.filter((arg, index) => index !== atIndex && index !== atIndex + 1)[0];
   if (!source) {
-    console.error("usage: replay-decision.ts <bundle.json | ->");
+    console.error("usage: replay-decision.ts <bundle.json | -> [--at <epoch ms>]");
     process.exit(2);
   }
   const raw = source === "-" ? readFileSync(0, "utf8") : readFileSync(source, "utf8");
-  const { outcome, error } = runReplayBundle(raw);
+  const { outcome, error } = runReplayBundle(raw, atRaw === undefined ? undefined : Number(atRaw));
   if (!outcome) {
     console.error(`replay-decision: ${error}`);
     process.exit(2);

src/queue/ai-review-orchestration.ts

@@ -23,6 +23,7 @@ import { buildPullRequestAdvisory } from "../rules/advisory";
 import { recordAuditEvent, getDecryptedRepositoryAiKey, getRepository, listCheckSummaries, listPullRequestFiles } from "../db/repositories";
 import { registerHeldLock, unregisterHeldLock } from "./held-lock-registry";
 import { recordRoutingShadow } from "../services/reviewer-routing";
+import { scoreJudgmentAgreement } from "../review/judgment-agreement";
 import { createInstallationToken } from "../github/app";
 import type { AgentActionMode } from "../settings/agent-execution";
 import { buildAiReviewDiff } from "../review/review-diff";
@@ -889,6 +890,10 @@ export async function runAiReviewForAdvisory(
     // the REAL reviewer identities and the REAL system prompt into DecisionRecord instead of hardcoding null.
     const aiJudgmentModelIds = parsedReviewModelIds(result.reviewDiagnostics ?? []);
     const aiJudgmentPromptDigest = result.systemPromptDigest;
+    // #8834: inter-run agreement over the stances this review ALREADY produced (#8229's reviewerVotes) —
+    // zero additional AI spend. Computed once and attached to whichever AI-judgment finding is built below,
+    // so the decision record carries a per-decision confidence signal for the calibration set (#8835).
+    const aiJudgmentAgreement = (verbalizedConfidence: number) => scoreJudgmentAgreement(result.reviewerVotes, verbalizedConfidence);
     if (result.consensusDefect) {
       findings.push({
         code: "ai_consensus_defect",
@@ -913,6 +918,7 @@ export async function runAiReviewForAdvisory(
         confidence: result.consensusDefect.confidence,
         modelIds: aiJudgmentModelIds,
         promptDigest: aiJudgmentPromptDigest,
+        agreement: aiJudgmentAgreement(result.consensusDefect.confidence),
       });
     } else if (result.split) {
       // The reviewers DISAGREED — exactly one flagged a blocking defect. reviewbot's quorum treats any reviewer
@@ -940,6 +946,10 @@ export async function runAiReviewForAdvisory(
           : {}),
         modelIds: aiJudgmentModelIds,
         promptDigest: aiJudgmentPromptDigest,
+        // A split IS the disagreement case: the stances differ, so agreement scores strictly below unanimity
+        // and the recorded confidence falls with it. #8834's "disagreement routes to hold" is already this
+        // finding's existing behavior via the confidence floor; this measures it rather than re-routing it.
+        agreement: aiJudgmentAgreement(result.splitConfidence ?? 1),
       });
     } else if (result.inconclusive) {
       // Fail-CLOSED (#ai-fail-closed): block-mode AI could not return a usable verdict. Hold the PR for a human

src/queue/processors.ts

@@ -663,6 +663,7 @@ import {
 import { AI_JUDGMENT_BLOCKER_CODES } from "../rules/advisory";
 import { computeSalvageabilityForTarget } from "../review/salvageability-wire";
 import { deriveDecisionReasonCode, persistDecisionReplayInputForGate } from "../review/decision-replay";
+import { isWithinFreshRebaseWindow, type DecisionClockCapture } from "../review/staleness-clock";
 import { recordVerdictFlip } from "../review/verdict-flip-store";
 import { resolveAutomaticCloseConfidence } from "../review/risk-control-wire";
 import { maybeApplyCloseAuditHoldout } from "../review/close-audit-holdout";
@@ -3175,6 +3176,12 @@ async function runAgentMaintenancePlanAndExecute(
   },
 ): Promise<void> {
   const { installationId, repoFullName, pr, settings, otherOpenPullRequests, deliveryId, gate } = args;
+  // #9028: ONE wall-clock read for this whole decision pass, recorded into the replay input below and passed
+  // to every clock-dependent gate rule (today `gate.requireFreshRebaseWindow` via maybeForceFreshRebase)
+  // instead of each of them calling Date.now() independently. Two rules reading the clock at two different
+  // moments cannot both be replayed from one recorded instant — and an unrecorded instant is an unrecorded
+  // decision INPUT, which is exactly what makes a time-dependent decision unreplayable.
+  const decisionClock: DecisionClockCapture = { nowMs: Date.now() };
 
   // Convergence safety: feed the planner the PR's changed paths + the repo's hard-guardrail globs so guarded
   // paths force manual review, and flag owner-authored PRs so they are never auto-closed (standing rule).
@@ -3756,18 +3763,22 @@ async function runAgentMaintenancePlanAndExecute(
       modelIds: aiJudgment?.modelIds ?? null,
       promptDigest: aiJudgment?.promptDigest ?? null,
       aiConfidence: aiJudgment?.confidence ?? null,
+      // #8834: the inter-run agreement signal computed at review time (ai-review-orchestration.ts) and
+      // carried on the finding, so the record captures HOW REPRODUCIBLY the judgment was reached, not just
+      // what the model claimed. null for a rule-only decision, exactly like the fields above.
+      aiAgreement: aiJudgment?.agreement ?? null,
       salvageability,
       // #9135: legible on the record's own face — see maybeApplyCloseAuditHoldout's doc comment.
       divertedByHoldout: closeAuditHoldout?.diverted ?? false,
     });
     const recordId = await persistDecisionRecord(env, record, recordDigest);
-    // #8838: persist the evaluation's own exact inputs beside the record (PRIVATE sibling, migration 0181)
+    // #8838: persist the evaluation's own exact inputs beside the record (PRIVATE sibling, migration 0182)
     // so the replay harness can re-derive this decision bit-exactly. Best-effort, like the record itself;
     // the no-replay no-op (synthetic content-lane/bridge evaluations) lives inside the helper. Keyed to the
     // id persistDecisionRecord actually wrote (#9123: a supersession at the same head gets a revisioned id,
     // not the base one this used to always recompute independently). #9135: the holdout outcome rides along
     // so `holdout_consistency` has something to check the public record against.
-    if (recordId !== null) await persistDecisionReplayInputForGate(env, recordId, gate, policyCloseKind ?? null, closeAuditHoldout ?? null);
+    if (recordId !== null) await persistDecisionReplayInputForGate(env, recordId, gate, policyCloseKind ?? null, closeAuditHoldout ?? null, decisionClock);
   }
   // #2349 (PR 1): additive per-contributor calibration data, gated identically to recordNativeGateDecision
   // above -- see src/review/contributor-calibration.ts's doc comment. Currently write-only; nothing reads
@@ -3897,6 +3908,7 @@ async function runAgentMaintenancePlanAndExecute(
       token,
       admissionKey,
       deliveryId,
+      nowMs: decisionClock.nowMs,
     }))
   ) {
     return;
@@ -4806,17 +4818,20 @@ async function maybeForceFreshRebase(
     token: string | undefined;
     admissionKey: GitHubRateLimitAdmissionKey | undefined;
     deliveryId: string;
+    // #9028: the decision pass's single captured instant — this rule reads it instead of the clock, so the
+    // window comparison is replayable from `decision_replay_inputs.replay_json`.
+    nowMs: number;
   },
 ): Promise<boolean> {
-  const { installationId, repoFullName, pr, settings, windowMinutes, baseRef, token, admissionKey, deliveryId } = args;
+  const { installationId, repoFullName, pr, settings, windowMinutes, baseRef, token, admissionKey, deliveryId, nowMs } = args;
   /* v8 ignore next -- structurally unreachable: the caller only invokes this after confirming
    * (liveMergeState ?? pr.mergeableState) === "clean", which GitHub can never compute for a PR with no
    * head commit; the null check is belt-and-suspenders against the field's optional TS type. */
   if (!pr.headSha) return false;
   const advancedAt = await fetchLiveBaseBranchAdvancedAt(env, repoFullName, baseRef, token, admissionKey);
   if (!advancedAt) return false; // fail-open: unreadable base commit -> no forced rebase
   const advancedAtMs = Date.parse(advancedAt);
-  if (!Number.isFinite(advancedAtMs) || Date.now() - advancedAtMs >= windowMinutes * 60_000) return false;
+  if (!isWithinFreshRebaseWindow({ baseAdvancedAtMs: advancedAtMs, windowMinutes, nowMs })) return false;
 
   const countKey = freshRebaseForceCountKey(repoFullName, pr.number);
   const storedCount = Number(await getTransientKey(env, countKey));

src/review/decision-record.ts

@@ -39,7 +39,7 @@
 import { errorMessage, nowIso } from "../utils/json";
 
 /** Bump when the record's FIELD SET changes meaning — consumers compare records only within a version. */
-export const DECISION_RECORD_SCHEMA_VERSION = "4"; // v4 (#9124/#9135): configDigest digests the resolved policy (+ settingsDigest split out), promptDigest digests the actual sent prompt, modelId -> modelIds (real identities), ciState populated, + divertedByHoldout; v3 (#8962): + salvageability {score, factors}; v2 (#8834): + aiConfidence, model/prompt commitments
+export const DECISION_RECORD_SCHEMA_VERSION = "5"; // v5 (#8834): + aiAgreement (inter-run agreement folded with the verbalized confidence); v4 (#9124/#9135): configDigest digests the resolved policy (+ settingsDigest split out), promptDigest digests the actual sent prompt, modelId -> modelIds (real identities), ciState populated, + divertedByHoldout; v3 (#8962): + salvageability {score, factors}; v2 (#8834): + aiConfidence, model/prompt commitments
 
 /**
  * Canonical JSON: recursively key-sorted, no insignificant whitespace — the ONE serialization every digest
@@ -118,6 +118,12 @@ export type DecisionRecord = {
    *  defect / split), null when no AI judgment contributed. Persisted so every decision joins the
    *  risk-control calibration set (#8835) with its confidence attached. */
   aiConfidence: number | null;
+  /** #8834: the per-decision confidence signal — inter-run agreement across the reviewer stances that
+   *  produced the AI judgment, folded together with that judgment's verbalized confidence (see
+   *  src/review/judgment-agreement.ts). `aiConfidence` above records what the model SAID; this records how
+   *  reproducibly the reviewers reached it, which is the input a calibrated abstention threshold (#8835)
+   *  needs. null when no AI judgment contributed, and for every record predating v5. */
+  aiAgreement: { agreement: number; confidence: number; sampleCount: number; uncorroborated: boolean } | null;
   /** #8962: the deterministic salvageability score + its named factors when an AI judgment shaped the
    *  decision — the second-axis evidence for auditing the close/hold boundary. null for rule-only decisions
    *  (and for reconstructed/backfilled records predating v3). */
@@ -135,12 +141,13 @@ export type DecisionRecord = {
 /** Assemble the record and its own content digest. PURE given pre-computed digests. Normalizes the
  *  optional-shaped caller fields (undefined -> null) HERE so call sites carry no fallback arms of their own. */
 export async function buildDecisionRecord(
-  input: Omit<DecisionRecord, "schemaVersion" | "decidedAt" | "gatePack" | "ciState" | "baseSha" | "aiConfidence" | "salvageability" | "settingsDigest" | "divertedByHoldout"> & {
+  input: Omit<DecisionRecord, "schemaVersion" | "decidedAt" | "gatePack" | "ciState" | "baseSha" | "aiConfidence" | "aiAgreement" | "salvageability" | "settingsDigest" | "divertedByHoldout"> & {
     decidedAt?: string;
     gatePack?: string | null | undefined;
     ciState?: string | null | undefined;
     baseSha?: string | null | undefined;
     aiConfidence?: number | null | undefined;
+    aiAgreement?: { agreement: number; confidence: number; sampleCount: number; uncorroborated: boolean } | null | undefined;
     salvageability?: { score: number; factors: string[] } | null | undefined;
     settingsDigest?: string | null | undefined;
     divertedByHoldout?: boolean | undefined;
@@ -154,6 +161,7 @@ export async function buildDecisionRecord(
     ciState: input.ciState ?? null,
     baseSha: input.baseSha ?? null,
     aiConfidence: input.aiConfidence ?? null,
+    aiAgreement: input.aiAgreement ?? null,
     salvageability: input.salvageability ?? null,
     settingsDigest: input.settingsDigest ?? null,
     divertedByHoldout: input.divertedByHoldout ?? false,