fix(e2e): reconcile uncertain workflow dispatches

.agents/skills/nemoclaw-maintainer-day/MERGE-GATE.md

@@ -102,6 +102,9 @@ The retry must apply to the PR SHA and base SHA.
 - `child-cancelled` — Rerun CI when the child workflow was cancelled.
   You can also rerun it when every listed non-passing job was cancelled.
 - `evidence-download` — Rerun CI when a successful child's evidence download failed, was cancelled, or was skipped.
+- `dispatch-not-observed` — Rerun CI only after the controller completed its bounded zero-match window and recorded a validated dispatch receipt on a trusted GitHub Actions check.
+  Never resubmit the child workflow manually.
+  The controller must recheck the old correlation before it creates a replacement check and fresh correlation.
 
 The controller keeps each completed coordination check as audit history.
 For a retry, it creates an `in_progress` check for the same PR SHA and base SHA.
@@ -113,10 +116,14 @@ Do not retry these terminal failures on the same SHA:
 - Selected-job or typed-target product failures.
 - Assertion failures.
 - Evidence policy or integrity failures.
-- Reconciliation or controller errors.
+- Ambiguous, incomplete, or identity-invalid reconciliation.
+- Controller errors.
 - Unknown states.
 - Failures recorded before retry reasons existed.
 
+A validated `dispatch-not-observed` receipt on a trusted GitHub Actions check is the only retryable reconciliation result.
+If a late child, incomplete inventory, or contradiction appears while the old correlation is rechecked, stop and investigate rather than dispatching again.
+
 Push a change to create another SHA, and then run CI again.
 A passing controller does not override a failing required job.
 
@@ -155,6 +162,8 @@ The controller reads the approval history and requires one approval for that env
 The environment's required reviewers are the authorization allowlist.
 The controller also checks the PR number, head repository, PR SHA, base SHA, plan, pending coordination check, compatible `main`, and PR state.
 Immediately before dispatch, it confirms that the PR is open and that the PR SHA, base SHA, and coordination identity still match.
+It also performs a bounded direct check read and requires the coordination check to remain in the exact expected phase, regardless of a matching, stale, or missing list result.
+If the run-specific authorization response is lost, the controller accepts only an exact persisted child binding and otherwise attempts to revoke coordination before requesting child cancellation.
 
 Approval returns coordination to `Running <count> E2E check(s)`.
 The gate passes only after the selected jobs and targets return verified passing evidence.

test/e2e/README.md

@@ -572,11 +572,44 @@ projects each controller-selected target into a fixed target ID and hosted
 runner mapping. The generated live matrix must exactly match those trusted IDs
 and runners, and only the trusted projection can configure credential-bearing
 typed-target jobs. Ordinary branch dispatch is not an acceptable substitute.
-The controller uses GitHub's returned run ID for
-waiting, evidence download, and completion, then revalidates that the PR is
-still open with the PR SHA, base SHA, and coordination identity before
-recording a final result. The native observer revalidates the live revision
-before mirroring that terminal result.
+Immediately before the dispatch request, the controller also requires the
+coordination check to remain in the exact phase expected by that path.
+Regardless of the list result, a bounded direct check read must confirm the
+same identity and exact phase; pending, reserved, stale, or mismatched
+authorization state fails closed.
+The normal dispatch path uses GitHub's returned run ID for waiting, evidence
+download, and completion. A transport failure, HTTP 5xx response, invalid JSON,
+or malformed success response instead starts a 45-second read-only
+reconciliation window. The dispatch request is capped at 10 seconds, each
+inventory read is capped at 5 seconds, and the bounded adoption checks plus
+authorization publication keep the full path within the child's authorization
+window. The controller never sends a second dispatch request in that attempt.
+It waits for the full settling window and adopts exactly one
+correlated child only after validating its title, workflow path and SHA, `main`
+branch, first attempt, repository, canonical URLs, creation time, and GitHub
+Actions actor identities. It then rereads that run and revalidates the exact PR
+head, base, head repository, and unchanged pre-dispatch coordination check
+before publishing the run-specific authorization. If the authorization update
+response is lost or malformed, a bounded direct read accepts only the exact
+persisted child binding. Otherwise, the controller attempts to revoke
+coordination before requesting cancellation of the candidate child. If
+revocation itself fails,
+the controller still attempts cancellation and reports both failures. HTTP 4xx
+responses remain definitive dispatch rejections.
+
+Zero correlated runs produce the retryable `Workflow dispatch was not
+observed` result and a versioned receipt. Multiple or malformed correlated
+runs, incomplete or failed inventory reads, and identity drift fail closed as
+terminal reconciliation errors. Dispatch-reconciliation logs contain only the
+correlation ID, failure kind, optional HTTP status and safe GitHub request ID,
+and the `reconciling`, `adopted`, or `not-observed` result. API response
+excerpts are normalized to one line and limited to 512 characters. Dispatch
+inputs and tokens are not logged.
+
+After a child is selected, the controller revalidates that the PR is still open
+with the PR SHA, base SHA, and coordination identity before recording a final
+result. The native observer revalidates the live revision before mirroring that
+terminal result.
 
 An internal revision whose control-plane matches include a file outside the
 trusted controller and observer boundaries leaves coordination in progress
@@ -618,18 +651,23 @@ maintainer or administrator chooses **Run workflow** on `main`, selects
 as `expected_head_sha`, current 40-character base SHA as `expected_base_sha`,
 and a specific 10–500-character `review_reason`. This path additionally
 revalidates the triggering actor's `maintain` or `admin` permission and uses the
-same exact-revision and deterministic-plan checks. If authorization
-fails before a child run is dispatched, the controller restores the
-authorization title and leaves coordination in progress. A maintainer can then
-launch a fresh first-attempt manual authorization. To use the protected path
-again, correct its environment configuration, update the PR to create a new
-head, and trigger fresh PR CI. After a child is dispatched, a startup failure
-requests cancellation. Whether or not cancellation is confirmed, the
-controller completes coordination as
-`Authorized E2E run requires reconciliation`; that authorization for the PR/base SHA pair
-cannot be retried because the child may still execute and a retry could start
-duplicate credential-bearing work. Inspect the linked run, then update the PR
-and run fresh CI before authorizing again.
+same exact-revision and deterministic-plan checks. An ordinary validation
+failure before dispatch restores the authorization title and leaves
+coordination in progress. A maintainer can then launch a fresh first-attempt
+manual authorization. To use the protected path again, correct its environment
+configuration, update the PR to create a new head, and trigger fresh PR CI.
+
+An uncertain dispatch that completes the bounded window with zero correlated
+runs instead completes coordination as `Workflow dispatch was not observed`
+with a validated receipt on the trusted GitHub Actions-owned coordination
+check. Ambiguous reconciliation or a startup failure after a possible child
+dispatch completes coordination as `Authorized E2E run requires
+reconciliation`. The controller requests cancellation for every fully validated
+candidate child and every schema-valid run ID that carries the correlation,
+including identity-invalid candidates. That authorization for the PR/base SHA
+pair cannot be reused because a child may still execute and another dispatch
+could start duplicate credential-bearing work. Inspect any linked or candidate
+run, then update the PR and run fresh CI before authorizing again.
 The native required job treats authorization and running titles as intermediate
 waiting states only while coordination remains in progress. It also keeps
 polling when the current PR/base SHA coordination check is a completed failure
@@ -640,21 +678,28 @@ later eligible `CI / Pull Request` run can create a fresh coordination check for
 the same unchanged open head and base only when the newest failed coordination
 check carries a current-version retry reason:
 `prerequisite-ci` after the later CI run succeeds, `child-cancelled` after a
-conclusively cancelled child, or `evidence-download` after a successful child
-whose evidence download failed, was cancelled, or was skipped. The trusted
-controller leaves the completed check as audit history, creates and validates a
-new `in_progress` check with the same PR/base SHA external identity, and rebuilds
-the deterministic plan before exposing a fresh authorization state. The
+conclusively cancelled child, `evidence-download` after a successful child
+whose evidence download failed, was cancelled, or was skipped, or
+`dispatch-not-observed` after an uncertain dispatch completes its bounded
+zero-match window and records a validated receipt. Before replacing a
+`dispatch-not-observed` check, the next controller rereads the old correlation
+and fails closed if a late child, incomplete inventory, or contradiction
+appears. The trusted controller leaves the completed check as audit history,
+creates and validates a new `in_progress` check with the same PR/base SHA
+external identity and a fresh correlation, and rebuilds the deterministic plan
+before exposing a fresh authorization state. The
 controller and native observer select the highest check-run ID only when every
 older duplicate is a completed failure with a recognized versioned retry
 marker. An unexpected app or mismatched mutation identity, duplicate ID, older
 unmarked or otherwise non-retryable terminal state, or multiple active
 candidates fails closed. Selected-job product or
 assertion failures, evidence policy or integrity failures, schema or identity
-mismatches, traversal or provenance failures, reconciliation, controller
-errors, unknown states, and failures recorded before retry reasons existed
-remain terminal for that PR/base SHA pair. Update the PR and run fresh CI for
-terminal outcomes. The
+mismatches, traversal or provenance failures, ambiguous, incomplete, or
+identity-invalid reconciliation, controller errors, unknown states, and
+failures recorded before retry reasons existed remain terminal for that
+PR/base SHA pair. A validated `dispatch-not-observed` receipt on a trusted
+GitHub Actions check is the only retryable reconciliation result. Update the PR
+and run fresh CI for terminal outcomes. The
 normal wait, evidence download, and finish path is the only path that can record
 success; the authorization itself cannot make the gate green. A changed head or
 base requires a new authorization.

test/github-api.test.ts

@@ -0,0 +1,123 @@
+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+// SPDX-License-Identifier: Apache-2.0
+
+import { afterEach, describe, expect, it, vi } from "vitest";
+
+import { GitHubApiError, githubApi, githubApiWithResponse } from "../tools/advisors/github.mts";
+
+afterEach(() => {
+  vi.unstubAllGlobals();
+});
+
+describe("GitHub API response identity", () => {
+  it("returns successful response status and a safe GitHub request ID when requested", async () => {
+    vi.stubGlobal(
+      "fetch",
+      vi.fn().mockResolvedValue(
+        new Response('{"workflow_run_id":23}', {
+          status: 200,
+          headers: { "x-github-request-id": "SUCCESS:1234" },
+        }),
+      ),
+    );
+
+    await expect(
+      githubApiWithResponse<{ workflow_run_id: number }>(
+        "repos/NVIDIA/NemoClaw/actions/workflows/e2e.yaml/dispatches",
+        "token",
+      ),
+    ).resolves.toEqual({
+      data: { workflow_run_id: 23 },
+      status: 200,
+      requestId: "SUCCESS:1234",
+    });
+  });
+
+  it("preserves structured HTTP status and a safe GitHub request ID", async () => {
+    vi.stubGlobal(
+      "fetch",
+      vi.fn().mockResolvedValue(
+        new Response("dispatch unavailable", {
+          status: 500,
+          headers: { "x-github-request-id": "ABCD:1234:EFGH" },
+        }),
+      ),
+    );
+
+    const request = githubApi(
+      "repos/NVIDIA/NemoClaw/actions/workflows/e2e.yaml/dispatches",
+      "token",
+      {
+        method: "POST",
+      },
+    );
+
+    await expect(request).rejects.toMatchObject({
+      name: "GitHubApiError",
+      kind: "http",
+      method: "POST",
+      apiPath: "repos/NVIDIA/NemoClaw/actions/workflows/e2e.yaml/dispatches",
+      status: 500,
+      requestId: "ABCD:1234:EFGH",
+      responseExcerpt: "dispatch unavailable",
+    });
+    await expect(request).rejects.toThrow(
+      "GitHub API repos/NVIDIA/NemoClaw/actions/workflows/e2e.yaml/dispatches failed: 500 dispatch unavailable",
+    );
+  });
+
+  it("rejects unsafe request IDs and bounds single-line response diagnostics", async () => {
+    vi.stubGlobal(
+      "fetch",
+      vi.fn().mockResolvedValue(
+        new Response(`first line\n${"x".repeat(600)}`, {
+          status: 503,
+          headers: { "x-github-request-id": "unsafe request id" },
+        }),
+      ),
+    );
+
+    const error = await githubApi("repos/NVIDIA/NemoClaw/check-runs", "token").catch(
+      (caught: unknown) => caught,
+    );
+
+    expect(error).toBeInstanceOf(GitHubApiError);
+    expect(error).toMatchObject({
+      kind: "http",
+      status: 503,
+      requestId: undefined,
+    });
+    expect((error as GitHubApiError).responseExcerpt).toHaveLength(512);
+    expect((error as GitHubApiError).responseExcerpt).not.toMatch(/[\r\n\t]/u);
+  });
+
+  it("retains response identity when a successful response contains invalid JSON", async () => {
+    vi.stubGlobal(
+      "fetch",
+      vi.fn().mockResolvedValue(
+        new Response("{", {
+          status: 200,
+          headers: { "x-github-request-id": "DECODE:1234" },
+        }),
+      ),
+    );
+
+    await expect(githubApi("repos/NVIDIA/NemoClaw/pulls/7593", "token")).rejects.toMatchObject({
+      name: "GitHubApiError",
+      kind: "decode",
+      method: "GET",
+      status: 200,
+      requestId: "DECODE:1234",
+      responseExcerpt: "{",
+    });
+  });
+
+  it("preserves transport failures without inventing an HTTP response", async () => {
+    const transportError = new TypeError("fetch failed");
+    vi.stubGlobal("fetch", vi.fn().mockRejectedValue(transportError));
+
+    await expect(githubApi("repos/NVIDIA/NemoClaw/pulls/7593", "token")).rejects.toBe(
+      transportError,
+    );
+  });
+});