test(e2e): harden progress coverage boundaries

.github/workflows/candidate-compatibility.yaml

@@ -348,6 +348,13 @@ jobs:
             '
           fi
 
+      - id: artifact_safety
+        name: Validate final OpenShell gateway auth contract artifacts
+        if: ${{ always() }}
+        env:
+          E2E_ARTIFACT_DIR: ${{ github.workspace }}/candidate-source/e2e-artifacts/live/openshell-gateway-auth-contract
+        run: node --experimental-strip-types --no-warnings controller/tools/e2e/openshell-gateway-auth-artifact-safety.mts "$E2E_ARTIFACT_DIR"
+
       - name: Upload live evidence
         if: ${{ always() }}
         uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
@@ -356,7 +363,7 @@ jobs:
           path: |
             candidate-results/live-openshell-gateway-auth-contract.json
             candidate-live-observed.json
-            candidate-source/e2e-artifacts/live/openshell-gateway-auth-contract/
+            ${{ steps.artifact_safety.outcome == 'success' && steps.artifact_safety.outputs.approved_path || '' }}
           if-no-files-found: error
           retention-days: 30
 

.github/workflows/e2e.yaml

@@ -521,9 +521,17 @@ jobs:
           "$OPENSHELL_GATEWAY_BIN" --version
           npx tsx tools/e2e/live-vitest-invocation.mts run --test-path test/e2e/live/openshell-gateway-auth-source-contract.test.ts
 
-      - name: Upload OpenShell gateway auth contract artifacts
+      - id: artifact_safety
+        name: Validate final OpenShell gateway auth contract artifacts
         if: always()
+        run: node --experimental-strip-types --no-warnings tools/e2e/openshell-gateway-auth-artifact-safety.mts "$E2E_ARTIFACT_DIR"
+
+      - name: Upload OpenShell gateway auth contract artifacts
+        if: ${{ always() && steps.artifact_safety.outcome == 'success' && steps.artifact_safety.outputs.approved_path != '' }}
         uses: NVIDIA/NemoClaw/.github/actions/upload-e2e-artifacts@7768e15eb90d3ee2d33432f481dfe8747e4f6d57
+        with:
+          name: e2e-openshell-gateway-auth-contract
+          path: ${{ steps.artifact_safety.outputs.approved_path }}
 
       - name: Clean up Docker auth
         if: always()

.pre-commit-config.yaml

@@ -283,11 +283,11 @@ repos:
   - repo: local
     hooks:
       - id: e2e-semantic-phase-plans
-        name: Live E2E semantic phase plans
+        name: E2E semantic phase plans
         entry: npm run test:e2e-phases:check
         language: system
         pass_filenames: false
-        files: ^(test/e2e/live/.*\.ts|test/e2e/fixtures/(e2e-test|paths|progress)\.ts|tools/e2e/check-semantic-phases\.mts|vitest\.config\.ts|package\.json)$
+        files: ^(\.github/workflows/e2e\.yaml|test/(gateway-drift-preflight|vllm-docker-storage)\.test\.ts|test/e2e/live/.*\.ts|test/e2e/fixtures/.*\.ts|test/e2e/risk-signal-reporter\.ts|test/e2e/support/(e2e-semantic-phase-check|workflow-e2e-progress)\.test\.ts|tools/e2e/(check-semantic-phases|credential-free-tests|workflow-boundary|workflow-plan)\.mts|vitest\.config\.ts|package\.json)$
         priority: 20
 
       - id: test-cli

CONTRIBUTING.md

@@ -211,7 +211,7 @@ These are the primary npm scripts for day-to-day development:
 | `npm run test:watch` | Watch the CLI, plugin, and E2E-support projects and rerun affected tests |
 | `npm run test:shuffle` | Shuffle test order in the focused source projects without collecting coverage |
 | `npm run test:diagnose:leaks` | Report async-resource leaks and diagnose a Vitest process that hangs during shutdown |
-| `npm run test:e2e-phases:check` | Validate semantic phase plans for every live E2E test without executing live bodies |
+| `npm run test:e2e-phases:check` | Validate semantic phase plans for every live E2E test and workflow-selected credential-free integration test without executing test bodies |
 | `npm run test:runtime-audit -- <artifact-dir> [...]` | Rank captured live E2E runs by median, p95, variability, and slowest phase |
 | `npm run test:integration` | Clean-build the CLI and run root integration and installer tests |
 | `npm run test:package` | Clean-build CLI/plugin artifacts and run compiled-package contracts |
@@ -234,16 +234,29 @@ npx vitest run --project e2e-support
 This project is fast and does not run live targets. Live E2E remains opt-in through
 `npm run test:live-e2e` or the applicable GitHub Actions workflow.
 
-Every `e2e-live` test must declare its ordered, behavior-specific phase plan in
-`meta.e2ePhases`, call `progress.phase("literal phase label")` at those
-boundaries, and reach the final test-declared phase on every passing path. The
-harness then appends `release registered E2E resources` so cleanup duration and
-failures have their own phase. Run `npm run test:e2e-phases:check` after adding
-or changing a live E2E case; collection validates the plans without running
-infrastructure-mutating test bodies. See
+Every `e2e-live` test, plus every credential-free integration test selected by
+the shared E2E workflow planner, must declare its ordered, behavior-specific
+phase plan in `meta.e2ePhases`, call
+`progress.phase("literal phase label")` at those boundaries, and reach the final
+test-declared phase on every passing path. Live tests import the shared
+`e2e-test` fixture, which appends `release registered E2E resources` so cleanup
+duration and failures have their own phase. Workflow-selected integration tests
+import `workflow-e2e-test` and declare their own final release phase. Run
+`npm run test:e2e-phases:check` after changing either coverage set or its
+workflow selection; collection validates the union without running test bodies. See
 [`test/e2e/docs/README.md`](test/e2e/docs/README.md) for the logging and artifact
 contract.
 
+Use the shared `ShellProbe` for E2E child processes. The semantic-phase check
+also follows shared E2E helpers and rejects new direct asynchronous process
+boundaries unless they are explicitly audited for content-free activity and
+timestamp-only output reporting. Synchronous process calls must have a positive
+timeout shorter than the first heartbeat and use `killSignal: "SIGKILL"` so the
+child cannot ignore that bound; write child contents only through the redacted
+artifact sink. Pass the auto fixture's frozen, canonical `progress` capability
+through unchanged; custom, copied, or no-op progress adapters are rejected at
+audited subprocess boundaries.
+
 ### Test Declarative Behavior
 
 Do not read a shipped YAML, JSON, manifest, workflow, or E2E runtime file only to assert its keys,

test/candidate-compat.test.ts

@@ -95,15 +95,27 @@ describe("OpenShell candidate compatibility contract", () => {
             if?: string;
             name?: string;
             run?: string;
+            with?: Record<string, string>;
+            "working-directory"?: string;
           }>;
         }
       >;
       on: { workflow_dispatch: { inputs: Record<string, unknown> } };
       permissions: Record<string, string>;
     };
     const evidence = workflow.jobs.evidence;
+    const liveSteps = workflow.jobs.live?.steps ?? [];
     const finalize = evidence?.steps?.find((step) => step.name === "Finalize auditable evidence");
     const enforce = evidence?.steps?.find((step) => step.name === "Enforce aggregate result");
+    const uploadLiveEvidence = workflow.jobs.live?.steps?.find(
+      (step) => step.name === "Upload live evidence",
+    );
+    const artifactSafety = workflow.jobs.live?.steps?.find(
+      (step) => step.name === "Validate final OpenShell gateway auth contract artifacts",
+    );
+    const recordLiveResult = workflow.jobs.live?.steps?.find(
+      (step) => step.name === "Record receipt-bound live result",
+    );
     expect(Object.keys(workflow.on.workflow_dispatch.inputs).sort()).toEqual([
       "candidate",
       "component",
@@ -122,6 +134,28 @@ describe("OpenShell candidate compatibility contract", () => {
     expect(source).toContain("RESOLUTION_ID: ${{ needs.resolve.outputs.resolution_id }}");
     expect(source).toContain("verify-invocations");
     expect(source).toContain("openshell-gateway-auth-source-contract.test.ts");
+    expect(artifactSafety).toMatchObject({
+      env: {
+        E2E_ARTIFACT_DIR:
+          "${{ github.workspace }}/candidate-source/e2e-artifacts/live/openshell-gateway-auth-contract",
+      },
+      id: "artifact_safety",
+      if: "${{ always() }}",
+      run: 'node --experimental-strip-types --no-warnings controller/tools/e2e/openshell-gateway-auth-artifact-safety.mts "$E2E_ARTIFACT_DIR"',
+    });
+    expect(liveSteps.findIndex((step) => step.id === "live_test")).toBeLessThan(
+      liveSteps.indexOf(recordLiveResult!),
+    );
+    expect(liveSteps.indexOf(recordLiveResult!)).toBeLessThan(liveSteps.indexOf(artifactSafety!));
+    expect(liveSteps.indexOf(artifactSafety!)).toBeLessThan(liveSteps.indexOf(uploadLiveEvidence!));
+    expect(uploadLiveEvidence?.with?.path).toBe(
+      [
+        "candidate-results/live-openshell-gateway-auth-contract.json",
+        "candidate-live-observed.json",
+        "${{ steps.artifact_safety.outcome == 'success' && steps.artifact_safety.outputs.approved_path || '' }}",
+        "",
+      ].join("\n"),
+    );
     expect(evidence?.permissions).toEqual({ actions: "read", contents: "read" });
     expect(finalize?.id).toBe("finalize");
     expect(finalize?.env).toMatchObject({

test/compatible-endpoint-context-probe.test.ts

@@ -17,9 +17,20 @@ import {
   type FakeOpenAiCompatibleServer,
   startFakeOpenAiCompatibleServer,
 } from "./e2e/fixtures/fake-openai-compatible";
+import { startTestProgress, type TestProgress } from "./e2e/fixtures/progress.ts";
 import { testTimeout } from "./helpers/timeouts";
 
 const MODEL = "nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4";
+let progress: TestProgress | null = null;
+
+function testProgress(): TestProgress {
+  progress ??= startTestProgress(
+    "compatible endpoint support",
+    ["serve compatible endpoint", "verify compatible endpoint"],
+    { targetId: "compatible-endpoint-context-probe" },
+  );
+  return progress;
+}
 
 // The fake server binds to loopback (127.0.0.1). Loopback is an allowed
 // host-side probe target (a locally-run vLLM/Ollama endpoint), so these
@@ -43,16 +54,28 @@ function fetchFromServer(apiKey: string): () => unknown | null {
 }
 
 afterEach(async () => {
-  await server?.close();
-  server = null;
+  try {
+    await server?.close();
+  } finally {
+    server = null;
+    progress?.stop();
+    progress = null;
+  }
 });
 
 describe("compatible-endpoint context probe against a real server (#6177)", {
   timeout: testTimeout(60_000),
 }, () => {
   it("reads max_model_len from a live /v1/models endpoint into NEMOCLAW_CONTEXT_WINDOW (#6177)", async () => {
-    server = await startFakeOpenAiCompatibleServer({ model: MODEL, maxModelLen: 65_536 });
+    const testRun = testProgress();
+    testRun.phase("serve compatible endpoint");
+    server = await startFakeOpenAiCompatibleServer({
+      model: MODEL,
+      maxModelLen: 65_536,
+      progress: testRun,
+    });
 
+    testRun.phase("verify compatible endpoint");
     const models = fetchCompatibleEndpointModels(server.baseUrl, "");
     expect(models).toMatchObject({ data: [{ id: MODEL, max_model_len: 65_536 }] });
 
@@ -66,12 +89,16 @@ describe("compatible-endpoint context probe against a real server (#6177)", {
   });
 
   it("sends the endpoint credential through curl's --config auth flow (#6177)", async () => {
+    const testRun = testProgress();
+    testRun.phase("serve compatible endpoint");
     server = await startFakeOpenAiCompatibleServer({
       model: MODEL,
       maxModelLen: 32_768,
       apiKey: "secret-key",
+      progress: testRun,
     });
 
+    testRun.phase("verify compatible endpoint");
     const env: NodeJS.ProcessEnv = {};
     await applyCompatibleEndpointContextWindow(PUBLIC_ENDPOINT_URL, MODEL, {
       env,
@@ -89,13 +116,17 @@ describe("compatible-endpoint context probe against a real server (#6177)", {
   });
 
   it("enforces auth on /v1/models: sets the window with the key, skips it without (#6177)", async () => {
+    const testRun = testProgress();
+    testRun.phase("serve compatible endpoint");
     server = await startFakeOpenAiCompatibleServer({
       model: MODEL,
       maxModelLen: 65_536,
       apiKey: "secret-key",
+      progress: testRun,
       requireAuthModels: true,
     });
 
+    testRun.phase("verify compatible endpoint");
     // Wrong/absent credential → the endpoint 401s → no window is set.
     const noKeyEnv: NodeJS.ProcessEnv = {};
     await applyCompatibleEndpointContextWindow(PUBLIC_ENDPOINT_URL, MODEL, {
@@ -131,7 +162,14 @@ describe("compatible-endpoint context probe against a real server (#6177)", {
     // so the source-boundary guard exempts loopback (mirroring the chat probe)
     // and the real curl fetcher must run and propagate the window. Non-loopback
     // private targets stay blocked — see the unit-test rejection cases.
-    server = await startFakeOpenAiCompatibleServer({ model: MODEL, maxModelLen: 65_536 });
+    const testRun = testProgress();
+    testRun.phase("serve compatible endpoint");
+    server = await startFakeOpenAiCompatibleServer({
+      model: MODEL,
+      maxModelLen: 65_536,
+      progress: testRun,
+    });
+    testRun.phase("verify compatible endpoint");
     expect(new URL(server.baseUrl).hostname).toBe("127.0.0.1");
     const modelsRequestsBefore = server
       .requests()
@@ -151,8 +189,11 @@ describe("compatible-endpoint context probe against a real server (#6177)", {
   });
 
   it("keeps the default context window when the endpoint omits max_model_len (#6177)", async () => {
-    server = await startFakeOpenAiCompatibleServer({ model: MODEL });
+    const testRun = testProgress();
+    testRun.phase("serve compatible endpoint");
+    server = await startFakeOpenAiCompatibleServer({ model: MODEL, progress: testRun });
 
+    testRun.phase("verify compatible endpoint");
     const env: NodeJS.ProcessEnv = {};
     await applyCompatibleEndpointContextWindow(PUBLIC_ENDPOINT_URL, MODEL, {
       env,

test/e2e/README.md

@@ -86,35 +86,43 @@ artifact so baseline aggregation stays stable.
 Older issue references to Vitest target artifacts under `e2e-artifacts/vitest/`
 map to this consolidated `e2e-artifacts/live/` registry-target artifact layout.
 
-Every `e2e-live` test declares an ordered semantic phase plan in
-`meta.e2ePhases` and uses the automatic progress fixture. Normal live output
-shows only the phase number, semantic label, and the completed phase's outcome
-and duration. It deliberately does not repeat the test identity, test start or
-finish, or total test elapsed time already reported by Vitest and GitHub
-Actions. A transition looks like:
+Every `e2e-live` test and every credential-free integration test selected by
+the shared E2E workflow planner declares an ordered semantic phase plan in
+`meta.e2ePhases` and uses its automatic progress fixture. Normal E2E output
+identifies the workflow target and test scenario, then shows immediate phase
+start and completion lines with both phase and total elapsed time. A transition
+looks like:
 
 ```text
-[e2e phase 2/4] onboard the sandbox — passed in 2m 14s; next 3/4: verify hosted inference
+[e2e target="cloud-onboard" scenario="onboards a hosted sandbox"] [phase 2/4] completed: onboard the sandbox — passed in 2m 14s (total 2m 21s)
+[e2e target="cloud-onboard" scenario="onboards a hosted sandbox"] [phase 3/4] started: verify hosted inference (total 2m 21s; phase 0s)
 ```
 
-The harness appends `release registered E2E resources` after the test-declared
-plan, so the displayed phase count includes that terminal phase. Registered
-cleanup duration, failures, and stall diagnostics are attributed there. Soft
-assertion failures remain attributed to the semantic phase in which they
-occurred rather than being reassigned to resource release.
+For `e2e-live`, the stateful fixture appends `release registered E2E resources`
+after the test-declared plan, so the displayed phase count includes that
+terminal phase. Registered cleanup duration, failures, and stall diagnostics
+are attributed there. Workflow-selected integration tests instead declare and
+enter their own final release phase. Soft assertion failures remain attributed
+to the semantic phase in which they occurred rather than being reassigned to
+resource release.
 
 If one phase remains active for five minutes, a content-free diagnostic adds
-the phase duration, age of the last child output, current redacted command
-activity, and runner resources. It repeats every ten minutes while that same
-phase remains active. Child output contents are never forwarded by progress
-logging.
+the target/scenario identity, total and phase duration, age of the last child
+output, current redacted command or cleanup activity, and runner resources. It
+repeats every ten minutes while that same phase remains active. Automatic child
+output observation forwards only a timestamp and stream name, never contents.
+Operations with bounded retries may emit immediate content-free
+`progress.event(...)` lines for a timeout, cleanup, backoff, or retry; event
+labels are explicitly logged and must never contain child output, request data,
+credentials, or tokens.
 
 During fixture teardown, the fixture writes `test-progress.json` into each
 test's existing artifact directory for passing and failing tests. The summary
 keeps the test identity and overall timestamps, plus each recorded phase's
-timestamps, duration, outcome, output-event count, and last-output timestamp.
-It also records `E2E_TARGET_ID` and `NEMOCLAW_E2E_SHARD` when those values are
-set. Compare extracted artifacts from multiple runs with:
+timestamps, duration, outcome, child-output event count, and last-output timestamp.
+It records the target from `E2E_TARGET_ID`, falling back to the Actions
+`GITHUB_JOB` identity, and records `NEMOCLAW_E2E_SHARD` when set. Compare
+extracted artifacts from multiple runs with:
 
 ```bash
 npm run test:runtime-audit -- path/to/run-1 path/to/run-2
@@ -126,15 +134,28 @@ and outcome. Scheduled and ordinary manual runs include the same table for that
 run in the GitHub Actions scorecard summary. Keep phase
 labels specific to test behavior, call `progress.phase("literal phase label")`
 at the declared boundaries in order, and transition through the final
-test-declared phase on every passing path. The fixture rejects a passing live
-test that never reaches that phase; the harness enters the resource-release
-phase automatically.
-Validate phase coverage without executing live test bodies with:
+test-declared phase on every passing path. Both fixtures reject a passing test
+that never reaches that phase; only the stateful live fixture enters its
+resource-release phase automatically.
+Validate phase coverage without executing test bodies with:
 
 ```bash
 npm run test:e2e-phases:check
 ```
 
+The checker preserves coverage for every file under `test/e2e/live/` and adds
+workflow-selected integration files from the authoritative shared-job planner.
+Live modules import `fixtures/e2e-test.ts`; selected integration modules import
+`fixtures/workflow-e2e-test.ts` and declare their final release phase explicitly.
+It also follows shared E2E runtime helpers. Run child processes through
+`ShellProbe` or an existing audited progress-aware boundary; new direct async
+process boundaries fail the check. Synchronous calls require both a positive
+timeout shorter than the first heartbeat and `killSignal: "SIGKILL"`. Keep child
+contents in redacted artifacts and report only timestamp-based output activity
+to the console. Pass the fixture-provided frozen, canonical `progress`
+capability unchanged to an audited subprocess boundary; do not replace it with
+a custom, copied, or no-op adapter.
+
 ## PR E2E gate
 
 The controller, coordination check, and required job deliberately use