test(e2e): add semantic live progress phases

.github/workflows/e2e.yaml

@@ -356,6 +356,7 @@ jobs:
             e2e-artifacts/live/${{ matrix.id }}/run-plan.json
             e2e-artifacts/live/${{ matrix.id }}/target.json
             e2e-artifacts/live/${{ matrix.id }}/target-result.json
+            e2e-artifacts/live/${{ matrix.id }}/test-progress.json
             e2e-artifacts/live/${{ matrix.id }}/environment.result.json
             e2e-artifacts/live/${{ matrix.id }}/onboarding.result.json
             e2e-artifacts/live/${{ matrix.id }}/state-validation.result.json
@@ -776,6 +777,7 @@ jobs:
             e2e-artifacts/live/skill-agent/*/cleanup-skill-agent-summary.json
             e2e-artifacts/live/skill-agent/*/target.json
             e2e-artifacts/live/skill-agent/*/target-result.json
+            e2e-artifacts/live/skill-agent/*/test-progress.json
             e2e-artifacts/live/skill-agent/*/shell/*.result.json
             e2e-artifacts/live/skill-agent/*/shell/*.stdout.txt
             e2e-artifacts/live/skill-agent/*/shell/*.stderr.txt
@@ -1625,12 +1627,16 @@ jobs:
       fail-fast: false
       max-parallel: 1
       matrix:
-        scenario: [native, fallback, compatibility-only]
+        include:
+          - scenario: native
+          - scenario: fallback
+          - scenario: compatibility-only
     env:
       E2E_JOB: "1"
       E2E_TARGET_ID: "hermes-gpu-startup"
       E2E_ARTIFACT_DIR: ${{ github.workspace }}/e2e-artifacts/live/hermes-gpu-startup/${{ matrix.scenario }}
       E2E_HERMES_GPU_STARTUP_SCENARIO: ${{ matrix.scenario }}
+      NEMOCLAW_E2E_SHARD: ${{ matrix.scenario }}
       NEMOCLAW_CLI_BIN: ${{ github.workspace }}/bin/nemoclaw.js
       NEMOCLAW_RUN_LIVE_E2E: "1"
       NEMOCLAW_ACCEPT_THIRD_PARTY_SOFTWARE: "1"

.pre-commit-config.yaml

@@ -282,6 +282,14 @@ repos:
   # ── Priority 20: project-level checks (full coverage is manual) ────────────
   - repo: local
     hooks:
+      - id: e2e-semantic-phase-plans
+        name: Live E2E semantic phase plans
+        entry: npm run test:e2e-phases:check
+        language: system
+        pass_filenames: false
+        files: ^(test/e2e/live/.*\.ts|test/e2e/fixtures/(e2e-test|paths|progress)\.ts|tools/e2e/check-semantic-phases\.mts|vitest\.config\.ts|package\.json)$
+        priority: 20
+
       - id: test-cli
         name: Test (CLI)
         entry: npm run test:coverage:cli

CONTRIBUTING.md

@@ -211,6 +211,8 @@ These are the primary npm scripts for day-to-day development:
 | `npm run test:watch` | Watch the CLI, plugin, and E2E-support projects and rerun affected tests |
 | `npm run test:shuffle` | Shuffle test order in the focused source projects without collecting coverage |
 | `npm run test:diagnose:leaks` | Report async-resource leaks and diagnose a Vitest process that hangs during shutdown |
+| `npm run test:e2e-phases:check` | Validate semantic phase plans for every live E2E test without executing live bodies |
+| `npm run test:runtime-audit -- <artifact-dir> [...]` | Rank captured live E2E runs by median, p95, variability, and slowest phase |
 | `npm run test:integration` | Clean-build the CLI and run root integration and installer tests |
 | `npm run test:package` | Clean-build CLI/plugin artifacts and run compiled-package contracts |
 | `npm run test:live-e2e` | Opt into live E2E scenarios (mutates real external state) |
@@ -232,6 +234,16 @@ npx vitest run --project e2e-support
 This project is fast and does not run live targets. Live E2E remains opt-in through
 `npm run test:live-e2e` or the applicable GitHub Actions workflow.
 
+Every `e2e-live` test must declare its ordered, behavior-specific phase plan in
+`meta.e2ePhases`, call `progress.phase("literal phase label")` at those
+boundaries, and reach the final test-declared phase on every passing path. The
+harness then appends `release registered E2E resources` so cleanup duration and
+failures have their own phase. Run `npm run test:e2e-phases:check` after adding
+or changing a live E2E case; collection validates the plans without running
+infrastructure-mutating test bodies. See
+[`test/e2e/docs/README.md`](test/e2e/docs/README.md) for the logging and artifact
+contract.
+
 ### Test Declarative Behavior
 
 Do not read a shipped YAML, JSON, manifest, workflow, or E2E runtime file only to assert its keys,

package.json

@@ -30,6 +30,8 @@
     "test:watch": "vitest watch --project cli --project plugin --project e2e-support",
     "test:shuffle": "vitest run --project cli --project plugin --project e2e-support --sequence.shuffle.tests --coverage=false",
     "test:diagnose:leaks": "vitest run --project cli --project plugin --project e2e-support --detectAsyncLeaks --coverage=false --reporter=default --reporter=hanging-process",
+    "test:e2e-phases:check": "node --experimental-strip-types --no-warnings tools/e2e/check-semantic-phases.mts",
+    "test:runtime-audit": "tsx scripts/audit-test-runtime.mts",
     "test:integration": "npm run clean:cli && npm run build:cli && vitest run --project integration --project installer-integration",
     "test:package": "npm run clean:cli && npm --prefix nemoclaw run clean && npm run build:cli && npm --prefix nemoclaw run build && vitest run --project package-contract",
     "test:coverage:cli": "npm run clean:cli && npm run build:cli && tsx scripts/check-dist-sourcemaps.mts dist && vitest run --project cli --project integration --coverage --coverage.reporter=text-summary --coverage.reporter=json-summary --coverage.reportsDirectory=coverage/cli --coverage.include=\"bin/**/*.js\" --coverage.include=\"src/**/*.ts\" --coverage.exclude=\"test/**/*.js\" --coverage.exclude=\"test/**/*.ts\" && tsx scripts/check-coverage-ratchet.mts coverage/cli/coverage-summary.json ci/coverage-threshold-cli.json \"CLI coverage\"",

scripts/audit-test-runtime.mts

@@ -0,0 +1,152 @@
+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+// SPDX-License-Identifier: Apache-2.0
+
+import fs from "node:fs";
+import path from "node:path";
+import { pathToFileURL } from "node:url";
+
+import type { ProgressSummary } from "../test/e2e/fixtures/progress.ts";
+
+export interface RuntimeAuditRow {
+  target: string;
+  scenario: string;
+  runs: number;
+  medianMs: number;
+  p95Ms: number;
+  maxMs: number;
+  variabilityMs: number;
+  slowestPhase: string;
+  slowestPhaseMs: number;
+}
+
+function isProgressSummary(value: unknown): value is ProgressSummary {
+  if (!value || typeof value !== "object" || Array.isArray(value)) return false;
+  const summary = value as Partial<ProgressSummary>;
+  return (
+    summary.version === 1 &&
+    typeof summary.scenario === "string" &&
+    summary.scenario.length > 0 &&
+    (summary.targetId === undefined || typeof summary.targetId === "string") &&
+    (summary.shardId === undefined || typeof summary.shardId === "string") &&
+    typeof summary.durationMs === "number" &&
+    Number.isFinite(summary.durationMs) &&
+    summary.durationMs >= 0 &&
+    Array.isArray(summary.phases) &&
+    summary.phases.every(
+      (phase) =>
+        phase &&
+        typeof phase.label === "string" &&
+        typeof phase.durationMs === "number" &&
+        Number.isFinite(phase.durationMs) &&
+        phase.durationMs >= 0,
+    )
+  );
+}
+
+function progressFiles(root: string): string[] {
+  const result: string[] = [];
+  const pending = [path.resolve(root)];
+  while (pending.length > 0) {
+    const current = pending.pop();
+    if (!current || !fs.existsSync(current)) continue;
+    const stat = fs.lstatSync(current);
+    if (stat.isSymbolicLink()) continue;
+    if (stat.isFile()) {
+      if (path.basename(current) === "test-progress.json") result.push(current);
+      continue;
+    }
+    if (!stat.isDirectory()) continue;
+    for (const entry of fs.readdirSync(current, { withFileTypes: true })) {
+      if (!entry.isSymbolicLink()) pending.push(path.join(current, entry.name));
+    }
+  }
+  return result.sort();
+}
+
+function percentile(sorted: readonly number[], fraction: number): number {
+  const index = Math.max(0, Math.ceil(sorted.length * fraction) - 1);
+  return sorted[index] ?? 0;
+}
+
+function median(sorted: readonly number[]): number {
+  const middle = Math.floor(sorted.length / 2);
+  if (sorted.length % 2 === 0) {
+    return ((sorted[middle - 1] ?? 0) + (sorted[middle] ?? 0)) / 2;
+  }
+  return sorted[middle] ?? 0;
+}
+
+export function auditTestRuntime(roots: readonly string[]): RuntimeAuditRow[] {
+  const summaries = roots.flatMap(progressFiles).map((file) => {
+    const parsed: unknown = JSON.parse(fs.readFileSync(file, "utf8"));
+    if (!isProgressSummary(parsed)) throw new Error(`${file}: invalid test progress summary`);
+    return parsed;
+  });
+  const grouped = new Map<string, ProgressSummary[]>();
+  for (const summary of summaries) {
+    const key = JSON.stringify([
+      summary.targetId ?? "unlabeled",
+      summary.shardId,
+      summary.scenario,
+    ]);
+    const group = grouped.get(key) ?? [];
+    group.push(summary);
+    grouped.set(key, group);
+  }
+
+  return [...grouped.entries()]
+    .map(([, runs]): RuntimeAuditRow => {
+      const first = runs[0];
+      if (!first) throw new Error("runtime audit group is unexpectedly empty");
+      const durations = runs.map((run) => run.durationMs as number).sort((a, b) => a - b);
+      const phases = runs.flatMap((run) => run.phases);
+      const slowestPhase = phases.reduce(
+        (slowest, phase) => (phase.durationMs > slowest.durationMs ? phase : slowest),
+        { label: "n/a", durationMs: 0 },
+      );
+      const medianMs = median(durations);
+      const p95Ms = percentile(durations, 0.95);
+      return {
+        target: [first.targetId ?? "unlabeled", first.shardId].filter(Boolean).join("/"),
+        scenario: first.scenario,
+        runs: runs.length,
+        medianMs,
+        p95Ms,
+        maxMs: durations.at(-1) ?? 0,
+        variabilityMs: Math.max(0, p95Ms - medianMs),
+        slowestPhase: slowestPhase.label,
+        slowestPhaseMs: slowestPhase.durationMs,
+      };
+    })
+    .sort((a, b) => b.p95Ms - a.p95Ms || b.variabilityMs - a.variabilityMs);
+}
+
+function seconds(milliseconds: number): string {
+  return (milliseconds / 1_000).toFixed(1);
+}
+
+export function formatRuntimeAudit(rows: readonly RuntimeAuditRow[]): string {
+  const lines = [
+    "| Target | Scenario | Runs | Median | p95 | Max | p95 - median | Slowest observed phase |",
+    "| --- | --- | ---: | ---: | ---: | ---: | ---: | --- |",
+  ];
+  for (const row of rows) {
+    lines.push(
+      `| ${row.target.replaceAll("|", "\\|")} | ${row.scenario.replaceAll("|", "\\|")} | ${row.runs} | ${seconds(row.medianMs)}s | ${seconds(row.p95Ms)}s | ${seconds(row.maxMs)}s | ${seconds(row.variabilityMs)}s | ${row.slowestPhase.replaceAll("|", "\\|")} (${seconds(row.slowestPhaseMs)}s) |`,
+    );
+  }
+  return `${lines.join("\n")}\n`;
+}
+
+function main(argv: readonly string[]): void {
+  const roots = argv.length > 0 ? argv : [".e2e/live"];
+  const rows = auditTestRuntime(roots);
+  if (rows.length === 0) {
+    throw new Error(`no test-progress.json files found under: ${roots.join(", ")}`);
+  }
+  process.stdout.write(formatRuntimeAudit(rows));
+}
+
+if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
+  main(process.argv.slice(2));
+}

test/e2e/README.md

@@ -85,6 +85,53 @@ artifact so baseline aggregation stays stable.
 Older issue references to Vitest target artifacts under `e2e-artifacts/vitest/`
 map to this consolidated `e2e-artifacts/live/` registry-target artifact layout.
 
+Every `e2e-live` test declares an ordered semantic phase plan in
+`meta.e2ePhases` and uses the automatic progress fixture. Normal live output
+shows only the phase number, semantic label, and the completed phase's outcome
+and duration. It deliberately does not repeat the test identity, test start or
+finish, or total test elapsed time already reported by Vitest and GitHub
+Actions. A transition looks like:
+
+```text
+[e2e phase 2/4] onboard the sandbox — passed in 2m 14s; next 3/4: verify hosted inference
+```
+
+The harness appends `release registered E2E resources` after the test-declared
+plan, so the displayed phase count includes that terminal phase. Registered
+cleanup duration, failures, and stall diagnostics are attributed there. Soft
+assertion failures remain attributed to the semantic phase in which they
+occurred rather than being reassigned to resource release.
+
+If one phase remains active for five minutes, a content-free diagnostic adds
+the phase duration, age of the last child output, current redacted command
+activity, and runner resources. It repeats every ten minutes while that same
+phase remains active. Child output contents are never forwarded by progress
+logging.
+
+During fixture teardown, the fixture writes `test-progress.json` into each
+test's existing artifact directory for passing and failing tests. The summary
+keeps the test identity and overall timestamps, plus each recorded phase's
+timestamps, duration, outcome, output-event count, and last-output timestamp.
+It also records `E2E_TARGET_ID` and `NEMOCLAW_E2E_SHARD` when those values are
+set. Compare extracted artifacts from multiple runs with:
+
+```bash
+npm run test:runtime-audit -- path/to/run-1 path/to/run-2
+```
+
+The audit groups each test by target and optional shard, ranks the groups by
+p95 runtime, and reports variability and the slowest observed phase. Keep phase
+labels specific to test behavior, call `progress.phase("literal phase label")`
+at the declared boundaries in order, and transition through the final
+test-declared phase on every passing path. The fixture rejects a passing live
+test that never reaches that phase; the harness enters the resource-release
+phase automatically.
+Validate phase coverage without executing live test bodies with:
+
+```bash
+npm run test:e2e-phases:check
+```
+
 ## PR E2E gate
 
 The controller, coordination check, and required job deliberately use