fix(e2e): stabilize cancelled live-test lifecycles

[apurvvkumaria]

Summary

Keeps live-E2E cleanup usable after a test is cancelled, records content-free child-process outcomes for postmortems, and reserves explicit teardown time beneath job deadlines. This addresses the leaked-port failure reproduced locally while improving evidence for future runner disappearances without treating ordinary assertions as infrastructure loss.

The follow-up fixes also recognize GitHub’s exact shutdown-plus-exit-143 representation and keep the one-time retry bound to the selected child when GitHub preserves its canonical source-check URL.

Related Issue

Part of #7146. Parent epic: #7140.

Changes

• Run cleanup callbacks with a fresh, signal-aware ten-minute deadline instead of reusing the test's already-aborted signal.
• Latch the cleanup signal per shell or Docker command and await cleanup callbacks sequentially so resource disposal completes deterministically.
• Record ordinal child start and terminal outcomes without commands, arguments, environment, working directory, process IDs, raw output, exit codes, or signal names.
• Bound gateway-health and credential-free tests beneath their job limits, and reduce the Hermes rebuild test budget to 70 minutes beneath its 90-minute job deadline.
• Extend semantic and workflow-boundary guards for the new lifecycle evidence and cleanup reserves.
• Retry the exact GitHub-hosted shutdown plus exit-143 failed-step representation once, while rejecting ordinary process failures.
• Bind retry authorization, history, revalidation, and cleanup to the controller-selected child through either its exact Actions URL or GitHub’s canonical source-check URL.
• Keep legacy stranded-step retries bound to the canonical lost-communication annotation; generic shutdown-log evidence still requires a terminalized workload step and completion-time alignment.

Type of Change

• Code change (feature, bug fix, or refactor)
• Code change with doc updates
• Doc only (prose changes, no code sample modifications)
• Doc only (includes code sample changes)

Quality Gates

• Tests added or updated for changed behavior
• Existing tests cover changed behavior — justification:
• Tests not applicable — justification:
• Docs updated for user-facing behavior changes
• Docs not applicable — justification: The internal test/e2e/README.md now documents the retry-controller evidence contract. No public docs/ page applies because no user-facing command, configuration, policy, API, or operator procedure changes.
• Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
• Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: Independent focused review confirmed the cleanup signal is operation-scoped and bounded, lifecycle records are content-free and terminal-once, and the deadline changes retain explicit teardown reserve.
• Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Documentation Writer Review

• Documentation writer subagent reviewed the completed changes
• Result: docs-updated
• Evidence: test/e2e/README.md now documents both authenticated terminal-shutdown forms (cancelled plus operation-cancelled, and failed plus exit 143), their strict annotation/step/log alignment, and exact selected-child binding through either the child Actions URL or GitHub’s canonical source-check URL. No public docs/ page changes are needed because the behavior is confined to the internal E2E harness and retry controller; no user-facing command, configuration, policy, API, or operator procedure changes.
• Agent: Codex Desktop

DGX Station Hardware Evidence

• Tested on DGX Station
• Tested commit: Not applicable
• Station profile/scenario: Not applicable
• Result: Not applicable
• Supporting evidence: Not applicable

Verification

• PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
• Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run check:diff passed when hooks were skipped or unavailable
• Targeted behavior tests pass for the current change set, or tests are marked not applicable above — exact-head validation passed 122 focused tests across the cleanup, child-lifecycle, semantic-boundary, workflow-boundary, and runner-loss contracts; semantic validation passed 123 tests across 82 files; Vitest membership is exact across 1,764 candidate files; CLI typecheck, check:diff, normal commit hooks, and pre-push hooks passed.
• Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — command/result: Targeted harness, workflow, semantic, type, and diff-scoped gates passed; GitHub CI is authoritative for the repository-wide gate.
• Quality Gates section completed with required justifications or waivers
• No secrets, API keys, or credentials committed
• npm run docs builds without warnings (doc changes only)
• Doc pages follow the style guide (doc changes only)
• New doc pages include SPDX header and frontmatter (new pages only)

---

Signed-off-by: Apurv Kumaria akumaria@nvidia.com

Summary by CodeRabbit

• New Features
  • Cleanup execution now supports shared, timeout-based cancellation and rejects overlapping runs.
  • Shell and Docker probes now accept lazy/latchable signal sources resolved per command.
  • E2E progress reporting and observed-child semantics now include child lifecycle tracking with finer-grained lifecycle outcomes.
• Bug Fixes
  • Improved child lifecycle auditing and lifecycle classification across spawn/launch errors, exits, signals, and close behavior.
  • PR gate runner-loss retry/evidence validation tightened, including hosted shutdown handling for exit code 143.
• Tests
  • Added E2E cleanup-signal/timeout/concurrency coverage; expanded semantic-phase/lifecycle/progress audits; standardized live E2E timeouts.
• Documentation
  • Updated PR E2E gate documentation to reflect authenticated cancellation/retry and stricter evidence contracts.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

Cleanup execution now supports abort signals, shared deadlines, and concurrent-run protection. Child-process progress records lifecycle outcomes, semantic audits enforce lifecycle contracts, and PR gate runner-loss validation adds exit-143 evidence and stricter retry lineage checks.

Changes

E2E lifecycle control

Layer / File(s)	Summary
Cleanup signal and deadline flow test/e2e/fixtures/cleanup.ts, test/e2e/fixtures/*-probe.ts, test/e2e/fixtures/e2e-test.ts, test/e2e/support/e2e-cleanup-signal.test.ts	Cleanup runs use per-run controllers, configurable deadlines, concurrent-run guards, and dynamically resolved shell and Docker signals.
Child lifecycle progress reporting test/e2e/fixtures/progress.ts, test/e2e/fixtures/observed-child-process.ts, test/e2e/support/observed-child-process.test.ts	Progress records frozen lifecycle reporters, and observed child processes classify spawn failures, exits, signals, and unknown closes.
Semantic lifecycle contract enforcement tools/e2e/check-semantic-phases.mts, test/e2e/support/e2e-semantic-phase-check.test.ts	Semantic analysis validates lifecycle listeners, progress contracts, canonical usage, and direct-child evidence.
E2E timeout coverage test/e2e/live/*, test/e2e/support/shared-e2e-workflow-boundary.test.ts	Live tests receive explicit timeout values and workflow-boundary validation uses a configured timeout.

PR gate runner-loss evidence

Layer / File(s)	Summary
Runner-loss evidence contract test/e2e/README.md	Retry classification, terminal log evidence, annotation requirements, and selected-run linkage rules are tightened.
Runner-loss controller validation tools/e2e/pr-e2e-gate.mts	Runner-loss lineage uses selected-run URLs, exit-143 shutdown evidence is authenticated, and retry source checks use derived child-run URLs.
Runner-loss classifier and retry coverage test/pr-e2e-gate-runner-loss-*.test.ts	Classifier and retry tests cover canonical and rejected exit-143 evidence and retry authorization cases.

Estimated code review effort: 4 (Complex) | ~75 minutes

Sequence Diagram(s)

sequenceDiagram
  participant E2ETest
  participant CleanupRegistry
  participant Probe
  participant ChildProcess
  participant TestProgress
  E2ETest->>CleanupRegistry: runAll()
  CleanupRegistry->>Probe: run with current abort signal
  Probe->>ChildProcess: supervise command
  ChildProcess-->>Probe: spawn, error, or close
  Probe->>TestProgress: report lifecycle outcome
  CleanupRegistry-->>E2ETest: cleanup results

Loading

Possibly related PRs

• NVIDIA/NemoClaw#7100: Extends related E2E progress and semantic-phase lifecycle coverage.
• NVIDIA/NemoClaw#7397: Modifies the same semantic-phase auditing and child-process progress validation areas.

Suggested labels: bug-fix

Suggested reviewers: cv, prekshivyas

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title matches the main E2E cancellation and lifecycle stabilization work and is specific enough to convey the change.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/e2e-process-lifecycle-stability

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-code-quality]

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in commit f85baab in the codex/e2e-process-li... branch remains at 96%, unchanged from commit b51168d in the main branch.

TypeScript / code-coverage/cli

The overall coverage in commit f85baab in the codex/e2e-process-li... branch remains at 80%, unchanged from commit b51168d in the main branch.

Show a code coverage summary of the most impacted files.

File	main b51168d	codex/e2e-process-li... f85baab	+/-
src/lib/onboard...ndbox-create.ts	83%	33%	-50%
src/lib/inferen...lama/process.ts	100%	50%	-50%
src/lib/onboard...-create-plan.ts	88%	75%	-13%
src/lib/onboard...ndbox-create.ts	91%	83%	-8%
src/lib/inferen...er-lifecycle.ts	71%	65%	-6%
src/lib/state/g...way-registry.ts	95%	94%	-1%
src/lib/sandbox...rce-identity.ts	87%	87%	0%
src/lib/shields/index.ts	71%	72%	+1%
src/lib/state/m...-acquisition.ts	86%	87%	+1%
src/lib/domain/.../connect-env.ts	89%	97%	+8%

---

_{Updated July 24, 2026 04:08 UTC}

[github-actions]

PR Review Advisor — Informational

Advisor assessment: Informational / high confidence
Next action: No advisor follow-up needed.
Findings: 0 blockers · 0 warnings · 0 suggestions
Status: No actionable findings remain in the canonical review ledger.

Model lanes

• GPT-5.6 Terra (primary): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
• Nemotron 3 Ultra (second opinion): Completed · high confidence · 0 blockers · 4 warnings · 2 suggestions
• Model comparison: normalized findings differ; normalized E2E selections match; Nemotron reported the same number of blockers, 4 more warnings, 2 more suggestions.

Nemotron output stays in workflow artifacts and does not change the assessment above.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: cloud-onboard, credential-sanitization, security-posture, gateway-health-honest, onboard-negative-paths, openshell-version-pin, rebuild-hermes, rebuild-hermes-stale-base, ubuntu-repo-cli-smoke

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (1)

test/pr-e2e-gate-runner-loss-classifier.test.ts (1)

376-424: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Rejection matrix correctly exercises each new failure mode.

Each case (missing preceding shutdown, wrong exit code, missing log evidence, mismatched terminal timestamp) isolates exactly one invariant from the README's tightened contract. The "timestamp does not match" case recomputes exit143HostedRunnerLossJob() twice just to read .steps before passing it back into an override — harmless, but could be simplified to a single local const to avoid the redundant call.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/pr-e2e-gate-runner-loss-classifier.test.ts` around lines 376 - 424, The
timestamp-mismatch case redundantly invokes exit143HostedRunnerLossJob() while
preparing its steps override. Simplify that test setup by creating one local
base job value, deriving the modified steps from its steps, and passing the
resulting override to exit143HostedRunnerLossJob without changing the exercised
invariant or assertions.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tools/e2e/pr-e2e-gate.mts`:
- Around line 2546-2555: Update the interrupted-step validation around
interruptedSteps and interruptedStep to support the legacy stranded-step case
accepted at the later branch: recognize the single in_progress step with a null
conclusion separately from a completed step concluded as
marker.interruptedStepConclusion, and do not require completedAt for the
stranded step. Preserve the existing completed-cancelled validation and
downstream retry/log behavior for both evidence paths.

---

Nitpick comments:
In `@test/pr-e2e-gate-runner-loss-classifier.test.ts`:
- Around line 376-424: The timestamp-mismatch case redundantly invokes
exit143HostedRunnerLossJob() while preparing its steps override. Simplify that
test setup by creating one local base job value, deriving the modified steps
from its steps, and passing the resulting override to exit143HostedRunnerLossJob
without changing the exercised invariant or assertions.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 6e757e6f-ebff-410c-85b6-16c1709d9e7d

📥 Commits

Reviewing files that changed from the base of the PR and between cd5c91b and 85b1272.

📒 Files selected for processing (4)

• test/e2e/README.md
• test/pr-e2e-gate-runner-loss-classifier.test.ts
• test/pr-e2e-gate-runner-loss-retry.test.ts
• tools/e2e/pr-e2e-gate.mts

[prekshivyas]

Blocking approval at exact head bc13e8ef13d413788b5708478dcf64c2edc6140a:

1. The documentation-writer receipt is stale. The PR records reviewed head 6c7f567af, while the exact-head receipt reports headShaMatches: false and status: invalid. Rerun the documentation-writer review at the current head and refresh the hidden receipt metadata, as required by AGENTS.md.
2. Current-head CI and E2E coordination are still in progress. Please let the exact-head gates finish green before approval.

The implementation/security pass found no blocking code issue. The merge-from-main result was revalidated at this exact head: 122 focused tests passed, git diff --check passed, and all commits are GitHub-verified.

[prekshivyas]

Approved at exact head f85baab against base b51168d. The merge-gate audit passes all 56 checks; focused lifecycle/retry tests, documentation receipt, security review, CodeRabbit, both PR advisors, and the selected exact-SHA E2E plan are green. No unresolved review threads or product-scope blockers remain.