Skip to content

fix(ci): recover hosted runner loss once#7530

Merged
cv merged 13 commits into
mainfrom
codex/7140-hosted-runner-recovery-v2
Jul 26, 2026
Merged

fix(ci): recover hosted runner loss once#7530
cv merged 13 commits into
mainfrom
codex/7140-hosted-runner-recovery-v2

Conversation

@apurvvkumaria

@apurvvkumaria apurvvkumaria commented Jul 25, 2026

Copy link
Copy Markdown
Collaborator

Summary

Add a trusted-main recovery controller that reruns an eligible workflow once only when fail-closed evidence proves the latest first attempt lost a GitHub-hosted runner. Real test failures, mixed failures, superseded runs, manual reruns, and already-recovered runs remain ineligible.

Related Issue

Part of #7146 and parent epic #7140.

Changes

  • Extract hosted-runner loss classification and GitHub evidence collection from the PR E2E gate into shared, testable modules.
  • Require an eligible trusted-main source workflow, exact latest run/head identity, first-attempt failure, hosted-runner-loss-only evidence, and a second matching fingerprint immediately before mutation.
  • Add an actions: write controller that performs one full workflow rerun, preserving job dependencies and refusing mixed or ordinary failures.
  • Cover internal-error evidence, API pagination, retry history, event trust, workflow permissions, race handling, and one-rerun limits with executable tests.
  • Accept GitHub's valid null annotation title and detail fields by normalizing them to empty strings while retaining type and 16 KiB field bounds.
  • Document the recovery policy, supported workflows, evidence boundary, operator diagnostics, and a measurable 30-day retirement condition that resets after any accepted recovery.

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Quality Gates

  • Tests added or updated for changed behavior
  • Existing tests cover changed behavior — justification:
  • Tests not applicable — justification:
  • Docs updated for user-facing behavior changes
  • Docs not applicable — justification:
  • Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
  • Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: Independent nine-category review passed at exact head beedbfd922c91f2e360bb7b2423d97a7bb9320a8; trusted source identity, least privilege, double evidence fingerprints, mixed-failure rejection, nullable annotation normalization, token isolation, and the one-rerun ceiling were verified with no findings.
  • Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Documentation Writer Review

  • Documentation writer subagent reviewed the completed changes
  • Result: docs-updated
  • Evidence: test/e2e/README.md documents the internal recovery-controller contract, trust boundary, operator diagnostics, null-or-empty annotation fields, and 30-day retirement/reset condition; no end-user docs/ page is needed.
  • Agent: Codex Desktop

DGX Station Hardware Evidence

  • Tested on DGX Station
  • Tested commit:
  • Station profile/scenario:
  • Result:
  • Supporting evidence:

Verification

  • PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
  • Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run check:diff passed when hooks were skipped or unavailable
  • Targeted behavior tests pass for the current change set, or tests are marked not applicable above — command/result or justification: 18 affected integration files passed 398 tests; the exact nullable-annotation suite passed 89/89 and the independent security subset passed 195/195; npm run build:cli, npm run typecheck:cli, and the title/project/source-shape/test-size contracts passed. After the current-main sync, six focused recovery and controller files passed 206 tests; merge, commit, and push hooks passed.
  • Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — command/result: Not applicable; this is an isolated trusted-workflow controller covered by its full affected import surface and diff-scoped hooks.
  • Quality Gates section completed with required justifications or waivers
  • No secrets, API keys, or credentials committed
  • npm run docs builds without warnings (doc changes only)
  • Doc pages follow the style guide (doc changes only)
  • New doc pages include SPDX header and frontmatter (new pages only)

Signed-off-by: Apurv Kumaria akumaria@nvidia.com

Summary by CodeRabbit

  • New Features
    • Added “Hosted-runner recovery” GitHub workflow to request a one-time rerun when eligible first-attempt hosted-runner-loss evidence is stable, using strict identity gating and least-privilege permissions.
  • Bug Fixes
    • Tightened runner-loss/evidence verification with double-snapshot consistency, safer rejection for mixed/incomplete/ambiguous evidence, and support for null/empty runner-loss annotation title/details.
  • Documentation
    • Documented recovery workflow inputs, rerun scope, and evidence completeness requirements.
  • Tests
    • Added hosted-runner recovery and internal-error E2E/Vitest suites; updated e2e contracts/watch triggers and test budget exceptions.

Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
@apurvvkumaria apurvvkumaria self-assigned this Jul 25, 2026
@copy-pr-bot

copy-pr-bot Bot commented Jul 25, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@coderabbitai

coderabbitai Bot commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 444cbf93-36da-4e58-950c-8b1e8ef6696b

📥 Commits

Reviewing files that changed from the base of the PR and between beedbfd and 2e69364.

📒 Files selected for processing (1)
  • test/e2e/README.md

📝 Walkthrough

Walkthrough

Adds a guarded hosted-runner recovery workflow, shared runner-loss evidence modules, stable-snapshot rerun orchestration, PR gate integration, and extensive classifier, controller, workflow, and trigger tests.

Changes

Hosted-runner recovery

Layer / File(s) Summary
Runner-loss evidence contracts and detection
tools/e2e/hosted-runner-loss.mts, test/hosted-runner-loss-internal-error.test.ts
Defines strict evidence contracts, policy validation, hosted-runner markers, shutdown-log parsing, and verification tests.
GitHub evidence collection and fingerprinting
tools/e2e/hosted-runner-loss-github.mts
Validates GitHub job, annotation, check-run, and log responses; collects bounded evidence and fingerprints snapshots.
Recovery eligibility and stable rerun orchestration
tools/e2e/hosted-runner-recovery.mts, test/hosted-runner-recovery.test.ts
Matches approved workflow runs, validates latest-run evidence, compares snapshots, and requests reruns only after stable evidence.
Workflow boundary and contract enforcement
.github/workflows/hosted-runner-recovery.yaml, test/hosted-runner-recovery-workflow.test.ts, test/e2e/README.md, ci/source-shape-test-budget.json
Adds the fail-closed workflow, trusted controller execution, static summary output, boundary tests, documentation, and source-shape coverage.
PR gate evidence centralization
tools/e2e/pr-e2e-gate.mts, test/pr-e2e-gate-*.test.ts
Centralizes runner-loss collection and verification behind shared modules and threads the PR policy through retry and finalization paths.
Contract and watch-trigger integration
test/helpers/*, test/vitest-watch-triggers.test.ts
Registers the recovery contract and updates workflow-to-test trigger mappings.

Estimated code review effort: 5 (Critical) | ~120 minutes

Sequence Diagram(s)

sequenceDiagram
  participant SourceWorkflow
  participant HostedRunnerRecoveryWorkflow
  participant RecoveryController
  participant GitHubActionsAPI
  SourceWorkflow->>HostedRunnerRecoveryWorkflow: completed workflow_run event
  HostedRunnerRecoveryWorkflow->>RecoveryController: run with SOURCE_RUN_ID
  RecoveryController->>GitHubActionsAPI: collect and verify runner-loss evidence
  RecoveryController->>GitHubActionsAPI: compare repeated snapshots
  RecoveryController->>GitHubActionsAPI: request full source rerun
Loading

Possibly related PRs

  • NVIDIA/NemoClaw#7392: Uses the same hosted-runner-loss classifier and evidence primitives for an E2E main-run retry flow.
  • NVIDIA/NemoClaw#7450: Modifies the PR gate’s hosted-runner-loss retry and shutdown evidence logic.

Suggested labels: area: ci, area: e2e, area: security

Suggested reviewers: cjagwani, jyaunches, cv, senthilr-nv, prekshivyas

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: adding CI recovery for hosted runner loss with a one-time rerun.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/7140-hosted-runner-recovery-v2

Comment @coderabbitai help to get the list of available commands.

@github-code-quality

github-code-quality Bot commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in commit 2e69364 in the codex/7140-hosted-ru... branch remains at 96%, unchanged from commit baebfcb in the main branch.


Updated July 26, 2026 01:29 UTC

@github-actions

github-actions Bot commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor — Informational

Advisor assessment: Informational / low confidence
Next action: No advisor follow-up needed.
Findings: 0 blockers · 0 warnings · 0 suggestions
Status: PR review advisor failed: PR review advisor SDK execution failed: session: omitted required tool result(s): pr_review_metadata, pr_review_response_schema, pr_review_read_ledger; turn: synthesize-json: omitted required tool result(s): pr_review_metadata, pr_review_response_schema, pr_review_read_ledger

Model lanes

  • GPT-5.6 Terra (primary): Failed
  • Nemotron 3 Ultra (second opinion): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions

Nemotron output stays in workflow artifacts and does not change the assessment above.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: cloud-onboard, credential-sanitization, security-posture

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

@apurvvkumaria
apurvvkumaria requested a review from cjagwani July 25, 2026 20:20
@apurvvkumaria apurvvkumaria added the v0.0.96 Release target label Jul 25, 2026
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
@apurvvkumaria

Copy link
Copy Markdown
Collaborator Author

The refreshed branch-specific guard is green: codebase-growth-guardrails now passes, and the new commit is GitHub Verified.

The remaining build-typecheck failure is not in this PR's diff. Its package-contract lane checked out base 2ac4a0d9f and failed test/package-contract/openshell-policy-boundary.test.ts because the packaged sandbox-policy validator cannot load the newly referenced network-policy schema. The equivalent base-branch repair is already open as #7531.

I am keeping that policy/package repair out of this runner-recovery PR. Once #7531 lands, I will refresh this PR against the repaired base and let the normal CI run validate the combined merge revision.

Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
@apurvvkumaria

Copy link
Copy Markdown
Collaborator Author

Independent security review — PASS

Reviewed exact head aa117ebf67b972f5505060c60c23391d51b74227 against the repository security-review categories. No blockers, warnings, or actionable suggestions remain.

  • Trust is restricted to NVIDIA/NemoClaw, exact approved workflow identities, trusted main, first source/controller attempts, and the default-branch controller at github.workflow_sha.
  • The workflow uses job-scoped actions: write, read-only checks/contents access, SHA-pinned actions, no repository secrets, no source-run checkout, and a static summary.
  • The controller re-authenticates run/workflow/job/check identities, rejects incomplete, duplicate, changing, mixed, or non-hosted evidence, and requires two identical bounded fingerprints before the sole full-rerun request.
  • Signed job-log downloads use a strict HTTPS Azure Actions host/path boundary, no credential forwarding, no redirects, a strong ETag plus If-Match, an exact 64 KiB maximum range, a 30-second abort, and fail-closed parsing.
  • Source-workflow concurrency and attempt guards prevent repeated automatic recovery. The documented retirement condition does not weaken enforcement.

Residual non-blocking risks are limited to the GitHub API lacking compare-and-set for a trusted manual rerun race after the final snapshot, fail-closed availability if GitHub schemas or canonical markers drift, and floating GitHub-managed ubuntu-latest/Node 22 channels. These do not expand authorization.

Validation: exact-head recovery/security contracts pass 106/106 locally; an independent dependency-light pass completed 150 additional tests. aa117ebf6 is a verified, signed current-main merge, and the feature files are byte-identical to reviewed head 8a5cfdca1.

@apurvvkumaria
apurvvkumaria marked this pull request as ready for review July 25, 2026 21:32

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (7)
tools/e2e/hosted-runner-loss.mts (1)

591-599: 🚀 Performance & Scalability | 🔵 Trivial | 💤 Low value

Evaluate the trusted-marker predicate once per job.

hasTrustedMarker runs twice for every job (filter then some), so shutdown-log parsing over the 64 KiB tail plus all annotation/check matching is repeated. Compute once and derive both values.

♻️ Proposed refactor
-  const runnerLostMarkerCount = options.jobs.filter(hasTrustedMarker).length;
-  const otherNonPassingEvidencePresent = options.jobs.some((job) => !hasTrustedMarker(job));
+  const trustedMarkers = options.jobs.map(hasTrustedMarker);
+  const runnerLostMarkerCount = trustedMarkers.filter(Boolean).length;
+  const otherNonPassingEvidencePresent = runnerLostMarkerCount !== options.jobs.length;
   return {
     terminalClassificationPresent: otherNonPassingEvidencePresent,
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tools/e2e/hosted-runner-loss.mts` around lines 591 - 599, Update the
classification logic around hasTrustedMarker to evaluate the predicate once for
each job, cache the resulting booleans, and derive both runnerLostMarkerCount
and otherNonPassingEvidencePresent from that cached collection. Preserve the
existing terminalClassificationPresent and jobConclusion behavior.
test/hosted-runner-recovery.test.ts (2)

731-755: 🔒 Security & Privacy | 🔵 Trivial | ⚡ Quick win

Add coverage for the repository and token trust guards.

recoverHostedRunnerLoss rejects any repository other than NVIDIA/NemoClaw and an empty token (hosted-runner-recovery.mts Lines 456-459), but no test pins that behavior. These are the outermost trust-boundary checks for an actions: write controller, so a regression there would be silent.

💚 Suggested tests
it("refuses to recover outside the trusted repository (`#7140`)", async () => {
  const requests = setupRoutes();
  await expect(
    recoverHostedRunnerLoss({ ...recoveryRequest(), repository: "attacker/NemoClaw" }),
  ).rejects.toThrow(/restricted to NVIDIA\/NemoClaw/u);
  expect(requests).toEqual([]);
});

it("refuses to recover without a token (`#7140`)", async () => {
  const requests = setupRoutes();
  await expect(recoverHostedRunnerLoss({ ...recoveryRequest(), token: "" })).rejects.toThrow(
    /GitHub token is required/u,
  );
  expect(requests).toEqual([]);
});
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/hosted-runner-recovery.test.ts` around lines 731 - 755, Add tests
alongside the existing recoverHostedRunnerLoss guard and mutation tests for the
repository and token trust checks. Verify a non-NVIDIA/NemoClaw repository
rejects with the restricted-repository error and an empty token rejects with the
required-token error; assert both cases produce no requests.

253-276: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Fixture-integrity expect() calls inside route handlers can be swallowed.

These assertions run inside the code under test's async call stack; production try/catch blocks (e.g. the shutdown-log path in hosted-runner-loss-github.mts) can absorb them and turn a broken fixture into a passing-but-meaningless run. Throwing a plain Error here does not fix the swallow, but at least keeps the failure textually distinguishable from a Vitest assertion. Prefer asserting fixture completeness before setupRoutes returns.

As per path instructions: "Flag copied production algorithms, broad mocks that bypass the behavior under test, and conditionals that make a test pass without exercising its claim."

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/hosted-runner-recovery.test.ts` around lines 253 - 276, Move the
fixture-completeness assertions for check-run and annotation jobs out of the
githubFetchRoute handlers and perform them before setupRoutes returns, so
production error handling cannot swallow them. Validate that every referenced
job ID exists in lastJobs during route setup, then let handlers perform only
lookups and responses without Vitest expect calls.

Source: Path instructions

tools/e2e/hosted-runner-loss-github.mts (2)

554-561: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Include the swallowed error in the warning.

This catch disables recovery silently; without the cause, host-pattern/schema drift is undiagnosable from CI logs.

♻️ Suggested change
-            } catch {
+            } catch (error) {
               console.warn(
-                `Could not authenticate hosted-runner shutdown log for job ${job.id}; automatic retry remains disabled`,
+                `Could not authenticate hosted-runner shutdown log for job ${job.id}; automatic retry remains disabled: ${
+                  error instanceof Error ? error.message : String(error)
+                }`,
               );
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tools/e2e/hosted-runner-loss-github.mts` around lines 554 - 561, Update the
catch around downloadWorkflowJobLogTail in the hosted-runner job handling flow
to bind the thrown error and include its details in the console.warn message,
while preserving the existing job ID context and retry-disabled behavior.

531-533: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Extract the non-passing predicate.

The ["success","skipped","neutral"] filter is duplicated at Lines 531-533 and 572; divergence here silently changes what counts as evidence.

♻️ Suggested extraction
+const PASSING_CONCLUSIONS = new Set(["success", "skipped", "neutral"]);
+const isNonPassing = (job: WorkflowJob): boolean => !PASSING_CONCLUSIONS.has(job.conclusion ?? "");

Also applies to: 571-574

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tools/e2e/hosted-runner-loss-github.mts` around lines 531 - 533, Extract the
shared non-passing job predicate from the jobs filtering logic into a reusable
local helper or constant, then use it in both the filters around the current
nonPassingJobs calculation and the later duplicate block. Preserve the existing
success, skipped, and neutral exclusions so both checks remain consistent.
tools/e2e/hosted-runner-recovery.mts (2)

336-343: 🩺 Stability & Availability | 🔵 Trivial | 💤 Low value

Deep/stale source runs throw instead of being ignored.

If the source run has scrolled past MAX_WORKFLOW_RUN_PAGES (1000 runs) or its listing page is short before it appears, the controller exits non-zero rather than reporting a benign "not eligible". That is fail-closed for mutation, but it turns an expected steady-state condition into a red recovery job. Consider distinguishing "source older than the searched window" (ignore) from genuine listing inconsistencies (throw).

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tools/e2e/hosted-runner-recovery.mts` around lines 336 - 343, The workflow
run lookup should treat a source run beyond the searched window, including short
pages encountered before it appears, as not eligible rather than throwing.
Update the recovery logic around the page-limit and incomplete-listing checks to
return the existing benign skip/ignore outcome for stale runs, while preserving
errors for genuine listing inconsistencies such as exhausting all runs without
finding the source.

371-380: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Pass source.runAttempt instead of the literal 1.

Eligibility already pins runAttempt === 1 (Line 258), so the literal is correct today but silently decouples the two if that rule ever relaxes.

♻️ Suggested change
-    options.sourceRunId,
-    1,
+    options.sourceRunId,
+    source.runAttempt,
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tools/e2e/hosted-runner-recovery.mts` around lines 371 - 380, Update the
listNonPassingWorkflowJobs call in the recovery flow to pass source.runAttempt
instead of the hardcoded 1, preserving the existing argument order and
hostedRunnerLossPolicy options.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tools/e2e/hosted-runner-loss-github.mts`:
- Around line 156-161: Update the annotation validation around the nullable text
fields title and raw_details to accept null values and treat them as empty
strings before checking byte length. Preserve rejection of non-string, non-null
values and ensure listing continues without aborting recovery when either field
is null.

---

Nitpick comments:
In `@test/hosted-runner-recovery.test.ts`:
- Around line 731-755: Add tests alongside the existing recoverHostedRunnerLoss
guard and mutation tests for the repository and token trust checks. Verify a
non-NVIDIA/NemoClaw repository rejects with the restricted-repository error and
an empty token rejects with the required-token error; assert both cases produce
no requests.
- Around line 253-276: Move the fixture-completeness assertions for check-run
and annotation jobs out of the githubFetchRoute handlers and perform them before
setupRoutes returns, so production error handling cannot swallow them. Validate
that every referenced job ID exists in lastJobs during route setup, then let
handlers perform only lookups and responses without Vitest expect calls.

In `@tools/e2e/hosted-runner-loss-github.mts`:
- Around line 554-561: Update the catch around downloadWorkflowJobLogTail in the
hosted-runner job handling flow to bind the thrown error and include its details
in the console.warn message, while preserving the existing job ID context and
retry-disabled behavior.
- Around line 531-533: Extract the shared non-passing job predicate from the
jobs filtering logic into a reusable local helper or constant, then use it in
both the filters around the current nonPassingJobs calculation and the later
duplicate block. Preserve the existing success, skipped, and neutral exclusions
so both checks remain consistent.

In `@tools/e2e/hosted-runner-loss.mts`:
- Around line 591-599: Update the classification logic around hasTrustedMarker
to evaluate the predicate once for each job, cache the resulting booleans, and
derive both runnerLostMarkerCount and otherNonPassingEvidencePresent from that
cached collection. Preserve the existing terminalClassificationPresent and
jobConclusion behavior.

In `@tools/e2e/hosted-runner-recovery.mts`:
- Around line 336-343: The workflow run lookup should treat a source run beyond
the searched window, including short pages encountered before it appears, as not
eligible rather than throwing. Update the recovery logic around the page-limit
and incomplete-listing checks to return the existing benign skip/ignore outcome
for stale runs, while preserving errors for genuine listing inconsistencies such
as exhausting all runs without finding the source.
- Around line 371-380: Update the listNonPassingWorkflowJobs call in the
recovery flow to pass source.runAttempt instead of the hardcoded 1, preserving
the existing argument order and hostedRunnerLossPolicy options.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: f00a94a3-dcda-4489-8a31-b0fe1b10efe1

📥 Commits

Reviewing files that changed from the base of the PR and between 8b7a22e and aa117eb.

📒 Files selected for processing (15)
  • .github/workflows/hosted-runner-recovery.yaml
  • ci/source-shape-test-budget.json
  • test/e2e/README.md
  • test/helpers/e2e-workflow-contract.ts
  • test/helpers/vitest-watch-triggers.ts
  • test/hosted-runner-loss-internal-error.test.ts
  • test/hosted-runner-recovery-workflow.test.ts
  • test/hosted-runner-recovery.test.ts
  • test/pr-e2e-gate-runner-loss-classifier.test.ts
  • test/pr-e2e-gate-runner-loss-retry.test.ts
  • test/vitest-watch-triggers.test.ts
  • tools/e2e/hosted-runner-loss-github.mts
  • tools/e2e/hosted-runner-loss.mts
  • tools/e2e/hosted-runner-recovery.mts
  • tools/e2e/pr-e2e-gate.mts

Comment thread tools/e2e/hosted-runner-loss-github.mts Outdated
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
cv pushed a commit that referenced this pull request Jul 26, 2026
<!-- markdownlint-disable MD041 -->
## Summary
<!-- 1-3 plain sentences: what changes and why. Describe
before-and-after behavior when it applies. Follow the NemoClaw Writing
Guide: https://github.com/NVIDIA/NemoClaw/blob/main/WRITING.md. Do not
add unrelated prose cleanup. -->

Prevent a transient registry or transport failure while pulling a
published base image from silently cascading into the memory- and
disk-heavy local Hermes build. Positively classified transient pulls
receive at most three attempts; exhaustion stops with a distinct
temporary-failure status, while deterministic candidate absence and
invalid-image failures retain their existing behavior.

## Related Issue
<!-- Fixes #NNN or Closes #NNN. Remove this section if none. -->

Part of #7144 and parent epic #7140.

## Changes
<!-- List concrete changes. If this adds an abstraction, configuration,
fallback, migration, or compatibility path, name its current requirement
and consumer, explain why a direct change is insufficient, and identify
the test that protects it. -->

- Retry an allowlisted transient Docker pull failure at most three times
with fixed one- and two-second backoff.
- Keep this command-level registry/transport recovery separate from
#7146/#7530 whole-workflow hosted-runner-loss recovery.
- Refuse retries for manifest or repository absence, authentication and
authorization failures, non-429 HTTP 4xx responses, digest or integrity
failures, invalid references, platform mismatch, certificate validation
failures, and unknown diagnostics.
- Drain the full Docker stderr stream while retaining only its final 64
KiB in a private temporary directory; discard an ambiguous leading tail
record and fail closed when truncated output cannot be classified.
- Sanitize retained diagnostics before writing them to GitHub logs,
including URL credentials, arbitrarily long query keys, authentication
headers and folded values, cookies, known token forms, CR records, ANSI
CSI/OSC sequences, invalid bytes, and log-command injection.
- Exit with `EX_TEMPFAIL` 75 after transient exhaustion so the current
resolver callers cannot reinterpret the failure as a missing candidate
and launch a local Hermes build.
- Cover transient recovery and exhaustion, deterministic single-attempt
behavior, adversarial diagnostic redaction, bounded capture, and the
no-local-build caller contract.

## Type of Change

- [x] Code change (feature, bug fix, or refactor)
- [ ] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates
<!-- Check one tests line and one docs line. Check other lines when
applicable. Add every requested justification or approval reference. -->
- [x] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [ ] Tests not applicable — justification:
- [ ] Docs updated for user-facing behavior changes
- [x] Docs not applicable — justification: This changes internal CI
base-image resolution and failure classification; it does not change
CLI, configuration, API, policy, or supported runtime behavior.
- [x] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [x] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification: Independent
nine-category security review passed exact head `4edb0c05d` with no
findings. It covered classification precedence, status preservation,
credential and log-command redaction, bounded memory/disk use, FIFO
cleanup/cancellation, fail-closed truncation and collector failures, and
the no-local-build exhaustion contract.
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Documentation Writer Review
<!-- Required for code and documentation changes after the changes and
applicable validation are complete. Keep one review checkbox and one
instance of each visible or hidden field. For Evidence, list changed
documentation paths. For documentation-only changes, also state that the
writing rules and documentation style were reviewed. For other results,
explain why no documentation change is needed or why the review is
blocked. For Agent, use a consistent product and surface name, such as
Codex Desktop, Codex CLI, Claude Code, or Cursor. After committing all
review changes, put `git rev-parse --short HEAD` and `git rev-parse
--short HEAD:AGENTS.md` in the hidden metadata below. Rerun the review
and refresh that metadata after any new commit. This receipt is advisory
during the data-collection pilot. -->
- [x] Documentation writer subagent reviewed the completed changes
- Result: `no-docs-needed`
- Evidence: The exact three-commit diff changes only
`.github/actions/base-image-resolver.sh` and
`test/base-image-resolver-helper.test.ts`. It changes internal GitHub
Actions failure handling and does not alter the CLI, configuration, API,
sandbox runtime, or a user/operator workflow.
- Agent: Codex Desktop
<!-- docs-review-head-sha: 4edb0c0 -->
<!-- docs-review-agents-blob-sha: be20a09 -->

## DGX Station Hardware Evidence
<!-- Required only when scripts/prepare-dgx-station-host.sh changes.
Maintainers must review the linked evidence before approving or merging.
This is human-reviewed evidence, not authenticated hardware provenance.
Exceptional bypasses use existing repository governance and must be
documented on the PR. -->
- [ ] Tested on DGX Station
- Tested commit:
- Station profile/scenario:
- Result:
- Supporting evidence:

## Verification
<!-- Check each applicable item only when supported by the requested
evidence. Run targeted tests once per relevant change set and rerun
after later edits or hook autofixes that can affect the tested behavior.
Do not rerun hook-covered checks. -->
- [x] PR description includes a `Signed-off-by:` line and every commit
appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run check:diff` passed when hooks were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — `npx vitest run --project integration
test/base-image-resolver-helper.test.ts`: 61/61 passed; shfmt,
ShellCheck, Biome, Bash syntax, and `git diff --check` passed.
- [ ] Applicable broad gate passed — `npm test` for broad
runtime/test-harness changes; `npm run check` for repo-wide
validation/coverage changes — command/result: Not applicable; the change
is isolated to one shared resolver helper and its caller-contract tests.
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only)
- [ ] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

---
<!-- DCO sign-off is required in this PR description, and every commit
must appear as Verified in GitHub. Run: git config user.name && git
config user.email -->
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved container image pull recovery with automatic retries for
temporary registry failures.
* Prevented retries for permanent failures and avoided unsafe local
fallback builds when diagnostics are incomplete.
* Improved error messages by limiting output and removing sensitive
credentials and control characters.
* Ensured temporary diagnostic files are cleaned up after failed or
interrupted pulls.

* **Tests**
* Added coverage for retry behavior, failure classification, redaction,
truncation, cleanup, and fallback prevention.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
@cv
cv merged commit 55299f6 into main Jul 26, 2026
40 of 45 checks passed
@cv
cv deleted the codex/7140-hosted-runner-recovery-v2 branch July 26, 2026 01:29
@coderabbitai coderabbitai Bot mentioned this pull request Jul 26, 2026
23 tasks
@cv cv mentioned this pull request Jul 26, 2026
23 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

v0.0.96 Release target

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants