fix(ci): retry transient base image pulls

[apurvvkumaria]

Summary

Prevent a transient registry or transport failure while pulling a published base image from silently cascading into the memory- and disk-heavy local Hermes build. Positively classified transient pulls receive at most three attempts; exhaustion stops with a distinct temporary-failure status, while deterministic candidate absence and invalid-image failures retain their existing behavior.

Related Issue

Part of #7144 and parent epic #7140.

Changes

• Retry an allowlisted transient Docker pull failure at most three times with fixed one- and two-second backoff.
• Keep this command-level registry/transport recovery separate from ci(e2e): classify hosted-runner resource pressure and infrastructure loss #7146/fix(ci): recover hosted runner loss once #7530 whole-workflow hosted-runner-loss recovery.
• Refuse retries for manifest or repository absence, authentication and authorization failures, non-429 HTTP 4xx responses, digest or integrity failures, invalid references, platform mismatch, certificate validation failures, and unknown diagnostics.
• Drain the full Docker stderr stream while retaining only its final 64 KiB in a private temporary directory; discard an ambiguous leading tail record and fail closed when truncated output cannot be classified.
• Sanitize retained diagnostics before writing them to GitHub logs, including URL credentials, arbitrarily long query keys, authentication headers and folded values, cookies, known token forms, CR records, ANSI CSI/OSC sequences, invalid bytes, and log-command injection.
• Exit with EX_TEMPFAIL 75 after transient exhaustion so the current resolver callers cannot reinterpret the failure as a missing candidate and launch a local Hermes build.
• Cover transient recovery and exhaustion, deterministic single-attempt behavior, adversarial diagnostic redaction, bounded capture, and the no-local-build caller contract.

Type of Change

• Code change (feature, bug fix, or refactor)
• Code change with doc updates
• Doc only (prose changes, no code sample modifications)
• Doc only (includes code sample changes)

Quality Gates

• Tests added or updated for changed behavior
• Existing tests cover changed behavior — justification:
• Tests not applicable — justification:
• Docs updated for user-facing behavior changes
• Docs not applicable — justification: This changes internal CI base-image resolution and failure classification; it does not change CLI, configuration, API, policy, or supported runtime behavior.
• Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
• Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: Independent nine-category security review passed exact head 4edb0c05d with no findings. It covered classification precedence, status preservation, credential and log-command redaction, bounded memory/disk use, FIFO cleanup/cancellation, fail-closed truncation and collector failures, and the no-local-build exhaustion contract.
• Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Documentation Writer Review

• Documentation writer subagent reviewed the completed changes
• Result: no-docs-needed
• Evidence: The exact three-commit diff changes only .github/actions/base-image-resolver.sh and test/base-image-resolver-helper.test.ts. It changes internal GitHub Actions failure handling and does not alter the CLI, configuration, API, sandbox runtime, or a user/operator workflow.
• Agent: Codex Desktop

DGX Station Hardware Evidence

• Tested on DGX Station
• Tested commit:
• Station profile/scenario:
• Result:
• Supporting evidence:

Verification

• PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
• Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run check:diff passed when hooks were skipped or unavailable
• Targeted behavior tests pass for the current change set, or tests are marked not applicable above — npx vitest run --project integration test/base-image-resolver-helper.test.ts: 61/61 passed; shfmt, ShellCheck, Biome, Bash syntax, and git diff --check passed.
• Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — command/result: Not applicable; the change is isolated to one shared resolver helper and its caller-contract tests.
• Quality Gates section completed with required justifications or waivers
• No secrets, API keys, or credentials committed
• npm run docs builds without warnings (doc changes only)
• Doc pages follow the style guide (doc changes only)
• New doc pages include SPDX header and frontmatter (new pages only)

---

Signed-off-by: Apurv Kumaria akumaria@nvidia.com

Summary by CodeRabbit

• Bug Fixes

  • Improved container image pull recovery with automatic retries for temporary registry failures.
  • Prevented retries for permanent failures and avoided unsafe local fallback builds when diagnostics are incomplete.
  • Improved error messages by limiting output and removing sensitive credentials and control characters.
  • Ensured temporary diagnostic files are cleaned up after failed or interrupted pulls.

• Tests

  • Added coverage for retry behavior, failure classification, redaction, truncation, cleanup, and fallback prevention.

[coderabbitai]

📝 Walkthrough

Walkthrough

The resolver now securely captures and sanitizes Docker pull diagnostics, classifies failures as deterministic or transient, retries transient failures up to three times, and handles bounded or unclassifiable output without local fallback. Tests cover retries, redaction, truncation, status handling, and cleanup.

Changes

Docker pull recovery

Layer / File(s)	Summary
Diagnostic canonicalization and sanitization .github/actions/base-image-resolver.sh	Docker diagnostics are normalized, sensitive values are redacted, records are flattened, and output is bounded before emission.
Failure classification .github/actions/base-image-resolver.sh	Diagnostic patterns classify pull failures as deterministic or transient.
Bounded diagnostic capture .github/actions/base-image-resolver.sh	FIFO-based collectors continuously drain Docker stderr, enforce byte limits, handle truncation, validate metadata, and clean temporary artifacts.
Retry orchestration and validation .github/actions/base-image-resolver.sh, test/base-image-resolver-helper.test.ts	resolver_pull retries transient failures, stops on deterministic failures, and rejects unclassifiable exhausted output; tests verify retry timing, redaction, truncation, statuses, fallback behavior, and cleanup.

Estimated code review effort: 4 (Complex) | ~60 minutes

Suggested labels: area: security, bug-fix

Suggested reviewers: cjagwani

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main CI change: retrying transient base image pulls.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/7140-base-image-pull-retry

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-code-quality]

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in commit 4edb0c0 in the codex/7140-base-imag... branch remains at 96%, unchanged from commit 3e54a83 in the main branch.

TypeScript / code-coverage/cli

The overall coverage in commit 4edb0c0 in the codex/7140-base-imag... branch is 80%. The coverage in commit 3e54a83 in the main branch is 81%.

Show a code coverage summary of the most impacted files.

File	main 3e54a83	codex/7140-base-imag... 4edb0c0	+/-
src/lib/onboard/docker-cdi.ts	80%	70%	-10%
src/lib/shields/index.ts	72%	67%	-5%
src/lib/onboard...e-patch-flow.ts	76%	71%	-5%
src/lib/onboard...host-anchors.ts	94%	90%	-4%
src/lib/onboard/preflight.ts	77%	78%	+1%
src/lib/actions...light-guards.ts	86%	90%	+4%
src/lib/onboard...box-gpu-mode.ts	92%	97%	+5%
src/lib/actions...confirmation.ts	69%	79%	+10%
src/lib/actions...ocker-health.ts	65%	82%	+17%
src/lib/onboard...box-prebuild.ts	74%	92%	+18%

---

_{Updated July 26, 2026 01:18 UTC}

[github-actions]

PR Review Advisor — No blocking findings reported

Advisor assessment: No blocking advisor findings reported
Next action: No advisor follow-up needed.
Findings: 0 blockers · 0 warnings · 0 suggestions

Model lanes

• GPT-5.6 Terra (primary): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
• Nemotron 3 Ultra (second opinion): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
• Model comparison: normalized findings match; normalized E2E selections differ; severity counts match.

Nemotron output stays in workflow artifacts and does not change the assessment above.

Since last review: 0 prior items resolved · 0 still apply · 0 new items found

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: None

1 optional E2E recommendation

• rebuild-hermes

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

[coderabbitai]

🧹 Nitpick comments (4)

.github/actions/base-image-resolver.sh (2)

271-283: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win

Collector failure aborts the job even when docker pull succeeded.

If the diagnostic collector fails (line 277) or the byte-count metadata is malformed (lines 279-280) after status=0, the capture exits 74 and resolver_pull escalates to exit 75, killing the workflow despite a successful pull. Diagnostics only matter for the failure path; consider short-circuiting when status == 0.

♻️ Proposed short-circuit

   collector_pid=""
+  if ((status == 0)); then
+    printf '%s\n%s\n' "$status" 0
+    exit 0
+  fi
   ((collector_status == 0)) || exit 74

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/actions/base-image-resolver.sh around lines 271 - 283, Update the
post-collection handling in resolver_pull so a successful pull status
short-circuits before collector failure or byte-count validation can exit 74.
Preserve the existing diagnostic collection and validation behavior for failed
pulls, including truncation handling and resolver_pull’s failure escalation.

---

330-345: 🎯 Functional Correctness | 🔵 Trivial | 💤 Low value

Add an explicit terminal return after the retry loop.

Every path inside the loop currently returns, exits, or sleeps, so line 345 is unreachable today. If the attempt == 3 guard is ever changed, the function falls out of the loop and returns 0, silently reporting a failed pull as success.

🛡️ Defensive terminal return

   done
+
+  echo "::error::Base-image pull retry loop exited without a decision; refusing a local build fallback" >&2
+  exit 75
 }

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/actions/base-image-resolver.sh around lines 330 - 345, In the
base-image pull retry function, add an explicit nonzero terminal return
immediately after the retry loop so any future path that exits the loop still
reports pull failure rather than success. Preserve the existing exit 75 behavior
and error handling for current retry and sleep failures.

test/base-image-resolver-helper.test.ts (2)

624-631: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

observed-max can pass vacuously if no capture file is present.

When the glob matches nothing (or only non-regular entries such as FIFOs), max_size stays 0 and line 650 passes without ever observing the on-disk bound. Asserting the sample was non-zero keeps this from silently degrading into a no-op if the capture layer changes shape.

♻️ Make the observation mandatory

+    expect(Number(readFileSync(observedMax, "utf8"))).toBeGreaterThan(0);
     expect(Number(readFileSync(observedMax, "utf8"))).toBeLessThanOrEqual(65_536);

Note the capture path may drain through FIFOs, so if -f legitimately never matches, prefer asserting on the bounded regular file the collector writes instead.

Also applies to: 650-650

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/base-image-resolver-helper.test.ts` around lines 624 - 631, Make the
observed-max validation in the capture-size test mandatory rather than allowing
a missing or empty capture set to leave max_size at zero. Update the loop and
assertion around OBSERVED_MAX to verify that a bounded regular capture file was
actually observed, while accounting for FIFO-based draining when selecting the
file to validate.

Source: Path instructions

---

384-384: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Test title contradicts the behavior it asserts.

These cases are transient diagnostics that get retried three times and then exhaust to EX_TEMPFAIL; calling them "terminal" reads as the deterministic path, which is exactly the distinction this suite exists to pin down.

♻️ Suggested title

-  ])("retries a terminal %s after more than 64 KiB of output", (_name, diagnostic) => {
+  ])("retries and exhausts a transient %s after more than 64 KiB of output", (_name, diagnostic) => {

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/base-image-resolver-helper.test.ts` at line 384, Rename the
parameterized test title in the retry diagnostic suite from “terminal” to
“transient” so it accurately describes diagnostics retried three times before
returning EX_TEMPFAIL; leave the test behavior unchanged.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In @.github/actions/base-image-resolver.sh:
- Around line 271-283: Update the post-collection handling in resolver_pull so a
successful pull status short-circuits before collector failure or byte-count
validation can exit 74. Preserve the existing diagnostic collection and
validation behavior for failed pulls, including truncation handling and
resolver_pull’s failure escalation.
- Around line 330-345: In the base-image pull retry function, add an explicit
nonzero terminal return immediately after the retry loop so any future path that
exits the loop still reports pull failure rather than success. Preserve the
existing exit 75 behavior and error handling for current retry and sleep
failures.

In `@test/base-image-resolver-helper.test.ts`:
- Around line 624-631: Make the observed-max validation in the capture-size test
mandatory rather than allowing a missing or empty capture set to leave max_size
at zero. Update the loop and assertion around OBSERVED_MAX to verify that a
bounded regular capture file was actually observed, while accounting for
FIFO-based draining when selecting the file to validate.
- Line 384: Rename the parameterized test title in the retry diagnostic suite
from “terminal” to “transient” so it accurately describes diagnostics retried
three times before returning EX_TEMPFAIL; leave the test behavior unchanged.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 469bcc3e-43c8-44d7-b78d-de62018fd490

📥 Commits

Reviewing files that changed from the base of the PR and between 3e54a83 and 4edb0c0.

📒 Files selected for processing (2)

• .github/actions/base-image-resolver.sh
• test/base-image-resolver-helper.test.ts