Skip to content

ci(e2e): centralize larger runner routing#7391

Merged
cv merged 12 commits into
mainfrom
codex/7145-larger-runner-routing-refresh
Jul 23, 2026
Merged

ci(e2e): centralize larger runner routing#7391
cv merged 12 commits into
mainfrom
codex/7145-larger-runner-routing-refresh

Conversation

@apurvvkumaria

@apurvvkumaria apurvvkumaria commented Jul 22, 2026

Copy link
Copy Markdown
Collaborator

Summary

Rebuilds #7245 on current main and centralizes larger-runner selection for resource-heavy E2E lanes. Trusted final-main executions can use one administrator-configured label, while unset, PR-checkout, and untrusted contexts continue to use ubuntu-latest. Exact-head PR gates now add a bounded swap fallback for the hosted Hermes image-building lanes that intentionally remain on standard runners.

Related Issue

Part of #7145. Supersedes #7245, whose protected head branch is stacked on obsolete #7244 history.

Changes

  • Build one validated runner map in generate-matrix before target code is checked out, consuming E2E_LARGER_RUNNER_LABEL only for NVIDIA/NemoClaw on refs/heads/main with no alternate checkout SHA.
  • Route common egress, Hermes rebuilds, heavy MCP shards, and the Hermes lanes repeatedly interrupted in run 29897237525 through that map.
  • Keep the OpenClaw matrix counterparts, MCP OpenClaw/dev lanes, and unrelated jobs on standard runners.
  • For exact-head PR gates only, reuse at least 32 GiB of active swap or provision one fixed 32 GiB file before the hosted Hermes image-building lanes. The setup uses absolute non-interactive executables in a clean environment and fails closed before Vitest.
  • Exclude scheduled/manual main, larger-runner, workflow-managed rebuild, dedicated-runner, MCP dev, and non-Hermes executions from the swap fallback.
  • Document the runner sizing, concurrency, comparison, fallback, and rollback requirements.
  • Protect the centralized rollout, exact-head fallback, and trust boundary with executable tests. A workflow-only swap step cannot help an alternate checkout until that workflow revision reaches trusted main.

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Quality Gates

  • Tests added or updated for changed behavior
  • Existing tests cover changed behavior — justification:
  • Tests not applicable — justification:
  • Docs updated for user-facing behavior changes
  • Docs not applicable — justification:
  • Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
  • Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: Independent focused review passed with no blockers after exact-head-only activation, absolute privileged executables, a clean fixed environment, and active-swap-safe cleanup were enforced at af1cb1c77.
  • Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Documentation Writer Review

  • Documentation writer subagent reviewed the completed implementation
  • Result: docs-updated
  • Evidence: test/e2e/README.md documents the larger-runner trust boundary and the bounded exact-head swap fallback, selected Hermes lanes, fail-closed behavior, exclusions, sizing, measurement, and rollback requirements.
  • Agent: Codex Desktop
  • PR: ci(e2e): centralize larger runner routing #7391

DGX Station Hardware Evidence

  • Tested on DGX Station
  • Tested commit:
  • Station profile/scenario:
  • Result:
  • Supporting evidence:

Verification

  • PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
  • Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run check:diff passed when hooks were skipped or unavailable
  • Targeted behavior tests pass for the current change set, or tests are marked not applicable above — 59 focused E2E-support tests passed after the current main sync.
  • Applicable broad gate passed — npm run checks, npm run typecheck:cli, and npm run docs passed; Fern reported 0 errors and 2 non-fatal warnings.
  • Quality Gates section completed with required justifications or waivers
  • No secrets, API keys, or credentials committed
  • npm run docs builds without warnings (doc changes only)
  • Doc pages follow the style guide (doc changes only)
  • New doc pages include SPDX header and frontmatter (new pages only)

Signed-off-by: Apurv Kumaria akumaria@nvidia.com

Summary by CodeRabbit

  • New Features
    • E2E workflows now generate a trusted runner-routing map and use it to route selected “heavy” lanes to a configurable larger Ubuntu runner, with validation and safe fallback to ubuntu-latest.
    • Live Vitest E2E runs can now provision a bounded Hermes swap file beforehand when required, and fail closed if provisioning fails.
  • Documentation
    • Added guidance for enabling, validating boundaries, and rolling back larger-runner routing.
  • Tests
    • Added e2e boundary coverage for runner routing and expanded Hermes swap provisioning/selection coverage.
  • CI / Chores
    • Updated workflow boundary checks and routing assertions to match the new trusted routing behavior.

cjagwani and others added 2 commits July 22, 2026 10:06
Signed-off-by: Charan Jagwani <cjagwani@nvidia.com>
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
@apurvvkumaria apurvvkumaria added area: ci CI workflows, checks, release automation, or GitHub Actions area: e2e End-to-end tests, nightly failures, or validation infrastructure chore Build, CI, dependency, or tooling maintenance integration: dcode LangChain Deep Code integration behavior v0.0.92 labels Jul 22, 2026
@copy-pr-bot

copy-pr-bot Bot commented Jul 22, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@apurvvkumaria apurvvkumaria self-assigned this Jul 22, 2026
@coderabbitai

coderabbitai Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

The E2E workflow now computes trusted larger-runner routing for selected jobs, validates its contract, and documents and tests fallback and trusted-main behavior. Hermes live Vitest execution also provisions swap for eligible exact-head runs, with failure handling and selection tests.

Changes

Larger-runner E2E routing

Layer / File(s) Summary
Runner routing implementation
.github/workflows/e2e.yaml
The workflow validates E2E_LARGER_RUNNER_LABEL, emits routing JSON, and uses it for selected E2E jobs.
Routing contract validation
tools/e2e/workflow-boundary.mts, tools/e2e/*-workflow-boundary.mts
Validation enforces the trusted routing step, output bindings, eligible job expressions, and prohibited direct usage while allowing routed jobs to vary from ubuntu-latest.
Routing tests and documentation
test/e2e/support/*, test/e2e/README.md, ci/source-shape-test-budget.json
Tests cover fallback, trusted-main, malformed-label, and workflow-mutation cases; documentation and source-shape exceptions are updated.

Hermes E2E swap setup

Layer / File(s) Summary
Swap provisioning orchestration
tools/e2e/live-vitest-invocation.mts
Eligible Hermes live tests provision and validate swap through sudo before Vitest execution, preserving existing swap and returning setup failures.
Swap behavior validation
test/e2e/support/live-vitest-invocation.test.ts
Tests verify provisioning commands, fail-closed behavior, cleanup handling, and eligibility decisions across paths and environment values.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant GenerateMatrix
  participant RunnerRouting
  participant RoutedJobs
  GenerateMatrix->>RunnerRouting: evaluate trusted context and runner label
  RunnerRouting->>GenerateMatrix: emit runner_routing JSON
  GenerateMatrix->>RoutedJobs: provide runner labels through job expressions
Loading
sequenceDiagram
  participant LiveVitestCommand
  participant SwapProvisioner
  participant Vitest
  LiveVitestCommand->>SwapProvisioner: provision swap for eligible Hermes run
  SwapProvisioner->>LiveVitestCommand: return setup status
  LiveVitestCommand->>Vitest: execute Vitest after successful setup
Loading

Possibly related PRs

  • NVIDIA/NemoClaw#7399 — Both changes extend workflow boundary validation through the same validator entrypoint.

Suggested labels: feature, integration: hermes

Suggested reviewers: cv, prekshivyas

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the main change: centralizing larger-runner routing for E2E workflows.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/7145-larger-runner-routing-refresh

Comment @coderabbitai help to get the list of available commands.

@github-code-quality

github-code-quality Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in commit 0d92d12 in the codex/7145-larger-ru... branch remains at 96%, unchanged from commit cce9540 in the main branch.


Updated July 23, 2026 14:37 UTC

Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
@github-actions

github-actions Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor — Blocking findings reported

Advisor assessment: Blockers require maintainer review
Next action: Review the blockers below.
Findings: 1 blocker · 0 warnings · 0 suggestions

Model lanes

  • GPT-5.6 Terra (primary): Completed · high confidence · 1 blocker · 0 warnings · 0 suggestions
  • Nemotron 3 Ultra (second opinion): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
  • Model comparison: normalized findings differ; normalized E2E selections differ; Nemotron reported 1 fewer blocker, the same number of warnings, the same number of suggestions.

Nemotron output stays in workflow artifacts and does not change the assessment above.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: cloud-onboard, credential-sanitization, security-posture

5 optional E2E recommendations
  • hermes-e2e
  • hermes-dashboard
  • hermes-inference-switch
  • hermes-shields-config
  • mcp-bridge

Blockers

PRA-1 Blocker — Keep swap provisioning code in the trusted workflow boundary

  • Location: tools/e2e/live-vitest-invocation.mts:267
  • Category: security
  • Problem: Exact-head E2E jobs check out `inputs.checkout_sha` before the live-Vitest helper runs. The helper passes its `HERMES_E2E_SWAP_SCRIPT` string to `/usr/bin/sudo -n`, so PR-authored code can change the root shell program for those jobs.
  • Impact: A PR author can gain root execution on an exact-head E2E runner and access job resources, including credentials that affected jobs provide to the live test step.
  • Fix: Move swap provisioning to an immutable trusted workflow-side action or script. Pass only fixed, validated arguments from the PR checkout.
  • Verification: Inspect the exact-head checkout in `.github/workflows/e2e.yaml` and confirm that the command passed to sudo cannot be modified by `inputs.checkout_sha`.
  • Test coverage: Add a workflow-boundary test that mutates the PR-side helper fixture and proves the privileged swap command still comes from the trusted workflow revision.
  • Evidence: `.github/workflows/e2e.yaml` checks out `ref: ${{ inputs.checkout_sha || github.sha }}` before live jobs execute the helper. `tools/e2e/live-vitest-invocation.mts` defines `HERMES_E2E_SWAP_SCRIPT` and passes it as the `-c` argument to `/usr/bin/sudo -n /usr/bin/env -i /bin/bash`. The exact-head guard validates only the SHA format and selects when setup runs; it does not authenticate the helper source.

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

@apurvvkumaria

Copy link
Copy Markdown
Collaborator Author

/nvskills-ci

@apurvvkumaria
apurvvkumaria marked this pull request as ready for review July 22, 2026 18:50

@prekshivyas prekshivyas left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed exact head 4aa12a6 against base 1d418f5.

Security verdict: PASS, with no findings across all 9 changed files.

  • Secrets and credentials: PASS
  • Input validation and sanitization: PASS
  • Authentication and authorization: PASS
  • Dependencies and third-party libraries: PASS
  • Error handling and logging: PASS
  • Cryptography and data protection: PASS
  • Configuration and security headers: PASS
  • Security testing: PASS
  • Holistic security posture: PASS

The trusted-main eligibility checks, fixed routing map, pre-checkout execution, standard-runner fallback, and exact workflow-boundary validation preserve the existing PR and fork trust model.

Validation: 4 focused E2E-support files passed, 50/50 tests. The current E2E red is a maintainer-authorization timeout; exact-SHA credentialed E2E remains required.

@cv cv left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed exact head 6cc245e against current main 1bb15a0. Both base refreshes are Verified, conflict-free, and leave the nine-file effective routing diff unchanged. The merged documentation-receipt workflow does not alter this PR's runtime semantics; prior product-scope and nine-category security PASS conclusions remain valid. Awaiting fresh GitHub-hosted gates.

@apurvvkumaria

Copy link
Copy Markdown
Collaborator Author

E2E diagnosis for run 29964501381: Hermes security-posture job 89073364455 lost GitHub-hosted runner 1021276374 during phase 2/8. The log reports a runner shutdown followed by cancellation; there is no assertion failure, artifact upload, or cleanup. All other selected jobs passed.

This standard-runner failure reinforces the larger-runner/stability work in this PR. It also exposed a #7146 compatibility gap: GitHub now terminalizes the interrupted step as completed/cancelled, while the current detector only accepts an in_progress/null step, so no one-time retry was authorized. The strict API-shape fix and automatic retry follow-up are being developed separately so this routing PR stays focused.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
.github/workflows/e2e.yaml (1)

5189-5189: 🔒 Security & Privacy | 🟠 Major | ⚡ Quick win

Pass needs as data, not JavaScript source.

toJSON(needs) is interpolated into the github-script program. Upstream job outputs can therefore become executable source before the script runs. Parse an environment variable instead.

Proposed fix
         env:
+          NEEDS_JSON: ${{ toJSON(needs) }}
           EXPLICIT_ONLY_JOBS: ${{ needs.generate-matrix.outputs.explicit_only_jobs }}

-            const needs = ${{ toJSON(needs) }};
+            const needs = JSON.parse(process.env.NEEDS_JSON || '{}');

As per path instructions, trusted automation must pass untrusted values as data rather than interpolating them into programs.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/e2e.yaml at line 5189, Update the github-script setup so
needs is passed through an environment variable rather than interpolated into
the JavaScript source. In the script containing the needs assignment, read the
environment value and parse it as JSON before use, preserving the existing needs
data while preventing upstream outputs from becoming executable code.

Sources: Path instructions, Linters/SAST tools

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/e2e/support/live-vitest-invocation.test.ts`:
- Around line 418-422: Update the “a main larger-runner execution” case in the
test matrix so its environment remains distinct after merging with exactHeadEnv,
including the larger-runner label or selector that identifies this execution
path; otherwise remove the duplicate case. Ensure it independently exercises the
larger-runner main-runner conditional rather than matching the manual
main-run-without-SHA case.

---

Outside diff comments:
In @.github/workflows/e2e.yaml:
- Line 5189: Update the github-script setup so needs is passed through an
environment variable rather than interpolated into the JavaScript source. In the
script containing the needs assignment, read the environment value and parse it
as JSON before use, preserving the existing needs data while preventing upstream
outputs from becoming executable code.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: e0992243-d29f-4687-92e2-c4d84adef458

📥 Commits

Reviewing files that changed from the base of the PR and between e135865 and af1cb1c.

📒 Files selected for processing (4)
  • .github/workflows/e2e.yaml
  • test/e2e/README.md
  • test/e2e/support/live-vitest-invocation.test.ts
  • tools/e2e/live-vitest-invocation.mts
🚧 Files skipped from review as they are similar to previous changes (1)
  • test/e2e/README.md

Comment thread test/e2e/support/live-vitest-invocation.test.ts Outdated
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
test/e2e/support/live-vitest-invocation.test.ts (1)

489-526: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Assert cleanup outcomes instead of complete command traces.

These exact fake-command logs lock the tests to probing details unrelated to the claimed behavior. Make the fake fallocate/rm update a test-scoped swap file or marker, then assert removal after successful swapoff and preservation after swapoff or query failure; retain only a narrow ordering assertion if needed.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/e2e/support/live-vitest-invocation.test.ts` around lines 489 - 526,
Update the swap cleanup tests around runHermesSwapScriptFailure to use a
test-scoped fake swap file or marker that the mocked fallocate/rm operations
modify, rather than asserting complete result.calls traces. Assert that the
marker is removed after successful swapoff and preserved when swapoff fails or
active-swap querying fails, retaining only narrowly necessary ordering checks.

Source: Path instructions

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@test/e2e/support/live-vitest-invocation.test.ts`:
- Around line 489-526: Update the swap cleanup tests around
runHermesSwapScriptFailure to use a test-scoped fake swap file or marker that
the mocked fallocate/rm operations modify, rather than asserting complete
result.calls traces. Assert that the marker is removed after successful swapoff
and preserved when swapoff fails or active-swap querying fails, retaining only
narrowly necessary ordering checks.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 619809bc-9167-4c0a-a9e5-9101f63ee24d

📥 Commits

Reviewing files that changed from the base of the PR and between af1cb1c and 73596b8.

📒 Files selected for processing (3)
  • test/e2e/README.md
  • test/e2e/support/live-vitest-invocation.test.ts
  • tools/e2e/live-vitest-invocation.mts
🚧 Files skipped from review as they are similar to previous changes (2)
  • test/e2e/README.md
  • tools/e2e/live-vitest-invocation.mts

@apurvvkumaria

Copy link
Copy Markdown
Collaborator Author

Review follow-up:

  • I verified the outside-diff toJSON(needs) source-interpolation finding. Both sites are present on the PR base, so this PR did not introduce them; the hardening action is now recorded under Extract executable GitHub Actions logic into tested helpers #6952: Extract executable GitHub Actions logic into tested helpers #6952 (comment).
  • The remaining complete-trace assertion note is a non-blocking maintainability nit. For this privileged cleanup path, the command order and the absence of any rm after failed swapoff/state query are the security regression contract. The current executable tests prove those outcomes, so changing their harness is safe to fix forward if the helper contract evolves and does not justify another gate reset here.

@cv
cv merged commit 9ac7ab2 into main Jul 23, 2026
77 of 79 checks passed
@cv
cv deleted the codex/7145-larger-runner-routing-refresh branch July 23, 2026 14:44
@apurvvkumaria

Copy link
Copy Markdown
Collaborator Author

The post-merge PRA-1 trust-boundary finding is addressed by #7444. The fix moves exact-head Hermes swap setup into the trusted workflow before candidate checkout, removes privileged execution from the candidate-side helper, and adds exact workflow-boundary plus failure-cleanup tests. Focused security re-review found no remaining blocker.

cv pushed a commit that referenced this pull request Jul 23, 2026
## Summary

Records bounded start/end CPU, memory, and workspace evidence for every
#7145 larger-runner route so the standard-versus-larger experiment can
compare like-for-like jobs. This is the current-`main` replacement for
the unique telemetry portion of closed #7247; it does not change runner
routing or duplicate #7100 phase, stall, timing, or resource
diagnostics.

## Related Issue

Related to #7145. Supersedes the telemetry portion of #7247.

## Changes

- Write one canonical private numeric sample after workspace preparation
and one from an `always()` finalizer before artifact scanning or upload.
- Cover 12 routed workflow lane identities and 13 concrete executions;
`hermes-inference-switch` contributes its `hosted` and `anthropic`
matrix entries.
- Keep the OpenClaw entries for `mcp-bridge`, `channels-stop-start`, and
`security-posture` uninstrumented because #7391 leaves those entries on
standard runners.
- Summarize measured-window CPU utilization, host-memory endpoints and
root-cgroup lifetime peak, plus workspace endpoints and growth without
invoking Docker, `ps`, or background samplers.
- Reject unknown fields, duplicate JSON keys, malformed timestamps,
identity/counter drift, oversized evidence, symlink/hardlink final-file
substitution, matrix topology drift, and workflow scope/order drift.
- Document comparison controls and limitations. Standard-runner
baselines can begin before #7391 changes routing; the same evidence
remains after larger-runner activation.

## Type of Change

- [ ] Code change (feature, bug fix, or refactor)
- [x] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates

- [x] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [ ] Tests not applicable — justification:
- [x] Docs updated for user-facing behavior changes
- [ ] Docs not applicable — justification:
- [x] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [x] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification: independent review
confirmed the exact trusted-main/no-alternate-checkout boundary,
Hermes-only matrix guards, bounded private schema, and
finalizer-before-scan/upload ordering.
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Documentation Writer Review

- [x] Documentation writer subagent reviewed the completed
implementation
- Result: `docs-updated`
- Evidence: `test/e2e/README.md` documents the 12 routed workflow lane
identities, 13 concrete executions, OpenClaw exclusions, sample
limitations, missing-summary interpretation, and comparison
requirements. The exact-head refresh confirmed the final test-only
review fix requires no further documentation change.
- Agent: Codex Desktop
- PR: #7399
<!-- docs-review-head-sha: c09137d -->
<!-- docs-review-agents-blob-sha: 9061453 -->

## DGX Station Hardware Evidence

- [ ] Tested on DGX Station
- Tested commit: Not applicable
- Station profile/scenario: Not applicable
- Result: Not applicable
- Supporting evidence: Not applicable

## Verification

- [x] PR description includes a `Signed-off-by:` line and every commit
appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run check:diff` passed when hooks were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — focused E2E-support validation passed
102 tests; an independent rerun of the four directly affected suites
passed 94 tests; the final matrix-assertion review fix passed its
focused 28-test suite.
- [ ] Applicable broad gate passed — Full `e2e-support` previously
reached 1,381 passed and 9 skipped; two unrelated existing local
failures remained. GitHub CI is authoritative for the Linux gate.
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only) —
completed with 0 errors and 2 existing Fern warnings
- [ ] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only) — no `docs/` page changed
- [ ] New doc pages include SPDX header and frontmatter (new pages only)
— no new doc page

---

Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>

---------

Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com>
Co-authored-by: Prekshi Vyas <prekshiv@nvidia.com>
@cv cv mentioned this pull request Jul 23, 2026
23 tasks
cv added a commit that referenced this pull request Jul 23, 2026
<!-- markdownlint-disable MD041 -->
## Summary

Adds the canonical dated `## v0.0.93` release entry to
`docs/changelog/2026-07-23.mdx`.
The entry records user-visible behavior, release validation, and
documentation controls merged after `v0.0.92`, while preserving the
pending DGX OS `7.6.x` Station Express qualification caveat.

## Changes

- Adds the parser-safe dated release entry with a summary, grouped
details, and published-route links.
- Reconciles the `v0.0.92..origin/main` commit range with merged
`v0.0.93` PRs.
- Records that no-OTA DGX OS `7.6.x` passed bounded host preflight,
while full Station Express end-to-end qualification remains pending.
- Leaves existing product pages unchanged because the source PRs already
document their supported behavior.

### Source summary

- #7285 -> `docs/changelog/2026-07-23.mdx`: Records the existing-vLLM
ownership choice and resumable Station handoff.
- #7419 -> `docs/changelog/2026-07-23.mdx`: Records bounded no-OTA DGX
OS `7.6.x` recognition and its pending end-to-end qualification.
- #7268 -> `docs/changelog/2026-07-23.mdx`: Records optional Hugging
Face authentication, output sanitization, and resumable HTTP `429`
recovery.
- #7442 -> `docs/changelog/2026-07-23.mdx`: Records clean SIGINT
handling at hidden credential prompts.
- #7299 -> `docs/changelog/2026-07-23.mdx`: Records Intel macOS
rejection before ref resolution or network work.
- #7296 -> `docs/changelog/2026-07-23.mdx`: Records the DGX Spark
non-interactive local-vLLM selection order.
- #7342 -> `docs/changelog/2026-07-23.mdx`: Records delegated protected
E2E approvals in the grouped release-validation bullet.
- #7373 -> `docs/changelog/2026-07-23.mdx`: Records base-image
publication gating before final-main fanout.
- #7388 -> `docs/changelog/2026-07-23.mdx`: Records semantic phase
runtime summaries.
- #7397 -> `docs/changelog/2026-07-23.mdx`: Records progress coverage
hardening.
- #7391 -> `docs/changelog/2026-07-23.mdx`: Records centralized
larger-runner routing.
- #7423 -> `docs/changelog/2026-07-23.mdx`: Records one retry for
confirmed hosted-runner loss.
- #7399 -> `docs/changelog/2026-07-23.mdx`: Records runner-comparison
telemetry.
- #7270 -> `docs/changelog/2026-07-23.mdx`: Records staging Brev
Launchable validation.
- #7426 -> `docs/changelog/2026-07-23.mdx`: Records filtering of
irrelevant base-image run history.
- #7333 -> `docs/changelog/2026-07-23.mdx`: Records aligned Quickstart
platform guidance.
- #7343 -> `docs/changelog/2026-07-23.mdx`: Records documentation-writer
receipt collection.
- #7400 -> `docs/changelog/2026-07-23.mdx`: Records the
documentation-writer receipt requirement for docs-only PRs.
- #7413 -> `docs/changelog/2026-07-23.mdx`: Records removal of redundant
receipt PR metadata.
- #7405 -> `docs/changelog/2026-07-23.mdx`: Records corrected inference
CLI references.
- #7389 -> `docs/changelog/2026-07-23.mdx`: Records completion of the
v0.0.91 documentation audit.

`#7384` is an internal refactor with no intended runtime behavior
change.
`#7401` updates internal CodeQL Actions dependencies.
`#7376` is already contained in `v0.0.92`, so it is outside the
release-entry scan range despite its retained planning label.

## Type of Change

- [ ] Code change (feature, bug fix, or refactor)
- [ ] Code change with doc updates
- [x] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates

- [ ] Tests added or updated for changed behavior
- [x] Existing tests cover changed behavior — justification:
`test/changelog-docs.test.ts` validates dated changelog structure, SPDX
syntax, and version headings.
- [ ] Tests not applicable — justification:
- [x] Docs updated for user-facing behavior changes
- [ ] Docs not applicable — justification:
- [ ] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [ ] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification:
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Documentation Writer Review

- [x] Documentation writer subagent reviewed the completed changes
- Result: `docs-updated`
- Evidence: Reviewed `docs/changelog/2026-07-23.mdx` against
`WRITING.md`, `docs/CONTRIBUTING.md`, `docs/.docs-skip`,
`docs/index.yml`, the six user-visible source PRs, and the remaining
grouped release commits. The review corrected an ambiguous qualification
claim, confirmed all published routes, preserved the DGX OS `7.6.x`
caveat, and found no remaining action.
- Agent: Codex Desktop
<!-- docs-review-head-sha: ec0a866 -->
<!-- docs-review-agents-blob-sha: 9c9b36d -->

## DGX Station Hardware Evidence

- [ ] Tested on DGX Station
- Tested commit: Not applicable. This PR does not change
`scripts/prepare-dgx-station-host.sh`.
- Station profile/scenario: Not applicable.
- Result: Not applicable.
- Supporting evidence: Not applicable.

## Verification

- [x] PR description includes a `Signed-off-by:` line and every commit
appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run check:diff` passed when hooks were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — `npx vitest run
test/changelog-docs.test.ts`: 1 file and 6 tests passed.
- [ ] Applicable broad gate passed — Not applicable to one native
changelog file.
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only) —
completed with 0 errors and 2 existing Fern warnings.
- [x] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)
— not applicable because native changelog entries use a parser-safe MDX
SPDX comment without frontmatter.

---
Signed-off-by: Carlos Villela <cvillela@nvidia.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Documentation**
* Added the v0.0.93 changelog covering onboarding and validation
improvements.
* Documented support for additional DGX Station Express workstation
releases and clearer handling of existing vLLM workloads.
* Added guidance for optional Hugging Face authentication, resumable
rate-limit recovery, and DGX Spark provider selection.
* Clarified installer behavior on Intel macOS, release validation
requirements, hosted-runner retries, documentation checks, and supported
CLI quickstart paths.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
prekshivyas added a commit that referenced this pull request Jul 24, 2026
<!-- markdownlint-disable MD041 -->
## Summary

Stages exact-head Hermes swap provisioning in the trusted `main`
workflow before checkout. This phase keeps the reviewed candidate helper
only so the workflow change can validate under the older trusted
workflow currently on `main`; after this lands, a focused follow-up will
remove that compatibility helper and complete the #7391 trust-boundary
fix.

## Related Issue

Part of #7145. Security fix-forward for #7391.

## Changes

- Provision at least 32 GiB of usable swap before checkout for the eight
exact-head, GitHub-hosted Hermes lanes that rebuild or exercise the
Hermes image.
- Require the canonical repository, `workflow_dispatch`,
`refs/heads/main`, an exact trusted workflow SHA, the generated
controller matrix, and explicit trusted Hermes target selection before
privileged setup can run.
- Use a fixed trusted program with absolute executables, a clean shell
environment, a root-owned randomized file, a 16 GiB disk reserve,
bounded activation-visibility polling, and fail-closed cleanup.
- Temporarily retain the exact-head candidate helper for rollout
compatibility, but allocate 32 GiB + 4,096 bytes so formatting still
leaves the required 32 GiB usable. Once the trusted workflow is on
`main`, the helper observes sufficient swap and exits before creating
its fixed file.
- Pin the eligible job inventory, ordering, controller dependency, step
shape, environment, and exact program bytes with a workflow-boundary
validator.
- Add mutation and executable cleanup tests proving candidate changes
cannot alter the root command and partial failures do not delete active
swap.
- Document the trusted boundary and hosted-runner capacity behavior in
`test/e2e/README.md`.

The larger-runner route cannot directly replace this fallback for
exact-head validation because the trusted controller intentionally
refuses candidate-selected runner labels. Pre-merge live validation
executes the older trusted workflow from `main`, so this rollout phase
retains the reviewed candidate helper while adding the trusted
pre-checkout step. After this phase lands, a focused follow-up will
delete the helper and its tests; that follow-up's exact-head E2E will
execute the trusted setup from `main`.

## Type of Change

- [ ] Code change (feature, bug fix, or refactor)
- [x] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates

- [x] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [ ] Tests not applicable — justification:
- [ ] Docs updated for user-facing behavior changes
- [x] Docs not applicable — justification: Internal E2E maintainer
behavior is documented in `test/e2e/README.md`; no user-facing behavior
or `docs/` page changes.
- [x] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [x] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification: Independent focused
re-review found no blocker after trusted target selection, controller
sequencing, exact step-shape validation, bounded sizing, and
active-swap-safe failure cleanup were enforced at `947375403`;
`5874616a3` and `1ac224771` add the requested reuse and
successful-provisioning regression tests without changing runtime
behavior.
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Documentation Writer Review

- [x] Documentation writer subagent reviewed the completed changes
- Result: `docs-updated`
- Evidence: `test/e2e/README.md` documents the two-phase rollout,
trusted pre-checkout boundary, temporary exact-head compatibility
helper, 32 GiB usable / 32 GiB + 4,096-byte allocation distinction, five
bounded activation observations, fail-closed cleanup, protected lanes,
and required follow-up removal; no canonical user-doc change is needed.
- Agent: Codex Desktop
<!-- docs-review-head-sha: 763fc6a -->
<!-- docs-review-agents-blob-sha:
9c9b36d -->

## DGX Station Hardware Evidence

- [ ] Tested on DGX Station
- Tested commit:
- Station profile/scenario:
- Result:
- Supporting evidence:

## Verification

- [x] PR description includes a `Signed-off-by:` line and every commit
appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run check:diff` passed when hooks were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — after the rollout, delayed-visibility,
and supported util-linux syntax fixes: trusted boundary 11/11, live
helper 52/52, and E2E workflow contracts 42/42 passed independently;
project membership, source-shape, repository checks, CLI
build/typecheck, docs validation, diff validation, and push hooks
passed.
- [x] Applicable broad gate passed — `npm run checks`, `npm run
typecheck:cli`, `npm run source-shape:check`, `npm run
test:titles:check`, `npm run test:projects:check`, and `npm run
test-size:check` passed.
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only)
- [ ] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

---

Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added “trusted Hermes E2E swap” provisioning for eligible Hermes
end-to-end jobs, gated to trusted main-branch dispatches and run before
code checkout.
* **Documentation**
* Updated the E2E README with trusted-swap routing, disk/swap planning,
and cleanup fail-closed semantics.
* **Bug Fixes**
* Enforced fail-closed workflow boundaries to prevent candidate code
from influencing privileged swap provisioning or step ordering.
* **Tests**
* Added a workflow-boundary suite for trusted Hermes swap
stability/cleanup behavior and updated E2E watch-trigger coverage;
adjusted live Vitest invocation tests.
* **Refactor**
* Simplified the live Vitest invocation flow by removing Hermes swap
provisioning logic from the CLI path.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com>
Co-authored-by: Prekshi Vyas <34834085+prekshivyas@users.noreply.github.com>
Co-authored-by: Prekshi Vyas <prekshiv@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: ci CI workflows, checks, release automation, or GitHub Actions area: e2e End-to-end tests, nightly failures, or validation infrastructure chore Build, CI, dependency, or tooling maintenance integration: dcode LangChain Deep Code integration behavior

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants