Skip to content

perf(e2e): consolidate Hermes credentialless rebuild#7577

Merged
cv merged 2 commits into
mainfrom
codex/7144-hermes-credential-rebuild-consolidation
Jul 26, 2026
Merged

perf(e2e): consolidate Hermes credentialless rebuild#7577
cv merged 2 commits into
mainfrom
codex/7144-hermes-credential-rebuild-consolidation

Conversation

@apurvvkumaria

@apurvvkumaria apurvvkumaria commented Jul 26, 2026

Copy link
Copy Markdown
Collaborator

Summary

The Hermes Discord lane currently performs a second full sandbox rebuild only to prove that stored inference credentials survive when host keys are absent. This change moves that assertion into the canonical rebuild-hermes operation and removes the duplicate Discord rebuild while retaining the lane's unique gateway, token-isolation, REST, placeholder, and cleanup coverage.

Related Issue

Part of #7144
Refs #7140

Changes

  • Run the existing normal and stale-base Hermes rebuild operation without NVIDIA_INFERENCE_API_KEY, COMPATIBLE_API_KEY, or NVIDIA_API_KEY, while preserving the verified current-base reuse override.
  • Prove that the rebuild succeeds from persisted provider state and does not report a missing provider credential.
  • Remove only the redundant rebuild phase from hermes-discord; keep its native Discord Gateway rewrite, REST boundary, config schema, raw-token isolation, residue, and cleanup assertions.
  • Preserve the live-test phase contract and compact finalization artifact numbering.

The #7493 telemetry recorded the removed Discord rebuild at about 153 seconds; the full lane reached an approximately 11.6 GiB Docker/BuildKit process peak. This removes one high-memory build exposure without claiming that the retained initial install cannot reach a similar peak, adding a retry, or weakening product behavior coverage.

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Quality Gates

  • Tests added or updated for changed behavior
  • Existing tests cover changed behavior — justification:
  • Tests not applicable — justification:
  • Docs updated for user-facing behavior changes
  • Docs not applicable — justification: This consolidates internal live E2E ownership without changing NemoClaw commands, supported behavior, configuration, or user workflows.
  • Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
  • Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: Exact-head review at 6b306be12 verified the child-process credential boundary, persisted-provider setup, base-reuse override, retained Discord assertions, and cleanup behavior.
  • Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Documentation Writer Review

  • Documentation writer subagent reviewed the completed changes
  • Result: no-docs-needed
  • Evidence: Exact head 6b306be12 changes only internal live E2E ownership and its regression coverage; no user-facing documentation contract changes.
  • Agent: Codex Desktop

DGX Station Hardware Evidence

  • Tested on DGX Station
  • Tested commit: Not applicable
  • Station profile/scenario: Not applicable
  • Result: Not applicable
  • Supporting evidence: Not applicable

Verification

  • PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
  • Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run check:diff passed when hooks were skipped or unavailable
  • Targeted behavior tests pass for the current change set, or tests are marked not applicable above — 52 focused E2E-support tests passed; semantic phase collection validated 125 tests across 82 files; Biome and test-title checks passed.
  • Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — not run because the change is confined to two live targets and one focused E2E-support contract.
  • Quality Gates section completed with required justifications or waivers
  • No secrets, API keys, or credentials committed
  • npm run docs builds without warnings (doc changes only)
  • Doc pages follow the style guide (doc changes only)
  • New doc pages include SPDX header and frontmatter (new pages only)

Signed-off-by: Apurv Kumaria akumaria@nvidia.com

Summary by CodeRabbit

  • Tests
    • Updated Hermes rebuild validation to run without host inference credentials.
    • Added checks confirming inference credentials are excluded from rebuild environments.
    • Added validation that rebuilds do not report missing provider credentials.
    • Expanded compatibility coverage to ensure unsupported credentials are not forwarded.
    • Removed outdated Discord credential-reuse test coverage.

Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
@apurvvkumaria apurvvkumaria self-assigned this Jul 26, 2026
@apurvvkumaria apurvvkumaria added v0.0.96 Release target area: e2e End-to-end tests, nightly failures, or validation infrastructure area: performance Latency, throughput, resource use, benchmarks, or scaling integration: hermes Hermes integration behavior labels Jul 26, 2026
@coderabbitai

coderabbitai Bot commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 71f2eeb2-31e4-4eb4-a87a-078e1028887d

📥 Commits

Reviewing files that changed from the base of the PR and between f6347bd and 6b306be.

📒 Files selected for processing (3)
  • test/e2e/live/hermes-discord.test.ts
  • test/e2e/live/rebuild-hermes.test.ts
  • test/e2e/support/rebuild-hermes-env.test.ts

📝 Walkthrough

Walkthrough

Changes

Hermes rebuild validation

Layer / File(s) Summary
Rebuild environment filtering
test/e2e/support/rebuild-hermes-env.test.ts, test/e2e/live/rebuild-hermes.test.ts
Rebuild environment tests exclude unsupported inference and NVIDIA credential variables from forwarded child environment data.
Credential-free rebuild validation
test/e2e/live/rebuild-hermes.test.ts
The live Hermes rebuild records the credential-free rebuild boundary and verifies verbose output does not report a missing provider credential.
Discord test scope cleanup
test/e2e/live/hermes-discord.test.ts
The Hermes Discord test removes rebuild credential-reuse coverage, its assertion, and obsolete phase numbering.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Suggested reviewers: jyaunches, cjagwani, ericksoa

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately captures the main change: consolidating Hermes credentialless rebuild coverage and removing the redundant Discord rebuild path.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/7144-hermes-credential-rebuild-consolidation

Comment @coderabbitai help to get the list of available commands.

@github-code-quality

github-code-quality Bot commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in commit 6b306be in the codex/7144-hermes-cr... branch remains at 96%, unchanged from commit dc07773 in the main branch.

TypeScript / code-coverage/cli

The overall coverage in commit 6b306be in the codex/7144-hermes-cr... branch remains at 80%, unchanged from commit f6347bd in the main branch.

Show a code coverage summary of the most impacted files.
File main f6347bd codex/7144-hermes-cr... 6b306be +/-
src/lib/actions...-add-restart.ts 19% 10% -9%
src/lib/actions...on-readiness.ts 100% 91% -9%
src/lib/actions...x/mcp-bridge.ts 43% 36% -7%
src/lib/actions...lution-probe.ts 93% 88% -5%
src/lib/actions...e-validation.ts 84% 81% -3%
src/lib/onboard...shboard-port.ts 93% 90% -3%
src/lib/actions...dbox/destroy.ts 95% 93% -2%
src/lib/onboard...eway-service.ts 82% 81% -1%
src/lib/shields/index.ts 71% 72% +1%
src/lib/platform.ts 84% 89% +5%

Updated July 26, 2026 08:08 UTC

@github-actions

github-actions Bot commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor — No blocking findings reported

Advisor assessment: No blocking advisor findings reported
Next action: No advisor follow-up needed.
Findings: 0 blockers · 0 warnings · 0 suggestions

Model lanes

  • GPT-5.6 Terra (primary): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
  • Nemotron 3 Ultra (second opinion): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
  • Model comparison: normalized findings match; normalized E2E selections match; severity counts match.

Nemotron output stays in workflow artifacts and does not change the assessment above.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: cloud-onboard, credential-sanitization, security-posture, hermes-discord, rebuild-hermes, rebuild-hermes-stale-base

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

@apurvvkumaria

Copy link
Copy Markdown
Collaborator Author

Exact-head CI disposition for 6b306be128632b7726b3b1e2ee418cd05ddf043c:

  • All eight CLI shards, aggregate CLI tests, static/build/type checks, plugin tests, installer integration, and the WeChat runtime audit passed in run 30193745448.
  • The complete self-hosted image/sandbox suite passed in run 30193746678.
  • CodeRabbit and both advisor lanes reported no actionable findings; there are no review threads.
  • The sole primary failure is reviewed-npm-audit: npm returned incomplete metadata (summary and detail were empty) rather than a vulnerability report. The aggregate checks and E2E coordination failures are derivative.

This is the same shared registry failure affecting the adjacent runner-stability PRs and is unrelated to the three Hermes E2E files in this diff. I am not issuing another blind rerun; #7575 is the scoped controlled-retry follow-up.

@cv
cv merged commit b6fbf8f into main Jul 26, 2026
73 of 78 checks passed
@cv
cv deleted the codex/7144-hermes-credential-rebuild-consolidation branch July 26, 2026 08:49
@cv cv mentioned this pull request Jul 26, 2026
23 tasks
apurvvkumaria pushed a commit that referenced this pull request Jul 27, 2026
<!-- markdownlint-disable MD041 -->
## Summary

Add the canonical `docs/changelog/2026-07-25.mdx` release entry with the
exact `## v0.0.96` heading.
The entry reconciles all 90 first-parent commits since v0.0.95 with all
92 merged PRs in the live `v0.0.96` label ledger and groups the
user-visible changes by operator journey.

## Changes

- Add the parser-safe dated MDX changelog entry for v0.0.96 with
root-absolute links to the focused user guides.
- Source summary:
- [#7194](#7194) ->
`docs/changelog/2026-07-25.mdx`: Document persistent baseline network
policy exclusions and their inspection, rebuild, and snapshot behavior.
- [#7188](#7188),
[#7427](#7427), and
[#7546](#7546) ->
`docs/changelog/2026-07-25.mdx`: Document DNS-backed HTTPS inference
routing, keyless loopback endpoints, and provider-marker isolation.
- [#7238](#7238) ->
`docs/changelog/2026-07-25.mdx`: Document blueprint sandbox and provider
identifier validation before state writes or OpenShell calls, with
bounded terminal-safe rejection previews.
- [#7319](#7319),
[#7274](#7274),
[#7528](#7528),
[#7353](#7353), and
[#7560](#7560) ->
`docs/changelog/2026-07-25.mdx`: Document the managed default gateway
service, onboarding readiness, and container-runtime identity
safeguards.
- [#7349](#7349),
[#7498](#7498),
[#7406](#7406),
[#7196](#7196),
[#7559](#7559),
[#7421](#7421),
[#7510](#7510),
[#7295](#7295), and
[#7565](#7565) ->
`docs/changelog/2026-07-25.mdx`: Document gateway-scoped status,
lifecycle diagnostics, managed MCP recovery, delete-edge safeguards, and
fail-closed CLI prompt and command output.
- [#7591](#7591) ->
`docs/changelog/2026-07-25.mdx`: Document opt-in authenticated MCP
tool-name discovery, its bounded and names-only contract, probe
interaction, and rebuild requirement.
- [#7305](#7305),
[#7480](#7480),
[#7471](#7471),
[#7365](#7365), and
[#7541](#7541) ->
`docs/changelog/2026-07-25.mdx`: Document installer version checks,
version-tag reporting, license guidance, WSL Ollama selection, and DGX
Station vLLM detection.
- [#7482](#7482),
[#7466](#7466),
[#7208](#7208),
[#7434](#7434), and
[#7586](#7586) ->
`docs/changelog/2026-07-25.mdx`: Document Ollama resource details,
reasoning precedence, Hermes onboarding behavior, and preserved managed
Hermes BuildKit failures.

- [#6830](#6830),
[#7492](#7492),
[#7563](#7563), and
[#7582](#7582) ->
`docs/changelog/2026-07-25.mdx`: Document the authoritative OpenClaw
production lock, fixed managed-image dependencies, immutable Hermes base
adoption, and Hermes image-size reduction.
- [#7505](#7505),
[#7530](#7530),
[#7547](#7547),
[#7508](#7508),
[#7548](#7548),
[#7549](#7549),
[#7537](#7537),
[#7534](#7534),
[#7515](#7515),
[#7511](#7511),
[#7551](#7551),
[#7562](#7562),
[#7575](#7575),
[#7496](#7496),
[#7594](#7594),
[#7595](#7595), and
[#7599](#7599) ->
`docs/changelog/2026-07-25.mdx`: Summarize release validation, transient
and bounded dispatch reconciliation, exact pre-tag qualification,
identity revalidation, npm-audit retry, sharding, image reuse, timeout,
telemetry, and workflow-hardening changes.
- Reconciled without separate changelog prose:
- [#7539](#7539),
[#7526](#7526),
[#7507](#7507),
[#7506](#7506),
[#7519](#7519),
[#7516](#7516),
[#7396](#7396),
[#7254](#7254),
[#7583](#7583),
[#7596](#7596), and
[#7598](#7598): Test-harness or
fixture-only changes.
- [#7403](#7403),
[#7161](#7161),
[#6877](#6877),
[#7531](#7531),
[#7525](#7525),
[#7522](#7522),
[#7536](#7536),
[#7552](#7552),
[#7566](#7566),
[#7553](#7553),
[#7561](#7561),
[#7577](#7577),
[#7569](#7569),
[#7585](#7585),
[#7584](#7584),
[#7592](#7592),
[#7580](#7580),
[#7571](#7571),
[#7517](#7517),
[#7589](#7589),
[#7402](#7402),
[#7558](#7558),
[#7544](#7544), and
[#7601](#7601): Dependency,
internal recovery, validation, contributor-workflow, E2E optimization,
telemetry, or CI trust changes with no separate user-facing release
claim.
- [#7556](#7556),
[#7573](#7573),
[#7576](#7576), and
[#7578](#7578): Experimental
repository-maintainer conflict automation with no canonical user
documentation surface.

## Type of Change

- [ ] Code change (feature, bug fix, or refactor)
- [ ] Code change with doc updates
- [x] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates

- [ ] Tests added or updated for changed behavior
- [x] Existing tests cover changed behavior — justification:
`test/changelog-docs.test.ts` validates dated changelog structure,
version headings, and published links.
- [ ] Tests not applicable — justification:
- [x] Docs updated for user-facing behavior changes
- [ ] Docs not applicable — justification:
- [ ] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [ ] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification:
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Documentation Writer Review

- [x] Documentation writer subagent reviewed the completed changes
- Result: `docs-updated`
- Evidence: Reviewed `docs/changelog/2026-07-25.mdx` at exact head
`0f5dedb47` against 90 first-parent release commits and 92 merged PRs
labeled `v0.0.96`. Verified parser-safe MDX SPDX, the exact version
heading, literal CLI names, writing style, skip terms, all 20
root-absolute published links, and the accepted #7591 opt-in
authenticated discovery bounds. #7544, #7599, and #7601 remain internal
or CI-only release-ledger entries. Changelog tests passed 6/6, the docs
build passed with 0 errors and two pre-existing Fern warnings, and `npm
run check:diff` plus the final diff check passed.
- Agent: Codex Desktop documentation-writer subagent
<!-- docs-review-head-sha: 0f5dedb -->
<!-- docs-review-agents-blob-sha: be20a09 -->

## DGX Station Hardware Evidence

- [ ] Tested on DGX Station
- Tested commit:
- Station profile/scenario:
- Result:
- Supporting evidence:

## Verification

- [x] PR description includes a `Signed-off-by:` line and every commit
appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run check:diff` passed when hooks were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — `npx vitest run
test/changelog-docs.test.ts`: 6/6 passed.
- [ ] Applicable broad gate passed — `npm test` for broad
runtime/test-harness changes; `npm run check` for repo-wide
validation/coverage changes — command/result: Not applicable to this
prose-only changelog entry.
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only) — the
build passed with 0 errors and 2 existing Fern warnings; the
published-route check passed.
- [x] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)
— native changelog files use the required parser-safe MDX SPDX comment
and no frontmatter.

---
Signed-off-by: Carlos Villela <cvillela@nvidia.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Persistent network policy exclusions with consistent restore/exclusion
reporting across rebuilds/snapshots.
* Opt-in MCP tool discovery via `mcp status --tools` with bounded,
redacted authenticated traffic.
* Improved HTTPS inference switching for custom endpoints and refreshed
onboarding/model menu details.
* Refined OpenShell gateway defaults for port `8080`, including more
reliable readiness checks.
* **Bug Fixes**
* Prevent incorrect provider/model restoration after compatible-provider
update failures.
* Preserve managed MCP state after exec loss and tighten gateway/doctor
status scoping.
* **Tests**
* Stronger, fail-closed release validation with hardened
evidence/artifact handoff and bounded timeouts/retries.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com>
Co-authored-by: Prekshi Vyas <prekshiv@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: e2e End-to-end tests, nightly failures, or validation infrastructure area: performance Latency, throughput, resource use, benchmarks, or scaling integration: hermes Hermes integration behavior v0.0.96 Release target

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants