ci(e2e): isolate long-running scenario lanes#7448
Conversation
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
📝 WalkthroughWalkthroughE2E workflows now run ChangesE2E scenario execution and validation
Estimated code review effort: 3 (Moderate) | ~25 minutes Sequence Diagram(s)sequenceDiagram
participant GitHubActions
participant Vitest
participant ArtifactUpload
GitHubActions->>Vitest: Run matrix.selector for each scenario
Vitest->>ArtifactUpload: Produce scenario-scoped output
GitHubActions->>ArtifactUpload: Upload scenario-named artifacts
Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
Code Coverage OverviewLanguages: TypeScript TypeScript / code-coverage/pluginThe overall coverage in commit a409fcc in the TypeScript / code-coverage/cliThe overall coverage in commit a409fcc in the Show a code coverage summary of the most impacted files.
Updated |
PR Review Advisor — InformationalAdvisor assessment: Informational / high confidence Model lanes
Nemotron output stays in workflow artifacts and does not change the assessment above. E2E guidanceAdvisory only. E2E / PR Gate selects and runs jobs independently. Recommended E2E: 2 optional E2E recommendations
This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge. |
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
There was a problem hiding this comment.
🧹 Nitpick comments (1)
test/e2e/support/e2e-cleanup-resources.test.ts (1)
112-129: 📐 Maintainability & Code Quality | 🔵 Trivial | 🏗️ Heavy liftReplace source-text assertions with behavioral contract tests.
The
indexOfcomparison can pair unrelated occurrences, while the timeout checks lock in private constant spelling rather than proving cleanup reserve behavior.
test/e2e/support/e2e-cleanup-resources.test.ts#L112-L129: use a fake host/installer boundary and assert cleanup is registered before an installer failure.test/e2e/support/e2e-workflow.test.ts#L81-L82: assert the network-policy timing budget through an explicit, validated contract rather than file text.test/e2e/support/e2e-workflow.test.ts#L127-L128: assert the common-egress timing budget through the same public contract.As per path instructions, “Prefer observable outcomes through the public boundary over source-text, private-shape, or mock-call assertions.”
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@test/e2e/support/e2e-cleanup-resources.test.ts` around lines 112 - 129, Replace the source-text ordering test in test/e2e/support/e2e-cleanup-resources.test.ts:112-129 with a fake host/installer boundary that verifies cleanup registration occurs before an installer failure. In test/e2e/support/e2e-workflow.test.ts:81-82 and :127-128, replace timeout/source-text checks with explicit validated assertions through the public timing contract for the network-policy and common-egress reserve budgets, respectively; update each site as requested and avoid private constant names or mock-call assertions.Source: Path instructions
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@test/e2e/support/e2e-cleanup-resources.test.ts`:
- Around line 112-129: Replace the source-text ordering test in
test/e2e/support/e2e-cleanup-resources.test.ts:112-129 with a fake
host/installer boundary that verifies cleanup registration occurs before an
installer failure. In test/e2e/support/e2e-workflow.test.ts:81-82 and :127-128,
replace timeout/source-text checks with explicit validated assertions through
the public timing contract for the network-policy and common-egress reserve
budgets, respectively; update each site as requested and avoid private constant
names or mock-call assertions.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 42c41052-00ec-458e-a209-9567a54c6974
📒 Files selected for processing (10)
.github/workflows/e2e.yamltest/e2e/README.mdtest/e2e/support/e2e-cleanup-resources.test.tstest/e2e/support/e2e-workflow.test.tstest/e2e/support/prepare-e2e-workflow-boundary.test.tstest/e2e/support/upload-e2e-artifacts-workflow-boundary.test.tstest/pr-e2e-gate-signal-shards.test.tstools/e2e/prepare-e2e-workflow-boundary.mtstools/e2e/upload-e2e-artifacts-workflow-boundary.mtstools/e2e/workflow-boundary.mts
<!-- markdownlint-disable MD041 --> ## Summary This adds bounded periodic and phase-aware resource telemetry to the canonical runner-comparison ledger for trusted heavy E2E executions. It reuses the existing progress pulse and records only numeric or fixed-enum evidence, improving runner-pressure diagnosis without changing failure classification or retry policy. ## Related Issue Related to #7145 and #7146 (parent epic: #7140). ## Changes - Extend the canonical runner-comparison artifact to a bounded v2 lifecycle with private `0600` files, a reserved final sample, v1 read compatibility, and deterministic summaries. - Capture bounded host, cgroup, PSI, filesystem, Docker, container, and coarse process-class evidence at endpoints, phase boundaries, and an approximately 60-second cadence. - Reuse the existing progress timer with fixed deadlines, no catch-up bursts, per-scenario attribution boundaries, and a permanent fallback to the existing bounded collector if canonical sampling fails. - Add focused privacy, compatibility, timing, fallback, aggregation, summary, and workflow-boundary coverage, including the three-shard `common-egress-agent` topology merged in #7448. - Document the telemetry lifecycle, probe budgets, privacy limits, compatibility behavior, and the separation from classification and retry. This builds on the merged progress/timing work from #7100 and #7213 and extends the canonical ledger from #7399 rather than creating parallel infrastructure. ## Type of Change - [ ] Code change (feature, bug fix, or refactor) - [x] Code change with doc updates - [ ] Doc only (prose changes, no code sample modifications) - [ ] Doc only (includes code sample changes) ## Quality Gates <!-- Check one tests line and one docs line. Check other lines when applicable. Add every requested justification or approval reference. --> - [x] Tests added or updated for changed behavior - [ ] Existing tests cover changed behavior — justification: - [ ] Tests not applicable — justification: - [x] Docs updated for user-facing behavior changes - [ ] Docs not applicable — justification: - [x] Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging) - [x] Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: Independent exact-head privacy, schema, classifier, retry, and compatibility review at `419182d89` found no actionable issues. - [ ] Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue: ## Documentation Writer Review <!-- Required for code and documentation changes after the changes and applicable validation are complete. Keep one review checkbox and one instance of each visible or hidden field. For Evidence, list changed documentation paths. For documentation-only changes, also state that the writing rules and documentation style were reviewed. For other results, explain why no documentation change is needed or why the review is blocked. For Agent, use a consistent product and surface name, such as Codex Desktop, Codex CLI, Claude Code, or Cursor. After committing all review changes, put `git rev-parse --short HEAD` and `git rev-parse --short HEAD:AGENTS.md` in the hidden metadata below. Rerun the review and refresh that metadata after any new commit. This receipt is advisory during the data-collection pilot. --> - [x] Documentation writer subagent reviewed the completed changes - Result: `docs-updated` - Evidence: `test/e2e/README.md` - Agent: Codex Desktop <!-- docs-review-head-sha: e06653c --> <!-- docs-review-agents-blob-sha: 9c9b36d --> ## DGX Station Hardware Evidence <!-- Required only when scripts/prepare-dgx-station-host.sh changes. Maintainers must review the linked evidence before approving or merging. This is human-reviewed evidence, not authenticated hardware provenance. Exceptional bypasses use existing repository governance and must be documented on the PR. --> - [ ] Tested on DGX Station - Tested commit: - Station profile/scenario: - Result: - Supporting evidence: ## Verification <!-- Check each applicable item only when supported by the requested evidence. Run targeted tests once per relevant change set and rerun after later edits or hook autofixes that can affect the tested behavior. Do not rerun hook-covered checks. --> - [x] PR description includes a `Signed-off-by:` line and every commit appears as `Verified` in GitHub - [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or `npm run check:diff` passed when hooks were skipped or unavailable - [x] Targeted behavior tests pass for the current change set, or tests are marked not applicable above — command/result or justification: Exact-head serialized telemetry/progress/workflow suite passed 225/225 tests across 15 files; after the linear-test guardrail repair, the primary focused suite passed 165/165 across 9 files, Vitest membership was exact across 1,764 files, CLI typecheck passed, and `npm run check:diff` passed. - [ ] Applicable broad gate passed — `npm test` for broad runtime/test-harness changes; `npm run check` for repo-wide validation/coverage changes — command/result: A broad E2E-support run exposed only parallel-load timeouts and one macOS lifecycle failure reproduced unchanged on `origin/main`; scoped exact-head gates are listed above. - [x] Quality Gates section completed with required justifications or waivers - [x] No secrets, API keys, or credentials committed - [ ] `npm run docs` builds without warnings (doc changes only) - [x] Doc pages follow the [style guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md) (doc changes only) - [ ] New doc pages include SPDX header and frontmatter (new pages only) --- <!-- DCO sign-off is required in this PR description, and every commit must appear as Verified in GitHub. Run: git config user.name && git config user.email --> Signed-off-by: Apurv Kumaria <akumaria@nvidia.com> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added runner-comparison schema v2 telemetry as a bounded, ordered ledger with periodic, scenario-start, phase, and finalize samples. * Enabled live E2E runner-comparison sampling and added conditional periodic sampling in the E2E progress harness. * Introduced pulse-based progress resource sampling with configurable interval and best-effort phase/scenario capture. * **Bug Fixes / Improvements** * Switched memory diagnostics to available-memory reporting (with free-memory fallback labeling). * Hardened telemetry parsing/validation, evidence sanitization, and expanded runner-pressure/resource diagnostics. * **Documentation** * Documented v2 ordering/cadence, privacy/size limits, and diagnostic-only summary guidance. * **Tests** * Added/expanded E2E tests for v2 schema validation, timing/cadence, fallback behavior, and updated expected outputs. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Apurv Kumaria <akumaria@nvidia.com> Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com> Co-authored-by: Prekshi Vyas <prekshiv@nvidia.com>
<!-- markdownlint-disable MD041 --> ## Summary This PR adds the canonical dated release entry for NemoClaw v0.0.94 before the tag is cut. The entry reconciles all 26 commits since v0.0.93 and links each user-visible change to its owning documentation. ## Changes - Add `docs/changelog/2026-07-24.mdx` with the exact `## v0.0.94` heading, parser-safe SPDX comment, release summary, and detailed bullets. - Record sandbox restore and update behavior, onboarding and inference changes, network policy behavior, security evidence, Hermes build performance, DGX Station guidance, and E2E validation changes. - Preserve `docs/` as the source of truth without changing the AI-agent documentation routing skill. - Use [E2E run 30075443016](https://github.com/NVIDIA/NemoClaw/actions/runs/30075443016) for release QA evidence at exact tested SHA `04e6dfa883071dda9df429c66e73168e1a995cba`. ### Source summary - [#7461](#7461) -> `docs/changelog/2026-07-24.mdx`: Record the ownership-preserving Hermes image layer reduction and hosted timing comparison. - [#7460](#7460) -> `docs/changelog/2026-07-24.mdx`: Record removal of candidate Hermes swap setup from E2E validation. - [#7458](#7458) -> `docs/security/fern-5.80.1-dependency-review.md`, `docs/changelog/2026-07-24.mdx`: Record the reviewed Fern CLI update. - [#7457](#7457) -> `docs/changelog/2026-07-24.mdx`: Record periodic runner-pressure telemetry. - [#7455](#7455) -> `docs/changelog/2026-07-24.mdx`: Record non-blocking absent Fern previews. - [#7450](#7450) -> `docs/changelog/2026-07-24.mdx`: Record stable cancellation handling for live-test child processes. - [#7449](#7449) -> `docs/changelog/2026-07-24.mdx`: Record parallel plugin EXDEV coverage. - [#7448](#7448) -> `docs/changelog/2026-07-24.mdx`: Record isolated long-running E2E lanes. - [#7444](#7444) -> `docs/changelog/2026-07-24.mdx`: Record exact-head Hermes swap validation. - [#7437](#7437) -> `docs/manage-sandboxes/backup-restore.mdx`, `docs/changelog/2026-07-24.mdx`: Record gateway pairing and authenticated verification after cross-sandbox restore. - [#7436](#7436) -> `docs/manage-sandboxes/backup-restore.mdx`, `docs/reference/commands.mdx`, `docs/changelog/2026-07-24.mdx`: Record selected stale-state cleanup and Hermes virtual-environment access repair. - [#7385](#7385) -> `docs/network-policy/customize-network-policy.mdx`, `docs/changelog/2026-07-24.mdx`: Record the read-only agent-variant route check. - [#7371](#7371) -> `docs/changelog/2026-07-24.mdx`: Record host-artifact verification for session exports. - [#7359](#7359) -> `docs/changelog/2026-07-24.mdx`: Record platform validation for managed vLLM model overrides. - [#7356](#7356) -> `docs/changelog/2026-07-24.mdx`: Record token-shaped value redaction for `sandbox doctor --json`. - [#7354](#7354) -> `docs/security/advisory-early-warning.md`, `docs/changelog/2026-07-24.mdx`: Record advisory correlation and retained audit provenance. - [#7352](#7352) -> `docs/network-policy/customize-network-policy.mdx`, `docs/network-policy/integration-policy-examples.mdx`, `docs/reference/commands.mdx`, `docs/changelog/2026-07-24.mdx`: Record preset reapplication and bounded `tls: skip` guidance. - [#7345](#7345) -> `docs/security/openclaw-2026.6.10-dependency-review.md`, `docs/security/openclaw-2026.7.1-dependency-review.md`, `docs/changelog/2026-07-24.mdx`: Record reviewed npm audit exception enforcement. - [#7340](#7340) -> `docs/network-policy/customize-network-policy.mdx`, `docs/changelog/2026-07-24.mdx`: Record the repaired CLI-reference route. - [#7334](#7334) -> `docs/get-started/dgx-station-preparation.mdx`, `docs/changelog/2026-07-24.mdx`: Record the qualified OTA metadata fallback and narrowed override wording. - [#7322](#7322) -> `docs/changelog/2026-07-24.mdx`: Reconcile the gateway source tag added to plugin registration banners. - [#7284](#7284) -> `docs/manage-sandboxes/update-sandboxes.mdx`, `docs/changelog/2026-07-24.mdx`: Record read-only `upgrade-sandboxes --check` behavior and recorded-gateway selection. - [#7277](#7277) -> `docs/changelog/2026-07-24.mdx`: Reconcile deterministic gateway TCP refusal coverage. - [#7234](#7234) -> `docs/reference/troubleshooting.mdx`, `docs/changelog/2026-07-24.mdx`: Record preserved DGX Spark managed vLLM Express intent on resume. - [#7185](#7185) -> `docs/reference/troubleshooting.mdx`, `docs/changelog/2026-07-24.mdx`: Record IPv4 fallback DNS selection and exact resolver probing. - [#6820](#6820) -> `docs/reference/commands.mdx`, `docs/changelog/2026-07-24.mdx`: Record the versioned, redacted `--events=jsonl` onboarding stream. ## Type of Change - [ ] Code change (feature, bug fix, or refactor) - [ ] Code change with doc updates - [x] Doc only (prose changes, no code sample modifications) - [ ] Doc only (includes code sample changes) ## Quality Gates - [ ] Tests added or updated for changed behavior - [x] Existing tests cover changed behavior — justification: `npx vitest run test/changelog-docs.test.ts` passed 6/6 tests. - [ ] Tests not applicable — justification: - [x] Docs updated for user-facing behavior changes - [ ] Docs not applicable — justification: - [ ] Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging) - [ ] Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: - [ ] Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue: ## Documentation Writer Review - [x] Documentation writer subagent reviewed the completed changes - Result: `docs-updated` - Evidence: `docs/changelog/2026-07-24.mdx`; the writing rules, documentation style, exact release range, skip terms, published routes, and product scope were reviewed; the changelog test passed 6/6; `npm run docs` passed with route checking OK, zero errors, and two existing warnings. - Agent: Codex Desktop <!-- docs-review-head-sha: 65368f9 --> <!-- docs-review-agents-blob-sha: 9c9b36d --> ## DGX Station Hardware Evidence - [ ] Tested on DGX Station - Tested commit: Not applicable - Station profile/scenario: Not applicable - Result: Not applicable - Supporting evidence: Not applicable. This PR does not change `scripts/prepare-dgx-station-host.sh`. ## Verification - [x] PR description includes a `Signed-off-by:` line and every commit appears as `Verified` in GitHub - [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or `npm run check:diff` passed when hooks were skipped or unavailable - [x] Targeted behavior tests pass for the current change set, or tests are marked not applicable above — `npx vitest run test/changelog-docs.test.ts` passed 6/6 tests. - [ ] Applicable broad gate passed — `npm test` for broad runtime/test-harness changes; `npm run check` for repo-wide validation/coverage changes — command/result: Not applicable to the dated changelog entry. - [x] Quality Gates section completed with required justifications or waivers - [x] No secrets, API keys, or credentials committed - [ ] `npm run docs` builds without warnings (doc changes only). The build passed with zero errors and two existing Fern warnings. - [x] Doc pages follow the [style guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md) (doc changes only) - [ ] New doc pages include SPDX header and frontmatter (new pages only). Native dated changelog entries use the required parser-safe MDX SPDX comment and no frontmatter. --- Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Added the v0.0.94 release changelog. * Documented improvements to sandbox snapshot and restore behavior. * Added updates for gateway selection, policy comparisons, onboarding event output, and DGX recovery workflows. * Documented enhanced diagnostics redaction, npm audit provenance, image assembly performance, and validation stability improvements. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com>
Summary
Reduces the resource lifetime and failure blast radius of long-running network-policy and common-egress coverage, while removing one duplicate CLI build from security-posture setup. This is a draft for measuring the additional runner fan-out; it complements runner-loss handling but does not claim that sharding fixes hosted-runner disappearance.
Related Issue
Part of #7146. Parent epic: #7140.
Changes
test/e2e/README.md.Type of Change
Quality Gates
Documentation Writer Review
docs-updatedtest/e2e/README.mdaccurately documents the updated 15-execution telemetry contract and the three common-egress scenario shards; the remaining changes affect only CI workflows, E2E isolation, and internal tests, so no canonicaldocs/page is needed.DGX Station Hardware Evidence
Verification
Signed-off-by:line and every commit appears asVerifiedin GitHubpre-commit,commit-msg, andpre-pushhooks passed, ornpm run check:diffpassed when hooks were skipped or unavailablenpm testfor broad runtime/test-harness changes;npm run checkfor repo-wide validation/coverage changes — command/result: Targeted workflow, selector, semantic, type, Markdown, and diff-scoped gates passed; GitHub CI is authoritative for the repository-wide gate.npm run docsbuilds without warnings (doc changes only)Signed-off-by: Apurv Kumaria akumaria@nvidia.com
Summary by CodeRabbit
CI / E2E Improvements
Validation & Reliability