ci(e2e): record runner comparison telemetry#7399
Merged
Merged
Conversation
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
apurvvkumaria
added
area: ci
21 tasks
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdds bounded runner-comparison telemetry for selected E2E jobs, including secure JSONL artifacts, summary generation, workflow-boundary validation, integration tests, and documentation. ChangesRunner comparison telemetry
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant E2EJob
participant RunnerComparisonCLI
participant RunnerComparisonCore
participant ArtifactFiles
E2EJob->>RunnerComparisonCLI: initialize
RunnerComparisonCLI->>RunnerComparisonCore: collect initial sample
RunnerComparisonCLI->>ArtifactFiles: write private JSONL ledger
E2EJob->>RunnerComparisonCLI: finalize
RunnerComparisonCLI->>ArtifactFiles: read ledger
RunnerComparisonCLI->>RunnerComparisonCore: collect final sample and summarize
RunnerComparisonCLI->>ArtifactFiles: write ledger and summary
Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
Contributor
Code Coverage OverviewLanguages: TypeScript TypeScript / code-coverage/pluginThe overall coverage in commit c09137d in the TypeScript / code-coverage/cliThe overall coverage in commit c09137d in the Show a code coverage summary of the most impacted files.
Updated |
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@test/e2e/support/runner-comparison.test.ts`:
- Around line 302-319: Split the parametrized “rejects a ledger replacement”
test into two focused tests, one for symlink replacement and one for hardlink
replacement, removing the runtime if/else branch while preserving the existing
setup, finalize failure assertion, and cleanup behavior in each test.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: f801db50-f2a3-4fdc-abd1-eae10571c0d0
📒 Files selected for processing (9)
.github/workflows/e2e.yamltest/e2e/README.mdtest/e2e/support/runner-comparison-workflow-boundary.test.tstest/e2e/support/runner-comparison.test.tstools/e2e/private-file.mtstools/e2e/runner-comparison-core.mtstools/e2e/runner-comparison-workflow-boundary.mtstools/e2e/runner-comparison.mtstools/e2e/workflow-boundary.mts
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
Contributor
PR Review Advisor — InformationalAdvisor assessment: Informational / high confidence Model lanes
Nemotron output stays in workflow artifacts and does not change the assessment above. E2E guidanceAdvisory only. E2E / PR Gate selects and runs jobs independently. Recommended E2E: 3 optional E2E recommendations
This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge. |
This was referenced Jul 22, 2026
apurvvkumaria
added
area: observability
Collaborator
Author
|
E2E diagnosis for run 29965049603: the only failing job, Hermes security-posture 89074697099, did not report a test assertion. GitHub-hosted runner 1021277393 disappeared during the live Vitest step; GitHub recorded that step as completed/cancelled, skipped artifact upload and cleanup, and then closed the job as failure. This is the same hosted-runner-loss signature independently observed on #7391 run 29964501381. It is not caused by the telemetry change. The current #7168 classifier only recognizes the older in_progress/null step shape, so the controller omitted the intended runner-loss retry marker. The compatibility and automatic one-time retry gaps are being tracked under #7146. |
22 tasks
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
Contributor
There was a problem hiding this comment.
🧹 Nitpick comments (1)
test/e2e/support/runner-comparison-workflow-boundary.test.ts (1)
60-65: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAvoid duplicating the production matrix parser in the test.
matrixValuesmirrorstools/e2e/runner-comparison-workflow-boundary.mts:60-65. If direct-array orincludehandling regresses, this test can reproduce the same defect and still pass. Prefer independent assertions over the concrete workflow matrix values rather than copying the production algorithm.As per path instructions, E2E support tests should prioritize behavioral confidence and flag copied production algorithms.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@test/e2e/support/runner-comparison-workflow-boundary.test.ts` around lines 60 - 65, Remove the duplicated parsing logic from the test helper matrixValues. Replace its use with independent assertions against the concrete workflow matrix values, covering the expected direct-array and include entries without reproducing production selection or fallback behavior.Source: Path instructions
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@test/e2e/support/runner-comparison-workflow-boundary.test.ts`:
- Around line 60-65: Remove the duplicated parsing logic from the test helper
matrixValues. Replace its use with independent assertions against the concrete
workflow matrix values, covering the expected direct-array and include entries
without reproducing production selection or fallback behavior.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 44ad7027-1467-4baf-9b71-a306d88fc60f
📒 Files selected for processing (4)
.github/workflows/e2e.yamltest/e2e/README.mdtest/e2e/support/runner-comparison-workflow-boundary.test.tstools/e2e/runner-comparison-workflow-boundary.mts
🚧 Files skipped from review as they are similar to previous changes (2)
- test/e2e/README.md
- tools/e2e/runner-comparison-workflow-boundary.mts
Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com>
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
This was referenced Jul 23, 2026
23 tasks
cv
approved these changes
Jul 23, 2026
cv
added a commit
that referenced
this pull request
Jul 23, 2026
<!-- markdownlint-disable MD041 --> ## Summary The PR E2E gate now creates one fresh child run when the first child fails only because a standard GitHub-hosted runner disappeared. It recognizes both GitHub's canonical lost-communication annotation and the exact shutdown-log signature observed in the repeated Hermes cancellations. Assertion, timeout, OOM, policy, mixed, custom-runner, and incomplete-evidence failures remain terminal. ## Related Issue Part of #7146. Parent epic: #7140. ## Changes - Require a completed failed first child run, exact workflow identity, complete Jobs and Checks pagination, and either the canonical SHA-bound lost-runner annotation or one exact generic-cancellation annotation plus authenticated terminal shutdown evidence. - Authenticate the fallback through GitHub's job-log API without forwarding the repository token to storage. Allowlist the signed HTTPS result host, require a strong ETag and exact bounded range, and accept only the terminal runner shutdown/cleanup block bound to the cancelled step and job timestamps. Cap annotation count, per-field bytes, and normalized aggregate evidence. - Require every non-passing job to match the standard `ubuntu-latest` GitHub Actions runner, runner group, check, annotation, and terminal step-shape contract. Re-read and fingerprint all run, check, annotation, and optional log evidence before dispatch. - Create a fresh child run and replacement required check for the one retry. Preserve the first run, check, state, plan, and evidence as linked diagnostic history. - Revalidate the PR SHA, base SHA, workflow SHA, risk plan, state, and retry history before each privileged mutation. - Keep coordination checks immutable across base identities so a controller that resumes after a rapid retarget cannot fail the newer base's gate. - Terminalize the second attempt and clean partial replacement state after failure, cancellation, skipped setup, interruption, or response loss. The required observer follows the latest exact check. - Serialize gate mutations by repository, PR number, PR SHA, and base SHA. Keep up to 100 pending exact-identity mutations in GitHub's FIFO queue without canceling their owner. - Bound each child wait at 140 minutes, the controller at 330 minutes, the observer at 358 minutes, and the required job at 360 minutes. - Document the strict retry criteria, lineage, cleanup behavior, and timeout envelope in `test/e2e/README.md`. - Keep merged #7100 progress telemetry and #7399 resource telemetry unchanged. This PR does not add a second sampler or retry final-main runs. ## Type of Change - [ ] Code change (feature, bug fix, or refactor) - [x] Code change with doc updates - [ ] Doc only (prose changes, no code sample modifications) - [ ] Doc only (includes code sample changes) ## Quality Gates - [x] Tests added or updated for changed behavior - [ ] Existing tests cover changed behavior — justification: - [ ] Tests not applicable — justification: - [x] Docs updated for user-facing behavior changes - [ ] Docs not applicable — justification: - [x] Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging) - [x] Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: an independent 9-category audit passed at `697d78e5f`; it verified annotation and log authentication, token isolation, bounded transport, spoof rejection, immutable retry lineage, two-pass evidence revalidation, cross-base check ownership, cleanup, timeout arithmetic, and FIFO mutation ownership with no unresolved finding. - [ ] Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue: ## Documentation Writer Review - [x] Documentation writer subagent reviewed the completed changes - Result: `docs-updated` - Evidence: `test/e2e/README.md` - Agent: Codex Desktop - PR: #7423 <!-- docs-review-head-sha: d1f8247 --> <!-- docs-review-agents-blob-sha: 560ff38 --> ## DGX Station Hardware Evidence - [ ] Tested on DGX Station - Tested commit: Not applicable - Station profile/scenario: Not applicable - Result: Not applicable - Supporting evidence: Not applicable ## Verification - [x] PR description includes a `Signed-off-by:` line and every commit appears as `Verified` in GitHub - [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or `npm run check:diff` passed when hooks were skipped or unavailable - [x] Targeted behavior tests pass for the current change set, or tests are marked not applicable above — 14 PR-gate integration files passed 244 tests at `697d78e5f`; the focused classifier/transport suites passed 72 tests; the test-only review fix passed all 26 controller tests at `ac430dcf7`; the documented fallback lifecycle passed docs validation and all 39 classifier tests at `d1f824728`; CLI and JavaScript type checking, changed-file hooks, source-shape, test-size, Biome, YAML, Markdown, gitleaks, and diff checks passed. - [ ] Applicable broad gate passed — `npm test` for broad runtime/test-harness changes; `npm run check` for repo-wide validation/coverage changes — targeted controller coverage is authoritative; GitHub CI will run the repository gate. - [x] Quality Gates section completed with required justifications or waivers - [x] No secrets, API keys, or credentials committed - [ ] `npm run docs` builds without warnings (doc changes only) — passed with 0 errors and 2 existing Fern warnings - [ ] Doc pages follow the [style guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md) (doc changes only) — no `docs/` page changed - [ ] New doc pages include SPDX header and frontmatter (new pages only) — no new doc page --- Signed-off-by: Apurv Kumaria <akumaria@nvidia.com> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a one-time automated PR E2E retry for hosted-runner loss, including dedicated retry and abandon lifecycle steps with evidence download/verification. * Updated PR E2E gate coordination to run retry-sensitive paths only on the first workflow attempt using attempt-aware, per-PR/per-SHA concurrency isolation. * **Bug Fixes** * Hardened gate/approval execution to prevent cross-run collisions and incorrect retry/authorization, with improved fail-closed behavior. * Increased relevant time budgets and tightened “superseded/cancel” handling. * **Documentation** * Expanded PR E2E gate flow documentation for retries, evidence rules, and terminal outcomes. * **Tests** * Expanded end-to-end coverage for runner-loss classification, retry lifecycle, authorization, and command parsing edge cases. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Apurv Kumaria <akumaria@nvidia.com> Co-authored-by: Carlos Villela <cvillela@nvidia.com>
23 tasks
23 tasks
cv
added a commit
that referenced
this pull request
Jul 23, 2026
<!-- markdownlint-disable MD041 --> ## Summary Adds the canonical dated `## v0.0.93` release entry to `docs/changelog/2026-07-23.mdx`. The entry records user-visible behavior, release validation, and documentation controls merged after `v0.0.92`, while preserving the pending DGX OS `7.6.x` Station Express qualification caveat. ## Changes - Adds the parser-safe dated release entry with a summary, grouped details, and published-route links. - Reconciles the `v0.0.92..origin/main` commit range with merged `v0.0.93` PRs. - Records that no-OTA DGX OS `7.6.x` passed bounded host preflight, while full Station Express end-to-end qualification remains pending. - Leaves existing product pages unchanged because the source PRs already document their supported behavior. ### Source summary - #7285 -> `docs/changelog/2026-07-23.mdx`: Records the existing-vLLM ownership choice and resumable Station handoff. - #7419 -> `docs/changelog/2026-07-23.mdx`: Records bounded no-OTA DGX OS `7.6.x` recognition and its pending end-to-end qualification. - #7268 -> `docs/changelog/2026-07-23.mdx`: Records optional Hugging Face authentication, output sanitization, and resumable HTTP `429` recovery. - #7442 -> `docs/changelog/2026-07-23.mdx`: Records clean SIGINT handling at hidden credential prompts. - #7299 -> `docs/changelog/2026-07-23.mdx`: Records Intel macOS rejection before ref resolution or network work. - #7296 -> `docs/changelog/2026-07-23.mdx`: Records the DGX Spark non-interactive local-vLLM selection order. - #7342 -> `docs/changelog/2026-07-23.mdx`: Records delegated protected E2E approvals in the grouped release-validation bullet. - #7373 -> `docs/changelog/2026-07-23.mdx`: Records base-image publication gating before final-main fanout. - #7388 -> `docs/changelog/2026-07-23.mdx`: Records semantic phase runtime summaries. - #7397 -> `docs/changelog/2026-07-23.mdx`: Records progress coverage hardening. - #7391 -> `docs/changelog/2026-07-23.mdx`: Records centralized larger-runner routing. - #7423 -> `docs/changelog/2026-07-23.mdx`: Records one retry for confirmed hosted-runner loss. - #7399 -> `docs/changelog/2026-07-23.mdx`: Records runner-comparison telemetry. - #7270 -> `docs/changelog/2026-07-23.mdx`: Records staging Brev Launchable validation. - #7426 -> `docs/changelog/2026-07-23.mdx`: Records filtering of irrelevant base-image run history. - #7333 -> `docs/changelog/2026-07-23.mdx`: Records aligned Quickstart platform guidance. - #7343 -> `docs/changelog/2026-07-23.mdx`: Records documentation-writer receipt collection. - #7400 -> `docs/changelog/2026-07-23.mdx`: Records the documentation-writer receipt requirement for docs-only PRs. - #7413 -> `docs/changelog/2026-07-23.mdx`: Records removal of redundant receipt PR metadata. - #7405 -> `docs/changelog/2026-07-23.mdx`: Records corrected inference CLI references. - #7389 -> `docs/changelog/2026-07-23.mdx`: Records completion of the v0.0.91 documentation audit. `#7384` is an internal refactor with no intended runtime behavior change. `#7401` updates internal CodeQL Actions dependencies. `#7376` is already contained in `v0.0.92`, so it is outside the release-entry scan range despite its retained planning label. ## Type of Change - [ ] Code change (feature, bug fix, or refactor) - [ ] Code change with doc updates - [x] Doc only (prose changes, no code sample modifications) - [ ] Doc only (includes code sample changes) ## Quality Gates - [ ] Tests added or updated for changed behavior - [x] Existing tests cover changed behavior — justification: `test/changelog-docs.test.ts` validates dated changelog structure, SPDX syntax, and version headings. - [ ] Tests not applicable — justification: - [x] Docs updated for user-facing behavior changes - [ ] Docs not applicable — justification: - [ ] Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging) - [ ] Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: - [ ] Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue: ## Documentation Writer Review - [x] Documentation writer subagent reviewed the completed changes - Result: `docs-updated` - Evidence: Reviewed `docs/changelog/2026-07-23.mdx` against `WRITING.md`, `docs/CONTRIBUTING.md`, `docs/.docs-skip`, `docs/index.yml`, the six user-visible source PRs, and the remaining grouped release commits. The review corrected an ambiguous qualification claim, confirmed all published routes, preserved the DGX OS `7.6.x` caveat, and found no remaining action. - Agent: Codex Desktop <!-- docs-review-head-sha: ec0a866 --> <!-- docs-review-agents-blob-sha: 9c9b36d --> ## DGX Station Hardware Evidence - [ ] Tested on DGX Station - Tested commit: Not applicable. This PR does not change `scripts/prepare-dgx-station-host.sh`. - Station profile/scenario: Not applicable. - Result: Not applicable. - Supporting evidence: Not applicable. ## Verification - [x] PR description includes a `Signed-off-by:` line and every commit appears as `Verified` in GitHub - [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or `npm run check:diff` passed when hooks were skipped or unavailable - [x] Targeted behavior tests pass for the current change set, or tests are marked not applicable above — `npx vitest run test/changelog-docs.test.ts`: 1 file and 6 tests passed. - [ ] Applicable broad gate passed — Not applicable to one native changelog file. - [x] Quality Gates section completed with required justifications or waivers - [x] No secrets, API keys, or credentials committed - [ ] `npm run docs` builds without warnings (doc changes only) — completed with 0 errors and 2 existing Fern warnings. - [x] Doc pages follow the [style guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md) (doc changes only) - [ ] New doc pages include SPDX header and frontmatter (new pages only) — not applicable because native changelog entries use a parser-safe MDX SPDX comment without frontmatter. --- Signed-off-by: Carlos Villela <cvillela@nvidia.com> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Added the v0.0.93 changelog covering onboarding and validation improvements. * Documented support for additional DGX Station Express workstation releases and clearer handling of existing vLLM workloads. * Added guidance for optional Hugging Face authentication, resumable rate-limit recovery, and DGX Spark provider selection. * Clarified installer behavior on Intel macOS, release validation requirements, hosted-runner retries, documentation checks, and supported CLI quickstart paths. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
23 tasks
23 tasks
apurvvkumaria
added a commit
that referenced
this pull request
Jul 24, 2026
<!-- markdownlint-disable MD041 --> ## Summary This adds bounded periodic and phase-aware resource telemetry to the canonical runner-comparison ledger for trusted heavy E2E executions. It reuses the existing progress pulse and records only numeric or fixed-enum evidence, improving runner-pressure diagnosis without changing failure classification or retry policy. ## Related Issue Related to #7145 and #7146 (parent epic: #7140). ## Changes - Extend the canonical runner-comparison artifact to a bounded v2 lifecycle with private `0600` files, a reserved final sample, v1 read compatibility, and deterministic summaries. - Capture bounded host, cgroup, PSI, filesystem, Docker, container, and coarse process-class evidence at endpoints, phase boundaries, and an approximately 60-second cadence. - Reuse the existing progress timer with fixed deadlines, no catch-up bursts, per-scenario attribution boundaries, and a permanent fallback to the existing bounded collector if canonical sampling fails. - Add focused privacy, compatibility, timing, fallback, aggregation, summary, and workflow-boundary coverage, including the three-shard `common-egress-agent` topology merged in #7448. - Document the telemetry lifecycle, probe budgets, privacy limits, compatibility behavior, and the separation from classification and retry. This builds on the merged progress/timing work from #7100 and #7213 and extends the canonical ledger from #7399 rather than creating parallel infrastructure. ## Type of Change - [ ] Code change (feature, bug fix, or refactor) - [x] Code change with doc updates - [ ] Doc only (prose changes, no code sample modifications) - [ ] Doc only (includes code sample changes) ## Quality Gates <!-- Check one tests line and one docs line. Check other lines when applicable. Add every requested justification or approval reference. --> - [x] Tests added or updated for changed behavior - [ ] Existing tests cover changed behavior — justification: - [ ] Tests not applicable — justification: - [x] Docs updated for user-facing behavior changes - [ ] Docs not applicable — justification: - [x] Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging) - [x] Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: Independent exact-head privacy, schema, classifier, retry, and compatibility review at `419182d89` found no actionable issues. - [ ] Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue: ## Documentation Writer Review <!-- Required for code and documentation changes after the changes and applicable validation are complete. Keep one review checkbox and one instance of each visible or hidden field. For Evidence, list changed documentation paths. For documentation-only changes, also state that the writing rules and documentation style were reviewed. For other results, explain why no documentation change is needed or why the review is blocked. For Agent, use a consistent product and surface name, such as Codex Desktop, Codex CLI, Claude Code, or Cursor. After committing all review changes, put `git rev-parse --short HEAD` and `git rev-parse --short HEAD:AGENTS.md` in the hidden metadata below. Rerun the review and refresh that metadata after any new commit. This receipt is advisory during the data-collection pilot. --> - [x] Documentation writer subagent reviewed the completed changes - Result: `docs-updated` - Evidence: `test/e2e/README.md` - Agent: Codex Desktop <!-- docs-review-head-sha: e06653c --> <!-- docs-review-agents-blob-sha: 9c9b36d --> ## DGX Station Hardware Evidence <!-- Required only when scripts/prepare-dgx-station-host.sh changes. Maintainers must review the linked evidence before approving or merging. This is human-reviewed evidence, not authenticated hardware provenance. Exceptional bypasses use existing repository governance and must be documented on the PR. --> - [ ] Tested on DGX Station - Tested commit: - Station profile/scenario: - Result: - Supporting evidence: ## Verification <!-- Check each applicable item only when supported by the requested evidence. Run targeted tests once per relevant change set and rerun after later edits or hook autofixes that can affect the tested behavior. Do not rerun hook-covered checks. --> - [x] PR description includes a `Signed-off-by:` line and every commit appears as `Verified` in GitHub - [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or `npm run check:diff` passed when hooks were skipped or unavailable - [x] Targeted behavior tests pass for the current change set, or tests are marked not applicable above — command/result or justification: Exact-head serialized telemetry/progress/workflow suite passed 225/225 tests across 15 files; after the linear-test guardrail repair, the primary focused suite passed 165/165 across 9 files, Vitest membership was exact across 1,764 files, CLI typecheck passed, and `npm run check:diff` passed. - [ ] Applicable broad gate passed — `npm test` for broad runtime/test-harness changes; `npm run check` for repo-wide validation/coverage changes — command/result: A broad E2E-support run exposed only parallel-load timeouts and one macOS lifecycle failure reproduced unchanged on `origin/main`; scoped exact-head gates are listed above. - [x] Quality Gates section completed with required justifications or waivers - [x] No secrets, API keys, or credentials committed - [ ] `npm run docs` builds without warnings (doc changes only) - [x] Doc pages follow the [style guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md) (doc changes only) - [ ] New doc pages include SPDX header and frontmatter (new pages only) --- <!-- DCO sign-off is required in this PR description, and every commit must appear as Verified in GitHub. Run: git config user.name && git config user.email --> Signed-off-by: Apurv Kumaria <akumaria@nvidia.com> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added runner-comparison schema v2 telemetry as a bounded, ordered ledger with periodic, scenario-start, phase, and finalize samples. * Enabled live E2E runner-comparison sampling and added conditional periodic sampling in the E2E progress harness. * Introduced pulse-based progress resource sampling with configurable interval and best-effort phase/scenario capture. * **Bug Fixes / Improvements** * Switched memory diagnostics to available-memory reporting (with free-memory fallback labeling). * Hardened telemetry parsing/validation, evidence sanitization, and expanded runner-pressure/resource diagnostics. * **Documentation** * Documented v2 ordering/cadence, privacy/size limits, and diagnostic-only summary guidance. * **Tests** * Added/expanded E2E tests for v2 schema validation, timing/cadence, fallback behavior, and updated expected outputs. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Apurv Kumaria <akumaria@nvidia.com> Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com> Co-authored-by: Prekshi Vyas <prekshiv@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Records bounded start/end CPU, memory, and workspace evidence for every #7145 larger-runner route so the standard-versus-larger experiment can compare like-for-like jobs. This is the current-
mainreplacement for the unique telemetry portion of closed #7247; it does not change runner routing or duplicate #7100 phase, stall, timing, or resource diagnostics.Related Issue
Related to #7145. Supersedes the telemetry portion of #7247.
Changes
always()finalizer before artifact scanning or upload.hermes-inference-switchcontributes itshostedandanthropicmatrix entries.mcp-bridge,channels-stop-start, andsecurity-postureuninstrumented because ci(e2e): centralize larger runner routing #7391 leaves those entries on standard runners.ps, or background samplers.Type of Change
Quality Gates
Documentation Writer Review
docs-updatedtest/e2e/README.mddocuments the 12 routed workflow lane identities, 13 concrete executions, OpenClaw exclusions, sample limitations, missing-summary interpretation, and comparison requirements. The exact-head refresh confirmed the final test-only review fix requires no further documentation change.DGX Station Hardware Evidence
Verification
Signed-off-by:line and every commit appears asVerifiedin GitHubpre-commit,commit-msg, andpre-pushhooks passed, ornpm run check:diffpassed when hooks were skipped or unavailablee2e-supportpreviously reached 1,381 passed and 9 skipped; two unrelated existing local failures remained. GitHub CI is authoritative for the Linux gate.npm run docsbuilds without warnings (doc changes only) — completed with 0 errors and 2 existing Fern warningsdocs/page changedSigned-off-by: Apurv Kumaria akumaria@nvidia.com