ci(e2e): record runner comparison telemetry#7247
Conversation
Signed-off-by: Charan Jagwani <cjagwani@nvidia.com>
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
|
/nvskills-ci |
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
Code Coverage OverviewLanguages: TypeScript TypeScript / code-coverage/pluginThe overall coverage in commit 2f579c2 in the TypeScript / code-coverage/cliThe overall coverage in commit 2f579c2 in the Show a code coverage summary of the most impacted files.
Updated |
Signed-off-by: Charan Jagwani <cjagwani@nvidia.com>
|
/nvskills-ci |
wscurran
added
area: ci
PR Review Advisor — InformationalAdvisor assessment: Informational / high confidence Model lanes
Nemotron output stays in workflow artifacts and does not change the assessment above. E2E guidanceAdvisory only. E2E / PR Gate selects and runs jobs independently. Recommended E2E: 4 optional E2E recommendations
This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge. |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Signed-off-by: Charan Jagwani <cjagwani@nvidia.com>
|
/nvskills-ci |
|
Closing this draft because the stacked implementation is superseded in its current form: #7100 now provides the shared semantic-phase, stall, timing, and resource evidence, and the parent routing PR #7245 was closed without merging. The unique job-level CPU, memory, and disk comparison telemetry may still be useful for #7145; if that larger-runner experiment resumes, it should be salvaged as a smaller PR based on current main that reuses the #7100 infrastructure. This PR does not implement the separate rolling nightly runtime-history and GitHub Step Summary work. |
|
The unique runner-comparison telemetry has been rebuilt from current main as #7399. The replacement is limited to the five #7145 candidate executions, records only start/end CPU, memory, and workspace evidence, and does not duplicate #7100 phase/stall diagnostics or depend on #7391's routing changes. |
## Summary Records bounded start/end CPU, memory, and workspace evidence for every #7145 larger-runner route so the standard-versus-larger experiment can compare like-for-like jobs. This is the current-`main` replacement for the unique telemetry portion of closed #7247; it does not change runner routing or duplicate #7100 phase, stall, timing, or resource diagnostics. ## Related Issue Related to #7145. Supersedes the telemetry portion of #7247. ## Changes - Write one canonical private numeric sample after workspace preparation and one from an `always()` finalizer before artifact scanning or upload. - Cover 12 routed workflow lane identities and 13 concrete executions; `hermes-inference-switch` contributes its `hosted` and `anthropic` matrix entries. - Keep the OpenClaw entries for `mcp-bridge`, `channels-stop-start`, and `security-posture` uninstrumented because #7391 leaves those entries on standard runners. - Summarize measured-window CPU utilization, host-memory endpoints and root-cgroup lifetime peak, plus workspace endpoints and growth without invoking Docker, `ps`, or background samplers. - Reject unknown fields, duplicate JSON keys, malformed timestamps, identity/counter drift, oversized evidence, symlink/hardlink final-file substitution, matrix topology drift, and workflow scope/order drift. - Document comparison controls and limitations. Standard-runner baselines can begin before #7391 changes routing; the same evidence remains after larger-runner activation. ## Type of Change - [ ] Code change (feature, bug fix, or refactor) - [x] Code change with doc updates - [ ] Doc only (prose changes, no code sample modifications) - [ ] Doc only (includes code sample changes) ## Quality Gates - [x] Tests added or updated for changed behavior - [ ] Existing tests cover changed behavior — justification: - [ ] Tests not applicable — justification: - [x] Docs updated for user-facing behavior changes - [ ] Docs not applicable — justification: - [x] Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging) - [x] Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: independent review confirmed the exact trusted-main/no-alternate-checkout boundary, Hermes-only matrix guards, bounded private schema, and finalizer-before-scan/upload ordering. - [ ] Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue: ## Documentation Writer Review - [x] Documentation writer subagent reviewed the completed implementation - Result: `docs-updated` - Evidence: `test/e2e/README.md` documents the 12 routed workflow lane identities, 13 concrete executions, OpenClaw exclusions, sample limitations, missing-summary interpretation, and comparison requirements. The exact-head refresh confirmed the final test-only review fix requires no further documentation change. - Agent: Codex Desktop - PR: #7399 <!-- docs-review-head-sha: c09137d --> <!-- docs-review-agents-blob-sha: 9061453 --> ## DGX Station Hardware Evidence - [ ] Tested on DGX Station - Tested commit: Not applicable - Station profile/scenario: Not applicable - Result: Not applicable - Supporting evidence: Not applicable ## Verification - [x] PR description includes a `Signed-off-by:` line and every commit appears as `Verified` in GitHub - [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or `npm run check:diff` passed when hooks were skipped or unavailable - [x] Targeted behavior tests pass for the current change set, or tests are marked not applicable above — focused E2E-support validation passed 102 tests; an independent rerun of the four directly affected suites passed 94 tests; the final matrix-assertion review fix passed its focused 28-test suite. - [ ] Applicable broad gate passed — Full `e2e-support` previously reached 1,381 passed and 9 skipped; two unrelated existing local failures remained. GitHub CI is authoritative for the Linux gate. - [x] Quality Gates section completed with required justifications or waivers - [x] No secrets, API keys, or credentials committed - [ ] `npm run docs` builds without warnings (doc changes only) — completed with 0 errors and 2 existing Fern warnings - [ ] Doc pages follow the [style guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md) (doc changes only) — no `docs/` page changed - [ ] New doc pages include SPDX header and frontmatter (new pages only) — no new doc page --- Signed-off-by: Apurv Kumaria <akumaria@nvidia.com> --------- Signed-off-by: Apurv Kumaria <akumaria@nvidia.com> Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com> Co-authored-by: Prekshi Vyas <prekshiv@nvidia.com>
5 participants
Summary
Adds durable job-level runner telemetry to the five larger-runner candidates so the inactive #7145 experiment can compare standard and larger runners with the same evidence contract. Sampling reuses the existing progress heartbeat, stays best-effort, and does not change live-test outcomes.
Related Issue
Related to #7145
Changes
Type of Change
Quality Gates
Verification
Signed-off-by:line and every commit appears asVerifiedin GitHubpre-commit,commit-msg, andpre-pushhooks passed, ornpm run check:diffpassed when hooks were skipped or unavailablenpx vitest run --project e2e-supportfor the six telemetry/progress suites: 90/90 passed.e2e-supportwith four workers: 1,295 passed and 4 skipped; two existing fixed-timeout tests failed under aggregate load, then passed in isolation (4/4 and 31/31).npm run docsbuilds without warnings (doc changes only) — completed successfully with 0 errors and 2 existing Fern warnings.Signed-off-by: Charan Jagwani cjagwani@nvidia.com