ci(e2e): extend Hermes runner telemetry

[apurvvkumaria]

Summary

Agent-turn latency and the Hermes Bedrock compatibility lane already emitted semantic progress samples, but their workflows never initialized or finalized the bounded runner-comparison ledger. This change gives both trusted-main executions the same phase-attributed CPU, memory, disk, Docker, and pressure summary used by the existing heavy Hermes lanes.

Related Issue

Part of #7140. This PR does not close the epic because the optimized Hermes image, matched before/after cohorts, and remaining lane-specific work are still pending.

Changes

• Initialize runner-comparison telemetry immediately after workspace preparation for agent-turn-latency.
• Initialize telemetry only for the Hermes shard of bedrock-runtime-compatible-anthropic; the OpenClaw shard remains excluded.
• Always finalize telemetry immediately before each job publishes artifacts.
• Extend the fail-closed workflow boundary to require exact trusted-main guards, matrix topology, invocation count, and ordering.
• Document 14 routed diagnostic identities / 17 concrete executions and clarify that this change does not route either lane to a larger runner.
• Keep hermes-slack out of scope so fixed self-hosted runner capacity is not published without a separate maintainer decision.

Type of Change

• Code change (feature, bug fix, or refactor)
• Code change with doc updates
• Doc only (prose changes, no code sample modifications)
• Doc only (includes code sample changes)

Quality Gates

• Tests added or updated for changed behavior
• Existing tests cover changed behavior — justification:
• Tests not applicable — justification:
• Docs updated for user-facing behavior changes
• Docs not applicable — justification: This changes internal trusted-main CI telemetry only; test/e2e/README.md documents the operator-facing contract.
• Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
• Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: Independent review passed at exact head b60f48ccdd319b65575ea37f606083aeff616b08 against base 2f6298ee165f4821f635be962fedff5e165701ee. Repository, branch, alternate-checkout, matrix, numeric-schema, and best-effort finalizer boundaries remain fail closed; no permission, secret, dependency, or runner-routing expansion was found.
• Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Documentation Writer Review

• Documentation writer subagent reviewed the completed changes
• Result: no-docs-needed
• Evidence: test/e2e/README.md now records the internal operator contract; no CLI, configuration, API, supported behavior, or user procedure changes.
• Agent: Codex Desktop

DGX Station Hardware Evidence

• Tested on DGX Station
• Tested commit:
• Station profile/scenario:
• Result:
• Supporting evidence:

Verification

• PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
• Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run check:diff passed when hooks were skipped or unavailable
• Targeted behavior tests pass for the current change set, or tests are marked not applicable above — npx vitest run --project e2e-support test/e2e/support/runner-comparison-workflow-boundary.test.ts test/e2e/support/runner-comparison.test.ts test/e2e/support/e2e-workflow.test.ts test/e2e/support/upload-e2e-artifacts-workflow-boundary.test.ts test/e2e/support/trusted-hermes-swap-workflow-boundary.test.ts (158 passed)
• Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — command/result: Not applicable; this is a narrow workflow-boundary change covered by the focused suites. npm run typecheck:cli, npm run test:titles:check, npm run source-shape:check, Biome, and git diff --check passed.
• Quality Gates section completed with required justifications or waivers
• No secrets, API keys, or credentials committed
• npm run docs builds without warnings (doc changes only)
• Doc pages follow the style guide (doc changes only)
• New doc pages include SPDX header and frontmatter (new pages only)

---

Signed-off-by: Apurv Kumaria akumaria@nvidia.com

Summary by CodeRabbit

• New Features
  • Added runner-comparison telemetry lanes for agent-turn-latency and bedrock-runtime-compatible-anthropic.
  • Expanded routed-lane runner comparison coverage to 14 lanes / 17 executions.
• Documentation
  • Updated Scheduled operations “Runner comparison telemetry” contract (including non-instrumented lane guidance) and clarified PR gate behavior around controller binding and cache-window polling.
• Tests
  • Updated PR E2E gate and workflow-boundary tests to match the revised controller-check validation, including child run URL inclusion and stricter polling/cache expectations.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: b32cc72b-7cf7-426b-9f50-b8f5442af2d2

📥 Commits

Reviewing files that changed from the base of the PR and between f041224 and 429521c.

📒 Files selected for processing (2)

• test/e2e/support/e2e-operations-workflow-boundary.test.ts
• tools/e2e/operations-workflow-boundary.mts

🚧 Files skipped from review as they are similar to previous changes (2)

• tools/e2e/operations-workflow-boundary.mts
• test/e2e/support/e2e-operations-workflow-boundary.test.ts

---

📝 Walkthrough

Walkthrough

The PR revises controller child-run authorization to use check output summaries and cache-spanning polling, and adds runner-comparison telemetry to agent-turn-latency and the Bedrock Hermes lane. Boundary validators, tests, and E2E documentation reflect the expanded contract.

Changes

E2E workflow contracts

Layer / File(s)	Summary
Controller authorization contract .github/workflows/e2e.yaml, tools/e2e/..., test/e2e/support/..., test/pr-e2e-gate.test.ts	Controller validation now matches child-run URLs in check summaries, enforces no-cache polling over 60 seconds, and tests canonicalized check URLs and running-check updates.
Runner-comparison telemetry integration .github/workflows/e2e.yaml, tools/e2e/runner-comparison-workflow-boundary.mts, test/e2e/support/runner-comparison-workflow-boundary.test.ts	Adds guarded initialize/finalize telemetry to two jobs and updates lane, execution, matrix, and validation expectations.
Coverage contract documentation test/e2e/README.md	Documents 14 routed lanes, 17 executions, the added telemetry lanes, excluded matrix entries, and controller polling behavior.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant E2EWorkflow
  participant PRGate
  participant GitHubChecks
  participant ChildRun
  E2EWorkflow->>ChildRun: dispatch E2E workflow
  PRGate->>GitHubChecks: poll controller check output
  GitHubChecks-->>PRGate: return summary containing child run URL
  PRGate->>E2EWorkflow: authorize matching child run

Loading

sequenceDiagram
  participant AgentTurnLatency
  participant BedrockHermes
  participant RunnerComparison
  AgentTurnLatency->>RunnerComparison: initialize
  AgentTurnLatency->>RunnerComparison: finalize after live test
  BedrockHermes->>RunnerComparison: initialize
  BedrockHermes->>RunnerComparison: finalize after live test

Loading

Possibly related PRs

• NVIDIA/NemoClaw#7342: Modifies the same PR E2E gate controller check-run authorization contract.
• NVIDIA/NemoClaw#7450: Updates related child-run URL handling in PR gate check-run logic.
• NVIDIA/NemoClaw#7457: Implements the runner-comparison telemetry lifecycle used by the added workflow steps.

Suggested labels: area: observability

Suggested reviewers: cv, prekshivyas

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 25.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately captures the main theme of extending E2E telemetry, especially the Hermes runner additions.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/7140-hermes-telemetry-coverage

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-code-quality]

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in commit 429521c in the codex/7140-hermes-te... branch remains at 96%, unchanged from commit dbac677 in the main branch.

TypeScript / code-coverage/cli

The overall coverage in commit 429521c in the codex/7140-hermes-te... branch remains at 80%, unchanged from commit dbac677 in the main branch.

Show a code coverage summary of the most impacted files.

File	main dbac677	codex/7140-hermes-te... 429521c	+/-
src/lib/state/m...-acquisition.ts	89%	84%	-5%
src/lib/sandbox...rce-identity.ts	87%	87%	0%
src/lib/state/m...ock-identity.ts	95%	95%	0%
src/lib/state/m...lock-storage.ts	97%	97%	0%
src/lib/domain/.../connect-env.ts	89%	97%	+8%

---

_{Updated July 25, 2026 17:31 UTC}

[github-actions]

PR Review Advisor — Informational

Advisor assessment: Informational / high confidence
Next action: Review the warnings below.
Findings: 0 blockers · 1 warning · 0 suggestions
Status: Canonical ledger: 0 blocker(s), 1 warning(s), 0 suggestion(s).

Model lanes

• GPT-5.6 Terra (primary): Completed · high confidence · 0 blockers · 1 warning · 0 suggestions
• Nemotron 3 Ultra (second opinion): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
• Model comparison: normalized findings differ; normalized E2E selections differ; Nemotron reported the same number of blockers, 1 fewer warning, the same number of suggestions.

Nemotron output stays in workflow artifacts and does not change the assessment above.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: cloud-onboard, credential-sanitization, security-posture

1 warning · 0 suggestions

Warnings

Warnings do not block.

PRA-1 Warning — Cover delayed controller-check propagation

• Location: test/e2e/support/e2e-operations-workflow-boundary.test.ts:191
• Category: tests
• Problem: The changed acceptance test supplies the authorized check on its first curl response. It does not execute the new retry path that handles stale check-run state before GitHub publishes the child-run summary.
• Impact: A future change can preserve the polling constants and static fragments while breaking retry behavior, causing authorized credentialed E2E dispatches to fail closed after normal check propagation delay.
• Recommendation: Add a shell-level authentication test that returns a stale or mismatched check response before the exact authorized response. Assert that authentication succeeds only after retry and that each request retains the no-cache headers.
• Verification: Inspect the mocked curl behavior in the summary-bound-child test and confirm that it returns FAKE_CHECK on the first request rather than varying responses by attempt.
• Test coverage: A test for Authenticate controller dispatch that simulates at least one stale check response followed by the exact summary-bound check, then verifies success and no-cache request headers.
• Evidence: `.github/workflows/e2e.yaml` adds a 45-attempt polling loop to wait for fresh controller check state. The test at line 191 replaces curl with `printf '%s' "$FAKE_CHECK"`, so its first response already has the expected summary. Existing mutation tests verify constants and source fragments but do not execute delayed propagation.

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

[apurvvkumaria]

Exact-head credentialed E2E passed.

• PR head: b60f48ccdd319b65575ea37f606083aeff616b08
• Base/workflow SHA: 2f6298ee165f4821f635be962fedff5e165701ee
• Control-plane verification: https://github.com/NVIDIA/NemoClaw/actions/runs/30143172716
• Credentialed child run: https://github.com/NVIDIA/NemoClaw/actions/runs/30143187236
• Passed: cloud-onboard, credential-sanitization, OpenClaw security-posture, and Hermes security-posture
• Hosted-runner-loss retry: not used

Because GitHub dispatches the trusted workflow definition from main, this run validates the exact PR checkout/runtime path but does not execute the PR-authored YAML topology. The new agent-turn-latency and Hermes Bedrock telemetry placement is instead protected by the exact-head 158-test workflow-boundary suite and independent sensitive-path review. All required checks are now green; independent human review remains.

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tools/e2e/operations-workflow-boundary.mts`:
- Around line 206-222: Update the controller polling validation around
maxAttempts and pollSeconds to require exactly 45 attempts and 2-second delays,
while preserving the existing safe-integer and cache-spanning duration checks.
In test/e2e/support/e2e-operations-workflow-boundary.test.ts lines 181-199, add
independent mutations for each configuration value and assert the boundary
validator rejects both changes.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 7da21c8c-8a49-4f82-a87d-84fcb47f934b

📥 Commits

Reviewing files that changed from the base of the PR and between b3ba2c6 and f041224.

📒 Files selected for processing (6)

• .github/workflows/e2e.yaml
• test/e2e/README.md
• test/e2e/support/e2e-operations-workflow-boundary.test.ts
• test/pr-e2e-gate.test.ts
• tools/e2e/operations-workflow-boundary.mts
• tools/e2e/pr-e2e-gate.mts

🚧 Files skipped from review as they are similar to previous changes (1)

• test/e2e/README.md