test(e2e): add semantic live progress phases#7100
Open
cjagwani wants to merge 24 commits into
Open
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (2)
🚧 Files skipped from review as they are similar to previous changes (2)
📝 WalkthroughWalkthroughThis PR adds shared E2E phase tracking, shell instrumentation, persisted ChangesE2E runtime observability
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant LiveE2ETest
participant TestProgress
participant ShellProbe
participant E2EArtifact
participant RuntimeAudit
LiveE2ETest->>TestProgress: declare and advance semantic phases
LiveE2ETest->>ShellProbe: execute instrumented command
ShellProbe-->>TestProgress: report output and activity
TestProgress->>E2EArtifact: write test-progress.json
RuntimeAudit->>E2EArtifact: discover and validate summaries
RuntimeAudit-->>RuntimeAudit: rank p95, variability, max, and slowest phase
Possibly related issues
Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
Contributor
Code Coverage OverviewLanguages: TypeScript TypeScript / code-coverage/pluginThe overall coverage in commit 9601701 in the TypeScript / code-coverage/cliThe overall coverage in commit 9601701 in the Show a code coverage summary of the most impacted files.
Updated |
2 tasks
Contributor
PR Review Advisor — InformationalAdvisor assessment: Informational / high confidence Model lanes
Nemotron output stays in workflow artifacts and does not change the assessment above. E2E guidanceAdvisory only. E2E / PR Gate selects and runs jobs independently. Recommended E2E: 1 optional E2E recommendation
This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge. |
wscurran
added
area: ci
Contributor
|
🌿 Preview your docs: https://nvidia-preview-pr-7100.docs.buildwithfern.com/nemoclaw |
cjagwani
changed the title
cjagwani
marked this pull request as ready for review
July 20, 2026 14:53
…y-hang' into codex/diagnose-agent-turn-latency-hang
cjagwani
marked this pull request as draft
July 20, 2026 15:01
cjagwani
marked this pull request as ready for review
July 20, 2026 15:02
cjagwani
marked this pull request as draft
July 20, 2026 15:03
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@scripts/audit-test-runtime.mts`:
- Around line 99-100: Update scripts/audit-test-runtime.mts at lines 99-100 to
use a dedicated median helper that averages the two middle sorted durations for
even-sized run sets, while preserving the existing percentile calculation for
p95. Update test/e2e/support/test-runtime-audit.test.ts at lines 56-82 to expect
the two-run median, variability, and formatted row as 30.0s, 20.0s, and 20.0s.
In `@test/e2e/live/agent-turn-latency.test.ts`:
- Around line 50-93: Remove the explicit progress.stop() call associated with
the cleanup setup around cleanup.trackDisposable. Let cleanup.runAll() execute
while progress remains active so cleanup phases and output are preserved,
relying on the progress fixture’s finally block to stop itself.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 9573d819-5d22-4bb1-a36a-ae48ae47dc38
📒 Files selected for processing (17)
CONTRIBUTING.mdpackage.jsonscripts/audit-test-runtime.mtstest/e2e/README.mdtest/e2e/docs/README.mdtest/e2e/fixtures/e2e-test.tstest/e2e/fixtures/progress.tstest/e2e/fixtures/shell-probe.tstest/e2e/live/agent-turn-latency-helpers.tstest/e2e/live/agent-turn-latency.test.tstest/e2e/live/rebuild-hermes-progress.tstest/e2e/live/rebuild-hermes.test.tstest/e2e/support/agent-turn-latency-progress.test.tstest/e2e/support/e2e-fixture-context.test.tstest/e2e/support/rebuild-hermes-progress.test.tstest/e2e/support/rebuild-hermes-timing.test.tstest/e2e/support/test-runtime-audit.test.ts
💤 Files with no reviewable changes (1)
- test/e2e/live/rebuild-hermes-progress.ts
cjagwani
marked this pull request as ready for review
July 20, 2026 15:10
Collaborator
Author
|
Fresh advisor follow-up: PRA-1 is addressed in aecce15 with a fixture-backed teardown test that verifies a completed version-1 summary, scenario, target ID, shard ID, duration, and final phase. PRA-2 does not reproduce: the live matrix sets E2E_ARTIFACT_DIR to e2e-artifacts/live (not the target directory), liveTargetTestName(target) is exactly target.id, and createArtifactSink joins those into e2e-artifacts/live/${matrix.id}/test-progress.json. The existing artifact-path test also asserts that exact root. Changing the upload contract to */test-progress.json would therefore miss the registry target artifact. |
Collaborator
Author
|
The growth-guardrail failure is addressed in 950fd9c: the fixture regression now uses Vitest environment stubs and contains no added if statements. The focused fixture test and the repository test-conditionals scan both pass. |
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@test/e2e/support/e2e-progress-fixture.test.ts`:
- Around line 34-39: Consolidate the environment restoration logic in the test
fixture cleanup into a data-driven loop over the saved variable names and
values, using a single conditional to delete variables whose previous value is
undefined or restore defined values. Update the restoration block containing
previousArtifactDir, previousTargetId, and previousShardId while preserving each
variable’s existing mapping and behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: b64a9b47-60f5-4b69-9f15-2584904f691d
📒 Files selected for processing (1)
test/e2e/support/e2e-progress-fixture.test.ts
Signed-off-by: Charan Jagwani <cjagwani@nvidia.com>
Contributor
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@test/e2e/live/issue-2478-crash-loop-recovery.test.ts`:
- Line 429: Update the test title in the issue-2478 recovery test to remove the
leading “issue-2478:” prefix and append the issue reference as the final
“(`#2478`)” suffix, preserving the existing behavior-oriented wording.
In `@test/e2e/live/registry-targets.test.ts`:
- Around line 41-49: Ensure the test flow around the lifecycle handling at line
117 always advances through the declared lifecycle phase, even when
target.environment.lifecycle is absent. Keep lifecycle-specific work
conditional, but invoke the phase progression unconditionally so no-lifecycle
targets preserve the ordered REGISTRY_TARGET_PHASES plan and reach its final
phase.
In `@test/e2e/live/whatsapp-qr-compact.test.ts`:
- Around line 273-281: Update the E2E setup around compileProductionPreload and
the matching runtime package installation to request the host/shellProbe fixture
and route both direct child commands through it, preserving redaction and
lifecycle activity for stall diagnostics; alternatively, explicitly forward
equivalent redacted command activity.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 574adbbe-80b5-45a7-b124-e693ecfe474c
📒 Files selected for processing (104)
.pre-commit-config.yamlCONTRIBUTING.mdpackage.jsontest/e2e/README.mdtest/e2e/docs/README.mdtest/e2e/fixtures/e2e-test.tstest/e2e/fixtures/progress.tstest/e2e/fixtures/shell-probe.tstest/e2e/live/agent-turn-latency-helpers.tstest/e2e/live/agent-turn-latency.test.tstest/e2e/live/bedrock-runtime-compatible-anthropic.test.tstest/e2e/live/brave-search.test.tstest/e2e/live/channels-add-remove.test.tstest/e2e/live/channels-stop-start-helpers.tstest/e2e/live/channels-stop-start.test.tstest/e2e/live/cloud-inference.test.tstest/e2e/live/cloud-onboard.test.tstest/e2e/live/common-egress-agent.test.tstest/e2e/live/concurrent-gateway-ports.test.tstest/e2e/live/credential-migration.test.tstest/e2e/live/credential-sanitization.test.tstest/e2e/live/cron-preflight-inference-local.test.tstest/e2e/live/dashboard-remote-bind.test.tstest/e2e/live/device-auth-health.test.tstest/e2e/live/diagnostics.test.tstest/e2e/live/docs-validation.test.tstest/e2e/live/double-onboard.test.tstest/e2e/live/full-e2e.test.tstest/e2e/live/gateway-guard-recovery.test.tstest/e2e/live/gateway-health-honest.test.tstest/e2e/live/gpu-double-onboard.test.tstest/e2e/live/gpu-e2e.test.tstest/e2e/live/hermes-discord.test.tstest/e2e/live/hermes-e2e-phases.tstest/e2e/live/hermes-e2e.test.tstest/e2e/live/hermes-gpu-startup.test.tstest/e2e/live/hermes-inference-switch.test.tstest/e2e/live/hermes-root-entrypoint-smoke.test.tstest/e2e/live/hermes-sandbox-secret-boundary.test.tstest/e2e/live/hermes-shields-config.test.tstest/e2e/live/hermes-slack-e2e-helpers.tstest/e2e/live/hermes-slack-e2e.test.tstest/e2e/live/inference-routing-provider-smoke.test.tstest/e2e/live/inference-routing.test.tstest/e2e/live/issue-2478-crash-loop-recovery.test.tstest/e2e/live/issue-4434-tui-unreachable-inference.test.tstest/e2e/live/issue-4462-admin-approval-helper.tstest/e2e/live/issue-4462-scope-upgrade-approval.test.tstest/e2e/live/jetson-nvmap-gpu.test.tstest/e2e/live/kimi-inference-compat.test.tstest/e2e/live/launchable-smoke.test.tstest/e2e/live/mcp-bridge-phases.tstest/e2e/live/mcp-bridge.test.tstest/e2e/live/messaging-compatible-endpoint.test.tstest/e2e/live/messaging-providers.test.tstest/e2e/live/model-router-provider-routed-inference.test.tstest/e2e/live/network-policy.test.tstest/e2e/live/ollama-auth-proxy.test.tstest/e2e/live/onboard-negative-paths.test.tstest/e2e/live/onboard-repair.test.tstest/e2e/live/onboard-resume.test.tstest/e2e/live/openclaw-discord-pairing.test.tstest/e2e/live/openclaw-inference-switch.test.tstest/e2e/live/openclaw-plugin-runtime-exdev.test.tstest/e2e/live/openclaw-skill-cli.test.tstest/e2e/live/openclaw-slack-pairing.test.tstest/e2e/live/openclaw-tui-chat-correlation.test.tstest/e2e/live/openshell-credential-generation-window.test.tstest/e2e/live/openshell-gateway-auth-source-contract-helpers.tstest/e2e/live/openshell-gateway-auth-source-contract.test.tstest/e2e/live/openshell-gateway-upgrade.test.tstest/e2e/live/openshell-version-pin.test.tstest/e2e/live/overlayfs-autofix.test.tstest/e2e/live/rebuild-hermes.test.tstest/e2e/live/rebuild-openclaw.test.tstest/e2e/live/registry-targets.test.tstest/e2e/live/runtime-overrides.test.tstest/e2e/live/sandbox-operations.test.tstest/e2e/live/sandbox-rebuild.test.tstest/e2e/live/sandbox-rlimits-connect.test.tstest/e2e/live/sandbox-survival.test.tstest/e2e/live/sessions-agents-cli.test.tstest/e2e/live/shields-config.test.tstest/e2e/live/skill-agent.test.tstest/e2e/live/snapshot-commands.test.tstest/e2e/live/spark-install.test.tstest/e2e/live/state-backup-restore.test.tstest/e2e/live/state-dir-guard-metadata.test.tstest/e2e/live/telegram-injection.test.tstest/e2e/live/token-rotation.test.tstest/e2e/live/tunnel-lifecycle-helpers.tstest/e2e/live/tunnel-lifecycle.test.tstest/e2e/live/ubuntu-repo-cli-smoke.test.tstest/e2e/live/upgrade-stale-sandbox.test.tstest/e2e/live/whatsapp-qr-compact.test.tstest/e2e/support/agent-turn-latency-progress.test.tstest/e2e/support/e2e-progress-fixture.test.tstest/e2e/support/e2e-progress-outcome.test.tstest/e2e/support/e2e-semantic-phase-check.test.tstest/e2e/support/fixtures/e2e-progress-outcome.fixture.test.tstest/e2e/support/rebuild-hermes-progress.test.tstest/e2e/support/rebuild-hermes-timing.test.tstools/e2e/check-semantic-phases.mtsvitest.config.ts
🚧 Files skipped from review as they are similar to previous changes (3)
- package.json
- test/e2e/fixtures/shell-probe.ts
- test/e2e/fixtures/e2e-test.ts
Signed-off-by: Charan Jagwani <cjagwani@nvidia.com>
Collaborator
Author
|
@coderabbitai review |
Contributor
✅ Action performedReview finished.
|
Contributor
There was a problem hiding this comment.
♻️ Duplicate comments (1)
test/e2e/live/issue-2478-crash-loop-recovery.test.ts (1)
429-429: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winTest title still uses
issue-2478:prefix instead of the required(#2478)suffix.This was flagged in a previous review and remains unaddressed. Per coding guidelines, test titles must be behavior-oriented with local issue references placed in a final
(#1234)suffix.Proposed fix
-test("issue-2478: gateway recovery preserves guard chain and avoids crash loop", { +test("gateway recovery preserves guard chain and avoids crash loop (`#2478`)", {🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@test/e2e/live/issue-2478-crash-loop-recovery.test.ts` at line 429, Update the test title in the issue-2478 recovery test to describe the behavior without the `issue-2478:` prefix, and append the local issue reference as the final `(`#2478`)` suffix. Preserve the existing behavior description.Sources: Coding guidelines, Learnings
🧹 Nitpick comments (1)
test/e2e/support/e2e-semantic-phase-check.test.ts (1)
69-72: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAvoid hard-coding complete diagnostic formatting.
This assertion couples the test to wording, synthetic file paths, line numbers, and test names. Assert the failure count plus stable semantic fragments instead, so harmless diagnostic-format changes do not break behavioral coverage.
As per path instructions, tests should prefer observable outcomes through the public boundary over source-text or implementation-shape assertions.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@test/e2e/support/e2e-semantic-phase-check.test.ts` around lines 69 - 72, Update the assertion in the semantic phase check test to verify the expected failure count and stable diagnostic fragments, rather than matching complete messages with file paths, line numbers, or test names. Preserve coverage that each failure identifies the missing semantic phase and the relevant download scenarios while allowing diagnostic wording and formatting to change.Source: Path instructions
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Duplicate comments:
In `@test/e2e/live/issue-2478-crash-loop-recovery.test.ts`:
- Line 429: Update the test title in the issue-2478 recovery test to describe
the behavior without the `issue-2478:` prefix, and append the local issue
reference as the final `(`#2478`)` suffix. Preserve the existing behavior
description.
---
Nitpick comments:
In `@test/e2e/support/e2e-semantic-phase-check.test.ts`:
- Around line 69-72: Update the assertion in the semantic phase check test to
verify the expected failure count and stable diagnostic fragments, rather than
matching complete messages with file paths, line numbers, or test names.
Preserve coverage that each failure identifies the missing semantic phase and
the relevant download scenarios while allowing diagnostic wording and formatting
to change.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 8c645882-1ff1-4b6a-929f-70911712c70e
📒 Files selected for processing (106)
.pre-commit-config.yamlCONTRIBUTING.mdpackage.jsontest/e2e/README.mdtest/e2e/docs/README.mdtest/e2e/fixtures/e2e-test.tstest/e2e/fixtures/progress.tstest/e2e/fixtures/shell-probe.tstest/e2e/live/agent-turn-latency-helpers.tstest/e2e/live/agent-turn-latency.test.tstest/e2e/live/bedrock-runtime-compatible-anthropic.test.tstest/e2e/live/brave-search.test.tstest/e2e/live/channels-add-remove.test.tstest/e2e/live/channels-stop-start-helpers.tstest/e2e/live/channels-stop-start.test.tstest/e2e/live/cloud-inference.test.tstest/e2e/live/cloud-onboard.test.tstest/e2e/live/common-egress-agent.test.tstest/e2e/live/concurrent-gateway-ports.test.tstest/e2e/live/credential-migration.test.tstest/e2e/live/credential-sanitization.test.tstest/e2e/live/cron-preflight-inference-local.test.tstest/e2e/live/dashboard-remote-bind.test.tstest/e2e/live/device-auth-health.test.tstest/e2e/live/diagnostics.test.tstest/e2e/live/docs-validation.test.tstest/e2e/live/double-onboard.test.tstest/e2e/live/full-e2e.test.tstest/e2e/live/gateway-guard-recovery.test.tstest/e2e/live/gateway-health-honest.test.tstest/e2e/live/gpu-double-onboard.test.tstest/e2e/live/gpu-e2e.test.tstest/e2e/live/hermes-discord.test.tstest/e2e/live/hermes-e2e-phases.tstest/e2e/live/hermes-e2e.test.tstest/e2e/live/hermes-gpu-startup.test.tstest/e2e/live/hermes-inference-switch.test.tstest/e2e/live/hermes-root-entrypoint-smoke.test.tstest/e2e/live/hermes-sandbox-secret-boundary.test.tstest/e2e/live/hermes-shields-config.test.tstest/e2e/live/hermes-slack-e2e-helpers.tstest/e2e/live/hermes-slack-e2e.test.tstest/e2e/live/inference-routing-provider-smoke.test.tstest/e2e/live/inference-routing.test.tstest/e2e/live/issue-2478-crash-loop-recovery.test.tstest/e2e/live/issue-4434-tui-unreachable-inference.test.tstest/e2e/live/issue-4462-admin-approval-helper.tstest/e2e/live/issue-4462-scope-upgrade-approval.test.tstest/e2e/live/jetson-nvmap-gpu.test.tstest/e2e/live/kimi-inference-compat.test.tstest/e2e/live/launchable-smoke.test.tstest/e2e/live/mcp-bridge-phases.tstest/e2e/live/mcp-bridge.test.tstest/e2e/live/messaging-compatible-endpoint.test.tstest/e2e/live/messaging-providers.test.tstest/e2e/live/model-router-provider-routed-inference.test.tstest/e2e/live/network-policy.test.tstest/e2e/live/ollama-auth-proxy.test.tstest/e2e/live/onboard-negative-paths.test.tstest/e2e/live/onboard-repair.test.tstest/e2e/live/onboard-resume.test.tstest/e2e/live/openclaw-discord-pairing.test.tstest/e2e/live/openclaw-inference-switch.test.tstest/e2e/live/openclaw-plugin-runtime-exdev.test.tstest/e2e/live/openclaw-skill-cli.test.tstest/e2e/live/openclaw-slack-pairing.test.tstest/e2e/live/openclaw-tui-chat-correlation.test.tstest/e2e/live/openshell-credential-generation-window.test.tstest/e2e/live/openshell-gateway-auth-source-contract-helpers.tstest/e2e/live/openshell-gateway-auth-source-contract.test.tstest/e2e/live/openshell-gateway-upgrade.test.tstest/e2e/live/openshell-version-pin.test.tstest/e2e/live/overlayfs-autofix.test.tstest/e2e/live/rebuild-hermes.test.tstest/e2e/live/rebuild-openclaw.test.tstest/e2e/live/registry-targets.test.tstest/e2e/live/runtime-overrides.test.tstest/e2e/live/sandbox-operations.test.tstest/e2e/live/sandbox-rebuild.test.tstest/e2e/live/sandbox-rlimits-connect.test.tstest/e2e/live/sandbox-survival.test.tstest/e2e/live/sessions-agents-cli.test.tstest/e2e/live/shields-config.test.tstest/e2e/live/skill-agent.test.tstest/e2e/live/snapshot-commands.test.tstest/e2e/live/spark-install.test.tstest/e2e/live/state-backup-restore.test.tstest/e2e/live/state-dir-guard-metadata.test.tstest/e2e/live/telegram-injection.test.tstest/e2e/live/token-rotation.test.tstest/e2e/live/tunnel-lifecycle-helpers.tstest/e2e/live/tunnel-lifecycle.test.tstest/e2e/live/ubuntu-repo-cli-smoke.test.tstest/e2e/live/upgrade-stale-sandbox.test.tstest/e2e/live/whatsapp-qr-compact.test.tstest/e2e/mock-parity.jsontest/e2e/support/agent-turn-latency-progress.test.tstest/e2e/support/e2e-progress-fixture.test.tstest/e2e/support/e2e-progress-outcome.test.tstest/e2e/support/e2e-semantic-phase-check.test.tstest/e2e/support/fixtures/e2e-progress-outcome.fixture.test.tstest/e2e/support/fixtures/semantic-phase-invalid.fixture.tstest/e2e/support/rebuild-hermes-progress.test.tstest/e2e/support/rebuild-hermes-timing.test.tstools/e2e/check-semantic-phases.mtsvitest.config.ts
🚧 Files skipped from review as they are similar to previous changes (83)
- test/e2e/live/channels-stop-start.test.ts
- test/e2e/live/mcp-bridge-phases.ts
- test/e2e/live/tunnel-lifecycle.test.ts
- test/e2e/live/sandbox-rlimits-connect.test.ts
- package.json
- test/e2e/live/issue-4462-admin-approval-helper.ts
- test/e2e/live/spark-install.test.ts
- test/e2e/live/docs-validation.test.ts
- test/e2e/live/hermes-e2e-phases.ts
- test/e2e/live/openshell-gateway-auth-source-contract.test.ts
- test/e2e/live/dashboard-remote-bind.test.ts
- test/e2e/live/cloud-onboard.test.ts
- .pre-commit-config.yaml
- test/e2e/live/concurrent-gateway-ports.test.ts
- test/e2e/live/hermes-inference-switch.test.ts
- test/e2e/README.md
- test/e2e/live/brave-search.test.ts
- test/e2e/live/cron-preflight-inference-local.test.ts
- test/e2e/live/hermes-slack-e2e.test.ts
- test/e2e/live/openclaw-skill-cli.test.ts
- test/e2e/live/shields-config.test.ts
- test/e2e/live/openclaw-slack-pairing.test.ts
- test/e2e/live/double-onboard.test.ts
- test/e2e/live/full-e2e.test.ts
- test/e2e/live/channels-stop-start-helpers.ts
- test/e2e/live/ollama-auth-proxy.test.ts
- test/e2e/live/credential-sanitization.test.ts
- test/e2e/live/skill-agent.test.ts
- test/e2e/live/messaging-providers.test.ts
- test/e2e/live/state-backup-restore.test.ts
- test/e2e/live/openclaw-discord-pairing.test.ts
- test/e2e/live/model-router-provider-routed-inference.test.ts
- test/e2e/live/credential-migration.test.ts
- test/e2e/live/device-auth-health.test.ts
- test/e2e/live/hermes-root-entrypoint-smoke.test.ts
- test/e2e/live/kimi-inference-compat.test.ts
- test/e2e/live/jetson-nvmap-gpu.test.ts
- test/e2e/live/ubuntu-repo-cli-smoke.test.ts
- test/e2e/support/fixtures/e2e-progress-outcome.fixture.test.ts
- test/e2e/live/hermes-gpu-startup.test.ts
- test/e2e/live/gateway-health-honest.test.ts
- test/e2e/live/runtime-overrides.test.ts
- CONTRIBUTING.md
- test/e2e/fixtures/shell-probe.ts
- test/e2e/live/onboard-resume.test.ts
- test/e2e/live/openshell-credential-generation-window.test.ts
- test/e2e/live/telegram-injection.test.ts
- test/e2e/live/openshell-version-pin.test.ts
- test/e2e/live/issue-4462-scope-upgrade-approval.test.ts
- test/e2e/live/sandbox-operations.test.ts
- test/e2e/live/hermes-sandbox-secret-boundary.test.ts
- test/e2e/live/hermes-discord.test.ts
- test/e2e/live/openshell-gateway-auth-source-contract-helpers.ts
- test/e2e/live/sandbox-rebuild.test.ts
- test/e2e/live/sandbox-survival.test.ts
- test/e2e/support/rebuild-hermes-timing.test.ts
- test/e2e/live/hermes-slack-e2e-helpers.ts
- test/e2e/support/e2e-progress-fixture.test.ts
- test/e2e/live/tunnel-lifecycle-helpers.ts
- test/e2e/live/gateway-guard-recovery.test.ts
- test/e2e/live/token-rotation.test.ts
- test/e2e/live/rebuild-openclaw.test.ts
- test/e2e/live/cloud-inference.test.ts
- test/e2e/live/agent-turn-latency-helpers.ts
- test/e2e/live/inference-routing-provider-smoke.test.ts
- test/e2e/live/gpu-e2e.test.ts
- test/e2e/live/bedrock-runtime-compatible-anthropic.test.ts
- test/e2e/live/openclaw-inference-switch.test.ts
- test/e2e/live/snapshot-commands.test.ts
- test/e2e/live/messaging-compatible-endpoint.test.ts
- test/e2e/live/sessions-agents-cli.test.ts
- test/e2e/live/hermes-e2e.test.ts
- test/e2e/fixtures/progress.ts
- test/e2e/live/overlayfs-autofix.test.ts
- test/e2e/support/agent-turn-latency-progress.test.ts
- test/e2e/fixtures/e2e-test.ts
- test/e2e/live/rebuild-hermes.test.ts
- test/e2e/live/network-policy.test.ts
- test/e2e/live/onboard-repair.test.ts
- tools/e2e/check-semantic-phases.mts
- test/e2e/live/common-egress-agent.test.ts
- test/e2e/live/inference-routing.test.ts
- test/e2e/live/state-dir-guard-metadata.test.ts
Signed-off-by: Charan Jagwani <cjagwani@nvidia.com>
Collaborator
Author
|
@coderabbitai review |
Contributor
✅ Action performedReview finished.
|
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@test/e2e/live/openshell-version-pin.test.ts`:
- Line 383: Move the enterInspectionPhase() call in the installation test to
immediately after the expect(result.status).toBe(0) assertion, ensuring
inspection begins only when installer execution succeeds and failures remain
attributed to installation.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: f33b0067-abf0-411d-babc-5f68d4c7052d
📒 Files selected for processing (7)
.github/workflows/e2e.yamltest/e2e/docs/README.mdtest/e2e/live/openshell-version-pin.test.tstest/e2e/support/e2e-semantic-phase-check.test.tstest/e2e/support/upload-e2e-artifacts-workflow-boundary.test.tstools/e2e/check-semantic-phases.mtstools/e2e/upload-e2e-artifacts-workflow-boundary.mts
🚧 Files skipped from review as they are similar to previous changes (4)
- .github/workflows/e2e.yaml
- tools/e2e/upload-e2e-artifacts-workflow-boundary.mts
- test/e2e/docs/README.md
- tools/e2e/check-semantic-phases.mts
Signed-off-by: Charan Jagwani <cjagwani@nvidia.com>
Collaborator
Author
|
@coderabbitai review |
Signed-off-by: Charan Jagwani <cjagwani@nvidia.com>
Contributor
✅ Action performedReview finished.
|
Signed-off-by: Charan Jagwani <cjagwani@nvidia.com>
Collaborator
Author
|
@coderabbitai review |
Contributor
✅ Action performedReview finished.
|
Signed-off-by: Charan Jagwani <cjagwani@nvidia.com>
Signed-off-by: Charan Jagwani <cjagwani@nvidia.com>
Signed-off-by: Charan Jagwani <cjagwani@nvidia.com>
Collaborator
Author
|
@coderabbitai review |
Contributor
✅ Action performedReview finished.
|
Contributor
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
test/e2e/mock-parity.json (1)
49-55: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winMap the semantic-progress checks for this migrated live test.
launchable-smoke.test.tsnow declares and transitions semantic phases, but this entry omits the three phase-support tests included for the other migrated live entries. The parity validator only checks declared mappings, so a phase-only regression here can select unrelated fast tests instead.Proposed mapping update
"fast": [ "test/brev-launchable-ci-cpu-checksum.test.ts", "test/e2e/support/e2e-live-target-gating.test.ts", + "test/e2e/support/e2e-progress-fixture.test.ts", + "test/e2e/support/e2e-progress-outcome.test.ts", + "test/e2e/support/e2e-semantic-phase-check.test.ts", "test/e2e/support/prepare-e2e-workflow-boundary.test.ts",🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@test/e2e/mock-parity.json` around lines 49 - 55, Update the launchable-smoke.test.ts entry in the mock-parity mapping to include the same three semantic-phase support tests used by the other migrated live entries. Preserve the existing fast test mappings and copy the established support-test identifiers exactly so phase-only regressions select the correct checks.
🧹 Nitpick comments (1)
test/e2e/support/e2e-semantic-phase-check.test.ts (1)
45-78: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winCover empty and multiple forwarded-module lists.
The validator requires exactly one forwarded module, but this test only exercises valid and invalid single-entry lists. Add
[]and a two-entry list to verify the cardinality guard and prevent future weakening to “contains the expected target.”🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@test/e2e/support/e2e-semantic-phase-check.test.ts` around lines 45 - 78, Extend the “accepts only the exact bootstrap forwarding alias” test for validateCollectedSemanticPhaseModule with empty and two-entry forwardedTestModules cases. Assert both are rejected with the existing exact-forwarding-module error, preserving the valid single-entry case and preventing acceptance based only on containing the expected target.Source: Path instructions
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@test/e2e/mock-parity.json`:
- Around line 49-55: Update the launchable-smoke.test.ts entry in the
mock-parity mapping to include the same three semantic-phase support tests used
by the other migrated live entries. Preserve the existing fast test mappings and
copy the established support-test identifiers exactly so phase-only regressions
select the correct checks.
---
Nitpick comments:
In `@test/e2e/support/e2e-semantic-phase-check.test.ts`:
- Around line 45-78: Extend the “accepts only the exact bootstrap forwarding
alias” test for validateCollectedSemanticPhaseModule with empty and two-entry
forwardedTestModules cases. Assert both are rejected with the existing
exact-forwarding-module error, preserving the valid single-entry case and
preventing acceptance based only on containing the expected target.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: ab54d457-e9a0-4238-b667-32d160c8fb7d
📒 Files selected for processing (8)
.github/workflows/e2e.yamltest/e2e/live/launchable-smoke.test.tstest/e2e/live/registry-targets.test.tstest/e2e/mock-parity.jsontest/e2e/support/e2e-fixture-context.test.tstest/e2e/support/e2e-semantic-phase-check.test.tstools/e2e/check-semantic-phases.mtsvitest.config.ts
🚧 Files skipped from review as they are similar to previous changes (4)
- .github/workflows/e2e.yaml
- test/e2e/live/registry-targets.test.ts
- vitest.config.ts
- tools/e2e/check-semantic-phases.mts
Signed-off-by: Charan Jagwani <cjagwani@nvidia.com>
Collaborator
Author
|
@coderabbitai review |
Contributor
✅ Action performedReview finished.
|
21 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Live E2E jobs now report the semantic phase they are executing without repeating Vitest's test-level start, identity, elapsed time, or completion output. Every live test declares an individualized ordered plan, and each completed phase records its outcome and duration so a stalled or failed matrix job shows where time was spent.
Related Issue
Part of #7101
Changes
passed,failed, orskippedoutcomes and per-phase durations, then append a harness-owned resource-release phase so cleanup hangs and failures are timed independently.test-progress.json, including the dedicatedskill-agentjob upload.Type of Change
Quality Gates
Verification
Signed-off-by:line and every commit appears asVerifiedin GitHubpre-commit,commit-msg, andpre-pushhooks passed, ornpm run check:diffpassed when hooks were skipped or unavailablenpm run typecheck:cli, project membership, source-shape, file-size, changed-file Biome, andgit diff --checkpassednpm testfor broad runtime/test-harness changes;npm run checkfor repo-wide validation/coverage changes — command/result: required CI is authoritative for the broad sharded suitenpm run docsbuilds without warnings (doc changes only) — validation passed; Fern reported 2 existing warningsSigned-off-by: Charan Jagwani cjagwani@nvidia.com
Summary by CodeRabbit
New Features
test-progress.jsonartifact (including target/shard metadata when available).Documentation
Tests