Skip to content

ci(e2e): derive free-standing selectors from workflow#5370

Merged
cv merged 3 commits into
mainfrom
codex/derive-e2e-selectors-from-workflow
Jun 13, 2026
Merged

ci(e2e): derive free-standing selectors from workflow#5370
cv merged 3 commits into
mainfrom
codex/derive-e2e-selectors-from-workflow

Conversation

@cv

@cv cv commented Jun 12, 2026

Copy link
Copy Markdown
Collaborator

Summary

Derives free-standing E2E Vitest job and scenario selectors from metadata in .github/workflows/e2e-vitest-scenarios.yaml instead of the shared env-file registry. This removes the append-only selector file as a merge-conflict point while keeping CI selector validation strict.

Related Issue

Refs #5098

Changes

  • Add FREE_STANDING_VITEST_JOB and FREE_STANDING_SCENARIO_ID metadata to free-standing E2E jobs.
  • Add tools/e2e-scenarios/free-standing-workflow-inventory.mts to emit shell-safe selector inventory from workflow YAML.
  • Update workflow boundary helpers and tests to derive and validate selector inventory from workflow metadata.
  • Delete tools/e2e-scenarios/free-standing-jobs.env.

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Verification

  • npx prek run --all-files passes
  • npm test passes
  • Tests added or updated for new or changed behavior
  • No secrets, API keys, or credentials committed
  • Docs updated for user-facing behavior changes
  • npm run docs builds without warnings (doc changes only)
  • Doc pages follow the style guide (doc changes only)
  • New doc pages include SPDX header and frontmatter (new pages only)

Signed-off-by: Carlos Villela cvillela@nvidia.com

Summary by CodeRabbit

  • Chores

    • E2E workflow now derives free-standing test inventory dynamically from workflow definitions (removed reliance on a static allowlist file).
    • CI jobs are tagged with environment markers to identify free-standing Vitest runs and, where applicable, include a scenario ID.
  • Tests

    • Updated and added tests to validate the derived inventory, error handling for malformed workflow metadata, and boundary checks for newly marked free-standing jobs.

Signed-off-by: Carlos Villela <cvillela@nvidia.com>
@cv cv self-assigned this Jun 12, 2026
@coderabbitai

coderabbitai Bot commented Jun 12, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 67c66c30-7dc8-4a6e-9833-4846753eca9e

📥 Commits

Reviewing files that changed from the base of the PR and between 37ef225 and 0328dbe.

📒 Files selected for processing (3)
  • .github/workflows/e2e-vitest-scenarios.yaml
  • test/e2e-scenario/support-tests/e2e-scenarios-workflow.test.ts
  • tools/e2e-scenarios/workflow-boundary.mts
🚧 Files skipped from review as they are similar to previous changes (2)
  • tools/e2e-scenarios/workflow-boundary.mts
  • .github/workflows/e2e-vitest-scenarios.yaml

📝 Walkthrough

Walkthrough

Free-standing Vitest inventory is derived from workflow job YAML markers; a new TypeScript CLI exports it as shell key=value or JSON; the workflow’s matrix generator invokes and strictly validates that output; jobs are tagged with env markers; boundary validators and tests were updated to use the derived inventory.

Changes

Free-standing inventory migration

Layer / File(s) Summary
Inventory derivation and validation contracts
tools/e2e-scenarios/workflow-boundary.mts
Adds FREE_STANDING_WORKFLOW_INVENTORY_SCRIPT, marker constants, deriveFreeStandingJobsInventoryFromJobs, readWorkflowRecord, readFreeStandingJobsInventory, validateFreeStandingWorkflowInventory, and formatFreeStandingJobsInventoryForShell; integrates derived inventory into validators and coverage checks.
Workflow job metadata markers
.github/workflows/e2e-vitest-scenarios.yaml
Adds FREE_STANDING_VITEST_JOB: "1" and FREE_STANDING_SCENARIO_ID (where applicable) to free-standing Vitest jobs; moves NVIDIA_API_KEY to step-level env for one job.
CLI tool for inventory output
tools/e2e-scenarios/free-standing-workflow-inventory.mts
New CLI with usage, parseArgs for --shell/--workflow/--help, reads derived inventory, emits shell key=value lines or JSON, and reports errors as ::error:: plus non-zero exit code on failure.
Workflow matrix generation and validation
.github/workflows/e2e-vitest-scenarios.yaml
generate-matrix invokes the inventory CLI, parses stdout as strict data-only key=value lines, rejects duplicates/redefinitions, requires specific keys, validates allowed job IDs, scenario IDs, and scenario:job mappings, and produces the same matrix outputs.
Selector evaluation and coverage validation
tools/e2e-scenarios/workflow-boundary.mts
evaluateE2eVitestWorkflowDispatchSelectors, unknown-job checks, default selection, scenario→job lookup, and coverage validators now consume the derived FreeStandingJobsInventory instance.
Test updates
test/e2e-scenario/support-tests/e2e-scenarios-workflow.test.ts
Replace env-file validation with workflow-derived inventory tests, add malformed-workflow negative tests that assert validator errors and CLI ::error:: output, refactor runtime-overrides tests, and add boundary checks for newly marked free-standing jobs.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~50 minutes

Possibly related PRs

Suggested labels

area: ci, area: e2e, chore

Suggested reviewers

  • prekshivyas

Poem

🐰 I hopped from env files into YAML light,
Markers on jobs made inventory bright.
A tiny CLI sings key=value tunes,
The matrix parses, tests hum in tune,
Hooray — the workflow hops and runs tonight!

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title 'ci(e2e): derive free-standing selectors from workflow' clearly and concisely summarizes the main change: deriving E2E selectors from workflow metadata instead of an env file.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/derive-e2e-selectors-from-workflow

Comment @coderabbitai help to get the list of available commands and usage tips.

@github-actions

github-actions Bot commented Jun 12, 2026

Copy link
Copy Markdown
Contributor

E2E Advisor Recommendation

Required E2E: e2e-vitest-scenarios-generate-matrix, openshell-version-pin-vitest
Optional E2E: gateway-guard-recovery, onboard-negative-paths-vitest

Dispatch hint: openshell-version-pin-vitest

Workflow run

Full advisor summary

E2E Recommendation Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required E2E

  • e2e-vitest-scenarios-generate-matrix (low): Required because the changed workflow now executes tools/e2e-scenarios/free-standing-workflow-inventory.mts during matrix generation; dispatching a selected free-standing job validates the new inventory derivation, shell formatting, data-only parsing, allowed-jobs validation, and selector gating before any downstream job runs.
  • openshell-version-pin-vitest (low): Required as the lowest-cost existing free-standing Vitest E2E job to prove the updated workflow can route a jobs selector through the derived inventory and execute a selected free-standing job end-to-end without running the entire live scenario suite.

Optional E2E

  • gateway-guard-recovery (high): Optional because this PR specifically moves NVIDIA_API_KEY from the gateway-guard-recovery job environment to the run-step environment. Running it would validate that the recovery scenario still receives the secret where needed, but it is a heavier disruption/recovery lane and not necessary for basic workflow-inventory confidence.
  • onboard-negative-paths-vitest (low): Optional adjacent smoke for a CLI-building free-standing lane selected by the new inventory metadata; useful if reviewers want a second inexpensive selector/job shape beyond openshell-version-pin.

New E2E recommendations

  • E2E workflow selector self-validation (medium): Existing support tests validate selector behavior in-process, but there is no dedicated cheap workflow-dispatch E2E that exercises both jobs= and scenarios= selector inputs against a no-network free-standing job and asserts the expected job is selected from the derived workflow metadata.
    • Suggested test: Add a lightweight workflow-dispatch smoke or reusable check that dispatches e2e-vitest-scenarios with scenarios=openshell-version-pin and jobs=openshell-version-pin-vitest in separate dry-run-style validations and verifies the derived inventory selects only openshell-version-pin-vitest.

Dispatch hint

  • Workflow: .github/workflows/e2e-vitest-scenarios.yaml
  • jobs input: openshell-version-pin-vitest

@github-actions

github-actions Bot commented Jun 12, 2026

Copy link
Copy Markdown
Contributor

Vitest E2E Scenario Recommendation

Required Vitest E2E scenarios: e2e-scenarios-all
Optional Vitest E2E scenarios: None

Dispatch required Vitest E2E scenarios:

  • gh workflow run e2e-vitest-scenarios.yaml --ref <pr-head-ref>

Workflow run

Full Vitest E2E advisor summary

Vitest E2E Scenario Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required Vitest E2E scenarios

  • e2e-scenarios-all: The PR changes the shared Vitest scenario workflow machinery, matrix/free-standing job inventory derivation, and workflow boundary tests, so the full Vitest scenario fan-out should validate registry scenarios and free-standing jobs through the canonical workflow.
    • Dispatch: gh workflow run e2e-vitest-scenarios.yaml --ref <pr-head-ref>

Optional Vitest E2E scenarios

  • None.

Relevant changed files

  • .github/workflows/e2e-vitest-scenarios.yaml
  • test/e2e-scenario/support-tests/e2e-scenarios-workflow.test.ts
  • tools/e2e-scenarios/free-standing-jobs.env
  • tools/e2e-scenarios/free-standing-workflow-inventory.mts
  • tools/e2e-scenarios/workflow-boundary.mts

@github-actions

github-actions Bot commented Jun 12, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor

Findings: 0 needs attention, 1 worth checking, 0 nice ideas
Since last review: 0 prior items resolved, 1 still applies, 0 new items found

Review findings

🛠️ Needs attention

  • None.

🔎 Worth checking

  • Branch-controlled inventory helper gates E2E selectors (.github/workflows/e2e-vitest-scenarios.yaml:53): The workflow now derives the free-standing selector inventory by executing repo TypeScript in generate-matrix. This is not a confirmed vulnerability: generate-matrix does not receive secrets, top-level permissions are read-only, checkout credentials are disabled, dependencies install with npm ci --ignore-scripts, actions are pinned, and the helper output is revalidated as restrictive data-only key/value lines. Still, the source of truth moved from a static data file to executable branch-controlled code that influences which later secret-bearing E2E jobs are considered selected.
    • Recommendation: Keep the helper intentionally small and covered by workflow-boundary tests. Have maintainers explicitly accept that executing this branch-controlled TypeScript in generate-matrix is the intended replacement for the deleted data-only inventory, and keep future changes to this helper under the same workflow trusted-code-boundary review.
    • Evidence: generate-matrix sets inventory_output="$(npx tsx tools/e2e-scenarios/free-standing-workflow-inventory.mts --shell)", then validates allowed_jobs, free_standing_scenarios_csv, and free_standing_scenario_jobs_csv before selector routing. The new tests cover malformed marker metadata, selector drift, newly marked job boundary checks, and report-to-pr coverage.

🌱 Nice ideas

  • None.
Consider writing more tests for
  • **Runtime validation** — Runtime workflow_dispatch with jobs=openshell-version-pin-vitest produces matrix=[] and selects only openshell-version-pin-vitest among free-standing jobs.. Local support tests are strong and spawn the extracted generate-matrix Bash script, but the changed selector routing still depends on real GitHub Actions expression evaluation, needs/if routing, fromJSON behavior, and npx tsx execution after npm ci --ignore-scripts.
  • **Runtime validation** — Runtime workflow_dispatch with scenarios=openshell-version-pin produces matrix=[] and routes to openshell-version-pin-vitest without adding openshell-version-pin to the registry matrix.. Local support tests are strong and spawn the extracted generate-matrix Bash script, but the changed selector routing still depends on real GitHub Actions expression evaluation, needs/if routing, fromJSON behavior, and npx tsx execution after npm ci --ignore-scripts.
  • **Runtime validation** — Runtime workflow_dispatch with scenarios=network-policy,ubuntu-repo-cloud-openclaw routes network-policy-vitest and keeps only ubuntu-repo-cloud-openclaw in the registry matrix.. Local support tests are strong and spawn the extracted generate-matrix Bash script, but the changed selector routing still depends on real GitHub Actions expression evaluation, needs/if routing, fromJSON behavior, and npx tsx execution after npm ci --ignore-scripts.
  • **Runtime validation** — Runtime workflow_dispatch with malformed jobs=network-policy-vitest,../escape fails in generate-matrix before any secret-bearing E2E job is selected.. Local support tests are strong and spawn the extracted generate-matrix Bash script, but the changed selector routing still depends on real GitHub Actions expression evaluation, needs/if routing, fromJSON behavior, and npx tsx execution after npm ci --ignore-scripts.
  • **Runtime validation** — Runtime workflow_dispatch with both jobs and scenarios fails in generate-matrix before any secret-bearing E2E job is selected.. Local support tests are strong and spawn the extracted generate-matrix Bash script, but the changed selector routing still depends on real GitHub Actions expression evaluation, needs/if routing, fromJSON behavior, and npx tsx execution after npm ci --ignore-scripts.
  • **Acceptance clause:** Refs Epic: Migrate legacy bash E2E into the Vitest E2E system #5098 — add test evidence or identify existing coverage. The deterministic context did not provide issue Epic: Migrate legacy bash E2E into the Vitest E2E system #5098 body or comments and linkedIssues was empty, so issue-level acceptance clauses could not be extracted or mapped literally.
Since last review details

Current findings:

  • Branch-controlled inventory helper gates E2E selectors (.github/workflows/e2e-vitest-scenarios.yaml:53): The workflow now derives the free-standing selector inventory by executing repo TypeScript in generate-matrix. This is not a confirmed vulnerability: generate-matrix does not receive secrets, top-level permissions are read-only, checkout credentials are disabled, dependencies install with npm ci --ignore-scripts, actions are pinned, and the helper output is revalidated as restrictive data-only key/value lines. Still, the source of truth moved from a static data file to executable branch-controlled code that influences which later secret-bearing E2E jobs are considered selected.
    • Recommendation: Keep the helper intentionally small and covered by workflow-boundary tests. Have maintainers explicitly accept that executing this branch-controlled TypeScript in generate-matrix is the intended replacement for the deleted data-only inventory, and keep future changes to this helper under the same workflow trusted-code-boundary review.
    • Evidence: generate-matrix sets inventory_output="$(npx tsx tools/e2e-scenarios/free-standing-workflow-inventory.mts --shell)", then validates allowed_jobs, free_standing_scenarios_csv, and free_standing_scenario_jobs_csv before selector routing. The new tests cover malformed marker metadata, selector drift, newly marked job boundary checks, and report-to-pr coverage.

Workflow run details

This is an automated advisory review. A human maintainer must make the final merge decision.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tools/e2e-scenarios/workflow-boundary.mts`:
- Around line 1910-1912: The inventory-derived jobs returned by
deriveFreeStandingJobsInventoryFromJobs are only passed to
validateFreeStandingInventoryCoverage and therefore skip boundary checks (needs,
shared selector if, pinned actions, secret placement) enforced by
validateE2eVitestScenariosWorkflowBoundary; update the flow so that
freeStandingInventory (from deriveFreeStandingJobsInventoryFromJobs) is also
passed through validateFreeStandingJobSelector and any existing lane-specific
validators used for explicit free-standing jobs (e.g. the secret-bearing lanes)
— add calls to validateFreeStandingJobSelector(freeStandingInventory, ...) and
route secret-bearing inventory entries to the dedicated validators used by
validateE2eVitestScenariosWorkflowBoundary so those checks (needs, if selectors,
pinned actions, secret placement) run on derived-inventory jobs as well.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: fc4022c5-12aa-46ea-af25-3a96426397b1

📥 Commits

Reviewing files that changed from the base of the PR and between ebd6127 and 37ef225.

📒 Files selected for processing (5)
  • .github/workflows/e2e-vitest-scenarios.yaml
  • test/e2e-scenario/support-tests/e2e-scenarios-workflow.test.ts
  • tools/e2e-scenarios/free-standing-jobs.env
  • tools/e2e-scenarios/free-standing-workflow-inventory.mts
  • tools/e2e-scenarios/workflow-boundary.mts
💤 Files with no reviewable changes (1)
  • tools/e2e-scenarios/free-standing-jobs.env

Comment thread tools/e2e-scenarios/workflow-boundary.mts
@cv cv added the v0.0.65 Release target label Jun 13, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Vitest E2E Scenario Results — ❌ Some jobs failed

Run: 27449811792
Workflow ref: codex/derive-e2e-selectors-from-workflow
Requested scenarios: (default — all supported)
Requested jobs: (default — all free-standing when no scenarios are requested)
Summary: 17 passed, 7 failed, 0 skipped

Job Result
credential-migration-vitest ✅ success
credential-sanitization-vitest ❌ failure
double-onboard-vitest ✅ success
gateway-guard-recovery ✅ success
generate-matrix ✅ success
hermes-e2e-vitest ✅ success
hermes-root-entrypoint-smoke-vitest ✅ success
inference-routing-vitest ✅ success
issue-4434-tui-unreachable-inference-vitest ❌ failure
launchable-smoke-vitest ✅ success
live-scenarios ❌ failure
messaging-providers-vitest ❌ failure
model-router-provider-routed-inference-vitest ✅ success
network-policy-vitest ❌ failure
onboard-negative-paths-vitest ✅ success
openclaw-tui-chat-correlation-vitest ❌ failure
openshell-version-pin-vitest ✅ success
rebuild-openclaw-vitest ✅ success
runtime-overrides-vitest ✅ success
sandbox-rebuild-vitest ✅ success
sandbox-survival-vitest ❌ failure
shields-config-vitest ✅ success
skill-agent-vitest ✅ success
token-rotation-vitest ✅ success

Failed jobs: credential-sanitization-vitest, issue-4434-tui-unreachable-inference-vitest, live-scenarios, messaging-providers-vitest, network-policy-vitest, openclaw-tui-chat-correlation-vitest, sandbox-survival-vitest. Check run artifacts for logs.

@github-actions

Copy link
Copy Markdown
Contributor

Vitest E2E Scenario Results — ❌ Some jobs failed

Run: 27450569888
Workflow ref: codex/derive-e2e-selectors-from-workflow
Requested scenarios: (default — all supported)
Requested jobs: (default — all free-standing when no scenarios are requested)
Summary: 19 passed, 5 failed, 0 skipped

Job Result
credential-migration-vitest ✅ success
credential-sanitization-vitest ✅ success
double-onboard-vitest ✅ success
gateway-guard-recovery ❌ failure
generate-matrix ✅ success
hermes-e2e-vitest ✅ success
hermes-root-entrypoint-smoke-vitest ✅ success
inference-routing-vitest ✅ success
issue-4434-tui-unreachable-inference-vitest ❌ failure
launchable-smoke-vitest ✅ success
live-scenarios ❌ failure
messaging-providers-vitest ✅ success
model-router-provider-routed-inference-vitest ✅ success
network-policy-vitest ✅ success
onboard-negative-paths-vitest ✅ success
openclaw-tui-chat-correlation-vitest ❌ failure
openshell-version-pin-vitest ✅ success
rebuild-openclaw-vitest ✅ success
runtime-overrides-vitest ✅ success
sandbox-rebuild-vitest ✅ success
sandbox-survival-vitest ❌ failure
shields-config-vitest ✅ success
skill-agent-vitest ✅ success
token-rotation-vitest ✅ success

Failed jobs: gateway-guard-recovery, issue-4434-tui-unreachable-inference-vitest, live-scenarios, openclaw-tui-chat-correlation-vitest, sandbox-survival-vitest. Check run artifacts for logs.

@github-actions

Copy link
Copy Markdown
Contributor

Vitest E2E Scenario Results — ✅ All jobs passed

Run: 27450569888
Workflow ref: codex/derive-e2e-selectors-from-workflow
Requested scenarios: (default — all supported)
Requested jobs: (default — all free-standing when no scenarios are requested)
Summary: 24 passed, 0 failed, 0 skipped

Job Result
credential-migration-vitest ✅ success
credential-sanitization-vitest ✅ success
double-onboard-vitest ✅ success
gateway-guard-recovery ✅ success
generate-matrix ✅ success
hermes-e2e-vitest ✅ success
hermes-root-entrypoint-smoke-vitest ✅ success
inference-routing-vitest ✅ success
issue-4434-tui-unreachable-inference-vitest ✅ success
launchable-smoke-vitest ✅ success
live-scenarios ✅ success
messaging-providers-vitest ✅ success
model-router-provider-routed-inference-vitest ✅ success
network-policy-vitest ✅ success
onboard-negative-paths-vitest ✅ success
openclaw-tui-chat-correlation-vitest ✅ success
openshell-version-pin-vitest ✅ success
rebuild-openclaw-vitest ✅ success
runtime-overrides-vitest ✅ success
sandbox-rebuild-vitest ✅ success
sandbox-survival-vitest ✅ success
shields-config-vitest ✅ success
skill-agent-vitest ✅ success
token-rotation-vitest ✅ success

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

v0.0.65 Release target

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants