Epic: reduce E2E wall time, runner cost, and failure surface

[apurvvkumaria]

Problem

NemoClaw's full scheduled E2E workflow takes too long even with broad parallelism, consumes substantial runner capacity, and exposes the run to many independent failure opportunities. Adding more standard runners alone will not address the main bottleneck because recent queue time is low; the cost is dominated by long-running test setup and execution.

A recent full scheduled run took about 29.6 minutes wall-clock, consumed about 592.5 runner-minutes, and scheduled 83 non-skipped jobs. Across five sampled scheduled runs, median wall time was about 32.1 minutes and median runner consumption was about 595.4 minutes. All five sampled runs failed, which also indicates an excessive failure surface.

Reference run: https://github.com/NVIDIA/NemoClaw/actions/runs/30503498077

Scope boundaries with #7614 and #7451

This epic is a follow-on to existing E2E cleanup and feedback-loop work, not a duplicate.

Owner	Scope
#7614	Determine which E2E checks do not protect a unique live boundary; relocate useful assertions into fast projects; retire the corresponding jobs, targets, and matrix rows.
#7451	Reconstruct the specific mixed #7262 diff as focused PRs, including its platform prerequisites and sharding, Hermes production-image reuse, main-workflow scheduling, orchestration timeouts, Brev disposition, protected authorization documentation, and direct PR-gate architecture.
#7912	Improve the runner infrastructure, implementation time, and selective scheduling of the live E2E journeys retained after #7614 and the completed #7451 slices.

The following work is already owned by #7614 and its Priority 2 child #7615, with implementation in #7665, and is explicitly out of scope here:

• Retiring sandbox-rebuild and upgrade-stale-sandbox.
• Retiring hosted OpenClaw and Hermes inference-switch rows.
• Retiring the Hermes Bedrock row while retaining the canonical OpenClaw seam.
• Moving the associated configuration, classification, output, and state assertions into fast tests.

The following work is already owned by #7451 and is explicitly out of scope here:

• Hosted-runner-loss retry, landed in fix(ci): retry confirmed hosted runner loss once #7423.
• macOS and WSL platform sharding, landed in ci(platform): shard macOS and WSL Vitest suites #7548.
• Hermes production-image build-once producer/consumer reuse and no-rebuild enforcement, landed in ci(e2e): reuse the Hermes production image #7508 and test(e2e): reject continued Hermes consumer builds #7553.
• Explicit E2E orchestration job timeouts, landed in ci(e2e): add orchestration job timeouts #7537.
• Starting sandbox image validation after main-workflow preflight, landed in ci(main): start sandbox validation after preflight #7552.
• Legacy Brev source-install disposition, owned by ci(e2e): migrate or retire legacy Brev source-install validation #7490 and implemented by ci(e2e): retire legacy Brev source-install validation #7632.
• Protected internal E2E authorization documentation and direct E2E / PR Gate check architecture that remain tracked by ci: split E2E feedback-loop work into focused PRs #7451.

Guardrails:

1. Do not create a Epic: reduce E2E wall time, runner cost, and failure surface #7912 child issue that reimplements work listed in Epic: move non-live E2E checks into fast test projects #7614, Priority 2: consolidate overlapping live E2E seams #7615, test(e2e): consolidate priority 2 live seams #7665, or ci: split E2E feedback-loop work into focused PRs #7451.
2. Establish the Epic: reduce E2E wall time, runner cost, and failure surface #7912 baseline only after test(e2e): consolidate priority 2 live seams #7665 is merged or its final retained/removal decisions are reflected.
3. If Epic: reduce E2E wall time, runner cost, and failure surface #7912 analysis finds that a check is not a unique live boundary, route its relocation or retirement back to Epic: move non-live E2E checks into fast test projects #7614 rather than duplicating that work here.
4. If a proposal reconstructs or completes a perf(ci): shorten main and E2E feedback loops #7262 slice, route it to ci: split E2E feedback-loop work into focused PRs #7451 rather than duplicating it here.
5. Epic: reduce E2E wall time, runner cost, and failure surface #7912 child issues may optimize or selectively schedule a retained seam, but must name that seam and record why it remains live.

Goal

Reduce feedback time and runner cost for the retained live E2E suite while preserving every distinct real process, platform, security, migration, and external-system boundary.

Initial measurable targets:

• Nightly E2E p95 wall time: 15–20 minutes over at least five consecutive runs.
• Nightly median runner consumption: under 300 runner-minutes.
• PR-selected E2E: normally 15 minutes or less.
• Weekly full compatibility suite: 45 minutes or less, excluding separate release/Brev validation.

Workstream 1: infrastructure for retained live journeys

• Activate the existing larger-runner routing as a controlled A/B test. Start with the documented 8-vCPU, 32-GB RAM, 300-GB-disk profile and maximum concurrency of four.
• Route only telemetry-proven CPU-, memory-, or disk-heavy retained jobs to larger runners.
• Do not add standard-runner capacity unless p95 queue time remains above five minutes across several runs. Recent queue time is too low to justify that as the first fix.
• Publish digest-pinned reusable legacy/versioned images, plugin fixtures, and exact-commit CLI artifacts so retained jobs do not rebuild identical inputs. This does not reopen the Hermes production-image producer/consumer reuse completed by ci(e2e): reuse the Hermes production image #7508/test(e2e): reject continued Hermes consumer builds #7553.
• Consider ephemeral self-hosted runners only for trusted-main/release workloads if larger hosted runners still show infrastructure loss.
• Keep phase timing, CPU, memory, disk, Docker, queue, retry, and runner-loss telemetry available for comparison.

Workstream 2: make retained live journeys faster

Prioritize the persistent critical paths that remain after #7665:

• Split the two independent DeepAgents MCP bridge scenarios into parallel jobs or share their setup; target about 12 minutes wall-clock.
• Optimize the retained openclaw-plugin-runtime-exdev runtime boundary so it does not perform unnecessary repeated setup; target 15–18 minutes.
• Consume reusable digest-pinned historical OpenClaw and gateway fixtures where doing so preserves the retained live contract.
• Optimize the retained token-rotation transition and no-op paths; any decision to move or retire non-live provider-matrix assertions remains owned by Epic: move non-live E2E checks into fast test projects #7614.
• Separate retained functional crash recovery from long soak execution: run a short recovery path nightly and the full repeated-cycle soak weekly.
• Build the NemoClaw CLI once per workflow and distribute a hashed, exact-commit artifact to retained jobs instead of rebuilding it repeatedly.
• Replace fixed sleeps with state-driven readiness checks except when elapsed soak time is itself the contract.

Recent slow paths include openclaw-plugin-runtime-exdev, DeepAgents MCP bridge, legacy gateway upgrade, token rotation, rebuild-openclaw, channel stop/start, and crash-loop recovery. upgrade-stale-sandbox is excluded because its retirement is already owned by #7614/#7665.

Workstream 3: selectively schedule retained live coverage

Classify the live journeys retained after #7614 into execution tiers:

• Pull requests: use changed-area mapping and run only one to three canonical retained live journeys.
• Nightly: run roughly 10–15 core retained product journeys.
• Weekly: run retained compatibility variants, historical-version coverage, provider variants, and soak tests.
• Release: run the exact-commit retained full suite plus staging/Brev launch validation.

Candidate scheduling changes:

• Keep one current retained gateway migration row nightly; move other retained architecture or historical variants weekly.
• Run retained release-baseline plugin compatibility weekly and for releases.
• Keep one representative retained messaging lifecycle nightly; run provider-specific live variants selectively or weekly.
• Run retained issue-specific regression soaks weekly or when their owning files change.
• Support-layer changes should normally run e2e-support; map a changed live test file to its canonical retained job instead of broad unrelated E2E groups.

This workstream changes when retained live coverage runs. It does not own assertion relocation or test retirement from #7614, and it does not own main-workflow sandbox scheduling or required-check architecture from #7451.

Proposed child-issue groups

Child issues will be created separately after #7665 is accounted for. Expected groups are:

1. Larger-runner A/B experiment and post-test(e2e): consolidate priority 2 live seams #7665 telemetry baseline.
2. Reusable legacy/versioned images and exact-commit build artifacts for retained jobs, excluding the Hermes production-image reuse completed under ci: split E2E feedback-loop work into focused PRs #7451.
3. DeepAgents MCP bridge parallelization.
4. Retained OpenClaw plugin EXDEV path optimization.
5. Retained token-rotation path optimization.
6. Recovery-versus-soak scheduling split.
7. Retained gateway and compatibility scheduling tiers.
8. Changed-area dispatch for retained live journeys.
9. Runtime-budget and scheduling policy for retained live E2E.

Acceptance criteria

• test(e2e): consolidate priority 2 live seams #7665 is merged or its final retained/removal decisions are reflected before child work begins.
• No child issue duplicates Epic: move non-live E2E checks into fast test projects #7614/Priority 2: consolidate overlapping live E2E seams #7615/test(e2e): consolidate priority 2 live seams #7665 assertion-relocation or retirement work.
• No child issue duplicates a perf(ci): shorten main and E2E feedback loops #7262 slice or another deliverable owned by ci: split E2E feedback-loop work into focused PRs #7451.
• Each child issue names the retained live boundary, baseline, expected savings, owner, and validation plan.
• Nightly p95 wall time is 20 minutes or less over five consecutive runs.
• Nightly median runner consumption is under 300 runner-minutes over the same sample.
• PR-selected E2E normally completes in 15 minutes or less.
• Unique process, platform, security, migration, and external-system boundaries remain covered.
• Full compatibility and soak coverage has an explicit weekly or release schedule.
• Runner-loss failures are distinguishable from product/test failures through telemetry.