perf(e2e): reduce rebuild-hermes runtime with pinned fixtures and reusable build cache

[apurvvkumaria]

Parent epic: #7140

Problem

The rebuild-hermes E2E lanes spend most of their wall time preparing images and dependencies rather than exercising the rebuild behavior.

Recent evidence recorded in #7140 shows:

• normal lane: about 16m19s;
• stale-base lane: about 18m22s;
• roughly 90% of the time is image, installer, and dependency preparation; and
• the rebuild operation itself is roughly 59–80 seconds.

The normal lane currently builds the current Hermes base during setup and then reaches a forced-build path during rebuild because the synthetic old sandbox has no resolvable base-image metadata. The same expensive dependency layers can therefore be prepared twice in one job.

Scope

• Remove the redundant normal-lane current-base build, or pass immutable image/ref metadata from setup into rebuild so the exact prepared image is reused.
• Publish the fixed old Hermes E2E fixture as an immutable digest and pull it instead of rebuilding it in every job.
• Evaluate an immutable current-base fixture where that does not bypass the behavior under test.
• Persist reusable BuildKit cache across jobs and add cache mounts for npm and uv downloads while preserving lockfile and digest validation.
• Separate installer/bootstrap contract coverage from rebuild-state coverage where the full installer is not part of the assertion.
• Keep the stale-base lane's real regression boundary: it must still prove stale local metadata is detected and refreshed.
• Coordinate any general prebuilt-artifact contract with Support reusable prebuilt sandbox artifacts in nemoclaw onboard #6402 without expanding this issue into product-facing onboarding behavior.

Acceptance criteria

• The normal lane constructs or downloads the current Hermes base once, not twice.
• The stale fixture is pinned by immutable digest and its provenance is verified before use.
• Both lanes still verify state backup/restore, sandbox replacement, readiness, and the expected final base digest.
• Before/after timings are collected on the same runner class across repeated runs.
• Median wall time improves by at least 25% without weakening assertions or increasing retry count.
• Cache misses and fixture publication failures fail clearly and do not silently fall back to an unpinned image.

Non-goals

• Changing Hermes product behavior.
• Masking deterministic test failures with retries.
• Removing the stale-base regression lane.

[atulya-singh]

Tracking the incremental PRs for this issue (each is scoped to land independently; none closes the issue on its own):

• perf(e2e): drop redundant current-base build from rebuild-hermes normal lane #7148 — perf(e2e): drop the redundant normal-lane current-base build so the current Hermes base is constructed exactly once (acceptance criterion 1, normal lane). Phase 6's real forced-build path is preserved.
• test(e2e): capture rebuild-hermes phase timings for perf comparison #7149 — test(e2e): capture per-phase and total wall time as a rebuild-hermes-timing.json artifact tagged with a runner-class fingerprint, establishing the before/after baseline required by acceptance criterion 4.

Still open, sequenced after the baseline above:

• Pin the old Hermes fixture by immutable digest and pull it (criterion 2) — needs coordination with Support reusable prebuilt sandbox artifacts in nemoclaw onboard #6402 on the prebuilt-artifact contract.
• Reusable build cache (scope item 4) — blocked on a maintainer design decision: requireDockerEngineRebuilds in tools/e2e/workflow-boundary.mts intentionally rejects any external buildx builder / registry cache in rebuild-* jobs ("must keep rebuild builds on the Docker engine cache"). Landing cross-job BuildKit cache would require changing that security-classified boundary.
• Separate installer/bootstrap contract coverage from rebuild-state coverage (scope item 5).

The stale-base regression boundary is preserved throughout.