Add secret-safe phase progress diagnostics to every E2E test

[cjagwani]

Problem Statement

The E2E workflow can spend tens of minutes inside a test without showing which operation is still running. During the v0.0.85 release qualification, agent-turn-latency remained active for roughly 50 minutes with no phase-level evidence, so maintainers could not tell whether onboarding, sandbox creation, route validation, an agent turn, or cleanup was slow or hung.

Command output is correctly kept in redacted artifacts, but the absence of safe progress signals forces maintainers to wait for the outer timeout before diagnosis can begin. The same blind spot can occur in any E2E target, especially large matrix targets and historically flaky scenarios.

Desired Behavior

Every E2E test represented by .github/workflows/e2e.yaml should emit consistent, secret-safe progress diagnostics from start through cleanup.

At minimum, logs should show:

• the E2E target/scenario and active semantic phase;
• total and phase elapsed time;
• how long it has been since the child process produced output, without forwarding that output;
• runner resource context such as free memory, test-process RSS, workspace capacity, and load when available;
• immediate phase-start/phase-finish events plus a bounded heartbeat while a phase remains active; and
• explicit completion or cleanup progress so teardown hangs are distinguishable from test-body hangs.

For example, instead of an otherwise silent 50-minute job, maintainers should see that agent-turn-latency has spent 12 minutes in Hermes install attempt 1, last observed child output 40 seconds ago, and has adequate runner memory and disk.

Completion means every test selected by the authoritative E2E workflow is accounted for. The rollout may prioritize the slowest and flakiest targets first, but it is not complete after instrumenting only those targets.

Constraints and Non-goals

• Do not print raw child stdout/stderr, request bodies, credentials, tokens, or other potentially sensitive values.
• Do not raise timeouts, add retries, or otherwise mask a real hang; this work makes stalls diagnosable rather than changing acceptance behavior.
• Do not assume shared harness instrumentation is sufficient. A common fixture or command boundary may reduce duplication only when it provides phase names that are meaningful for that specific test.
• Keep independent per-test instrumentation in scope, including instrumenting every test explicitly if the coverage audit shows that is necessary to expose each scenario's semantic phases.
• Keep heartbeat volume bounded; phase changes should log immediately and stable phases should report at a consistent interval.
• Preserve redacted failure artifacts as the detailed post-failure evidence source.

Implementation Idea

1. Build an inventory from .github/workflows/e2e.yaml and the generated workflow plan, mapping every selected target to its live test or script entrypoint. This is the coverage ledger and prevents silent omissions.
2. Audit recent run data for duration, timeouts, failures, and retries. Rank targets by p95/max runtime and failure frequency to choose rollout order, while retaining all targets in the ledger.
3. Establish one shared diagnostic contract: scenario, phase, total/phase elapsed time, timestamp-only output liveness, resource snapshot, heartbeat interval, and idempotent stop behavior.
4. Evaluate common host-command, sandbox-command, fixture lifecycle, and cleanup boundaries as optional reuse points; verify their phase labels remain meaningful for each consuming test rather than assuming shared instrumentation solves coverage.
5. Add explicit phase transitions independently to every test that needs them. This may mean instrumenting every E2E test directly. Instrument large/flaky targets first, then complete the remaining workflow inventory in batches without narrowing the final scope.
6. Add E2E-support contract tests that prove diagnostics never include child payloads and that every workflow-selected test has progress coverage. The coverage check should fail when a new E2E target is added without diagnostics.
7. Validate representative targets with selective live runs and record before/after evidence showing that a slow or hung phase can be identified before the outer timeout.
8. Document the progress contract in the E2E contributor guide once the shared integration boundary is stable.

Draft PR #7100 is the first implementation slice: it adds the shared secret-safe heartbeat and applies it to agent-turn-latency, while reusing it for the existing rebuild-Hermes reporter.

Category

Testing

Checklist

• I searched existing issues and this is not a duplicate
• I described the problem and desired behavior