historic-fleet: retry transient network failures at the harness level

[davekilleen]

Problem

Three consecutive formal historic-fleet-darwin runs each failed exactly one of 202 cases on a momentary runner network blip, at different starting tags each time:

• run 31048691941 and run 31066105126 — one case each, including a DNS failure ("Could not resolve host: github.com") during a delivery hop
• run 31156498438 — starting tag dist/archive/v1.64.0-aa005a9 failed at phase follow-up-delivery with failure_reason: "network-unavailable", elapsed_ms: 134, retaining follow-up-delivery-failure.diagnostic.json

A 134 ms failure is a connectivity blip, not a real upgrade regression, yet it costs a full multi-hour fleet rerun.

Why PR #417 cannot cover this

The retry fix merged in #417 lives in the updater code inside releases. The fleet's delivery step runs the installed foundation release's (v1.81.16-era) updater via the fixture lifecycle runtime — code frozen at whatever the foundation shipped, which predates that fix. No in-release retry can ever protect the delivery of pre-fix foundations, so the protection has to live in the harness that drives them.

Retry design (harness level, executor-owned)

scripts/release_fleet_executor.py now wraps the two per-case delivery hops — the foundation bridge and the follow-up deliver_latest_release — in a bounded transient-network retry:

• Classification: a delivery attempt is retried only when it is classifiably a momentary network outage — the closed network-unavailable delivery reason, or an error message matching DNS-resolution / connection-reset / connection-refused / connection-timeout / unreachable-network fragments. Everything else (evidence-invalid, route drift, approval violations, runtime timeouts, …) raises exactly as before on the first attempt.
• Bound: at most 3 total attempts per hop, with 10 s then 30 s backoff.
• Unchanged acceptance semantics: a case that still fails after retries fails precisely as today (same diagnostic file, same ExecutorError, same 202/202 bar). Public-route drift is checked per attempt and never retried. Bridge approvals are reset per attempt so a retried bridge still enforces the exact approval word and count.
• Evidence honesty: the journey transcript's bridge-foundation and foundation-preview events now carry journey_transport: {"attempt_count": N} (mirroring the existing Doctor transport evidence), and the failure diagnostic's delivery section gains attempt_count — so a retried-then-passed case is distinguishable from a first-try pass, and an exhausted-retries failure is distinguishable from a single-shot failure.

Because the journey protocol pins the executor's exact bytes, core/update/journey-protocol-v1.json is regenerated via scripts/generate-update-journey-protocol.py.

Tests

• New executor tests: transient evidence-reason retry (passes on attempt 3, backoff schedule 10 s/30 s asserted via patched sleep), transient DNS ExecutorError retry, non-network delivery failure never retried, bounded exhaustion still fails with attempt_count: 3 in the diagnostic, bridge network blip retried with fresh exact approvals, non-network bridge failure never retried, plus direct classifier coverage.
• Updated failure-diagnostic assertions for the new attempt_count field.
• Local run (Linux): core/tests/test_release_fleet_executor.py, test_release_fleet_failure_diagnostics.py, test_update_journey_protocol.py — 60 passed; test_release_fleet.py, test_release_fleet_acceptance.py, test_historic_fleet_darwin_workflow.py — 115 passed, 3 pre-existing macOS-only failures also present on clean main; test_distribution_artifacts.py — 82 passed. Full macOS fleet behavior is exercised by the PR canary.

🤖 Generated with Claude Code

[github-actions]

What this pull request touches

• build & release — feeds turning a reviewed contribution into a safe Dex update.
• tests — feeds catching regressions before contributors and users encounter them.

Gates that will judge this change

• Personal-data gate: added lines must not expose real identities or personal vault content.
• Change-aware gates: source changes are checked for tests, path-contract use, documentation drift, and touched-file coverage.
• Tests and coverage: the Python, MCP, migration, hook, and script suites must remain healthy.
• Safety and quality: security, lint, distribution, path consistency, and large-vault checks still apply.

Based on 4 changed files.