Define clean-install HA qualification

[ankitgoswami]

Reviewable diff: +215/-24 across 3 files (excludes generated, test, and story files).

Summary

Defines the evidence required before the fixed three-host Fleet application, database, DCS, and VIP profile is called supported at a recorded fleet scale and topology. The report remains pending until the packaged release passes every gate on the three Raspberry Pis; update qualification and deferred Fleet Node, scheduler-recovery, and alert-delivery behavior are separate.

Stack: #887 -> #888 -> #889 -> #890 -> #891 -> #892. This documentation-only diff is relative to installer PR #888. Update qualification is deferred to #892.

How it works

Operators install the same packaged release concurrently on three clean hosts and record the release bundle and deployment-manifest digests, container image identities, host-package versions, host hardware, miner count, plugin mix, connection topology, command backlog, and timing evidence. The checklist verifies reboot persistence, independent and combined Fleet/database host loss, acknowledged-write durability, direct host-pinned writer exclusivity, passive request rejection, demotion cancellation, observable firewall behavior, stale command completion rejection, an active curtailment target through failover, repeated failovers, and a 24-hour soak. Every injected Fleet, PostgreSQL, etcd, or VIP fault, including degraded-only failures, must also continue or resume MQTT curtailment and reach exact shed and restoration targets within 180 seconds. A controller-side command ledger requires every acknowledged PENDING command to succeed within 60 seconds and permits interruption failure only for commands proven PROCESSING when the fault began. Controller-clock probe windows conservatively bound possible VIP ownership, while a fleet-wide external recorder captures every command, plugin, and curtailment request with its device and source host independently of sampled health. Any evidence gap or overlapping old/new-holder work fails the gate. Any failed or missing gate leaves the exact artifact and host profile unsupported.

flowchart LR
  I["Clean install"] --> R["Reboot recovery"]
  R --> F["Failure matrix"]
  F --> C["Commands and curtailment"]
  C --> S["Repeated failover and soak"]
  S --> V{"Every gate passes?"}

Loading

Areas of the code involved

Area / package / file	What changed	Why it matters for review
deployment-files/ha/QUALIFICATION.md	Adds the redacted qualification checklist and result table	Check the gates match support claims and require durable evidence
deployment-files/ha/README.md	Distinguishes installer-compatible architectures from qualified hardware profiles	Check that support claims require recorded evidence
docs/rfcs/0002-active-passive-fleet-ha.md	Aligns the initial support contract with the epic's deferred Fleet Node, scheduler-recovery, and alert-delivery work	Check that local degraded status remains required

Key technical decisions & trade-offs

• Qualification uses a packaged release rather than a source checkout.
• Support is limited to the exact artifact identities, host-package versions, recorded host hardware, fleet scale and topology, and an HA segment restricted to the three hosts and trusted network infrastructure. Same-L2 peer or ARP spoofing is outside the profile.
• PROCESSING-command recovery covers server state; exactly-once device effects and device-side fencing remain outside the profile's support claim.
• Every injected Fleet, PostgreSQL, etcd, or VIP fault carries an active curtailment target, including degraded-readiness gates, and its separate 180-second shed and restoration bounds.
• The address recorder buffers subscriptions before boundary snapshots, replays the overlap, and expands each possible ownership interval across delivery uncertainty; ambiguous overlap fails rather than relying on event arrival order. Health polling measures availability, while the gap-free fleet-wide recorder proves that old- and new-holder work never overlaps anywhere in the qualified test fleet.
• Fleet Node reconnect scale, scheduler recovery, and alert delivery remain deferred; passive rejection and local degraded status stay in scope.
• Host source filtering is qualified against a non-peer using its assigned address; hostile same-L2 peer or ARP spoofing requires network enforcement outside this profile.
• The procedure records outcomes and timings without adding test-only runtime behavior.

Testing & validation

• The static HA profile check and diff check pass.
• Hardware failovers and the 180-second curtailment target at the recorded fleet scale, repetition counts, and the 24-hour soak are pending; therefore this PR remains draft.

[github-actions]

🔐 Codex Security Review

Note: This is an automated security-focused code review generated by Codex.
It should be used as a supplementary check alongside human review.
False positives are possible - use your judgment.

Scope summary

• Reviewed pull request diff only (24378e21c9cadf373d7f3413c11431406fc5ae19...b7277a07d82e12339f5dd06e4bafbdde316a70d9, exact PR three-dot diff)
• Model: gpt-5.6-sol

💡 Click "edited" above to see previous reviews for this PR.

---

Review Summary

Overall Risk: NONE

Findings

No actionable security, correctness, or reliability findings were identified in the changed hunks.

Notes

The executable change adds a fail-closed check before destructive etcd principal cleanup. The remaining changes conservatively document HA qualification boundaries and pending validation requirements. Review was limited to static inspection of the authoritative diff.

---

_{Generated by Codex Security Review |
Triggered by: @ankitgoswami |
Review workflow run}