diff --git a/deployment-files/ha/QUALIFICATION.md b/deployment-files/ha/QUALIFICATION.md new file mode 100644 index 000000000..b15862f38 --- /dev/null +++ b/deployment-files/ha/QUALIFICATION.md @@ -0,0 +1,197 @@ +# Proto Fleet HA qualification + +The tested release artifact, host package versions, and architecture are +supported only after they pass every gate in this document on three clean, +same-L2 hosts. Complete and publish one report for each required platform: + +| Operating system | Architecture | Hardware | +| --- | --- | --- | +| Debian 12 or 13 | arm64 | Raspberry Pi | +| Ubuntu 22.04 or 24.04 | amd64 | Qualified host model | +| 64-bit Raspberry Pi OS based on Debian 12 or 13 | arm64 | Raspberry Pi | + +Other derivatives remain installer-compatible but unqualified. Do not combine +results across platform reports. End each measurement at the gate's actual +recovery signal: VIP health for routing, +writable SQL for database recovery, durable command/device state for commands, +and telemetry or independent power measurement for curtailment. Do not include +addresses, certificates, passwords, device names, or customer data in the +committed report. + +This report qualifies the fixed Fleet application, database, DCS, and VIP +profile only on the recorded host hardware, fleet scale, and topology. +Adjacent application updates are not qualified here. Other host hardware, +Fleet Node HA, larger fleets, reconnect scale, schedule recovery during +failover, and alert delivery remain outside this support claim. + +The HA segment must be restricted to the three HA hosts and trusted network +infrastructure. This profile does not defend against a compromised same-L2 +host spoofing a peer address or claiming the VIP through ARP; environments +without an enforced trusted segment are unsupported. + +## Test identity + +| Field | Value | +| --- | --- | +| Release version | Pending | +| Commit SHA | Pending | +| Release bundle SHA-256 | Pending | +| `deployment-manifest.sha256` SHA-256 | Pending | +| Deployed API, client, and Patroni image IDs/digests on every host | Pending | +| Architecture | Pending | +| Host model and board revision | Pending | +| CPU, memory, and boot storage | Pending | +| Ethernet controller and driver | Pending | +| Operating system | Pending | +| Kernel and firmware versions | Pending | +| Page size | 4096 bytes | +| Docker, containerd, keepalived, nftables, and arping package versions | Pending | +| HA segment isolation control | Pending | +| Test miner count and plugin mix | Pending | +| Miner connection topology | Pending | +| Command backlog at curtailment | Pending | +| Started | Pending | +| Completed | Pending | + +## Clean installation + +1. Start with three freshly provisioned hosts for the report's platform. Do not + qualify by deleting directories from a previous HA installation; reimage + the hosts so no old services, containers, firewall rules, VIP, or data remain. +2. Copy the same release, host-specific `node.env`, and only that node's + matching secret directory to each host. Never copy the generated `offline` + directory to a running host. Copy the etcd root password separately to + `ha-a` only. +3. Run `fleet-ha install` concurrently on `ha-a`, `ha-b`, and `ha-c` as + described in [README.md](README.md). +4. Reboot all three hosts. On both database hosts, run + `sudo /opt/proto-fleet/deployment/ha/fleet-ha status + /etc/proto-fleet/ha/node.env`. +5. Confirm exactly one active Fleet, one passive Fleet, one Patroni primary, + one Patroni replica, three etcd members, and one VIP owner. +6. Confirm every host uses the recorded deployment manifest and container image + identities from the qualified release bundle. + +## Failure matrix + +Restore full readiness before starting each row. Run the external append-only +recorder described under Repetition and soak from before fault injection through +full recovery. Across the complete qualified test fleet, continuously submit +uniquely identified idempotent command probes and keep a controller-side ledger of every +acknowledged or durably observed command ID, including its state at fault +injection. Require every PENDING ID to reach SUCCESS within 60 seconds. Permit +FAILED only for an ID already PROCESSING at fault injection and only with the +expected interruption reason. Treat missing, duplicate, PENDING, or PROCESSING +rows after that bound as failures; record ambiguous submissions separately +rather than counting them as acknowledged. Also fail any row on possible VIP +or writable-primary overlap, overlapping active-only device work, a collection +gap, or a stale transition. Record the observed recovery time and a short +redacted evidence reference. Route or instrument every qualified test miner so +the external recorder captures every command, plugin, and curtailment request +from acceptance through response or connection close, including the device and +source host. Fail on a recorder gap or if work from the old holder remains in +flight when work from the new holder begins anywhere in the test fleet. + +For every row that injects a Fleet, PostgreSQL, etcd, or VIP failure, publish a +unique MQTT curtailment target and confirm shedding has started before injecting +the fault. Keep the fault active while requiring MQTT intake to continue or +resume, the target to be retained or reasserted, and measured load to reach the +exact target within 180 seconds. Then publish a distinct restoration target and +require measured load to follow it within 180 seconds. Apply these bounds +whether the fault causes takeover, writer promotion, or degraded readiness, and +in addition to the row's service recovery bound. For database isolation and +failover rows, also +record host-pinned writable SQL probe results against both database hosts at +100 ms or faster. Before isolating the old primary, begin a writable transaction +on it, write a unique probe identifier, and hold the transaction open. After the +peer is confirmed promoted and writable, begin a fresh transaction there and +write a different identifier. Use a controller barrier to attempt both commits +concurrently. Treat an ambiguous result as potentially committed and fail if +both transactions can commit. Record transaction start, promotion, commit, and +failure bounds on the controller clock. After the old primary rejoins, verify +every possibly committed probe identifier exists at most once and every +acknowledged identifier exists exactly once. + +| Gate | Required result | Duration | Result | Evidence | +| --- | --- | --- | --- | --- | +| Kill active Fleet process | Peer serves VIP within 15s | Pending | Pending | Pending | +| Power off the active Fleet host after confirming it is the Patroni replica | Peer serves VIP within 15s | Pending | Pending | Pending | +| Power off a host that is both active Fleet and Patroni primary | Peer restores writable VIP service within 30s | Pending | Pending | Pending | +| Abruptly lose the database primary after an acknowledged uniquely identified write | Writable primary recovers within 30s and retains the write | Pending | Pending | Pending | +| Isolate the database primary from DCS quorum | At most one primary accepts writes; the old primary rejoins without divergence | Pending | Pending | Pending | +| Stop database standby | Service remains usable; failover readiness is degraded | Pending | Pending | Pending | +| Stop etcd witness | Service remains usable; failover readiness is degraded | Pending | Pending | Pending | +| Break active DCS path | Old active stops serving; peer takes over | Pending | Pending | Pending | +| Break active database path | Old active stops serving; peer takes over | Pending | Pending | Pending | +| Remove active VIP/interface path | Old active stops serving; peer takes over | Pending | Pending | Pending | +| Fail one active-runtime job without killing Fleet | Active health fails, the process exits, and the peer takes over | Pending | Pending | Pending | +| Send Connect RPC, non-RPC HTTP, and ControlStream traffic directly to the passive host | Each product transport rejects it as not active; only health and local status remain available | Pending | Pending | Pending | +| Hold an active-only request across demotion | The old active cancels it before the peer serves active traffic | Pending | Pending | Pending | +| Record active-only device work across failover | No old-holder request remains in flight when new-holder work begins | Pending | Pending | Pending | +| Probe public health through the VIP | Public responses reveal no HA topology and `/api-proxy/health/ha` returns 404 | Pending | Pending | Pending | +| With Grafana and telemetry ingestion stopped, stop the etcd witness | Active health remains usable and local HA status still reports degraded failover readiness | Pending | Pending | Pending | +| From a fourth non-peer, probe each listening HA-only service port before and after reboot | Peer traffic remains healthy and every non-peer connection is blocked | Pending | Pending | Pending | +| Send a valid winning VRRP advertisement from a non-peer's assigned address | The host source filter rejects it and VIP ownership does not change | Pending | Pending | Pending | +| Fail over with PENDING command | New active dispatches the command | Pending | Pending | Pending | +| SIGSTOP the old active with a stalled PROCESSING plugin call, let the peer recover it, queue the old result, then SIGCONT | The resumed stale transition is rejected, exactly one terminal database result remains, and later work resumes | Pending | Pending | Pending | +| Fail over during firmware command | Transitional device state is cleared | Pending | Pending | Pending | +| Send command after failover | Command succeeds on the new active | Pending | Pending | Pending | +The 180-second curtailment result applies only to the miner count, plugin mix, +connection topology, and command backlog recorded above. + +The PROCESSING-command gate proves server-side recovery, not exactly-once +device effects. Device-side fencing is outside this profile's support claim. + +## Repetition and soak + +| Gate | Required result | Result | Evidence | +| --- | --- | --- | --- | +| Application failover | 5 consecutive passes | Pending | Pending | +| Database failover | 3 consecutive passes | Pending | Pending | +| Soak | 24h with no overlapping active-only device work, VIP overlap, or lost failover readiness | Pending | Pending | + +Before the soak, start an external append-only recorder on a separate +controller. Start and confirm both host subscriptions first, buffering +interface-address events. Timestamp every probe send, response, and event +arrival on that controller's monotonic clock; do not compare host clocks. Then +take timestamped state snapshots, replay all buffered events that can intersect +each snapshot, and begin the observation window. Fail if the collector cannot +prove that subscription, snapshot, replay, and timing bounds were gap-free. +Evidence stored only inside the HA database is insufficient. An event's receipt +time is not its occurrence time. For each host, conservatively bound every +possible VIP-owned interval from the send time of the last probe that confirmed +the VIP absent through the receipt time of the first later probe or event that +confirmed it absent again. Seed an initially owned interval at the observation +window start and extend a finally owned interval through the observation window +end. During each injected failure and VIP move, probe both hosts at least every +100 milliseconds; uncertainty that makes the two hosts' possible-owned +intervals overlap fails the gate rather than being resolved by event arrival +order. Fail if either boundary state is unknown. Treat an unreachable host, +write failure, or address-event collection gap as a failed soak. +For a power-off gate, also record the switched power outlet or an independent +power monitor on the same clock. The first confirmed power-off sample closes +that host's possible active and VIP intervals; network unreachability alone +does not. Stream termination after that sample is expected, not a collection +gap. Fail the gate if independent power-state evidence is unavailable. +Outside injected transitions, sample both hosts at least every two seconds for +availability. Record local +`sudo /opt/proto-fleet/deployment/ha/fleet-ha status /etc/proto-fleet/ha/node.env`, +and retain the status, health, and address streams. Run this probe outside shell +error-exit handling: capture stdout, stderr, and exit status, then continue +after the expected nonzero result for degraded failover readiness. Missing or +invalid JSON is a collection failure, not expected degradation. Fail if the +conservative possible-owned intervals overlap. Health polling is availability +evidence, not proof that every runtime transition was observed. Prove effect +exclusivity with the +fleet-wide recorder instead: it must capture every request for every qualified +test miner, and no old-holder work may remain in flight when new-holder work +begins anywhere in the fleet. Any recorder gap fails the soak. Export redacted +evidence after the soak. + +## Verdict + +**Pending.** Do not describe an artifact and hardware profile as supported +until every result above is `PASS` and the report records the tested release, +commit, host hardware, and package versions. A hardware or package-version +change requires requalification. Passing one required platform does not +qualify the other platforms or make an architecture-wide support claim. diff --git a/deployment-files/ha/README.md b/deployment-files/ha/README.md index 223eaf77c..70a42601d 100644 --- a/deployment-files/ha/README.md +++ b/deployment-files/ha/README.md @@ -195,4 +195,5 @@ go test ./server/internal/ha -run '^TestProductionHAProfile$' -v Repeat on `ha-b` without `HA_PROFILE_MIGRATE`. The emitted `HA_PROFILE_EVIDENCE` line proves that the deployment artifacts, etcd leader, Patroni primary, and connected PostgreSQL writer agree. -The qualification workflow owns the complete failure matrix and evidence. +The [qualification procedure](QUALIFICATION.md) owns the release qualification +matrix and redacted evidence. diff --git a/docs/rfcs/0002-active-passive-fleet-ha.md b/docs/rfcs/0002-active-passive-fleet-ha.md index 1e50940a1..11babacd1 100644 --- a/docs/rfcs/0002-active-passive-fleet-ha.md +++ b/docs/rfcs/0002-active-passive-fleet-ha.md @@ -3,13 +3,13 @@ - **Status**: approved - **Author(s)**: Ankit Goswami (@ankitgoswami) - **Created**: 2026-07-13 -- **Last updated**: 2026-08-04 +- **Last updated**: 2026-08-07 ## Summary Add a supported high-availability install mode for Proto Fleet where two warm Fleet app hosts share a self-managed Postgres/TimescaleDB HA cluster, but exactly one Fleet app instance is active for real-time control at a time. The active instance is selected by a Fleet-owned, epoch-fenced database lease. Postgres failover is handled by Patroni over a three-member quorum, and Fleet connects to the current writable database through a pgx/libpq-style multi-host DSN rather than a DB proxy. -This RFC deliberately scopes the HA promise to the real-time control plane. Curtailment dispatch, command execution, schedules, Fleet Node ControlStreams, MQTT curtailment intake, and the database state required for those flows must recover automatically after a single failure. Live HA alerting for failover readiness and control-plane health remains in scope. Historical telemetry, Grafana dashboards, alert history, logs, and cache-like artifacts may be stale, delayed, unavailable, or partially lost in degraded mode. +This RFC deliberately scopes the initial HA promise to Fleet application control. Curtailment dispatch, command execution, MQTT curtailment intake, and the database state required for those flows must recover automatically after a single failure. Local HA status must expose failover readiness and control-plane health. Schedule failover recovery, Fleet Node HA and reconnect-scale qualification, alert delivery, historical telemetry, Grafana dashboards, alert history, logs, and cache-like artifacts remain outside the initial supported profile. ## Decision summary @@ -41,9 +41,9 @@ The supported HA contract is intentionally narrower than "every subsystem stays | Class | Examples | HA guarantee | | ---- | -------- | ------------ | -| Critical control state | Curtailment events and targets, command queue/status, schedules, Fleet Node auth/pairing state, MQTT curtailment source config and runtime edge state, active Fleet lease | Durable while replication is healthy; required for RTO | -| Real-time runtime | Active Fleet app, Fleet Node ControlStreams, command executor, curtailment reconciler, scheduler, MQTT subscriber | One active instance; resumes automatically on another Fleet app host | -| Live HA alert state | Failover readiness, active Fleet holder count, DB primary/standby health, quorum, replication lag, VIP/load-balancer target | Must be emitted from control-plane HA status, independent of Grafana history | +| Critical control state | Curtailment events and targets, command queue/status, Fleet Node auth/pairing state, MQTT curtailment source config and runtime edge state, active Fleet lease | Durable while replication is healthy; required for RTO | +| Real-time runtime | Active Fleet app, command executor, curtailment reconciler, MQTT subscriber | One active instance; resumes automatically on another Fleet app host | +| Local HA status | Runtime role, observation freshness, endpoint health, control and failover readiness, and generic reason codes | Must be emitted independently of Grafana history; alert delivery is deferred | | Best-effort history | Raw telemetry samples, rollups, notification metric samples, Grafana dashboards, alert history, logs | May be stale, delayed, unavailable, or partially lost | | Local artifacts | Firmware files, command artifacts, cached downloads | Not v1 HA unless explicitly promoted to critical storage | @@ -102,13 +102,13 @@ For on-prem installs, the supported endpoint is a Fleet VIP managed by keepalive - both Fleet app hosts run keepalived with the same VIP; - keepalived advertises the VIP only on the host whose local Fleet app passes `/health/active`; - the VIP moves to the peer after the old active fails health and the peer acquires the Fleet lease; -- existing long-lived UI and ControlStream connections reconnect to the same stable endpoint after VIP movement. +- existing UI and API clients reconnect to the same stable endpoint after VIP movement. The VIP is an endpoint routing mechanism, not the correctness authority. Fleet's database lease remains the only source of truth for app activeness. The VIP must follow `/health/active`; it must not decide which Fleet app is allowed to dispatch commands or curtailment. Endpoint-adapter failures that make the active Fleet endpoint unavailable are active-readiness failures. In the on-prem VIP profile, if the current active host cannot maintain local VIP ownership or advertisement, it must fail `/health/active` and stop renewing or relinquish the Fleet lease, or provide an equivalent endpoint-adapter fencing mechanism that lets the peer take over. An active Fleet app that is no longer reachable through the supported stable endpoint must not keep the active lease indefinitely. -The stable endpoint must preserve Fleet's client-facing identity and network-security expectations in every environment. VIP movement must not make UI/API, Fleet Node, or ControlStream traffic reachable outside the intended private network. The supported on-prem VIP profile can rely on the site VPN/private network plus VIP ownership controls that restrict advertisement to the intended Fleet app hosts. Cloud or otherwise untrusted network paths require environment-appropriate transport security and server identity. +The stable endpoint must preserve Fleet's client-facing identity and network-security expectations in every environment. VIP movement must not make UI/API traffic reachable outside the intended private network. The supported on-prem VIP profile can rely on the site VPN/private network plus VIP ownership controls that restrict advertisement to the intended Fleet app hosts. Cloud or otherwise untrusted network paths require environment-appropriate transport security and server identity. MQTT curtailment intake is not Fleet VIP traffic. It is an active runtime responsibility: the old active subscriber must quiesce on failover, and the new active Fleet app must subscribe to the configured broker within the RTO target. @@ -232,7 +232,7 @@ Active-only work in v1 includes: - command execution and command-state repair; - telemetry polling and discovery work that feeds active control; -- schedule processing; +- schedule processing, with failover recovery qualification deferred; - curtailment reconciliation and MQTT intake; - cleanup/sweep work that mutates shared control state or external command artifacts. @@ -244,7 +244,7 @@ Always-on services: Activation must trigger required control reconciliation promptly enough to meet the RTO target. -### Request and ControlStream gating +### Request gating Add a single active-mode request gate for product traffic: @@ -252,14 +252,13 @@ Add a single active-mode request gate for product traffic: - In passive mode, Connect/gRPC product traffic fails with `Unavailable` and a machine-readable `not-active` detail; other product transports use the equivalent retryable status. - Load-balancer-safe health endpoints and loopback-only HA diagnostics bypass active gating. -Fleet Node ControlStreams are explicitly gated: +Fleet Node ControlStreams are also rejected on a passive Fleet instance as a +safety boundary: - Passive Fleet rejects ControlStreams with `not-active`. - Lease loss cancels active-scoped request contexts and closes already accepted product streams with `not-active`. -- Fleet Node clients treat `not-active` as a cheap redirect signal and reconnect quickly, with bounded jitter so large fleets do not reconnect in lockstep. -- Fleet Node transport must detect dead streams quickly enough to meet the RTO target. - -This preserves the RFC 0001 model where Fleet Nodes connect outbound to the server/Fleet app endpoint, while ensuring only the active Fleet app host owns command routing state. +Fleet Node failover, reconnect timing, and fleet-scale reconnect behavior are +not qualified by the initial supported profile. ## Health and operator status @@ -325,8 +324,8 @@ The exact installer flags, templates, compose files, and runbook commands belong | On-prem active Fleet host loses health | keepalived stops advertising the VIP; the peer advertises the same VIP after acquiring the Fleet lease and passing `/health/active`. | | On-prem active Fleet host loses VIP ownership | Active readiness fails or equivalent endpoint-adapter fencing triggers; the peer can acquire the Fleet lease and advertise the VIP. | | Endpoint routes to passive | Passive fails `/health/active` and rejects product traffic and ControlStreams. | -| Grafana fails | Real-time control continues; `/health/active` remains based on control readiness only; HA alert generation continues from control-plane status. | -| Historical telemetry ingestion stalls | Real-time control continues; history is stale/lossy and HA status/alerts still report control-plane degraded state. | +| Grafana fails | Real-time control continues; `/health/active` remains based on control readiness only; local HA status remains available. | +| Historical telemetry ingestion stalls | Real-time control continues; history is stale/lossy and local HA status reports control-plane degraded state. | ## Validation gates @@ -341,7 +340,6 @@ Activation and fencing: - Passive mode rejects all product traffic, including non-RPC HTTP routes, while preserving the explicit health and operator-status bypasses. - Lease loss terminates already accepted product streams and active-scoped request work. - A stalled active process cannot overwrite terminal command state or renew an expired lease with its old epoch. -- Fleet-scale reconnect tests avoid synchronized ControlStream reconnect storms during failover. Database and durability: @@ -363,7 +361,7 @@ Deployment and diagnostics: - Same-subnet on-prem validation proves the configured VIP is unused, moves only to the Fleet app host passing `/health/active`, remains reachable only through the intended private network, and resists unintended VIP ownership by non-HA hosts. - Same-subnet on-prem failover tests prove loss of local VIP ownership on the active host triggers active-readiness failure, lease release/expiry, or equivalent endpoint-adapter fencing so the peer can take over within the RTO target. - HA management-plane ports are restricted to HA peers and approved operator diagnostics paths; cloud or untrusted network profiles add authenticated transport. -- HA alerts fire from control-plane status for standby loss, quorum loss, active holder anomalies, replication lag, and endpoint targeting failures even when Grafana or telemetry history is unavailable. +- Local control-plane status reports standby loss, quorum loss, active holder anomalies, replication lag, and endpoint targeting failures even when Grafana or telemetry history is unavailable. - Telemetry/Grafana failures do not fail `/health/active`. - Public health endpoints do not expose HA topology, and `/health/ha` is reachable only on loopback. @@ -405,7 +403,7 @@ Deployment and diagnostics: | 1 | DB writer routing | Multi-host DSN, read-write targeting, stale pooled connection discard, failover retry classification | Fleet can reconnect to the current DB writer without HAProxy. | | 2 | Active lease | `fleet_runtime_lease`, sqlc queries, `ha.Coordinator`, `/health/active` | Fleet can decide active/passive state safely. | | 3 | Runtime supervision | Move active-only services behind a supervisor; immediate reconciler tick on activation | Passive Fleet stays warm but does not dispatch or mutate control state. | -| 4 | Passive gating and Fleet Node retry | Active-mode request gate, ControlStream `not-active`, Fleet Node fast retry and stream liveness tuning | Traffic can safely route only to active Fleet. | +| 4 | Passive gating | Active-mode request gate and ControlStream `not-active` rejection | Product traffic can safely route only to active Fleet; Fleet Node reconnect qualification is deferred. | | 5 | HA substrate | Patroni image/config, etcd, keepalived/VRRP VIP templates, peer-connectivity preflight, install/join flow | Operators can install the on-prem HA profile. | | 6 | Degraded-mode observability | Local status for standby loss, quorum loss, active count, replication lag, and failover readiness | Operators can distinguish full HA from control-only/degraded operation. | | 7 | Lab and cloud references | Repeated failover tests, partition tests, runbook, cloud deployment reference | The install mode can be marked supported after validation gates pass. |